Simultaneous Missing Value Imputation and Structure Learning with Groups
Authors: Pablo Morales-Alvarez
Wenbo Gong
Angus Lamb
Simon Woodhead
Simon Peyton Jones
Nick Pawlowski
Miltiadis Allamanis
Cheng Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: structure learning
missing value
variational inference
graph neural network

Abstract: Learning structures between groups of variables from data with missing values is an important task in the real world, yet difficult to solve. One typical scenario is discovering the structure among topics in the education domain to identify learning pathways. Here, the observations are student performances for questions under each topic which contain missing values. However, most existing methods focus on learning structures between a few individual variables from the complete data. In this work, we propose VISL, a novel scalable structure learning approach that can simultaneously infer structures between groups of variables under missing data and perform missing value imputations with deep learning. Particularly, we propose a generative model with a structured latent space and a graph neural network-based architecture, scaling to a large number of variables. Empirically, we conduct extensive experiments on synthetic, semi-synthetic, and real-world education data sets. We show improved performances on both imputation and structure learning accuracy compared to popular and recent approaches. 

Simultaneous Missing Value Imputation and Structure Learning with Groups
Pablo Morales-Alvarez* University of Granada
Wenbo Gong Microsoft Research
Angus Lamb G-Research
Simon Woodhead Eedi
Simon Peyton Jones Epic Games
Nick Pawlowski Microsoft Research
Miltiadis Allamanis Google
Cheng Zhang Microsoft Research
Abstract
Learning structures between groups variables from data with missing values is an important task in the real world, yet difficult to solve. One typical scenario is discovering the structure among topics in the education domain to identify learning pathways. Here, the observations are student performances for questions under each topic which contain missing values. However, most existing methods focus on learning structures between a few individual variables from the complete data. In this work, we propose VISI., a novel scalable structure learning approach that can simultaneously infer structures between groups of variables under missing data and perform missing value imputations with deep learning. Particularly, we propose a generative model with structured latent space and graph neural network-bat -bsed architecture, scaling to large number of variables. Empirically, we conduct extensive experiments synthetic, semi-synthetic, and real-world education data sets. We show improved performances on both imputation and structure learning accuracy compared to popular and recent approaches.
Introduction
Understanding the structural relationships among different variables provides critical insights in many real-world applications, such as medicine, economics and education [42] 62). Thus, learning graphs from observed data, known as structure learning, has recently made remarkable progress
100
61
63 64
For many applications, variables in the data can be gathered into semantically meaningful groups where useful insights are group level. For example, in finance, one may be interested in how financial situation influences different industries (i.e. groups) instead of individual companies (i.e. variables). Similarly, in education, the data can contain student responses to thousands of individual questions (i.e. variables), where each question belongs to broader topic (i.e. groups). Again, it is insightful to find relationships between topics instead of individual questions. Moreover, real-world data such as educational data is inherently sparse since it is not feasible to ask every question to every student; the dimensions of the data in terms of the number of variables and the number of observations very high, posing scalability challenge. Despite the progress in structure learning, no existing method can discover group-wise relationships given large-scale partially observed data.
In this work, we present VISL (missing value imputation with structural learning), novel approach simultaneously tackle group-wise structure learning and missing value imputations driven by the
Department of Statistics and Operation Research, University of Granada, Spain Work done while internship in Microsoft Research Cambridge. Correspondence to: Cheng Zhang <Cheng Zhang @ @microsoft.com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

1
Zn.2
Zn
G
T n2
Dn
Gas
20
X1
X2
X3
81
82
83
Xn
n,,
(a)
(b)
Figure 1: (a) Graphic representation of VISI.. VISL is variational auto-encoder based framework. Observations from each group are encoded into low dimensional latent variables. The structure is treated as global latent variable. A GNN based decoder used to decode the latent variables to ohservations. (b) Probabilistic graphical model for VISL. where the partial observation x is generated from its local latent variable z and the global latent variable G which characterizes the structures.
real-world topic relationship discovery in education setting. This is accomplished by combining variational inference with a generative model that leverages a structured latent space and a decoder based on message-passing Graph Neural Networks (GNN) [13]. Namely, the structured latent space endows each group of variables with its latent subspace, and the interactions between the subspaces regulated by GNN whose behavior depends the inferred graph from variational inference, see Fig. aa). VISL satisfies all the desired properties: it leverages continuous optimization of the structure learning to achieve scalability [63 64]; the VISL formulation naturally handles missing values, and it can discover relations at different levels of granularity with pre-defined groups. Empirically, we evaluate VISL on one synthetic and two real-world problems including the aforementioned education scenario. VISL shows improved performance in both missing data imputation and structure learning accuracy compared to popular and recent approaches for each task. We worked closely with an education domain expert to evaluate the learned topic relationships, and our model has provided insightful results as recognized by the domain experts.
2 Model Description
In the following, present the formulation of VISI. for scalable group-wise structure learning with partial observations using a novel deep generative model based framework.
2.1 Problem setting
Assume training data set X = [xx with x, E R.D. The observed and missing values are denoted as Xo and Xu, respectively, where we assume the data are missing completely random (MCAR) missing random (MAR). In|Appx. A] we explain how to handle MAR. In particular, variables can be gathered into M groups, where each can be denoted as Xn.m = [ In containing the variable indices belonging group (c.g.. In [4, 5, indicates group 2 includes the 4th 5th and 6th variables). One should note that each Im may have varying sizes for different m (i.e. varying group sizes). Throughout the paper, assume the group information is provided priori. this information unavailable, casual representation learning can be leveraged to represent low-level signals into groups. The goal of VISL is to (i) perform missing value imputation for test samples and (ii) infer structures between groups of variables. We use the adjacency matrix G [0, M represent graph, where Gij or 0 indicates whether there is directed edge from th to j-th group or not. In the context of the education domain, the above formulation can be rephrased follows: variable X,, containing the student's responses a of questions. zi,j = 1 represents student i has answered question j correctly. Groups can be defined as the topic associated with each question. Z,, contains the question IDs that belong to the same topic, and Xm represents group of responses related to that topic. Clearly, not all students can answer every question. Thus, Xo. X represent the existing responses and un-an answered questions, respectively. The goal of VISL is to (i) predict students' responses to un-answered questions, which by itself is important in the education domain [56] [7], and (ii) discover the relationships between topics, which can help education experts optimize the learning experience and the curriculum. For structure learning, we adopt a Bayesian approach for graphs [18. Namely, we seek to maximize the posterior probability of G given partially observed training data Xo within the space of all DAGs:

---

G. arg GEDA. Da p(Xo/G))(CC
To optimize over the structure with the DAG constraint in Eq. 1
we resort to recent continuous optimization techniques 25 63 64), where differentiable measure of 'DAG-ness', R(G)
tr(cGoG) D 1, was proposed and is zero if and only if G is a DAG. To leverage this DAG-ness characterisation, we follow Kyono al. [23]. Yu al. [61] and introduce regulariser based on RGG) to favour the DAG-ness of the solution, i.e.
G. arg maxG (p(Xo (G) XR(G))
(2)
In the following two sections, we present our detailed formulation, training and imputation algorithms of VISL, that allows the model to infer the latent structure G and impute missing values Xy in test sample R based on the observed Xo-
2.2 Generative model and variational inference
For the generation of observation X, adopt the latent vari- able model of Fig. I Particularly, given an inferred graph G and latent Z, the generative path from Z to X is provided in Fig. T] where we use a graph neural network (GNN) decoder that respects the learned graph structure G and the provided grouping structure. Then the joint model likelihood
Algorithm Generative process
Gi Bernoulli(ps)
for n E (1,2.... .J do Z, ~ I
X, ~ ~Noo(ZZ, o)
end for
(X, 2, G) = p(G) II, P(.,/Zn, G)p(Z.).
(3)
Using this marginal likelihood p( X) is consistent with Bayesian
score based causal discovery [18. As Kaiser and Sipos [22] and Reisach et al. 39 pointed out that other commonly used objectives, e.g. L2 loss in NOTEARS [63], are sensitive to data scaling and
result in learning directions towards the high variance nodes. Loh and Bühlmann 28 and Ng et al. 351 showed that using proper likelihood function can address these problems. Next, we leverage amortized variational inference sidestep the intractable marginalization of the joint likelihood.
Amortized variational inference. The true posterior distribution over Z and in Eq. 3 is intractable since we use complex deep learning architecture. Therefore, resort variational inference as in Kingma and Welling (24, Kingma et al. 23]. Here, we consider a fully
efficient amortized
factorized variational distribution q G 9 (G) IG) 9 Zn x..), where qo (Z, (x,) ) is Gaussian whose mean and (diagonal) covariance matrix are given by an encoder. For (G), we consider the product of independent Bernoulli distributions over the edges; that is, the presence of each edge from to j is associated with a probability Piş to be estimated. With the above formulation, the evidence lower bound (ELBO) is
ELBO = Σ.. {Eg, (Z., xx, (acg Zn. G) KLL(gs (Z, Xx, p [Z,)]] (LGGG)))
(4)
Next, we explain our choice of the generator (decoder), which uses a GNN over a learned graph G to model the interactions between latent variables, representing the information about each group. Then, we focus on the inference network (encoder), representing the mapping from the group of observed variables to its corresponding latent representation.
Generator. The generator (i.e., decoder) takes Z, and G as inputs and outputs the reconstructed x, fo (o., G), where 0 are the decoder parameters. In order to respect the pre-defined group structure, as shown Fig. I Z,, is partitioned into M parts, where Zn.m represents the latent variable for the group of observations Xn,m This defines group-wing wise structured latent space. We adopt two-step process for the generative path Z, to X (i) GNN message passing with respect to the leamned graph G between latent Z (ii) final read-out layer generate X.,
GNN message passing in the generator. In message passing, the information flows between nodes in T consecutive pode-to-edge (n2e) and edge-to-no node (e2n) operations [33] At the t-th step, we compute embedding b/-, for each edge i + j. called forward embedding, which summarizes the
3

---

information sent from node i to Specifically, the n2e/e2n operations in VISL are
(0/ n2e: h i-+ = MT.P/
(5)
c2n z
(t) MLP EEuf Gki k
(t).f
(6)
Here, / refers to the /-th iteration of message passing (that is, Z = Zn notice that we omit subindex for clarity). Finally, MLP and MLPe2e are MLPs to be trained.
Interestingly, the message passing updates indicate that the information flows between latent nodes if a directed edge specified in graph G. Hence, the inferred structure G directly defines relations for latent space Z which contains the information of pre-defined groups. We show that under certain conditions, the inferred graph G also represents the group-wise structure in observational space, and the corresponding model can be reformulated to general structural equation model (SEM) [37] (see Appx. B)
Read-out layer in the generator. After I' iterations of GNN message passing, we have Z Due to we allow ZTT) and x have different dimensions, apply final function that maps Z(T) to the reconstructed x. which also respects the pre-defined group structure. Since the ob- servation x = Ix..... Xm] may contain Xm with different dimensions, we adopt M different MLPs. one for each group as the final read-out layer, to respect the group structure. Namely, & = (g' (zT)).... ... (ZT) where gm represents the MLP for group m. Thus, the decoder parameters include the parameters of the following neural networks: MLP/, MLP and g" for m = -,..., M.
Inference network. As in standard VAEs, the encoder maps a sample x, to its latent representation Zn As discussed before, Zn is partitioned into M parts, where each Zn.m contains the information of the observation in group m. Similar to the read-out layer, we utilize the M MLPs to map groups of observations to the mean/variance of the latent variables:
nn (Hon (Xn,1), Porse (Xn.ar)
M
On 0 (Xn,1),....
M Dons (Xm,M))
T
Here, " and a are neural networks for group m. When missing values are present, we replace them with constant as in [34]. Under this formulation, VISL can infer latent variables Z,, from incomplete x,, for both training and test data. A graphic representation of how the encoder respects the structure of the latent space is shown in the appendix. Fig. 6b).
2.3 Training VISL
Given the model described above, we propose the training objective to minimize w.r.t. 0, & and G: [visL(O, o, G) = EEBBO + (E [R(G)],
(8)
where ELBO is given by Eq. 4 and the DAG regulariser R(G) was introduced in Eq. to favor the DAG-ness of learned graph
Evaluating the training loss LVISI. VISL can work with any type of data. The log-likelihood term (log po (x,, Iz,, G) in Fq.. 4) is defined according the data type. We use Gaussian likelihood for continuous variables and a Bernoulli likelihood for binary ones. For the inference of and G, the standard reparametrization trick used to sample Zn from the Gaussian q, (Z,, x,.) (2] 24 To backpropagate the gradients through the discrete variable G. we resort to the Gumbel-softmal trick to sample from (G) [2] 3T. The KL(gs (Z,,,,,) [p,))] and [L[q( [[]GG)))(() terms can be obtained in closed-form form since they are Gaussian distributions and independent Bernoulli distributions over the edges, respectively. This formulation brings additional advantages real- life applications since one can easily incorporate domain knowledge and prior information into the VISL framework. For example, the existence of a specific edge is known a priori, the edge probability can be set to 0/1 in the prior distribution. Finally, the DAG-loss regulariser in Eq. can be computed by evaluating the function on Gumbel-softmax sample from (G). To adapt the model to different missing levels in the training data X, we adopt the masking strategy [15] 30), which drops random percent

---

Algorithm 2 Training VISL.
Input :Training dataset X, possibly with missing values.
for each batch of samples (x,, J do
Drop a percentage of the data for each sample Xa.
Encode xs, through the reparamctrization trick to sample Zn ( using Eq
Use the Gumbel-softmar to sample from q(G).
Use decoder to reconstruct &., ZZ.. G).
Calculate the training loss LVisI. (Eq. 8)
Gradient step w.r.t. & (encoder parameters), (decoder parameters) and G (posterior edge probabilities).
Output Encoder parameters Φ), decoder parameters and posterior probabilities over the edges G.
of the observed values during training. The entire training procedure for VISL is summarised in Algorithm 2
Two-step training. After training, we obtain the posterior of the graph G, which respects the underlying structure of the groups as shown in Anpppx. BJ With the trained network, we can impute missing values in the groups where their ancestors contain some observations but if a group has no ancestors no information can be propagated during imputation. After learning the graph structure and to facilitate the imputation task, we introduce backwards edge: for an edge from j to i we denote the backwards edge information as hリ which codifies the information that the + j
edge lets flow from the j-th to the i-th node. It is defined in the same way as Eq. i.e..: h (1).6 i-+j
MLP ([z.
(t-1) (1x-1)) :2j 1. where MLP" is the backward MLP; and the e2n update (Eg6)
6 is modified
to 2, (t) MLPpe2n [Eife Gki { (0. [ + b222))
In summary, we propose two-stage training process, where the first stage described in previous sections focuses on discovering the edge directions between nodes without the MLP (i.c., we do train the MLP"). In the second stage, we fix the graph structure G and continue train the model with the backward MLP. This two-stage training process allows VISL to leverage the backward MLP for the imputation task without updating the graph structure.
Revisiting the learning objectives. The optimal graph of relationships, denoted as G. in Eq. is given by the estimated posterior probabilities of graph G. In addition, the regularizer R provides a way to evaluate if the resulting graph is a DAG. By tuning the regularizer strength X, one can ensure that the resulting G represents proper DAG.
For imputation, similar to Ma et al. [30] Nazabal et al. (34], the trained model can impute missing values for test instance X as
X) = (Z. Z G).
Therefore, the distribution over X (missing values) is obtained by applying the encoder and decoder with x as input. Previous work [30 34 also uses variational autoencoders for missing value imputation, which can have problems with overfitting due to spurious correlations (25) even with sufficient training data. One important distinction of VISL is the incorporation of the learned structure G, which helps the model to be robust to spurious correlations.
Special case: variable-wise relations. In the above formulation, have defined VISL for group- wise structure learning. Variable-wise relations can be regarded as a special case. In particular, we set M D and I,m = {m) (see Fig. 5 (a) in the appendix), i.e. each group only contains single variable. Through this modification, we can further simplify the encoder and read-out layer. Instead of using M different MLPs, single MLP can be shared across all variables since each group has dimension of 1. The mean function for the encoder is then defined as
pn (Mn())))) ...., (x(,
(10)
One can define encoder variance (Fig. (a) in the appendix) and the read-out layer g analogously.
Computational cost. The main computational bottleneck is the training of VISL, where we require a different inference network and read-out layer for each group E M. However, some weight sharing schemes can used reduce the computational cost. When each group has the same dimensionality, share the weights of the inference networks. In the case of different dimensionalities, can
5

---

infer the latent variables by parallelizing inference network forward passes to reduce computational the wall-clock time comparison for each of the methods for up to 512 variables. We can see that all deep learning based methods (i.e. NOTEARS, DAG-GNN, VISI, etc.) have similar time complexity.
cost. Tin appendix provides
3 Related Work
Since VISL simultaneously tackles missing value imputation and structure learning, we review both fields. Moreover, review recent works regarding improving the deep learning performance with structure learning. Finally, as one of the focused applications of this work is education, we review recent advances of Al in education.
Structure learning. Structure learning aims to infer the underlying structures associated with some observations. There are mainly three types of methods: constrained based,, score-based, and hybrid. Constraint-based ones exploit (conditional) independence tests to find the underlying structure, such as PC 47 and Fast Causal Inference (FCT) 48 They have recently been extended to handle partially observed data through test-wise deletion and adjustments
(50) [2). Score-based methods find the structure by optimizing a proper scoring function. The core difficulty lies in the number of possible graphs growing super-exponentialy with the number of nodes 4.. Thus, explicitly solving the
optimization can only be done up few nodes 77 36 46). Therefore, approximation have been proposed to ease the computational burden, including searching over topological ordering [4]] greedy search 38, coordinate descent [T].
Recently, continuous optimization, called Notears, has become very popular 663] Notears proposed differentiable characterization of the DAG learn the model parameters and graph structures jointly. Notears has inspired the development of other methods, Notears- MMLP and Notears-Sol [64], Grandag (26), and DAG-GNN [61], which extends to model nonlinear relationships between variables. However. their formulations cannot handle missing values and have been observed to be sensitive to data scaling [22. DAG-GNN also adopts a specially-designee GNN to perform structure learning [61]. There are three key distinctions compared to VISL: (i) our model handles the group-wise relationship, while DAG-GNN focuses on variables level: (ii) our model is capable of missing value imputation and group-wise structure learning simultancously, whereas the original formulation of DAG-GNN and related work require complete data; (iii) VISL adopts Bayesian view for the graphs, compared to point estimation. VISL is also related to Bayesian DAG learning. Viinikka et al. (53) proposed to sample the graph posterior using MCMC, but it suffers from high computation complexity. VISL, in contrary. can easily be scaled to high dimensional datasets. BCD (6] and DiBS (2 have recently leveraged the approximate inference for scalable Bayesian DAG learning. BCD nets focused on linear Gaussian SEM (v.s. nonlinear relationship by VISL). DiBS adopted a full Bayesian treatment and used a particle sampler to draw from a joint posterior. Compared to
VISL, it cannot handle missing values nor variable groups. Geffner et al. 12 recently proposed an end-to-end causal inference framework (DECI) combining causal discovery and inference with missing data. Compared VISL, DECI is built upon nonlinear additive noise models [19), whereas VISL uses an autoencoder structure with a GNN decoder. When the GNN reaches equilibrium state, Appendix B shows that VISL. represents general SEM, which includes additive noise models as a special case. Apart from the likelihood- based causal discovery, Rolland et al. [40] used score matching to extract causal relationships for non-linear additive noise models. Our work differs in four aspects: (1) structure learning (VISL) v.s. causal discovery;
(2) VISL is based on the auto-encoder framework; (3) VISL can handle MAR missing data and (4) VISI uses Bayesian view of graphs.
Structure deep learning. Continuous optimization for learning structures has been used to boost performance in classification. In CASTLE (25), structure learning is introduced for a classification model. This regulariser reconstructs only the most relevant features, leading to improved out-of-sample predictions. In SLAPS 10). the classification objective is supplemented with a self-supervised task that learns a graph of interactions between variables through a GNN. These works focused on leveraging the structure learning instead of advancing its performance.
regulariser
Missing values imputation. The relevance of missing data real-world problems has motivated a long history of research [9 [1] A popular approach is to estimate the missing values based on the observed ones through different techniques (45) e.g. Random Forest [49], Bayesian Ridge Regression (2]. Wu et al. [59 explored the use of generative model for missing value imputation,

---

although fully observed training data is required. This limitation is addressed in both Nazabal et al. [44] with eerotimpuiing strategy, and Ma et al. [3] with a permutation invariant set encoder. Mattei and Frellsen [32) proposed to use the importance weighted autoencoder, which enables a tighter
lower bound than EL.BO and leads to improved performance. Ivanov et al. (20] parameterized
the imputation as sampling from conditional distribution, and proposed a method for arbitrary
conditioning with VAEs.
Al in education. There has been tremendous progress in AI for educational applications, e.g. knowl- edge tracing [27 33 54]; grading students' performance [58]; generating feedback for students working on coding challenges [60]. In particular, most related to VISL is imputing missing values in students' responses. Wang et al. [56] adopts a partial VAE [0] to perform missing value imputation and personalization. However, partial VAE does not consider the structural relations between ques- tions/topics. With the additional insights from structure learning, VISI. can provide more information to teachers to help curriculum design than just imputations.
20
3:54
DACE
2-
DURN
BD
MM
X22
5
2
3
4
0:08 0.004
0.04
8517
50
(a)
(b)
Figure 2: (a): Structure simulated for one of the synthetic datasets with 5 variables. (b): Graph predicted by VISL (when the one on the left is used as the true one). VISL recovers the ground truth graph with one addition edge from 5 to 4.
Figure 3: Probability of edges obtained by VISL in the synthetic experiment. By using a 0.5 threshold, we get the predicted graph in Fig. 2). Item (i, refers to the probability of edge i + j.
RMSE
Major
0.54
Adjacency
Oriemaaionn
Causal
Mein
0:22::000661
Recall
Precision
Foose
Recall
Precision
Fj spore
aceuncy
MICE
0:14:000046
Misdurest
01131000022
PC
0.42:0.0056
0.61100066
0.49:00006
0.201000466
033:200061
0.26:20.051
0.33:::004
PVAR
014200043
GFS
0.45200044
0.57200006
0.49:20.388
0.25::0046
031:00033
0.2720.049
0:36:::005
VISL
0.12::0.004
0.19:00025
0.M1100099
0.271:0.36
0.151:0021
037:10.000
0.21::0.02
0.15100033
NOT (NL)
0.3320.030
0.49100051
0.3920.044
02810003
042:100043
0.33:20.355
0:281:00022
Table 1: Imputation re- sults for the synthetic experiment. Mean and standard error over 15 datasets.
DAG-GNN
0.44200064
0.51200222
0.46:20006
0.354:0.000
047:500002
0.37:20.049
0.35::0.000
VISL
0,78:00064
0.73:14.078
0,74200003
0.66:1.133
0.60:10.10
0.63:10110
0.66:10.13
Table 2: Structure discovery results for synthetic experiment (mean and std error over 1.5 datasets).
Experiments
We evaluate the performance of VISL in three different problems: a synthetic experiment where the data generation process is controlled, a comi-synthetic problem (simulated data from a real-world problem) (Neuropathic Pain), and the real-world problem that motivated the group-level structure learning (Fedi). The first two datasets are on the variable level. The last one focuses on the group level and weal-world usage, and have worked closely with the domain expert to evaluate the results. Additional experiments are presented in the appendix.
Baselines. We consider five baselines for the structure discovery task at the variable level. PC [48] and GES 3 are popular constraint-baser and score-based approaches, respectively. NOTEARS (NOT.) 63 the non-linear (NL) extension of NOTEARS [64], and DAG-GN GNN [6]] are the other three closely related baselines. Unlike VISL, these baselines cannot deal with missing values. Therefore, we work with complete training data in the first two sections. The Eedi real-worke data is only partially observed, where these baselines are not applicable. For the missing data imputation, we also consider five baselines. Mean Imputing (Mean), Majority Vote (Major) (refer Appx. for short descriptions), Missforest (49) and MICE 22 are four widely-used imputation algorithms, and PVAE [30] is recent algorithm based on amortized inference.
7

---

Accuracy
AUROC
AUPR
Major 0.9268.10.0003
0.5304.1..0003
0.1366.10..025
Mcan 0.9268::000003
0.8529::0.0011
0.3262++0.0034
MICE 0.9469=20.0077
0.9319=000010
0.6513-+0.0046
Missforest
0.9305:200000 0.8915:2000003
0.5227-1000003
PVAE
0.9415.10.0033 0.9270.10.0007
0.5034110.0046
VISL
0.9471-100006
0,9392:+0.0008
0.6597=000003
Table 3: Imputation results for neuropathic pain data (mean and std error over five runs).
Adjacency
Orientation
Causal
Precision
Fi-score
Precision
Fi-score
Accuracy
PC
0.046100.00
0.005.10.006
0.024-10.001 0.082.1..001
0.199 1..01
0.044.1.0022
0.058100.001
GES 0.110:100001
0.436:1:0008
0.176::0.022
0.082100001
0.523-1:0.099
0.1311:0.033
000000
NOT. (L.
0.006:10.000
0.011:10.001
0.008
0.001
0.0011100001
0.001110.000
0.001
NOT. (NI.)
0.011::0.001
0.644:100.25
0.0224-0.002
0.006:10.001
0.354::0.018
0.012410.001
0.006-10.001
DAG-GNN
0.272 0.2
0.128=000027
0.051=0.010
0.126=00.059
0.050=20.007
0.051=0.010
VISL
0.261=0.006
0.6374:0.099
0.370=000055
0.236=1.0077
0.573-10.005
0.334:20.006
0.245=000066
Table 4: Structure discovery results for neuropathic pain data (mean and std error over five runs).
Metrics. Imputation performance is evaluated with standard metrics such RMSE (continuous variables) and accuracy (binary variables). For binary variables, we also provide the area under the ROC and the Precision-Recal curves (AUROC and AUPR, respectively), which especially useful for imbalanced data (such as Neuropathic Pain). We follow common practice 14. 52 regarding structure discovery performance, and consider metries on the adjacency and orientation. While the former does not take into account the direction of the edges, the latter does. We report recall, precision and F Fcosc. We also provide causal accuracy, a discovery metric that considers orientation [5]
4.1 Synthetic experiment
We simulate fifteen synthetic datasets. For each simulated dataset, we first sample the true structure G; see Fig. 2[a) for an example. The appendix provides detailed generation mechanism, including a visualisation of the data in Fig. For each dataset, we simulate 5000 training and 1000 test samples.
see
Imputation performance. VISL outperforms the basclines terms imputation across all synthetic datasets (Talee The results grouped by the number of variables are presented by Table 8 in the appendix.
Structure discovery performance. VISL obtains better performance than the baselines, see Table 2] The results split by the number of variables are shown in the appendix, Table 10 Specifically, we observe VISL consistently outperforms the baseline method in all metrics considered with all datasets (see Table 10). In this small synthetic experiment, it is possible to visualize the predicted graph. Fig. shows the posterior probability of each edge (i.e. the estimated matrix G) for the simulated dataset in Fig. a). Using a threshold of 0.5, we obtain the predicted graph FFg. 2)b)) We observed that VISL recovers the ground truth graphs with one additional edge. The sources of its advanced performances are twofold: 1) VISL uses the ELBO (Eq. 4) as training objective, a surrogate for marginal likelihood. Kaiser and Sipos 22 Reisach et al. [39 reported that L2 loss used in NOTEARS is sensitive to data scaling and directs the learning of directions towards high variance nodes. This can be resolved by using a proper likelihood training objective [28 351; 2) instead of using observed variables to construct the structural equation model (NOTEARS) or designing a specialized GNN structure (DAG-GNN), VISL is more flexible in terms of transforming the observed variables into a latent distribution via the encoder, and adopting a general message-passing GNN decoder.
Finally, VISL can scale to large data in terms of data points and dimensionality. We demonstrate the computational efficiency with synthetic data ranging from 4 to 512 nodes in the appendix, Table II
4.2 Neuropathic pain dataset
We evaluate our method using a benchmark in healthcare applications 533 The dataset contains records of patients regarding the symptoms associated with neuropathic pain. There are 222 variables
8

---

Adjucency
Orientation
Accuracy
AUROC
AUPR
Expt
Expt
Expe I Expe
Majer
0.6260:1000000
0.6208:1000000
0.7465-10.0000
Mean
0.6200=0000000
0.6753+:000000
0.690==0.0000
Random
2.04
2.08
1.44
MICE
0.6794=:00005 0.7453=t0000007
0.7483=000010
DAG-GNN
2.04
2.32
1.68
1.68
Missforest
0.6849:10.0005 0.7219:10.00007
0.7478::0.0000
VISL
3.60
3.70
2.76
2.60
PVAE
0.7138:10.0005 0.7852-100.001
0.8204110.0002
Table 6: Average expert evaluation of the topic re-
VISL
0.7147:00.007
0.78154:00008
7.0000000006
lationships. Cohen's 's inter-annotator agreement is 0.72 for adjacency and 0.76 for orientation (sub- stantial agreement).
Table
Imputation results for Eedi topics dataset
(mean and standard error over five runs).
in this dataset. Unlike the previous experiment with continuous data, this dataset has binary variables indicating the symptoms. The train and test sets have 1000 and 500 patients, respectively.
Imputation performance. VISL shows competitive or superior performance when compared to the baselines, see Table 3 Notice that AUROC and AUPR allow for an appropriate threshold-fres frefre assessment this imbalanced scenario. Indeed, as expected from medical data, the minority of values are 1 (symptoms); here, the prevalence of symptoms is around 8% in the test set. Interestingly, it precisely AUPR where the differences between VISL and the rest of the baselines are larger except MICE, whose performance is very similar to VISL in this dataset.
Structure discovery results. As in the synthetic experiment, VISL outperforms the causality- based baselines; see Table 4, Notice that NOTEARS (NL) is slightly better in terms of adjacency adjacency-precision, precision, i.e the edges that it predicts are slightly more reliable. However, this is the expense of significantly lower capacity to detect true edges, see the recall and the trade-off between both (FI-score).
4.3 Eedi topics dataset
Finally, we evaluate our method on an even more challenging real-world dataset in education to discover the group-wiss structure between topics while the observations are question-answer answer pairs under these topics. This is an important real-world problem in the field of Al-powered educational systems [56] (7). This dataset very sparse, with 74.1% of the values missing. The dataset contains the responses from 6147 students to 948 mathematics questions. The 948 variables are binary (1: correct and 0 otherwise). These questions target specific mathematical concepts and are grouped into a meaningful hierarchy of topics; see Fig. 4jin [Appx. E) Here we apply VISL to find the structures among the topics using the third level hierarchy (Fig. 4). resulting in 57 group-level nodes.
Imputation results. VISI achieves competitive or superior performance when compared to the base- lines Tabll 5). Although the dataset is more balanced (54% of the values are 1), we provide AUROC and AUPR for completeness. Notice that this setting is more challenging since the information flows less granular level (i.c. group). Interestingly, even this case, VISL obtains similar or improved imputation results compared to the baselines.
Structure discovery results between groups. Most of the baselines used so far cannot be applied here because i) they cannot deal with missing data or ii) they cannot learn group-level relationships. DAG-GNN is the only one that can be adapted to satisfy both properties. For missing data, we adapt DAG-GNN following the same strategy as in VISL, i.e. replacing missing entries with a constant value. For the second one, we further adapt it by using group-specific mappings like VISL to cope with arbitrary groups We also include another baseline, Random, where the structures between topics are randomly sampled from a Bernoulli distribution. Due the lack of ground truth relationships, we ask two experts (teachers) to assess the validity of the relationships found by VISL, DAG-GNN, and Random. For each relationship, they are asked to rate a value from (strongly disagree) to (strongly agree) for the adjacency (whether it is sensible to connect the two topics) and the orientation (whether the first one is prerequisite for the second one). We release the the complete list of relationships and expert evaluations for VISL, DAG-GNN, and Random; see Table 15 Table 16 and Taale 17 respectively. Thus, the results is reproducible and allow the community to build on this data as a structure discovery benchmark. In summary, Table shows here the average evaluations: we see that the relationships discovered by VISI. score much more highly across both metrics than the baseline models.

---

Another interesting aspect is how the relationships between level-3 3 topies are distributed in higher- level topics. Intuitively, it expected that most of the relationships happen inside higher-level topies (e.g. Number-related concepts are more probably related to each other than to Geometry-rell
ones). [Table 7 in appendix shows the distribution for the compared methods. Indeed, notice that the percentage of inside-topic relationships is higher for VISL (82%) and DAG-GNN (42%) than for Random (34%). An analogous analysis for the 25 level-2 topics is provided in the appendix; see Table 12 (VISL), Table (DAGGGN), and Table 14 (Random). 6% of the connections happen inside level 2 topics for Random, is 14% for DAG-GNN and 36% for VISL.
Education Impact. Lastly, to make real-world impact, we have been provided with an additional education dataset in the same format as Eedi by an education organization help provide insight for math curriculum building. The final structure among all topics found by VISL is presented by figure Fig. in the appendix. This figure provides insights into which topics are foundational and need be covered earlier (topics with many originating edges), or later (topics with many incoming edges). This allowed us to re-evaluate the order of topics in a nationwide used secondary curriculum. Specifically, topics such "arithmetic" or "properties of shapes" were moved earlier in the curriculum, while topics such as "negative numbers" or "proportion and similarity" were moved to later stage in the curriculum. Another interesting example found by the domain expert are "Venn diagrams", which were ori originally taught in year 9/10 and are now suggested to be moved to year 7. Experts found that the topic "Venn diagram" has been a useful tool in teaching other topies which are currently taught before year 10. This emphasises the real-world impact our model can have in planning curricula.
5 Conclusions
We introduced VISL. novel approach that simultaneously performs group-wise structure discovery and learns to impute missing values. Both tasks are performed jointly: imputation is informed by the discovered relationships and vice-versa, leading to improved performance for both tasks. Moreover, motivated by a real-world problem, VISLshows its impact in the real-world education domain to aid domain experts in setting up curriculum. Despite of the improved performance on structure learning with missing data, VISL can be further extended several directions. First, one potential limitation is that the inferred structure from VISL is not causal. Appendix B shows that VISL satisfies the cauasl Markov assumption under equilibrium, which opens a door for potential causal claims. Another direction for future work is to extend the M(C)AR assumption random(MNAR), which can be more practical for real world usage. In the end, the performance can further be boosted by designing better graph posteriors beyond an independent Bernoulli distribution.
missing not at
Acknowledgments and Disclosure of Funding
PMA acknowledges project PY20_00288 funded by Junta de Andalucía.
References
[1] Bryon Aragam and Qing Zhou Concave penalized estimation of sparse gaussian bayesian networks. The Journal of Machine Learning Research, 16(1):2273-233 2015.
[2] S van Buuren and Karin equations in Journal of statistical software, pages 1-68, 2010.
mice: Multivariate imputation by chained
[3] David Maxwell Chickering. Optimal structure identification with greedy search. Journal of machine learning research, 3(Nov):507-555 55400 2002.
[4] Max Chickering, David Heckerman, and Chris Meek. Large-sample learning of bayesian networks is np-hard. Journal of Machine Learning Research, 5, 2004.
[5] Tom Claassen and Tom Heskes. A bayesian approach to constraint based causal inference. In Proceedings of the Twenty-Ein Eighth Conference on Uncertainty in Artificial Intelligence, pages 207-216, 2012.
[6] Chris Cundy, Aditya Grover, and Stefano Ermon. Bed nets: Scalable variational approaches for bayesian causal discovery. Advances in Neural Information Processing Systems, 34, 2021.
[7] James Cussens, David Haws, and Milan Studeny. Polyhedral aspects of score equivalence in bayesian network structure learning. Mathematical Programming, 164(1):285-322 2017.
10

---

[8] Hanjun Dai, Zoriitsa Kozareva, Bo Dai, Alex Smola, and Le Song. Learning steady-states of iterative algorithms over graphs. In International conference on machine learning. pages 1 1166-114, 14. PMLR, 2018.
[9] Arthur P Dempster. Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society: Series B (Methodological), 39(1):1-22, 1977.
[10] Bahare Fatemi, Layla El Asri, and Seyed Mehran Kazemi. Slaps: Self-supervision improves structure learning for graph neural networks. arXiv preprint arXiv:2102.05034, 2021.
[11] Fei Fu and Qing Zhou. Learning sparse causal gaussian networks with experimental intervention: regularization and coordinate descent. Journal of the American Statistical Association, 108 (501):288-300, 2013.
[12] Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Emre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, et al. Deep end-to-end causal inference. arXiv preprint arXiv:2202.02219 2022.
[13] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pages 1263-1272. PMLR, 2017.
[14] Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics, 10:524, 2019.
[15] Wenbo Gong, Sebastian Tschiatschek, Richard Turner, Sebastian Nowozin, José Miguel Hemmnndez-Lobato, and Cheng Zhang. Icebreaker: Element-wise active information acquisition with bayesian deep latent gaussian model. In Advances in Neural Information Processing Systems, 2019.
[16] Fangda Gu, Heng Chang, Wenwu Zhu, Somayeh Sojoudi. and Laurent El Ghaoui. Implicit graph neural networks. arXiv preprint arXiv:2009.06211, 2020.
[17] Jaaying Gu, Fei Fu, and Qing Zhou. Penalized estimation of directed acyclic graphs from discrete data. Statistics and Computing, 29(1):161-17)) 2019.
[18] David Heckerman, Christopher Meek, and Gregory Cooper. A bayesian approach to causal discovery. In Innovations in Machine Learning, pages 1-28. Springer, 2006.
[19] Patrik Hoyer, Dominik Janzing, Joris M Mooij. Jonas Peters, and Bernhard Schölkopf. Nonlinear causal discovery with additive noise models. Advances in neural information processing systems, 21:689-694 1968,666,2220 2008
[20] Oleg Ivanov, Michael Figurnov, and Dmitry Vetrov. Variational autoencoder with arbitrary conditioning, arXiv preprint 10X.0:1806.02382, 2018.
[21] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmm In International conference on learning representations, 2017.
[22] Marcus Kaiser and Maksim Sipos. Unsuitability of NOTEARS for causal graph discovery. CoRR, abs/2104.05441 2021. URL httpss//aaxxii.orr//b///000 .05441
[23] Diederik P Kingma, Max Welling, et al. An introduction to variational autoencoders. Founda- tions and Trends@ in Machine Learning, 12(4):307-39) 2019.
[24] D.P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv
2013.
[25] Trent Kyono, Yao Zhang, and Mihaela van der Schaar. Castle: Regularization via auxil- iary causal graph discovery. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1501-
1512. Curran Associates, Inc., 2020. URL http://pwpcccccoss.s.ooocccppppp 1168bceb193231er 25325fe2223344cc55554Pppppp
[26) Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient- based neural dag learning. arXiv preprint 17XXv:1900.0222666 2019.
[27] Andrew S Lan, Christoph Studer, and Richard G Baraniuk. Time-varying learning and content analyties via sparse factor analysis. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 452-461, 2014.
11

---

[28] Po-Ling Loh and Peter Bühlmann. High- fige--dimessioall learning of linear causal networks via inverse covariance estimation. The Journal of Machine Learning Research, 15()):3055-1155 2014.
[29] Lars Lorch, Jonas Rothfuss, Bernhard Schölkopf, and Andreas Krause. Dibs: Differentiable bayesian structure learning. Advances in Neural Information Processing Systems, 34, 2021.
[30] Chao Ma, Sebastian Tschiatschek, Konstantina Palla, Jose Miguel Hernandez-Lobato Sebastian Nowozin, and Cheng Zhang. EDDI: Efficient dynamic discovery of high-value information with partial VAE. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4234 4243. PMLR, 09-15 Jun 2019. URL http://prooceedinggssm press/v97/mal 19cml
[31] Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The conerete distribution: A continuous relaxation of discrete random variables. In International conference on learning representations, 2017.
[32] Pierre-Alexanrre Mattei and Jes Frellsen. Miwae: Deep generative modelling and imputation of incomplete data sets. In International conference on machine learning, pages 4413-4423, PMLR, 2019.
[33] Junpei Naito, Yukino Baba, Hisashi Kashima, Takenori Takaki, and Takuya Funo. Predictive modeling of learning continuation in preschool education using temporal patterns of devel- opment tests. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[34] Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using vaes. Puttern Recognition, 107:107501, 2020.
[35] Ignavier Ng, AmirEmad Ghassami, and Kun Zhang. On the role of sparsity and dag constraints for learning linear dags. Advances in Neural Information Processing Systems, 13:17743-17955 2020.
[36] Sascha Ott and Satoru Miyano. Finding optimal gene networks using biological constraints. Genome Informatics, 14: 123-1333 2003.
[37] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: founda- lions and learning algorithms. The MIT Press, 2017.
[38] Joseph Ramsey, Madelyn Glymour, Ruben Manchez-Roment and Clark Glymour. A million variables and more: the fast greedy equivalence search algorithm for learning high- digg--imnnsionll graphical causal models, with an application to functional magnetic resonance images. Interna- tional journal of data science and analytics, 3(2):121-129, 2017.
[39] Alexander G Reisach, Christof Seiler, and Sebastian Weichwald. Beware of the simulated dag! varsortability in additive noise models. arXiv preprint arXiv:21022 13647, 2021.
[40] Paul Rolland, Volkan Cevher, Mathhuss Kleindessner. Chris Russell, Dominik Janzing, Bernhard Schölkopf, and Francesco Locatello. Score matching enables causal discovery of nonlinear additive noise models. In International Conference on Machine Learning, pages 18741-18753. PMLR, 2022.
[41] Donald B Rubin. Inference and missing data. Biometrika, 63(3):581-592, 1976.
[42] Karen Sachs, Omar Perez, Dana Pe' Pe'er, Douglas A Lauffenburger, and Ganyy P Nolan. Causal protein-signaling networks derived from multiparameter single-cell data. Science, 308(5721): 523-529, 2005.
[43] Mauro Scanagatta, Cassio P de Campos, Giorgio Corani, and Marco Zaffalon. Learning bayesian networks with thousands of variables. In NIPS, pages 1864-1872, 2015.
[44] Mauro Scanagatta, Giorgio Corani, Cassio P De Campos, and Marco Zaffalon. Learning treewidth-bounded bayesian networks with thousands of variables. In NIPS, pages 1462-1470, 2016.
[45] Judi Scheffer.
Dealing with missing data. In Research Letters in the Information and Mathemat- ical Sciences, pages 153-160, 2002.
[46] Ajit P Singh and Andrew W Moore. Finding optimal Bayesian networks by dynamic program- ming. Citeseer, 2005.
12

---

[47] Peter Spirtes and Clark Glymour. An algorithm for fast recovery of sparse causal graphs. Social science computer review, 91):622-7, 1991.
[48] Peter Spirtes, Clark N Glymour, Richard Scheines, and David Heckerman. Causation, prediction, and search. MIT press, 2000.
[49] Daniel J Stekhoven and Peter Bühlmann. Missforest non--onnnnnnnn non-parametric missing value imputation for mixed-type data. Bioinformaties,
2012.
[50] Eric V Strobl, Shyam Visweswaran, and Peter L Spirtes. Fast causal inference with non-random missingness by test-wise deletion. International journal of data science and analytics, 6(1):
47-62, 2018.
[51] Marc Teyssier and Daphne Koller. Ordering-based search: A simple and effective algorithm for learning bayesian networks. arXiv preprint arXivv 1207. 1429, 2012.
[52] Ruibo Tu, Cheng Zhang, Paul Ackermann, Karthika Mohan, Hedvig Kjellström. and Kun Zhang. Causal discovery in the presence of missing data. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1762-1770. PMLR, 2019.
[53] Ruibo Tu, Kun Zhang, Bo Christer Bertilson, Hedvig Kjellström, and Cheng Zhang. Neuropathic pain diagnosis simulator for causal discovery algorithm evaluation. In 33rd Conference on Neural Information Processing Systems (NeurIPS), DEC 08-14, 20/9, Vancouver, Canada, volume 32. Neural Information Processing Systems (NIPS), 2019.
[54] Jill-Jênn Vie and Hisashi Kashima. Knowledge tracing machines: Factorization machines for knowledge tracing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 750-757, 2019.
[55] Jussi Viinikka, Antti Hyttinen, Johan Pensar, and Mikko Koivisto. Towards scalable bayesian learning of causal dags. Advances in Neural Information Processing Systems, 33:6584-6594, 2020.
[56] Zichao Wang, Sebastian Tschiatschek, Simon Woodhead, José Miguel Hernández-Lobato, Simon Peyton Jones, Richard Baraniuk, and Cheng Zhang. Educational question mining at scale: Prediction, analysis and personalization. arXiv preprint arXiv:2003.05990 2020.
[57] Zichao Wang, Angus Lamb, Evgeny Saveliev, Pashmina Cameron, Yordan Zaykov, Jose Miguel Hernandez-Lobato, Richard E Turner, Richard G Baraniuk, Craig Barton, Simon Peyton Jones, et al. Results and insights from diagnostic questions: The neurips 2020 education challenge. arXiv preprint urXiv: 21..21044..4444 2021.
[58] Andrew E Waters, David Tinapple, and Richard G Baraniuk. Bayesrank: A bayesian approach to ranked peer grading. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale, pages 177-183, 2015.
[59] Ga Wu, Justin Domke, and Scott Sanner. Conditional inference in pre-trained variational autoencoders via cross-coding. arXiv preprint 1805.07788 2018.
[60] Mike Wu, Milan Mosse, Noah Goodman, and Chris Piech. Zero shot learning for code education: Rubric sampling with deep learning inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 782-790, 2019.
[61] Yue Yu. Jie Chen, Tian Gao, and Mo Yu. Dag-gan Dag structure learning with graph neural networks. In Proceedings of the 36th International Conference on Machine Learning, 2019.
[62] Bin Zhang, Chris Gaiteri, Liviu-Gabriel Bodea, Zhi Wang, Joshua McElwee, Alexei A Podtelezh- nikov, Chunsheng Zhang, Tao Xie, Linh Tran, Radu Dobrin, et al. Integrated systems approach identifies genetic nodes and networks in late-onset alzheimer's disease. Cell, 153(3):707-720, 2013.
[63] Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. DAGs with NO TEARS: Continuous Optimization for Structure Learning. In Advances in Neural Information Processing Systems, 2018
[64] Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric P. Xing. Learning sparse nonparametric DAGs. In International Conference on Artificial Intelligence and Statisties, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See last paragraph Sec. I
(b) Did you describe the limitations of your work? [Yes] Include future work at the end of
Sec. 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [N/A]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes]] We will provide the main model code in the supplemental material. The full running code will be released after acceptance.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Appx D
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See|Sec.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See(Sec. D.2
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] The dataset details in Appx. D where we cite the creator.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] Only Eedi experiment involves human evaluation and provide detailed instructions in [Sec. 4.33
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "RLTBaPg5MXL", "reviewer": "Reviewer_bhzA", "summary": "This paper proposes a new approach for learning the structural dependencies between variables in the latent space that can also be used for imputing the missing values.", "questions": "* How to decide the number of groups $M$ when we do not know it in advance?\n* What if the missing values are also present in the training data?\n* How do the authors explain the huge difference between the performance of the proposed method and the baseline methods on the simulated data (Table 2)? Maybe an unfair simulation that favors the proposed method?\n* What is the computational complexity of the proposed method? How does it compare with baseline approaches?\n* What are the limitations of the proposed method?", "limitations": "The limitations of the proposed method are not discussed within the text and would be nice if the authors can add it to the revised version.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*** Strengths***\n* The text is well-written and very clear.\n* It addresses an important and well-defined problem.\n* The experimental results are compelling.\n\n*** Weaknesses***\n* The number of groups in data ($M$) is an essential hyperparameter for the proposed model. However, the authors provide no guideline for deciding it (within optimization? Model selection? Prior knowledge?). In a certain application, what if we do not know $M$ in advance?\n* It seems that we need to have access to complete data (without missing values) at the model training stage. This can be very restrictive in real applications. What if the missing values are also present in the training data? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IYK7zc6cvFq", "reviewer": "Reviewer_7oh7", "summary": "In this paper, the authors propose VISL, a structure learning method that simultaneously infers structures between groups of variables under missing data and performs missing value imputations.  The authors conduct extensive experiments on synthetic, semi-synthetic, and real-world education data sets and they show improved performances on both imputation and structure learning.\n", "questions": "The authors could add discussion and comparison with score matching based method [1]. \n\n[1] Rolland, Paul, et al. \"Score matching enables causal discovery of nonlinear additive noise models.\" arXiv preprint arXiv:2203.04413 (2022).\n\n\n", "limitations": "As mentioned above, the authors could present additional theoretical results on structure learning under missing values to make the paper stronger. Theoretical study on how missing values will affect the performance of structure learning could technically strengthen the paper. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\t1\tThe writing of the paper is good. Technical sections are easy to follow.\n\t3\tExtensive experiments have been conducted to validate the method.\n\nWeakness:\n\t1. The contribution and novelty of the paper are limited.  The model proposed in this paper basically combines existing techniques, i.e. structure learning regularization (NOTEARS) and missing value imputation, and uses variational inference for model parameter learning. \n\n        2. Technical depth of the paper is insufficient. The authors could present additional theoretical results on structure learning under missing values to make the paper stronger. Theoretical study on how missing values will affect the performance of structure learning could technically strengthen the paper. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-ad9Cn20mFz", "reviewer": "Reviewer_9NtP", "summary": "This work introduces VISL (missing value imputation with structural learning) which simultaneously should learn relations between groups of variables and do missing value imputation. The model consists of a Variational Autoencoder with a structure imposed on the latent space. Each group of variables gets its own latent space and the specific relation between the subspaces is treated as a latent variable itself. A GNN generator works on the latent variables to reconstruct the missing values.", "questions": "*Q1*:  \nI see some hints at interpreting the learnt structure causally, e.g.  \n\"However, it is commonly impossible to perform randomized controlled trials for many real-world applications due to ethical or cost considerations.\".  \nCan the learnt structure be interpreted causally, without any further assumptions? Could you be more clear about whether you are interpreting the structure causally and if so what allows you to do that?\n\n*Q2*:  \n\"where we assume the data are missing completely at random (MCAR) or missing at random (MAR)\"\nDo you have any notion whether your specific dataset is truly M(C)AR or if it could be MNAR? Could the missing mechanism depend on the missing values themselves? How would such a situation affect the learnt structure and the imputations?\n\n*Q3*:  \n\"To adapt the model to different missing levels in the training data X, we adopt the masking strategy [14, 27], which drops a random percent of the observed values during training.\" Does the model then need fully observed data for training?\n\n*Q4*:  \n\"it incorporates the learned structure G into the imputation, which helps the model avoid over-fitting due to spurious correlations [22]\". Is that really the case? Couldn't the model in a small data regime just overfit G as well?\n\nRegarding VAEs for missing data, perhaps you would find [1, 2] interesting as well.\n\n[1] Mattei, Pierre-Alexandre, and Jes Frellsen. \"MIWAE: Deep generative modelling and imputation of incomplete data sets.\" International conference on machine learning. PMLR, 2019.  \n[2] Ivanov, Oleg, Michael Figurnov, and Dmitry Vetrov. \"Variational autoencoder with arbitrary conditioning.\" arXiv preprint arXiv:1806.02382 (2018).  \n", "limitations": "The authors have addressed limitations", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Originality*:   \nThis work seems to connect density learning using VAEs with structure learning in the latent space in a novel way to do both structure learning and missing value imputation.\n\n*Quality*:  \nThe model and experimental setup is well described and most claims are backed up.\n\n*Clarity*:  \nI think the paper is well written and fairly well structured.\n\n*Significance*:  \nThe paper provides a way of handling missing values in structure learning, which is valuable in most real-life settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept