Neural Graphical Models
Authors: Harsh Shrivastava
Urszula Chajewska
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graphical models
Deep learning
Learning Representations

Abstract: Graphs are ubiquitous and are often used to understand the dynamics of a system. Probabilistic Graphical Models comprising Bayesian and Markov networks, and Conditional Independence graphs are some of the popular graph representation techniques. They can model relationships between features (nodes) together with the underlying distribution.  Although theoretically these models can represent very complex dependency functions, in practice often simplifying assumptions are made due to computational limitations associated with graph operations. This work introduces Neural Graphical Models (NGMs) which attempt to represent complex feature dependencies with reasonable computational costs. Specifically, given a graph of feature relationships and corresponding samples, we capture the dependency structure between the features along with their complex function representations by using neural networks as a multi-task learning framework. We provide efficient learning, inference and sampling algorithms for NGMs. Moreover, NGMs can fit generic graph structures including directed, undirected and mixed-edge graphs as well as support mixed input data types. We present empirical studies that show NGMs' capability to represent Gaussian graphical models, inference analysis of a lung cancer data and extract insights from a real world infant mortality data provided by CDC.

Under review as a conference paper at ICLR 2023
NEURAL GRAPHICAL MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Graphs are ubiquitous and are often used to understand the dynamics of system. Probabilistic Graphical Models comprising Bayesian and Markov networks, and Conditional Independence graphs some of the popular graph representation techniques. They can model relationships between features (nodes) together with the underlying distribution. Although theoretically these models can represent very complex dependency functions, in practice often simplifying assumptions are made due computational limitations associated with graph operations. This work introduces Neural Graphical Models (NGMs) which attempt to represent complex feature dependencies with reasonable computational costs. Specifically, given a graph of feature relationships and corresponding samples, we capture the dependency structure between the features along with their complex function representations by using neural networks as a multi-task learning framework. We provide efficient learning, inference and sampling algorithms for NGMs. Moreover, NGMs can fit generic graph structures including directed, undirected and mixed- edge graphs as well as support mixed imput data types. We present empirical studies that show NGMs' capability to represent Gaussian graphical models, inference analysis of a lung cancer data and extract insights from real world infant mortality data provided by CDC.
Keywords: Graphical models, Deep learning, Learning representations SoftwareNGGM code link (provided in Supplementary)
INTRODUCTION
Graphical models are a powerful tool to analyze data. They can represent the relationship between the features and provide underlying distributions that model functional dependencies between them. Probabilistic graphical models (PGMs) are quite popular and often used to describe various sys- tems from different domains. Bayesian networks (directed acyelic graphs) and Markov networks (undirected graphs) are able to represent many complex systems due to their generic mathematical formulation Pearl (1988); Koller Friedman (2009). These models rely on conditional independence assumptions to make representation of the domain and the probability distribution over it feasible.
Learning, inference and sampling are operations that make such graphical models useful for domain exploration. Learning. in broad sense, consists of fitting the distribution function parameters from data. Inference is the procedure of answering queries in the form of marginal distributions or reporting conditional distributions with from the underlying distribution defined by the graphical model. One of the common bottlenecks of graphical model representations having high computational complexities for one or more of these procedures. Figuring out approximate algorithms or coming up with analytically favorable underlying distributions have been lopies of interest the research community for the past few decades.
more observed variables. Sampling is the ability draw samples
In particular, various graphical models have placed restrictions on the set of distributions or types of variables in the domain. Some graphical models work with continuous variables only (or categorical variables only) or place restrictions on the graph structure (e.g., that continuous variables cannot be parents of categorical variables in DAG). Other restrictions affect the set of probability distributions the models are capable of representing, e.g., to multivariate Gaussian.
Practically, for graphical models to be widely adoptable, the following properties are desired:
Facilitate rich representations of complex underlying distributions.
Support various relationship representations including directed, undirected, mixed-edge graphs.

---

Under review as a conference paper at ICLR 2023
Fast and efficient algorithms for learning, inference and sampling.
Direct access to the learned underlying distributions for analysis.
Handle different input data types such categorical, continuous, images, text, and generic embed- ding representations.
In this work we propose Neural Graphical Models (NGMs) that satisfy the aforementioned desiderata in computationally efficient way. NGMs accept feature dependency structure that can be given by an expert or learned from data. The dependency structure may have the form of a graph with clearly defined semantics (e.g., Bayesian network graph or a Markov network graph) or an adjacency matrix. Note that the graph may either directed or undirected. Based on this dependency structure, NGMs represent the probability function over the domain using deep neural network. The parameterization of such a network can be learned from data efficiently, with a loss function that jointly optimizes adherence the given dependency structure and fit to the data. Probability functions represented by NGMs are unrestricted by any of the common restrictions inherent in other PGMs. They also support efficient inference and sampling.
The rest of this paper is organized as follows: in Section 2 we briefty review work most closely related to ours, in Section 3 we introduce Neural Graphical Models including representation, learning, inference, sampling and handling of extended data types. We present experiments, both on synthetic and real -lfee data in Section 4 and Appendix B, discuss design considerations and limitations of our framework in Appendix A and close with conclusions and directions for future work in Section 5.
RELATED WORK
Probabilistic graphical models aim to learn the underlying joint distribution from which input data is sampled. Often, to make learning of the distribution computationally feasible, inducing an independence graph structure between the features helps. In cases where this independence graph structure is provided by domain expert, the problem of fitting PGMs reduces to learning distributions over this graph. Alternatively, there are many methods traditionally used to jointly learn the structure as well as the parameters Heckerman et al. (1995); Spirtes & Meek (1995); Koller & Friedman (2009); Scanagatta et al. (2019) and have been widely used to analyse data in many domains Barton et (2012); Bielza & Larrañaga (2014); Borunda et al. (2016); Shrivastava et al. (2019a); Shrivastava (2020).
A few researchers explored discriminative PGMs, learning not joint probability distributions over domain, but an approximation to a conditional distribution P(Y - -Y) where Y is pre-selected subset of X, typically in the context of undirected graphs. The best known conditional random fields (CRF) Lafferty et al. (2001). Discriminative models are more flexible in ignoring complex dependencies between most of the variables in the domain and focusing on their impact on a small subset. They often have faster and more accurate inference, albeit restricted to the pre-sclected set of variables. Generative models have higher bias they make more assumptions about the form of the distribution. The bias helps with regularization and avoiding overfitting. However, generative models are poorer predictors than discriminative models. In this work, we attempt to combine the advantages of both methods by creating discriminative model capable of predicting the value of any variable in a domain.
Recently, many interesting deep learning based approaches for DAG recovery have been pro- posed Zheng et al. (2018; 2020); Lachapelle et al. (2019); Yu et al. (2019). These works primarily focus the structure learning but technically they are learning probabilistic graphical model. These works depend on the existing algorithms developed for the Bayesian networks for the inference and sampling tasks. A parallel line of work combining graphical models with deep learning Bayesian deep learning approaches: Variational AutoEncoders, Boltzmann Machines etc. (Wang & Yeung, 2020). The deep learning models have significantly more parameters than traditional Bayesian networks. Thus, using these deep graphical models for downstream tasks is computationally expensive and often impedes their adoption.
We would be remiss not to mention the technical similarities NGMs have with some recent research works. First, we found 'Learning sparse nonparametric DAGs' Zheng al. (2020) to be the closest in terms of representation ability. In one of their versions, they model each independence structure with a different neural network (MLP). However, their choice of modeling feature independence

---

Under review as a conference paper at ICLR 2023
fi (23, 224)
22
fa (233
a²s
fa 11 22, n4
R2
24
fa (I1, F3)
Ss (x3)
The inpul graph G (undirected) for given input data X E MRX Each feature = fa (z))) is function of the neighboring features. For a DAG, the functions between features will be defined by the Markov Blanket relationship fa (MB(2) The adjacency matrix (right) represents the associated dependency structure
Figure I: Graphical view of NGMs:
Hidden
Input
Output
fi fi
22 f2(#s)
22
33
23 fa 22,,
24 f1(21,
35
fs(ms)
W1
W
Figure 2: Neural view of NGMs: NN multitask learning architecture capturing non-linear dependencies is path the imput feature an oulput feature,
for features the undirected graph Fig. that indicates a dependency between them. The dependency matrix between the input and output of the neural network reduces a simple matrix multiplication operation S.n II; [W| = [W,l [Wa]. Note that not all the zeroed out weights of the MLP (in black-dashed lines) shown for the sake of clarity.
critcrion differs from NGM. They zero out the weights of the row in the first layer of the NN to induce independence between the input and output features. This type of formulation restricts them from sharing the NNs across different factors. Second, we found similar path norm formulations of using the product of NN weights for input to output connectivity for NGMs in Lachapelle et al. (2019). They use the path norm to parametrize the DAG constraint for continuous optimization, while Shrivastava al. (2020; 2022b) use the within unrolled algorithm framework to learn sparse gene regulatory networks.
There are methods that model the conditional independence graphs (Friedman al., 2008; Belilovsky et al., 2017; Shrivastava et al., 2019b; 2022a) which are a type of graphical models that are based on underlying multivariate Gaussian distribution. Probabilistic Circuits (Peharz al., 2020), Conditional Random Fields Markov Networks (Sutton et al., 2012)are some other popular formulations. These PGMs often make simplifying assumptions on the underlying distributions and have certain restrictions on the input data type that can be bandled. Real-world input data often consist of mixed datatypes (real, categorical, text, images ctc.) and is challenging for the cxisting graphical model formulations to handle.
NEURAL GRAPHICAL MODELS
We propose a new probabilistic graphical model type, called Neural Graphical Models (NGMs) and describe the associated learning, inference and sampling algorithms. Our model accepts all input types and avoids placing any restrictions on the form of underlying distributions.

---

Under review as a conference paper at ICLR 2023
3.1 PROBLEM SETTING
We given imput data X that have M sample points with each sample consisting of D features. An example of such data can be gene expression data, where data is a matrix of the microarray expression values (samples) and genes (features). Another example is mix of continuous and categorical data describing a patient's 's health in medical domain. We are also provided graph G which can be directed, undirected or have mixed-edge types that represents our belief about the feature dependency rclationships (in probabilistic sensc). Such graphs are often provided by experts and include inductive biases and domain knowledge about the underlying system functions. In cases where the graph is not provided, we make use of the late-of-the-are algorithms to recover DAGs or CI graphs, as described Sec. 2. The NGM input is the tuple (X, G).
3.2 REPRESENTATION
Algorithm 1: NGMs: Learning algorithm
Fig. shows a sample graph recovered and how we view the value of cach feature function of the values of its neighbors. case of directed graphs, each feature's 's value is represented as a function of its Markov blanket in the graph. We the graph G to understand the domain's 's dependency struc- ture, but ignore any potential parametrization associated with it.
Function prox prroxmal-iinis (X
fw + Init MIP using dimensions from S fw f-
the
arg minw Σk- |X x fx
(Using 'adam' optimizer for Ej epochs) return fwo
F Function fit-NGM (X, S, fwo, For e = 1,.., E2 do
We introduce 'neural' view which is another way of looking at G. represented in Fig. 2. These neural networks are multi-layer per- ceptrons with appropriate input and output dimensions that represent graph connections in NGMs. Specifically, we view the neural networks as an 'open-box' and focus on the paths from input to output. These paths repre- sent functional dependencies. Consider a neu- ral network with H number of layers having ReLU non-linearity Wa.., WH (X+)=
L1 Σ k= X
fw (Xx)
+x- -- (TI, [W,
1
W <- backprop L1, to update params
(optional À update)
x + [[[I,]W5 1
Detach X from the computational graph
return e, Z
Function NGM learning (X,S): fwo + www.iialliiiiiiiiooe -nit (X
(... (W2- ReLU(W, X b1) + ..).. The dimensions of the weights and biases are chosen such that the neural network input and output units are equal to The prod- uet of the weights of the neural networks
X (II, W 50
fw fit NGM (X, fwo, X9)
return fw
S.n = IT, W. = |W1 [Wa] x x |Wel gives us path dependencies. Sun [x,, ra] = then the output va does not depend on input 24. Increasing the layers and hidden dimensions of the NNs will provide us with richer dependence function complexities.
Representing categorical variables. Assume that in the input X. we have column X. having [C] different categorical entrics. One way to handle categorical input is to do one-hot encoding on the column X- and end up with C| different columns, X. = [Xe, _e..........................__.. .xeel We replace the single categorical column with the corresponding one-hot representation in the original data. The MLP capturing path dependencies S will need to be updated accordingly. Whatever connections where previously connected to the categorical column X, should be maintained for all the one-hot columns as well. Thus, connect all the one-hot columns represent the same path connections as the original categorical column.
3.3 LEARNING
Using the rich and compact functional representation achieved by using the 'neural' view, the learning task fit the neural networks to achieve the desired dependency structure S. along with fitting the regression to the input data X. Given the input data X we want to learn the functions as described by the NGMs 'graphical-view; Fig. These can be obtained by solving the multiple regression

---

Under review as a conference paper at ICLR 2023
problems shown in neural view, Fig. 2. We achieve this by considering the neural view multi-tase learning framework. The goal is to find the set of parameters (W) that minimize the loss expressed as the distance from X to Tw(X5, while maintaining the dependency structure provided in the input graph G. We can define the regression operation as follows:
El fw (x
arg W
s.t. [
Here, S represents the compliment of the matrix S, which essentially replaces 0 by and vice-versa. The A B represents the hadamard operator which does an element-wise matrix multiplication between the same dimension matrices A, B. Including the constraint as a lagrangian term with & penalty and a constant that acts a tradeoff between fitting the regression and matching the graph dependency structure, get the following optimization formulation
arg min W
Σ X fw(xt)) I(IL/W.)) * 1
Though the bias term not explicitly written in the optimization to avoid cluttering, we learn the weights {W;} and the biases (b;) while optimizing for Eq. 2. In our implementation, the individual weights are normalized using n-rn before taking the product. We normalize the regression loss and the structure loss term separately, that both the losses are on similar scale while training and recommend the range of >=[1e-2, Ic2]. Appropriate scaling is applied to the inpul data features.
Proximal Initialization strategy: To get good initialization for the NN parameters W and À we implement the following procedure. We solve the regression problem deseribed in Eqn. with- out the structure constraint. This gives good initial guess of the NN weights W We choose the value A || (n, (WT) 50 and update after each epoch. Experimentally, we found that this strategy may work optimally in few cases and in such cases we recommend fixing the value of the beginning of the optimiza- tion. The value of À can be chosen such that it brings the regression loss and the structure loss values to same scale.
Algorithm 2: NGMs: Inference algorithm
Function pradient-basee
(fw, Xy);
{X,,
X } <
X,, split the data
X₃ fixed tensor (known)
Xu + learnable tensor (unknown)
fw freeze weights
do
X, + {X,, Xu}
X = fw((1)
L ||Xplk] [
X, < updated by backprop on Lin
while L., > return X
The learned NGM describes the underlying graphical model distributions, as presented in Alg. 1. There are multiple benefits jointly optimizing in multi-task learning framework modeled by the neural view of NGMs, eq. 2. First, sharing of parameters across tasks helps in significantly reducing the number of learning parameters. also makes the regression task more robust towards noisy and anomalous data points. Second, we fully leverage the expressive power of the neural networks to model complex non-linear dependencies. Additionally, learning all the functional dependencies jointly allows us to leverage batch learning powered with GPU based scaling to get quicker runtimes.
Function mes message-passiln (fw, ) Xx X + X split the data t=
while | x1- do
(Xt; Xx) fw Xx))
t= 1=t+1 t+1
X + XK + +X return X
Function NGM- nov--infeee ference ( (sw, X Input: fw trained NGM model X E Rx1 (mean values for unknown)
X -essaag-passingg (fw. X
... or
X x g-rrddi ennt based (fw, return X
3.4 INFERENCE
Inference is the process of using the graphical model to answer queries. Calculation of marginal distri- butions and conditional distributions are key operations for inference. Since NGMs are discriminative
5

---

Under review as a conference paper at ICLR 2023
models, for the prior distributions, we follow the frequentist approach and directly calculate them from the input data. We consider two iterative procedures to answer conditional distribution queries over NGMs described in Alg. 2. We split the input data X Xxr f- X into two parts, k denotes the known (observed) variable values and u denotes the unknown (target) variables. The inference task is to predict the values of the unknown nodes based on the trained NGM model distributions. In the fist approach, we use the popular message passing algorithms that keeps the observed values of the features fixed and iteratively updates the values of the unknowns until convergence, We developed an alternative algorithm which is efficient and is our recommended approach to do inference in NGMs.
Gradient based approach: The weights of the trained NGM model are frozen once trained. The imput data is divided into fixed X₂ (observed) and learnable X. (target) tensors. We then define regression loss over the known attribute values as we want to make sure that the prediction matches values for the observed features. Using this loss we update the learnable input tensors until convergence to obtain the values of the target features. Since the NGM model is trained to match the output to the input, can view this procedure of iteratively updating the unknown features such that the input and output matches. Based on the convergence loss value reached after the optimization, one can assess the confidence in the inference. Furthermore, plotting the individual feature dependency functions also helps in gaining insights about predicted values.
Obaining probability distributions. It is of- ten desirable to get the full probability density function rather than just point value for any inference query. In case of categorical vari- ables, this is readily obtained as we output distribution over all the categories. For real or numerical features, we consider a binned input the input side and real value out- put. In this case, the regression term of the loss function, Eq. 3 will take binned input and output a real value for the real valued fea- (proj(X 2-xomm) (i--mmmm)))
Algorithm 3: NGMs: Sampling algorithm
Function get-sample (fw, Pa):
D = len(D))
X E Px1 (random init, learnable tensor)
Sample 1st feature value from empirical marginal distribution 31 (P(z)))
For i 2,,.., D do
Xx <- X(1
1] (fixed tensor) X. < X(i: D] (learnable tensor)
X + {Xx, Xx)
tures Σx=1 || In practice, given a distribution over different categories obtained during the NGM inference, we clip the individual values between [c. 1] and then divide by the total sum to get the final dis-
X +- NGM-inferencce(fw:
X ~ U(P(s;
return X
Function NGM-sampl sampl 1 ing (/w, G) Input: fw trained NGM model Randomly choose x, 'th feature D,=BFS(G.zz.. [undirected] queue the features ...
tribution.
3.5 SAMPLING
D, (opplooiccal-soot(() [DAGs] X f- gegt -smplee (sw, D.)
One common way of sampling is define cu- mulative density functions and then sample from them. This will not be possible for NGMs. So, instead, we propose a procedure akin to Gibbs sampling as described in Alg. 3.
return X
We based our sampling procedure to follow X, ~ U[fnn (xbrs(X)))), Note that nbrs(X)) will be MB(X,) for DAGss We start sampling by choosing a feature at random. To get the order in which the features will be sampled, we do a Breadth-first--eard (topological sort in DAGs) and arrange the nodes in D... In this way, the immediate neighbors are chosen first and then the sampling spreads over the graph away from the starting feature. As we go through the ordered fcatures in the sampling procedure, we sample the value of cach fcature from the conditional distribution based on previously assigned values and then keep it fixed for the subsequent iterations (feature is now observed). We then call the inference algorithm conditioned on these fixed features to get the distributions over the unknown features. This process is repeated till we get sample value of all the features.
Our sampling procedure differs from the Gibbs sampling with regards to conditional distribution calculations. Traditionally, in Gibbs sampling, sample X* is derived from the previous sample X by following a conditional distribution update. Specifically, the value of X is obtained according to the distribution specified by (X/1 X X 1 x 1 ..... x--1). The new sample of the NGM is not derived from the previous sample, hence we avoid the 'burn-in' period issue with Gibbs
6

---

Under review as a conference paper at ICLR 2023
Hidden
Output
Input
7
dj
Dec
21
fi 24)
271
Ern
272
e2
d2
22 fa(zs)
373
E3
d3
33 12
C4
d4
24 fa(z1, va)
375
Enes
E6
dg
Dees
25
fs(Øs)
W1 W
Figure 3: Neural view with Projection modules NGMs: The input X can one (categorical), image or in general an embedding (text, audio, speech and other data types). Projection modules (encoder + decoder) are used as wrapper around the neural view of NGMs. The architecture choice of the projection modules depends on the input data type and users' design choices. Note that the output of the encoder can be more than unit (e1 can be hypermode). In that case, wc just need adjust the graph dependency structure account for that many umits and the corresponding feature connections. Same will be the case with the decoder side of the architecture. The remaining details are similar to the ones described in Fig. 2
sampling where one has to ignore the initial set of samples. The conditional updates for the NGMs We keep on fixing the value of features and run inference on the remaining fcatures until we have obtained the values of all the features and thus get new sample. The inference algorithm of the NGM facilitates conditional inference on multiple unknown features over multiple observed features. We leverage this capability of the inference algorithm for faster sampling from NGMs.
are of the form, (X,. x X X .... x
3.6 EXTENSION TO GENERIC DATA TYPES
The learning, inference and sampling algorithms proposed for NGMs in the previous section be extended to any generic input data lype. This implies that the data X can be real, categorical, image or have an embedding based representation. We add Projection module consisting of an encoder and decoder that act as wrapper around the neural view of the NGMs. With a slight modification, we obtain the following optimization for generic data types,
M
Σ k=1
x - (w(proj(x))) + IIT W.)) S1
arg min W.proj
The Projection module can be jointly learned in the optimization, as shown in Eq. 3, or one can add fine-tuning layers to the pretrained versions depending on the data type and user preference.
Alternatively, one can extend the idea of soft-thresholding the connection patterns to the encoder and decoder networks. This will result an efficient training strategy that leverages batch processing.
M
x (wx/) + An. [ S
arg min W W Mya
[[T 5 + Xa [W,)) SSll,
where, the connectivity of the input v and the input the neural view is modeled by the & sparsity term for the encoder network's 's sparsity pattern Sy Similar procedure is applied to the decoder side.
If the Projection modules used, the number of nodes in the neural view input should be adjusted according to the output units of the encoder. Similar adjustment is needed for neural view output and the decoder. In real world applications, For instance, in the gene expression data, there can be additional meta information (categorical) or images associated with the genes. Optionally, one can desire utilize node embeddings from some other pretrained deep learning models. NGMs are designed to handle such mixed input data types simultaneously which are otherwise very tricky to accommodate the existing graphical models.
often find inputs consisting of mixed datatypes.

---

Under review as a conference paper at ICLR 2023
GGM Chain graph
CI graph recovered from NGM samples
Noural Graphical Model
Gaussian Graphical Model
n8
n9
n4
n3
no
n5
n6
n5
n4
n1
n2
no
no
n9
n1
n2
n8
Figure 4: The leftmost graph shows the chain graph G (partial correlations in green are positive, red are negative, thickness shows the correlations strength) obtained from the initialized partial correlation matrix. Samples X 122000x10 were drawn from the GGM. NGM was learned the input (X. G). The plots the right show the dependency functions NGM and GGM and negative correlations are reflected in the slope of the curve, as expected analytically. We then sampled from the learned NGM to obtain data Xs E It M, 10 The graph, second from the left, shows the recovered graph by running uGLAD Shrivastava al. (2022a) on Xs. We can observe that it missed some of the edges but most of the connections along with the correlations signs were retrieved from the NGM samples.
particular node by varying its neighbor's values. The positive
EXPERIMENTS
We evaluate NGMs on synthetic and real data. Appendix A contains some best practices that we developed while working with NGMs. In Appendix B, we present an analysis of CDC's Infant Mortality Data (of Health et al.) using NGMs, which highlights NGMs-generie architecture's 's ability to model mixed input datatypes.
4.1 MODELING GAUSSIAN GRAPHICAL MODELS
We designed a synthetic experiment to study the capability of NGMs to represent Gaussian graphical models. The aim of this experiment is to see (via plots and sampling) how close are the distributions learned by the NGMs the GGMs.
Table 1: The recovered CI graph from NGM samples compared with the CI graph defined by the GGMs precision matrix. Area under the ROC curve (AUC) and Area under the precision-recall curve (AUPR) values for 10 runs are reported, refer to Fig. 4.
Samples
AUPR
AUC
1000
0.84 = 0.03
0.91 = 0.002
2000
0.86
0.02
0.93
0.001
4000
0.96 0.00
0.99 0.003
Setup: Define the underlying graph. We defined a 'chain' (or path-graph) containing D nodes the underlying graph. We chose this graph as it allows for an easier study of dependency functions.
Fit GGM and get samples. Based the underlying graph structure, defined precision matrix θ that randomly samples its entries from Oz,; ~ (1((1, 10..-0.. 0.5) U (0.5, 1))- We then used this precision matrix as multivariate Gaussian distribution parameter to obtain the input sample data X. We get the corresponding partial correlation graph G by using the formula, Px..x, XDWI
Ve. 9,.
Fit NGM and get samples. We fit a NGM on the input (X, G). We chose H = 30 with 2 layers and non-linearity tanh for the neural view's MLP. Training done by optimizing eq. for the input, refer to Fig. 4. Then, we obtained data samples Xs from the learned NGM.
Analysis: 'How close are the GGM and NGM samples?' We recover the graph using the graph recovery algorithm uGLAD on the sampled data points from NGMs and compare it with the true CI graph. Table shows the graph recovery results of varying the number of samples from NGMs. We observe that increasing the number of samples improves the graph recovery, which is expected.
'Were the NGMs able model the underlying distributions? The functions plot (on the right) in Fig. 4 plots the resultant regression function for particular node as learned by NGM. This straight line with the slope corresponding to the partial correlation value is what we expect theoretically for the GGM chain graph. This is also an indication that the learned NGMs were trained properly and reflect the desired underlying relations. Thus, NGMs are able to represent GGM models.

---

Under review as a conference paper at ICLR 2023
Figure 5: (left) The CI graph recovered by uGLAD for the Lung cancer data. Plots on the right show the condi- tional distribution for the features P(Lung cancer= aencr='Yes'll nbrs(Lung cancer)) and P(Smokingl nbrs(Smoking)) based on their neighbors. We used 2 layer NGM with hidden size H 30 and non linearity tanh. NGMs are able to capture the non-linear dependencies between the features. Interestingly the NGMs match the relation- ship trends discovered (positive and negative correlations) by the corresponding CI graph.
4.2 LUNG CANCER DATA ANALYSIS
We analysed a lung cancer data on Kaggle using NGMs. The effectiveness of cancer prediction system helps people know their cancer risk with low cost and also helps people take appropriate decisions based on their cancer risk status. This data contains 284 instances of patients and for cach patient features (Gender, Smoking, Anxiety, Lung cancer present, etc.) are collected. Each entry a binary entry (YES/NO) or in some cases (AGE), entries are binarized. Particularly, we used NGMs to study how different features are related and discover their underlying functional dependencies.
The input data along with the CI graph recovered using uGLAD were used to learn a NGM in Fig. 5. In order to gauge the regression quality of NGMs, gression to predict the probability of fcature values given the values of the remaining features. Table. 2 shows regression results of logistic regression (LR) and NGMs on different features, 'lung cancer' & "smoking". The prediction probability for NGMs were calculated by running inference each test datapoint, eg. P(lung-cancer saaenncrrrreesssl f: my Vi in test data). This experiment primarily demonstrates that single NGM model can robustly handle fitting multiple regressions and one can avoid training a separate regression model for each feature while maintaining al-par performance. Furthermore, we can obtain the dependency functions that bring in more interpretability for the predicted results, Fig. 5. Samples generated from this NGM model can be used for multiple downstream analyses.
Methods
Lung-cancer
Smoking
0.95 + 0.02 0.71 0.01
compare with logistic re- NGM
0.96 + 0.01
0.79 0.02
Table 2: 5-fold CV results.
5 CONCLUSIONS
This work attempts to improve the usefulness of probabilistic graphical models by extending the range of input data types and distribution forms such models can handle. Neural Graphical Models provide compact represenlation for a wide range of complex distributions and support efficient learning, inference and sampling. The experiments are carefully designed to systematically explore the various capabilities of NGMs. Though NGMs can leverage GPUs and distributed computing hardware, we do forsee some challenges in terms of scaling in number of features and performance very high-ddmensional data. Using NGMs for images & text based applications will be interesting to explore. We believe that NGMs is an interesting amalgam of the deep learning architectures' expressivity and Probabilistic Graphical models' representation capabilities.
Upcoming version: Discovering the dependency graph with NGMs. We are currently working on version of NGM that can jointly discover the feature dependency graph along with fitting the regression. One way can be to optimize this loss function,
arg min
x [w(proj(X [[ * Saiag l
(5)
W.proj
where Saing has diagonal entries 1. Essentially, we start with fully connected graph and then the term induces sparsity. This will be helpful in cases where input not provided.

---

Under review as a conference paper at ICLR 2023
REFERENCES
David N Barton, Sakari Kuikka, Olli Varis, Laura Uusitalo, Hans Jørgen Henriksen, Mark Borsuk, Africa de Hera, Raziyeh Farmani, Sandra Johnson, and John DC Linnell. Bayesian networks environmental and resource management. Integrated environmental assessment and management, 8(3):418-429, 2012.
Eugene Belilovsky, Kyle Kastner, Gaël Varoquaux, and Matthew B Blaschko. Learning to discover sparse graphical models. International Conference on Machine Learning, pp. 440-448. PMLR, 2017.
Sakyajit Bhattacharya, Vaibhav Rajan, and Harsh Shrivastava. Icu mortality prediction: classification algorithm for imbalanced datasets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Concha Bielza and Pedro Larrañaga. Bayesian networks in neuroscience: survey. Frontiers in computational neuroscience, 8:131, 2014.
Mónica Borunda, OA Jaramillo, Alberto Reyes, and Pablo Ibargüengoytia. Bayesian networks in renewable energy systems: A bibliographical survey. Renewable and Sustainable Energy Reviews, 62:32-45, 2016.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Biostatistics, 9(3):332-44) 2008
David Heckerman, Dan Geiger, and David M Chickering. Learning Bayesian networks: The combination of knowledge and statistical data. Machine learning, 20(3):19--43,, 1995.
Kaggle. Lung Cancer.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Sébastien Lachapelle, Philippe Brouillard, Tristan Deleu, and Simon Lacoste-Julien. Gradient- based neural dag learning. arXiv preprint arXiv: (rXXiv:906.0222)) 2019.
John D. Lafferty, Andrew McCallum, and Fernando Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, 2001.
United States Department of Health, Centers of Disease Control Human Services (US DHHS), and Division of Vital Statistics (DVS) Prevention (CDC), National Center for Health Statistics (NCHS). Birth Cohort Linked Birth Infant Death Data Files, 2004-2015. Compiled from data provided by the 57 vital statistics jurisdictions through the Vital Statistics Cooperative Program, on CDC WONDER On-line Database. Accessed ht https: www. edc. gov /oov/ / aataaaaaaaaasaaeaaaaaaaaaaa vitaistatsonling htm.
Judea Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 1988.
Robert Peharz, Steven Lang, Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Guy Van den Broeck, Kristian Kersting, and Zoubin Ghahramani. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In International Conference Machine Learning, pp- 7563-7574. PMLR, 2020.
Samyam Rajbhandari, Harsh Shrivastava, and Yuxiong He. Antman: Sparse low-rank compression to accelerate mnn inference. arXiv preprint arXiv:1910.01744 2019.
Mauro Scanagatta, Antonio Salmerón, and Fabio Stella. A survey bayesian network structure learning from data. Progress in Artificial Intelligence, 8(4):25-439, 2019.
10

---

Under review as a conference paper at ICLR 2023
Marco Scutari. Learning bayesian networks with the balearn R package. Journal of Statistical Software, 3-(3):1-2,, 2010. doi: 10. 00.18637/jss.v3335.03.
Harsh Shrivastava. On Using Inductive Biases for Designing Deep Learning Architectures. PhD thesis, Georgia Institute of Technology, 2020.
Harsh Shrivastava, Vijay Huddar, Sakyajit Bhattacharya, and Vaibhav Rajan. Classification with imbalance: A similarity-based method for predicting respiratory failure. In 2015 IEEE international conference on bioinformatics and biomedicine (BIBM), pp. 707-714. IEEE, 2015.
Harsh Shrivastava, Eugene Bart, Bob Price, Hanjun Dai, Bo Dai, and Srinivas Aluru, Cooperative neural networks (conn): Exploiting prior independence structure for improved classification. arXiv preprint arXiv:1906.0029) 2019a.
Harsh Shrivastava, Xinshi Chen, Binghong Chen, Guanghui Lan, Srinvas Aluru, Han Liu, and Le Song. GLAD: Learning sparse graph recovery. arXiv preprint arXiv: I 1906/000000 000711 2019h. 9b.
Harsh Shrivastava, Xiuwei Zhang, Srinivas Aluru, and Le Song. Grnular: Gene regulatory network reconstruction using unrolled algorithm from single cell ma-sequeneing data. bioRxiv, 2020.
Harsh Shrivastava, Urszula Chajewska, Robin Abraham, and Xinshi Chen. uGLAD: Sparse graph recovery by optimizing deep unrolled networks. arXiv preprint arXiv:2205. 2022a.
Harsh Shrivastava, Xiuwei Zhang, Le Song, and Srinivas Aluru. Grnular: A deep Icarning framework for recovering single-cell gene regulatory networks. Journal of Computational Biology, 29(1): 27 27-44, 2022b.
Peter Spirtes and Christopher Meek. Learning bayesian networks with discrete variables from data. In KDD, volume 1, pp. 294-299, 1995.
Charles Sutton, Andrew McCallum, et al. An introduction to conditional random fields. Foundations and Trends Machine Learning, 4(4):267-373, 2012.
Hao Wang and Dit- Yan Yeung. survey on bayesian deep learning. ACM Computing Surveys (CSUR), 53(5):1-37, 2020.
Yue Yu, Jie Chen. Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International Conference on Machine Learning, pp. 7154-7163. PMLR, 2019.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. DAGs with NO TEARS: Continuous optimization for structure learning. Advances Neural Information Processing Systems, 31:9472 0002-94833111 2018.
Xun Zheng, Chen Dan, Bryon Aragam, Pradeep Ravikumar, and Eric Xing. Learning sparse nonparametrie DAGs. International Conference on Artificial Intelligence and Statistics, pp- 3414-3425, PMLR, 2020.


Reviews:
[{"review_id": "bW9rhM4qiz", "reviewer": "Reviewer_1gso", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel class of probabilistic models, dubbed Neural Graphical Models (NGMs), which parametrize the distribution with neural networks. This setup supports mixed continuous/discrete distributions and arbitrary graphical structures.", "summary_of_the_review": "While this work is addressing a relevant problem and the key idea behind NGMs is reasonable, I cannot recommend the current version of the paper for publication.", "strength_and_weaknesses": "Strengths:\n+ Hybrid continuous/discrete probabilistic modelling has application in many scientific areas\n+ I think that parametrizing the factors of a PGM with neural networks is a sensible idea\n\nWeaknesses (see section below for details):\n- Some important claims are not supported\n- The presentation of the proposed approach should be improved\n- Relevant work in this area is not considered\n- The experimental section is weak", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TZ8g0woxyq0", "reviewer": "Reviewer_6f9Q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method of learning, performing inference in, and simulating from graphical models (undirected, directed, or a combination) using neural networks.  The authors claim that this greatly increases the flexibility while keeping computational complexity relatively low.\n\nI have some expertise in graphical models, but limited experience with neural networks, some of the comments below may be erroneous. ", "summary_of_the_review": "Overall, there just isn't enough here for me to recommend that the paper be accepted.  The method might be very good, but the authors haven't presented any evidence that it is, and the little they do provide suggests that it might not be that great.\n\n\n**Other Coments**\n\nPage3: \"conditional independence graphs...are a type of graphical models that are based on the multivariate Gaussian distribution.\" - This is not true, conditional independence graphs can represent any distribution, including non-parametric models.\n\n\n_Minor Points_\nMinor Points:\n - There are quite a lot of places where references should be in parentheses but are not.\n - page 3 - after (Sutton et al., 2012) missing a space.\n - \"$S^c$ essentially replaces 0 by 1 and vice versa\" - be precise here please.  If it does this other than on the diagonal, then say so.\n - In the next paragraph, \"Hadamard\" and \"Lagrangian\"\n - Equation (2) seems a bit basic.\n - don't write [1e-2, 1e2] but rather $[10^{-2}, 10^2]$.\n - page 6: \"fist\" -> \"first\"\n - \"we clip the individual values between $[\\epsilon, 1]$\": what is $\\epsilon$?\n - Why is it not possible to use CDFs with neural networks? (e.g. Chilinski and Silva, 2020)\n - page 6: \"till\" -> \"until\"\n - page 8: \"we randomly sample entries in the precision matrix between $(-1, -0.5) \\cup (0.5, 1)$...\" How do you guarantee positive definiteness?  Does this work precisely because the graph is a chain?\n - page 9: \"a lung cancer data on Kaggle\" -> \"a lung cancer dataset on Kaggle\"\n - page 9: please make your plots bigger; put most of them in the appendix and select a few to have in Figure 4.\n - several places on page 9: can you please write probabilities in mathmode? \n - Please ensure that the capitalization in your references is correct.  You can get it right by adding {} around words that are correctly written in the bibtex-entry.\n", "strength_and_weaknesses": "The paper is fairly clearly written, and comes with code that appears to run OK.  (Though there is a variable `model_NGM_trained` that is not supplied and breaks the attached Jupyter Notebook.) \n\nThe main weakness in my view is the lack of experiments comparing the method in the paper to the many others that the authors mention. \n\nThe only comparison that is made is learning a graph and its distribution from multivariate Gaussian data.  The recovered graph (done with uGLAD, an existing method) introduces three false edges, and the dependencies that the method in this paper discovers have the correct sign, but the magnitude of the dependence (seen in the thickness of edges) is very poorly recovered.  The accompanying plots are so small that the **massively** different scales on the y-axes are hard to see (and they are only given for two pairs of variables chosen by the authors).  This is generally not a good sign, and given that the sample size is 2000 it seems a little worrying (as does the edge between nodes 0 and 2, which is comparatively strong).\n\nThe claims made about speed and flexibility are not really supported in any experiments; no runtimes are given at all.  There are also not any experiments for inference or sampling with directed graphs, in spite of claims being made that this method can also deal with them.\n\nA lot of statements are made about benefits of joint minimization that are not supported by evidence (neither simulations nor citations).  \n\nMy **main feedback** is: do more comparisons (you can put some in the appendix) and give evidence for your claims.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Sa2VY7POwm", "reviewer": "Reviewer_3Bs7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides an approach to represent a probabilistic graphical model of relationships between features and a dataset over the same features using a neural network. This neural network can then be used draw samples from the joint distribution of the features, and it can be used to do conditional inference.\n\nThe contributions of the paper include the neural network learning algorithm, an algorithm to infer the values of unknown variables given some observed variables, and an algorithm to draw samples from the graph. The work is evaluated on a synthetic dataset by comparing pairwise correlations of sampled variables and on kaggle dataset by comparing to logistic regression.\n\n", "summary_of_the_review": "Nice tidy idea for representing PGMs in a neural network, but a lot of unclear or unsupported claims around representing uncertainty and mixed datatypes.\n", "strength_and_weaknesses": "The paper has clearly taken on an ambitious problem which attempts to unify generative and discriminative modeling with the efficiency of neural network inference. They provide a very intuitive representation of the graphical model in a neural network and devise easy to comprehend objective functions that are used for training the neural networks.\n\nThe paper doesn't seem to have hit the mark in terms of representing the uncertainty in a probabilistic graphical model. The description in the paragraph titled \"Obtaining probability distributions\" is completely unclear. How do you get a probabilistic output by inputting a binned value and what is a binned value in any case. The section on sampling, 3.5, describes drawing a full joint distribution and not really a conditional distribution. Even this description is somewhat unclear. The line, \"we sample the value of each feature from the conditional distribution based on previously assigned values\" is not clear. I understand how we can draw the value of one feature from the previously assigned value, but how do we get the conditional distribution of a feature? None of the experiments seem to cover the case of conditional distribution. Showing that joint samples retain pairwise correlations, Figure 5, is reassuring but doesn't prove anything about conditional distributions and neither does the accuracy results on a regression task, Table 2.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Cv87ZzqgLR", "reviewer": "Reviewer_ZfGg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nIn this paper the authors propose a new probabilistic graphical model representation called neural graphical models (NGMs). NGMs encode the joint probability distributions over random variables using multi-layer perceptrons. An NGM takes as input data and a graph that represents the probabilistic dependencies among the random variables. The graph structure is often provided by a domain expert or learned from the data itself.  A regression loss is minimized subject to the constraint that if xi and xj are neighbors in the input graph, then there is a path of non-zero weights from input xi to the corresponding output xj in the NGM.  As a result of such weight matrices, the outputs of NGMs represent regression functions over those input variables that are direct neighbors of the output variable. The main motivation around this representation is to be able to use a single representation that can accommodate various types of random variables as well as structural dependencies such as directed, undirected or mixed types. The authors describe inference and sampling procedures for these models and empirically test their performance on recovering a gaussian graphical model and on predicting lung cancer from a real world dataset. \n", "summary_of_the_review": "I found the empirical evaluations for this paper to be insufficient. A large body of work exists for learning various types of probabilistic graphical models and proposing a general representation like NGMs that can support all kinds of dependencies and data types as well as being efficiently learnable and queryable, requires more analysis. The authors have presented results on recovering a simple gaussian graphical model. These results don't seem to empirically validate the claim that these models can be very expressive and represent rich and complex distributions. An important question also arises on how the number of hidden layers affect the accuracy of the recovered distributions. \n\nThe lung cancer results seems to compare logistic regression accuracy to MLP accuracy on predicting cancer. The results are quite obvious. Unless more sophisticated inference tasks are carried out using NGMs, its hard to understand their expressive power. \nThe scenarios in which an NGM would be a better choice than a Bayesian network or a Markov network is not quite well understood. For example, Given the structure of a discrete Bayesian network, one can efficiently learn its parameters if the model has a bounded number of parents. Why would one use an NGM in such scenarios?   \n \nI am not sure why the authors have mentioned Gibbs sampling. The algorithm they mention generates samples in order similar to importance sampling (see Gogate, \"Sampling Algorithms for Probabilistic Graphical Models with Determinism\", PhD Thesis, 2009).\n\nSome other very important relevant work are:\nUria, Benigno, et al. \"Neural autoregressive distribution estimation.\" The Journal of Machine Learning Research 17.1 (2016): 7184-7220.\nUria, Benigno, Iain Murray, and Hugo Larochelle. \"RNADE: The real-valued neural autoregressive density-estimator.\" Advances in Neural Information Processing Systems 26 (2013).", "strength_and_weaknesses": "Strengths:\nA new representation to encode probability distributions using neural networks. \nThe model has the capacity to encode various types of dependencies and data types. \nThe authors have described both learning and inference algorithms. \nWeaknesses: \nWeak experimental evaluation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject