Title: Learning to Discover Sparse Graphical Models
Authors: Eugene Belilovsky
Kyle Kastner
Gael Varoquaux
Matthew B. Blaschko
Venue: ICLR 2017

Abstract: We consider structure discovery of undirected graphical models from observational data. Inferring likely structures from few examples is a complex task often requiring the formulation of priors and sophisticated inference procedures. In the setting of Gaussian Graphical Models (GGMs) a popular estimator is a maximum likelihood objective with a penalization on the precision matrix. Adapting this estimator to capture domain-specific knowledge as priors or a new data likelihood requires great effort. In addition, structure recovery is an indirect consequence of the data-fit term. By contrast, it may be easier to generate training samples of data that arise from graphs with the desired structure properties. We propose here to leverage this latter source of information as training data to learn a function mapping from empirical covariance matrices to estimated graph structures.  Learning this function brings two benefits: it implicitly models the desired structure or sparsity properties to form suitable priors, and it can be tailored to the specific problem of edge structure discovery, rather than maximizing data likelihood. We apply this framework to several real-world problems in structure discovery and show that it can be competitive to standard approaches such as graphical lasso, at a fraction of the execution speed. We use convolutional neural networks to parametrize our estimators due to the compositional structure of the problem. Experimentally, our learnable graph-discovery method trained on synthetic data generalizes well to different data: identifying relevant edges in real data, completely unknown at training time. We find that on genetics, brain imaging, and simulation data we obtain competitive(and generally superior) performance, compared with analytical methods. 

TL;DR: Sparse graphical model structure estimators make restrictive assumptions.  We show that empirical risk minimization can yield SOTA estimators for edge prediction across a wide range of graph structure distributions.  

Review discussion:
This paper proposes a new method for learning graphical models. Combined with a neural network architecture, some sparse edge structure is estimated via sampling methods. In introduction, the authors say that a problem in graphical lasso is model selection.
However, the proposed method still implicitly includes model selection. In the proposed method, $P(G)$ is a sparse prior, and should include some hyper-parameters. How do you tune the hyper-parameters? Is this tuning an equivalent problem to model section?
Therefore, I do not understand real advantage of this method over previous methods. What is the advantage of the proposed method?
Another concern is that this paper is unorganized. In Algorithm 1, first, G_i and \Sigma_i are sampled, and then x_j is sampled from N(0, \Sigma). Here, what is \Sigma? Is it different from \Sigma_i?
Furthermore, how do you construct (Y_i, \hat{\Sigma}_i) from (G_i, X_i )?
Finally, I have a simple question: Where is input data X (not sampled data) is used in Algorithm 1?
What is the definition of the receptive field in Proposition 2 and Proposition 3?
Thank you for your comments. We address each of them below
“In introduction, the authors say that a problem in graphical lasso is model selection.
However, the proposed method still implicitly includes model selection. In the proposed method, $P(G)$ is a sparse prior, and should include some hyper-parameters. How do you tune the hyper-parameters? Is this tuning an equivalent problem to model section?
Therefore, I do not understand real advantage of this method over previous methods. What is the advantage of the proposed method?”
In the Introduction Page 2, after Equation 1, we point to several problems  with a method such as the graphical lasso. a) It is difficult to embed new structured priors (e.g. small world graphs) within the penalty term and design optimization for them  b) model selection is unintuitive as it is controlled by the term \lambda in Equation 1. A final point (c) , from the abstract and Section 2.1, is that the edge recovery of graphical lasso and other methods is an indirect consequence of the objective, whereas we directly optimize an objective on edge recovery. This allows us to greatly outperform the graphical lasso.
With regard to (b), our point is simply that in many practical problems one has a reasonable range they can assume for the signal sparsity (or other hyperparameters such as number of hubs in small world graphs), yet there is no easy way to correspond these ranges to regularization terms and thus one often is forced to do cross-validation over the full range of lambda. In our setting, parameters such as the sparsity can be directly specified in an intuitive way (one can indicate for example they expect the signal to be 90-99% sparse).
We found that our method was extremely robust to model selection effects, but we do not rule out the possibility that further work on model selection in our framework could improve results even further. In our experiments, we were able to achieve excellent results, outperforming the graphical lasso and monte carlo based methods with a single trained model. We also note that for model selection we biased the graphical lasso against us in our experiments showing results for the optimal regularization parameter (on the test set).
“Another concern is that this paper is unorganized. “
We believe the organization to be reasonable but we are very open to correcting this if one can cite specific organizational issues.  We welcome further comments that can improve clarity.
“In Algorithm 1, first, G_i and \Sigma_i are sampled, and then x_j is sampled from N(0, \Sigma). Here, what is \Sigma? Is it different from \Sigma_i? “
We thank the reviewer for noting this indeed typographical mistake in Algorithm 1.
Indeed N(0,\Sigma) should read N(0, \Sigma_i.) We have corrected this error in the revision.
“Furthermore, how do you construct (Y_i, \hat{\Sigma}_i) from (G_i, X_i )?”
We try to maintain generality in the algorithm description.
Specifically in  the case of our experiments we construct  \hat{\Sigma}_i = X_i^TX_i/n and Y_i is defined from the graph G_i as in Equation (2). To put simply Y_i corresponds to the presence or absence of edges in the graph.
“ Finally, I have a simple question: Where is input data X (not sampled data) is used in Algorithm 1?”
Here is an important point of the paper. Algorithm 1 describes how to train the edge estimator using a prior distribution and does not utilize the final target input data at all.  At test time we show that this trained model can compute edges on a variety of data it has not seen before, and in fact in all our experiments we do not use any real data at all for training but obtain very impressive performance (in order magnitude less time) using both real data as well as synthetic data from completely different sampling packages that are often used to evaluate these algorithms.
“What is the definition of the receptive field in Proposition 2 and Proposition 3?”
We have added a definition to the revision. Here we refer to the part of the input seen at a "neuron" at a given layer. This is a commonly used term in some CNN literature but indeed should be defined.
Thank you again for your questions/comments and let me know if there is any other clarifications needed.
The paper proposes a novel algorithm to estimate graph structures by using a convolutional neural network to approximate the function that maps from empirical covariance matrix to the sparsity pattern of the graph. Compared with existing approaches, the new algorithm can adapt to different network structures, e.g. small-world networks, better under the same empirical risk minimization framework. Experiments on synthetic and real-world datasets show promising results compared with baselines.
In general, I think it is an interesting and novel paper. The idea of framing structure estimation as a learning problem is especially interesting and may inspire further research on related topics. The advantage of such an approach is that it allows easier adaptation to different network structure properties without designing specific regularization terms as in graph lasso.
The experiment results are also promising. In both synthetic and real-world datasets, the proposed algorithm outperforms other baselines in the small sample region.
However, the paper can be made clearer in describing the network architectures.
For example, in page 5, each o^k_{i,j} is said be a d-dimensional vector.
But from the context, it seems o^k_{i,j} is a scalar (from o^0_{i,j} = p_{i,j}). It is not clear what o^k_{i,j} is exactly and what d is. Is it the number of channels for the convolutional filters?
Figure 1 is also quite confusing. Why in (b) the table is 16 x 16 whereas in (a) there are only six nodes? And from the figure, it seems there is only one channel in each layer? What do the black squares represent and why are there three blocks of them. There are some descriptions in the text, but it is still not clear what they mean exactly.
For real-world data, how are the training data (Y, Sigma) generated? Are they generated in the same way as in the synthetic experiments where the entries are uniformly sparse? This is also related to the more general question of how to sample from the distribution P, in the case of real-world data.
Thank you for your review and comments.
With regards to clarity of the network architecture description in Section 2.3, we have made several revisions in the paper to improve the explanations and wording. You can find these in Figure 1 and where o^k_{i,j} is first introduced.
For o^k_{i,j} it is indeed a d-dimensional vector for all layers indexed above 0 (i.e l>0).
However in the initial layer it is simply from o^0_{i,j} = p_{i,j}, this is analogous to how CNNs in dense prediction tasks take in images while intermediate layers can be multidimensional. Here the dimensions of o^k_{i,j} correspond to filter outputs in a CNN.
Thus indeed d corresponds to channels in the CNN implementation.
For Figure 1 we have updated the description of (a) to be more explicit.  (a) is illustrating the nodes accessible by intermediate output o^1_{4,13}. In this first layer only edges from 6 nodes are used to construct o^1_{4,13}. Here the edge of interest (4,13) is shown in blue in Figure (a) and (b) and the edges processed to get o^1_{4,13} are shown in grey on both (a) and (b). This figure motivates pulling the entries near the diagonal as well instead of simply following the standard CNN structure since these allow enough information to determine, from o^1_{4,13}, a conditional independence of 4,13 given the adjacent nodes (5,14,3,12).
Finally we have decided to release a first version of the code (at this link https://github.com/eugenium/LearnGraphDiscovery), so that the implementation details can be ascertained.
With regard to training data. For the real-world data, we use the same training data and in fact for one case the same trained model as in the synthetic experiments. These models are trained using uniform sparsity. We considered this as performance on the synthetic task was quite strong and we wanted to show the generality of a trained model, reusing it for a variety of synthetic tasks and even real. In the case of real world data, both for our method and existing methods such as the graphical lasso, assumptions on the generating distribution must be (implicitly) made. In many cases these assumptions can come from prior knowledge about the problem (e.g. neuroscientific findings in the case of brain connectivity).  One possible interpretation of our results is that the generating distribution used seems to match well the real data as compared to the assumptions in graphical lasso.
Let me know if there are any other clarifications or questions.
I sincerely apologize for the late-arriving review.
This paper proposes to frame the problem of structure estimation as a supervised classification problem. The input is an empirical covariance matrix of the observed data, the output the binary decision whether or not two variables share a link. The paper is sufficiently clear, the goals are clear and everything is well described.
The main interesting point is the empirical results of the experimental section. The approach is simple and performs better than previous non-learning based methods. This observation is interesting and will be of interest in structure discovery problems.
I rate the specific construction of the supervised learning method as a reasonable attempt attempt to approach this problem. There is not very much technical novelty in this part. E.g., an algorithmic contribution would have been a method that is invariant to data permutation could have been a possible target for a technical contribution. The paper makes no claims on this technical part, as said, the method is well constructed and well executed.
It is good to precisely state the theoretical parts of a paper, the authors do this well. All results are rather straight-forward, I like that the claims are written down, but there is little surprise in the statements.
In summary, the paper makes a very interesting observation. Graph estimation can be posed as a supervised learning problem and training data from a separate source is sufficient to learn structure in novel and unseen test data from a new source. Practically this may be relevant, on one hand the empirical results are stronger with this method, on the other hand a practitioner who is interested in structural discovery may have side constraints about interpretability of the deriving method. From the Discussion and Conclusion I understand that the authors consider this as future work. It is a good first step, it could be stronger but also stands on its own already.
Thank you for your review and comments. We are largely in agreement regarding the contributions of the paper.
With regards to permutation invariance of the architecture, we did indeed consider several possibilities prior to our final model.
For example we looked at architectures with recurrent computations and attention mechanisms that can potentially be more invariant.
However, at least at this point such approaches are less efficient especially in larger graphs, and more importantly we have found that our current architecture naturally provides us an easy way to perform ensembling to additionally improve results. We can see in a new appendix (A.4) we have added that in fact the outputs for permutation have uncorrelated errors, while we have seen individually they still produce strong classification results.
At the same time the speed of our method makes ensembling a very practical approach.
With regards to interpretability of the method in practice, it is notable that graphical lasso for example has guarantees under restricted distribution assumptions.
On the other hand our method could be made to give empirical guarantees, potentially under more loose distribution assumptions or ones more interesting to the user, if they can construct a way to sample from them or obtain ground truth real data.  There are also notably many recent approaches which attempt to understand the predictions of neural networks and other “black box” models, including convolutional neural networks. Recent examples include [1] LIME and [2] the DeepVis toolbox. These techniques and future ones being developed in the community could be utilized with our method potentially helping explain for example which correlations help explain others.
It is also of note that in many applications one might be willing to sacrifice interpretability of the deriving process for speed as well as accuracy guarantees under a specific model. These include applications such as real-time connectivity analysis [3], robust estimation and portfolio optimization in finance. In other cases, there may not be an existing method that is scalable and allows the needed prior, thus an approach such as ours may be the only route.
[1]Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin. “Why should I trust you?” Explaining The Predictions of Any Classifier.  KDD 2016
[2] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding Neural Networks Through Deep Visualization. http://yosinski.com/deepvis
[3] Smith, Anne.  Near-real-time connectivity estimation for multivariate neural data.
We thank the reviewers for their comments and for providing us with excellent feedback. We have updated the paper with clarifications in Section 2.3 as well as an Appendix (A.4) with some additional experiments which analyze permuted inputs. We have also made an early release of our code available at https://github.com/eugenium/LearnGraphDiscovery.

Meta-review:
role: PC
data: comment: The authors provide a modern twist to the classical problem of graphical model selection. Traditionally, the sparsity priors to encourage selection of specific structures is hand-engineered. Instead, the authors propose using a neural network to train for these priors. Since graphical models are useful in the small-sample regime, using neural networks directly on the training data is not effective. Instead, the authors propose generating data based on the desired graph structures to train the neural network. 
 
 While this is a nice idea, the paper is not clear and convincing enough to be accepted to the conference, and instead, recommend it to the workshop track.

Decision: Invite to Workshop Track