GLAD: Learning Sparse Graph Recovery

Abstract: Recovering sparse conditional independence graphs from data is a fundamental problem in machine learning with wide applications. A popular formulation of the problem is an ℓ1 regularized maximum likelihood estimation. Many convex optimization algorithms have been designed to solve this formulation to recover the graph structure. Recently, there is a surge of interest to learn algorithms directly based on data, and in this case, learn to map empirical covariance to the sparse precision matrix. However, it is a challenging task in this case, since the symmetric positive definiteness (SPD) and sparsity of the matrix are not easy to enforce in learned algorithms, and a direct mapping from data to precision matrix may contain many parameters. We propose a deep learning architecture, GLAD, which uses an Alternating Minimization (AM) algorithm as our model inductive bias, and learns the model parameters via supervised learning. We show that GLAD learns a very compact and effective model for recovering sparse graphs from data.

Reviews and author responses:

We thank all the reviewers for their constructive comments and helping us improve our paper. Listing down a summary of changes made in the new version based on the recommendations received:
1. Made the explanation in our introduction more descriptive for clarifying the doubts raised about input and output of the problem.
2. As suggested, we cited the Belilovsky et al. (2017) paper in our related works section.
3. Added further clarifying points on the choice of loss function and the training methodology (penultimate paragraph of section2 and last two paragraphs of section3.3)
4. Added a new section 3.4 on explanation of why GLAD is more than just learning regularization parameters. We wrote that section based on the questions asked by the reviewers and using the constructive discussions following them.
5. We added to the main text on how we ensure SPD matrix in our experiments. (section 5.2, 1st paragraph.)
6. A new Appendix C.12 is added. It describes our approach on the real E.Coli dataset along with the experimental results (Table 4).
7. A new Appendix C.13 is added. It describes our proposed approaches to scale GLAD for large matrices.
8. We fixed some minor typos.

---------
Latest update: fixed some more typos and added references to the newly added appendices in the main text.

This paper proposes an approach to data driven edge recovery for sparse gaussian mrfs. The authors propose an AM procedure for solving the l1 regularized maximum likelihood which can be unrolled and parameterized. This method is shown to converge faster at inference time than other methods and it is also far more effective in terms of training time compared to an existing data driven method. The authors provide a theoretical analysis which explains how the AM procedure should succeed and some insights on how it can potentially converge better using an adaptive model (motivating the learning part).

Experiments:
Overall the experiments demonstrate the method is superior. I have a few comments/concerns however:
-Scalability to larger graphs. The graphs used here a relatively small. I would like to see how well this scales to larger graphs at least in principal if not experimentally. Is there any issues that make this difficult? For example more iterations might be needed for convergence and this becomes problematic for learning. Can this already compete  with large scale methods like BigQUIC.
-Are the training graphs always the same distribution as the test graphs (e.g. in terms of the sparsity level)? It would be good to evaluate how well the model works when the training conditions differ to testing, since applying it to real data would require this gap.
-Closely related to the above if I have understood correctly all the experiments including the gene networks are on synthetic data, it would be good however to see if synthetic data can help generalize to real data.
- How many iterations are used to train the model? Is the number of iterations ever more at inference than training? It seems the NMSE is increasing after hitting a bottom I am wondering if that is related to mismatch in the number of iterations in test/train
- (minor) the authors compare wall clock time per iteration in Table 2 however their method converges much faster, it would be good to also see the overall clock time for each method after some reasonable stopping crieria, to show how big the overall gain is. 

Related Work:
The overview of sparse graph recovery for Gaussian random variables is good and concise. However I found the high level motivations given in Introduction/Sec 3 are similar (at times even the wording) to those of Belilovsky et al 2017 which introduced/motivate the data driven approach to this problem. Although this reference is used in the experimental section, it would be appropriate to clarify the difference/contribution compared to this work in the Intro, Sec3, and/or related work as a naive reading of the paper incorrectly suggests it is the first to consider a data driven approach to this problem.


Other comments:
- In equation (9) \Theta^{*(i)} should there be an (i) index there, I suspect this is a typo but  if it is not can the authors explain how the target differs for different (i)
- The data generating process described in Sec 5, how does it assure SPD, the described procedure (sampling off diagonal entries U(-1,1) then randomly setting zeros) does not appear to me to assure SPD without further constraints.
- I appreciated Appendix C10/C11 the overview of other attempts to parametrize the inference procedure

Overall I found this work relevant, the formulation well motivated, and potentially of high impact for the community working on inference of sparse conditional independence structure. I would give the score at the moment between weak accept and accept. There is a few points which I would like the authors to clarify or correct in their rebuttal and I would be happy to increase my score.

-----Post Rebuttal
The authors have addressed my primary concerns and revised the text, I am thus increasing my score. I recommend the authors also itemize the main changes in the text from the initial manuscript as they are not easy to find right now using the openreview revision comparison system (which seems to be broken). For example its not clear if the results shown within text are in the new manuscript.

***Q4: How many iterations are used to train the model? Is the number of iterations ever more at inference than training? … 

One way to view GLAD model is to unroll the AM optimization to a fixed number of iterations and consider this whole unrolled architecture as a highly structured deep model. Hence, we use the same number of iterations for training as well as testing. If not mentioned otherwise, we unrolled for 30 iterations in our experiments.


***Q5: It seems the NMSE is increasing after hitting a bottom...

Yes, we also observed this issue. We think this might be caused by our choice of the discounting factor in equation 9. We choose $\gamma=1$ which is also a default choice in some related works (e.g., [1]). It gives the output at each step an equal weight. With a smaller $\gamma$, the outputs at later steps can gain more weights. In this case, it is possible that the trained network will output a progressively closer approximation of the ground truth and result in a smoother trajectory.

[1] Andrychowicz et al. “Learning to learn by gradient descent by gradient descent.” NeurIPS’16.


***Q6: (minor) the authors compare wall clock time per iteration …

We have added the time/iteration needed with varying number of nodes in Table2. To get a rough estimate of overall clock time for inference, we can estimate it from the number of iterations the algorithm runs. For instance, in Fig.4 the x-axis gives the number of iterations needed for different methods.  


***Related Work:

The reviewer says:
“However I found the high level motivations given in Introduction/Sec 3 are similar (at times even the wording) to those of Belilovsky et al 2017 which introduced/motivate the data driven approach to this problem.”
We analyse the similarities and differences of the motivation for our work from Belilovsky et al 2017. 

*Similarity: 
The motivation of learning a direct mapping from empirical covariance matrices to estimated graph structures and to use a data-driven approach for the same has been introduced by Belilovsky et al 2017. e have now added that acknowledgment that in our related works.

*Key Difference in our introduction/motivation:

Apart from motivating the use of data-driven approaches, we specifically list down the *challenges* of learning a direct mapping for the sparse graph recovery problem, including problem size, SPD constraints, permutation invariance, etc. The motivation of our design is to learn a mapping in a way which also addresses the aforementioned challenges. The architecture in Belilovsky et al 2017 do not directly address these challenges and thus, we can observe that some of these requirements/constraints are not inherent in their architecture. 


***Other comments:

- In Eq(9), that is a typo. It should only be \Theta^{*}. (Fixed)

- To ensure SPD,.. an appropriate multiple of the identity matrix was added to the current matrix, so that the resulting matrix had the smallest eigenvalue as 1. We had mentioned it in appendix C.1, and have now added it to the main text for the sake of completeness. 

- Thanks for your appreciation. We ran several other parameterization attempts and we believe that this information will be of help to researchers for further pursuing this line of approach. In our theoretical analysis, we also prove the linear convergence of the AM approach, so that we can use a fixed number of iterations to obtain results with reasonable error margins. This further facilitated its use as an unrolled algorithm.

We sincerely thank the reviewer for the diligent reading and constructive comments! Addressing the concerns raised:

Q1: Scalability to larger graphs. The graphs used here a relatively small. I would like to see how well this scales to larger graphs at least in principal if not experimentally?..
A1: This is an interesting question and we are actively looking for techniques to scale our method. With our current implementation, we have added results scaling up to 1081 nodes (Appendix C.12).

First, for the training stage, we can train our network on small graphs. Since there is no problem-size dependent component in our neural network design (refer Algorithm 1), we can directly apply the trained network to large graphs during testing. We have done some experiments to validate this viewpoint. (Sec 5.5 and newly added Appendix C.12).

Therefore, our primary focus is to scale the inference part. We proposed 2 scaling approaches in appendix C.13 which are at a nascent stage. An overview is given below:
1. Distributed Implementation of AM algorithm: Since, the sizes of the trained neural networks are extremely small, we can have multiple copies of it distributed among different processors. We are investigating into parallel MPI based algorithms for this task (https://stanford.edu/~boyd/admm.html is a good representative reference.)
2. Randomized algorithms for scaling: The underlying idea is to use length-squared sampling and approximate the computations involving very large matrices with their low rank approximations. We take inspiration from Kannan & Vempala “Randomized algorithms in numerical linear algebra.”, 2017 work.

We will highly appreciate your feedback and any other leads on scaling our method.


*** Q2: Are the training graphs always the same distribution as the test graphs(e.g. in terms of the sparsity level)? …

In the Ecoli subnetwork recovery experiment (section 5.5), the train and test distributions are different. The GLAD model was trained using SynTReN simulator on graphs of 25 nodes drawn from a particular sparsity & noise setting. The test graphs are Ecoli subnetworks which have 43 nodes and a different sparsity pattern. 


*** Q3:  ...to see if synthetic data can help generalize to real data?

You raise a valid concern about generalization. To see whether synthetic data can help generalize to real data, we added the results on the real gene expression data gathered from the microarray experiments of E.Coli bacteria (Appendix C.12). 

The E.coli dataset contains 4511 genes and 805 associated microarray experiments. The true underlying network has 2066 discovered edges and 150214 pairs of nodes do not have an edge between them. There is no data about the remaining edges. We take a subset of 1081 genes having non-zero degrees as the underlying gene network for the E.coli bacteria. Recovering this graph is a challenging task.

We trained GLAD on 50 node graphs using the SynTReN simulator and used it to predict the gene regulatory network for the E.Coli bacteria.The AUC values are reported below

Methods     BCD      GISTA      GLAD
AUC           0.548      0.541       0.572

Although GLAD performs better than BCD and GISTA, we understand that it is far from satisfactory. However, we also note that the gene network recovery becomes even more difficult due to the limited availability of microarray experiments (only 805 observations for more than 1000 genes). The results also depends on the simulator settings which needs to be properly adjusted. It will be an interesting study to see the performance improvement by training models using different synthetic simulators.


We sincerely thank the reviewer for the helpful comments. Addressing the concerns raised:

*** Q1: the meaning of the input and output of the problem, input covariance and output precision matrix.

Thank you for pointing out this clarity issue. Given a task (e.g. an optimization problem), an algorithm will solve it and output a solution. Thus we can view an algorithm as a function mapping, where the input is the *task-specific information* (i.e. the sample covariance matrix in our case) and the output is the *solution* (i.e. the estimated precision matrix in our case). 

This might be not obvious to audiences unfamiliar with data-driven algorithm design, so we have rephrased our statement and updated our draft accordingly.


*** Q2: why RNN and deep Q-learning is related

We view an algorithm as a function mapping between the input and output. We benefit from the expressive power of neural networks and use them to represent these function mappings. Many algorithms update the solution iteratively in a recursive fashion, which is similar to the structure of RNN updates. A straightforward choice of the DL model for learning an algorithm is an RNN, refer [1]. To solve discrete optimization problems, typically the algorithm needs to take a sequence of discrete actions. The result of sequential discrete actions are not easy to be modeled by a neural network which is continuous in nature. Therefore, it is formulated as RL problems and DQN has been used for learning the policy to solve discrete optimization problems (eg. [2]). We mentioned RNN and DQN because they are both used for data-driven algorithm design in existing literature.

[1] Andrychowicz, Marcin, et al. "Learning to learn by gradient descent by gradient descent." Advances in neural information processing systems. 2016.
[2] Khalil, Elias, et al. "Learning combinatorial optimization algorithms over graphs." NeurIPS’17.
 

*** Q3: It is not clear whether section 2 contains new elements or whether the new contribution is entirely in section 3.
In Section 2, we provide a succinct overview of the problem formulation, existing algorithms, and analysis. The new element in Section 2 is our justification on ‘why we need to use learning for the sparse graph recovery problem’, where we pointed out the limitations of the MLE formulation (Eq1): 
- The consistency of its solution requires carefully chosen conditions.
- There is a mismatch in the maximum likelihood objective (Eq 1) and the final recovery error (Eq. 9) as evident in our experiment in section 5.1

These highlighted drawbacks indicates the room for improvement and motivates us to pursue learning-based approach. 


*** Q4: Please motivate the use of CNN in section 3.1…

Section 3.1 highlights the challenges in designing the DL models for recovering the precision matrix from the input covariance matrix. We ruled out fully connected DNNs due to the quadratic scaling of number of parameters. The next obvious choice is to design CNN based architecture (which is also adopted in [Belilovsky et al. (2017)] for sparse graph recovery). Hence, we mention about CNN based approaches and rule them out because they fail to handle the permutation invariance and SPD constraint. 

In conclusion, we mention both DNN and CNN in Section 3.1 as examples to better illustrate the challenges in designing the learning model for this problem.


*** Q5:  eq (6): the methodology of this formulation can be better positioned with respect to the existing literature….

Thank you for your suggestion. Decoupling the optimization objective into two terms and then alternatively updating them is a popular technique to make the optimization problem easier to solve. We have added a reference of ADMM based methods in section 2 for readers to get familiar with the general idea of such techniques.


 The paper proposes a neural network architecture to address the problem of estimating a sparse precision matrix from data (and therefore inferring conditional independence if the random variables are gaussian).

The authors base their algorithm in the semidefinite relaxation by Banerjee et al. They add a regularization terms and penalization parameters, which they learn using neural networks. They consider an alternating minimization implementation similar to ADMM and the neural networks are only used to find the regularization parameters.

In order to learn the parameters, the training optimizes the regularization parameters that maximize the recovery objective function (meaning how far is the estimated precision matrices from the true given precision matrices) and doesn’t consider the sparsity. 

Something that is not a priori obvious is the setting of using a family of precision matrices from a family of graphs and trying to learn an underlying precision matrix (by averaging them?). Further explanation of beginning of section 3 would be useful. 

Something else that is not clear to this reviewer is the motivation for the loss (9). If the objective is to find the parameters that maximize the recovery objective without taking the sparsity into consideration then why not choose them that way in (1), why there should be learning involved? And what is the learning exactly pursuing? Is it trying to learn a way to combine the information from the different samples consistently? [I acknowledge this is probably a naive question, but maybe addressing this in section 3.3 will help understanding].

I think the overall idea is interesting. Regularization parameters are usually problematic because it is not obvious how to choose them. Having an automatic, data-driven way to choose them is a useful algorithm design tool. The objective pursued in the choice of the loss function is a key concept of the paper and I believe it is not clearly explained. Explaining this point in a convincing way will improve the paper and my assessment from weak reject to strong accept. I suggest cutting the introduction to half and use that space to justify and explain sections 3 and 3.3 in depth.  

---
Edit: I thank the authors and reviewer 1 for their explanations. I changed my rating to accept. I think it would be useful for the readers to include some of these remarks in the paper.


*** More than Finding Single Regularization Parameters *** 

We thank Reviewer1 for his comments and explanations. Elaborating on his response, our proposed method is not just a simple solver to Banerjee objective with custom regularization term. We used an unrolled AM algorithm for Eq (1) as the template for designing the architecture. However, the designed architecture, is more flexible than just learning the regularization parameters. 

First, the component in GLAD architecture corresponding to the regularization parameters are entry-wise and also adaptive to the input problem (covariance) and intermediate outputs. This is achieved by parametrizing them as the outputs of neural networks. One can also think that after learning, GLAD architecture can adaptively choose a matrix of regularization parameters. This task will be very challenging if the matrix of regularization parameters are tuned manually using cross-validation. In addition, after we designed GLAD architecture, we found a theoretical work [7] studying a matrix of adaptive regularization parameters, which show that this scheme is better than a single fixed parameters, and validates the reasonability of our design.

[7] Sun, Q., Tan, K. M., Liu, H., & Zhang, T. (2018). Graphical nonconvex optimization via an adaptive convex relaxation. In ICML.

Second, as mentioned in Sec 5.1, 5.3 that for other methods like GISTA, BCD, we also use the validation set and loss(9) to tune the regularization parameter, based on grid-search. If GLAD architecture is only tuning a single regularization parameter, its performance won’t be that much better than these baselines. The superior performance of GLAD architecture is due to the fact that it allows a more flexible set of adaptive regularization parameters produced by learned deep model. 


We thank the reviewer 4 & 1 for the helpful comments and discussions. 

*** Questions for Objective (9) ***

I. The learning paradigm in objective (9)

We appreciate Reviewer1’s careful reading of our work. The objective should be understood in a similar way as in Gregor & Lecun ICML10 (LISTA) [1], Belilovsky et al. ICML’17 [2], and Liu et al. ICLR’19 (ALISTA) [3], where deep architectures are designed to directly produce the sparse outputs. In this setting, the deep architecture will take a covariance matrix as input and directly output a sparse precision matrix. When the input covariance matrix is different, it will potentially produce a different output precision matrix. That is the deep architecture can be viewed as a learned algorithm/optimizer which can be applied to different sparse recovery problems. Other related papers include Andrychowicz et al. NeurIPS’16 [4] and Ke & Malik ICLR’17 [5]. 

[1] Gregor, Karol, and Yann LeCun. "Learning fast approximations of sparse coding." ICML’10.
[2]  Belilovsky, Eugene et al. “Learning to discover sparse graphical models.” ICML’17.
[3] Liu, Jialin et al. "ALISTA: Analytic weights are as good as learned weights in LISTA." ICLR’18.
[4] Andrychowicz et al. “Learning to learn by gradient descent by gradient descent.” NeurIPS’16.
[5] Ke & Malik. “Learning to optimize.” ICLR’17.
 
In this paradigm, a collection of input covariance matrix and ground truth sparse precision matrix pairs are available during training, either based on simulation or real data. Thus the objective in (9) is formed to directly compare the output of the deep architecture with the ground truth precision matrix. The goal is to train a deep architecture which can perform well for a family/distribution of input covariance matrix and ground truth sparse precision matrix pairs. The average in the objective function is *not* to average several precision matrices. It is averaging over different input covariance and precision matrix pairs such that the learned architecture is able to perform well over a family of problem instances.

Furthermore, each layer of our deep architecture outputs an intermediate prediction of the sparse precision matrix. The objective function (9) takes into account all these intermediate outputs, weights the loss according to the layer of the deep architecture, and tries to progressively bring these intermediate layer outputs closer and closer to the target ground truth. 

II. Comparison to previous paradigm in objective (1)

The objective in (1) is for recovery of the sparse precision matrix for a single problem instance. 
Prior to the data-driven paradigm for sparse recovery, since the target parameter $\Theta^*$ is unknown, the best precision matrix recovery method people can do is to resort to a surrogate objective function (1) (In the broader statistics literature, Eq (1) corresponds to the maximum penalized likelihood estimation).  Using Eq (1), we need to optimally choose an unknown tuning parameter $\lambda$, which is a very challenging problem. Making it less practical. Benefited by a large amount of simulation or real data, the neural network learning by optimizing the loss in Eq (9) solves the problem. So that the learned neural network automatically helps us to choose the unknown tuning parameter in an adaptive way. This hugely benefits our algorithmic approach.

In addition, our first set of experiments identified that there is a mismatch in the optimization objective and the recovery objective (last paragraph of section2 and figure 3 in expt 5.1). This is also a motivation to use learning and come up with a data-driven approach for the sparse graph recovery problem. 



It is not just a different regularization parameter for each entry. The rho/lambda is changing at each iteration throughout the inference procedure. I think thinking of it as a regularization parameter is causing confusion, the notation \pho and \lambad  is just a relic from how the naive AM algorithm is derived. Unlike the naive AM the final GLAD doesnt have an explicit  objective in terms of the input that one can write to describe its solution. 

The overall optimization objective over a set of a bunc of inference problems though can be described by (9) which optimizes the parameterized inference procedure. Note the objective of interest (e.g. ||Theta_true - Theta_pred||) is expressed in terms of the ground truth precision matrix while the typical approach (e.g.  (3)) doesnt have access to the ground truth precision and therefore can't optimize it directly. The parametrized nature of the inference combined with sampling allows to optimize (9) which is in terms of the ground truth. I would think of it more as like the works like Gregor & Lecun / ALISTA, the inference procedure is only insipired by an unrolled optimization procedure.  BTW to understand why there is a sum over K in (9) just think of it as asking the parametrized inference to output a progressively closer approximation of the ground truth at each step of the inference. 

Do you know how it helps to have a different regularization parameter for each entry of the decision matrix? What is the overall optimization objective?

I wanted to point out that I don't think this is simply solving the baneerje objective with a custom regularization term. The rho/lambda  are changing not just every iteration but also for every parameter of the precision matrix, thus the behavior is quite more complex than an optimal selection of regularization for the l1 regularized likelihood.

The authors propose a new method for graph recovery, which is a more data-driven approach by deep learning. It makes an original approach to this problem. In-depth theoretical results are provided in supplementary material. Good attention is also paid to hyper-parameter tuning.

However, some parts can be clarified and improved:

- In the introduction a number of phrases should be clarified: 
it is not entirely clear what the meaning is of the input and output of the problem, input covariance and output precision matrix. In the part of related works, it is difficult to understand in this stage why RNN and deep Q-learning is related to the scope of the paper.

- It is not clear whether section 2 contains new elements or whether the new contribution is entirely in section 3. 

- Please motivate the use of CNN in section 3.1. Why are convolutional layers important within this context?

- eq (6): the methodology of this formulation can be better positioned with respect to the existing literature. It appears to be based on principles of synchronization and consensus (in the term ||Z - Theta||_F). Additional explanation and references are needed at this point.


Meta-review:  The paper proposes a neural network architecture to address the problem of estimating a sparse precision matrix from data, which can be used for inferring conditional independence if the random variables are gaussian. The authors propose an Alternating Minimisation procedure for solving the l1 regularized maximum likelihood which can be unrolled and parameterized. This method is shown to converge faster at inference time than other methods and it is also far more effective in terms of training time compared to an existing data driven method.

Reviewers had good initial impressions of this paper, pointing out the significance of the idea and the soundness of the setup. After a productive rebuttal phase the authors significantly improved the readibility and successfully clarified the remaining concerns of the reviewers. This AC thus recommends acceptance. 

Decision: accept-poster