Title: Conditional Matrix Flows for Gaussian Graphical Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: normalizing flow

variational inference

graphical lasso

gaussian graphical model

bayesian inference
Abstract: Studying conditional independence among many variables with few observations is a challenging task.
Gaussian Graphical Models (GGMs) tackle this problem by encouraging sparsity in the precision matrix through $l_q$ regularization with $q\leq1$.
However, most GMMs rely on the $l_1$ norm because the objective is highly non-convex for sub-$l_1$ pseudo-norms.
In the frequentist formulation, the $l_1$ norm relaxation provides the solution path as a function of the shrinkage parameter $\lambda$.
In the Bayesian formulation, sparsity is instead encouraged through a Laplace prior, but posterior inference for different $\lambda$ requires repeated runs of expensive Gibbs samplers.
Here we propose a general framework for variational inference with matrix-variate Normalizing Flow in GGMs, which unifies the benefits of frequentist and Bayesian frameworks.
As a key improvement on previous work, we train with one flow a continuum of sparse regression models jointly for all regularization parameters $\lambda$ and all $l_q$ norms, including non-convex sub-$l_1$ pseudo-norms.
Within one model we thus have access to (i) the evolution of the posterior for any $\lambda$ and any $l_q$ (pseudo-) norm, (ii) the marginal log-likelihood for model selection, and (iii) the frequentist solution paths through simulated annealing in the MAP limit.

Conditional Matrix Flows for Gaussian Graphical Models

Abstract

Studying conditional independence among many variables with few observations is a challenging task. Gaussian Graphical Models (GGMs) tackle this problem by encouraging sparsity in the precision matrix through l q regularization with q ≤ 1 . However, most GMMs rely on the l 1 norm because the objective is highly non-convex for subl 1 pseudo-norms. In the frequentist formulation, the l 1 norm relaxation provides the solution path as a function of the shrinkage parameter λ . In the Bayesian formulation, sparsity is instead encouraged through a Laplace prior, but posterior inference for different λ requires repeated runs of expensive Gibbs samplers. Here we propose a general framework for variational inference with matrix-variate Normalizing Flow in GGMs, which unifies the benefits of frequentist and Bayesian frameworks. As a key improvement on previous work, we train with one flow a continuum of sparse regression models jointly for all regularization parameters λ and all l q norms, including non-convex subl 1 pseudo-norms. Within one model we thus have access to (i) the evolution of the posterior for any λ and any l q (pseudo-) norm, (ii) the marginal log-likelihood for model selection, and (iii) the frequentist solution paths through simulated annealing in the MAP limit.

1 Introduction

Estimating complex relationships between random variables is a central problem in science. When only few observations are available, the problem becomes even more challenging. Examples include functional connectivity in fMRI data [Smith et al., 2013], networks of interactions from microarray data [Castelo and Roverato, 2006], or correlation patterns in longitudinal studies [Diggle, 2002]. In such cases, it is particularly challening to infer the conditional independence among random variables. Graphical models are commonly used to represent such an independence structure in the form of network graphs. In this work, we focus specifically on Gaussian Graphical Models (GGMs). GGMs assume observations X ∈ R n × d to be generated from a multivariate Gaussian distribution X ∼ N ( µ , Σ ) with X i, : being realizations of X for i ∈ { 1 , . . . , n } . Here µ ∈ R d denotes the mean and Σ ∈ R d × d the covariance matrix, which is symmetric positive definite Σ ≻ 0 . GGMs provide a simple interpretation of conditional independence through the precision matrix Ω = Σ -1 , whenever Σ is non-singular. Specifically, Ω i,j = 0 implies that the pair of variables ( i, j ) is conditionally independent given all remaining variables. That is to say, there is no edge between nodes i and j in the underlying undirected graph.

Penalized likelihood formulation Given the centered observations X and the associated sample covariance matrix S = X T X , we are interested in reconstructing the precision matrix Ω , hence the underlying graph structure. This is particularly challenging when d > n because the sample covariance matrix becomes singular and the precision matrix can no longer be obtained by simply inverting the MLE of the covariance matrix. One way to overcome this problem is to consider likelihood-penalized models that encourage sparsity in the precision matrix. In other words, we trade off the likelihood with the number of zeros in Ω :

̸

where log det Ω -Tr ( 1 2 S Ω ) is the log-likelihood term and ∥ Ω ∥ 0 = ∑ i<j 1[ w ij = 0] is the l 0 norm, which counts the number of non-zero off-diagonal elements of Ω . The trade-off between the two terms is controlled by the parameter λ ≥ 0 . Note that the optimization must be performed over the space of symmetric positive definite matrices Ω ≻ 0 . In practice, the objective in Eq. (1) is highly non-convex and cannot be optimized easily. Even in the simpler linear regression setting, l 0 regularization can be computed exactly only for few features [Hastie et al., 2015]. Similar problems arise for all non-convex subl 1 pseudo-norms.

Lasso relaxation: Frequentist and Bayesian approaches In order to simplify Eq. (1), most approaches replace the highly non-convex l 0 norm with the closest convex norm, the l 1 norm: ∥ Ω ∥ 1 = ∑ i<j | w ij | . Meinshausen and Bühlmann [2006] first proposed using l 1 regularization in the Graphical Lasso model, which sparked the development of several likelihood penalized algorithms [Friedman et al., 2007, Yuan and Lin, 2007, Banerjee et al., 2007]. These frequentist approaches make it possible to elegantly compute the solution path as a function of the shrinkage parameter λ [Mazumder and Hastie, 2012]. The l 1 -relaxed problem can also be easily extended to a Bayesian formulation. Wang [2012] assumed a Laplace prior on the off-diagonal elements of the precision matrix and showed that the frequentist solution can be recovered as the maximum a posteriori estimate (MAP). In the Bayesian framework, it is possible to explore the full posterior distribution, to formulate the posterior predictive, and to use the marginal likelihood for model selection. But in high-dimensional settings, MCMC samplers suffer from poor mixing behaviour, which becomes increasingly difficult to diagnose in higher dimensions. Furthermore, different hyperparameters values, e.g., λ , require independent expensive Markov chains. In the literature, alternative priors have also been proposed [Li et al., 2019]. However, the dependence of Gibbs samplers on tractable posterior conditionals typically restricts the choice of priors significantly.

Variational Inference Variational inference approaches [Blei et al., 2017] approximate intractable densities with tractable parameterized distributions. Compared to MCMC samplers, variational inference turns a sampling problem into an optimization one, which generally requires less computational time. Let x be the observed variables and z the unobserved ones. Given a family Q = { q θ ( z ) | θ ∈ Θ } of parameterized distributions, variational inference involves finding the distribution q θ ∗ ( z ) that best approximates the posterior p ( z | x ) . The distance between the two distributions is usually measured as the Kullback-Leibler divergence, which defines the following optimization problem:

The goodness of q θ ∗ ( z ) as an approximation of the posterior p ( z | x ) creates a trade-off between the expressive power of the variational family Q and the tractability of the optimization in Eq. (2). Common approaches rely on the so-called mean field approximation, which assumes mutual independence among variables: q θ ( z ) = ∏ i q θ i ( z i ) . But this is not viable in GGMs as we intend to model precisely the dependence structure. Several approaches have been proposed for Bayesian Lasso [Alves et al., 2021] and for its group-sparse variant [Babacan et al., 2014] Yet, to the best of our knowledge, variational approaches have not been studied in the context of Bayesian GGMs.

Contribution We present a unified approach to sparse regression for the whole family of all l q (pseudo-) norms with 0 < q < ∞ , which allows for a fully Bayesian and frequentist-type interpretation. Specifically, we propose a very general framework for variational inference in Bayesian GGMs through a Normalizing Flow defined over the space of symmetric positive definite matrices. By conditioning the flow on λ and on q , we simultaneously train a continuum of sparse regression

models for all choices of shrinkage parameters 0 < λ < λ max and all l q (pseudo-) norms with 0 < q < q max, including the highly non-convex sub l 1 pseudo-norms. We use as prior the generalized Normal distribution, which for q = 1 recovers the Laplace prior and the Lasso regularization in the MAP limit. On the one hand, our approach inherits the advantages of the Bayesian framework while altogether avoiding problems arising from Gibbs sampling strategies. On the other hand, we can still recover the frequentist solution path in the MAP limit by training through simulated annealing.

In summary, our main contributions are the following:

We propose a general framework for variational inference in GGMs through a normalizing flow defined directly over the space of symmetric positive definite matrices. We condition such a flow on the shrinkage parameter λ and on q > 0 to model l q (pseudo-) norms.

We combine the advantages of the frequentist and Bayesian frameworks: in a single model we have access to posterior inference and to the marginal likelihood as a function of λ and q . We can further recover the frequentist solution path as the MAP by annealing the system.

To the best of our knowledge, we are the first to propose a unified framework for subl 1 pseudo-norms that does not require surrogate penalties and that enables consistent exploration of the full solution paths in terms of λ and q .

2 Related Work

In this section, we provide an overview of the Bayesian Graphical Lasso and illustrate the limitations of current inference methods. We then outline the limitations of Lasso regularization and briefly review existing approaches for sparsity with subl 1 pseudo-norms. Lastly, as an alternative to MCMC approaches, we briefly review variational inference with normalizing flows.

Bayesian Graphical Lasso Similarly to the Bayesian Lasso [Park and Casella, 2008], Wang [2012] provided a Bayesian interpretation of the Graphical Lasso (BGL) for posterior inference. Specifically, they showed that the l 1 -relaxed optimization of Eq. (1) is equivalent to the maximum a posteriori estimate (MAP) of the model defined by p ( Ω | S , λ ) ∝ p ( S | Ω ) p ( Ω | λ ) , with

where the indicator function I [ Ω ≻ 0] imposes positive definiteness on Ω . The likelihood term reflects that the sample covariance matrix S = X T X is distributed according to the Wishart distribution W d ( n, Ω -1 ) ∝ det( Ω ) n/ 2 det( S ) ( n -d -1) / 2 expTr( -1 2 Ω S ) . The prior is instead composed of two terms. The Wishart term W d ( d +1 , λ 1 d ) imposes symmetry and positive definiteness on Ω . In addition, the double exponential (or Laplace) prior DE( ω ij | λ ) = λ 2 exp {-λ | ω ij |} encourages sparsity on the off-diagonal elements of Ω .

Inference in the BGL, and more generally for Bayesian Lassos, is performed through Gibbs samplers. However, these MCMC strategies are computationally expensive and can suffer from high rejection rates, especially in high dimensions [Mohammadi and Wit, 2015]. Furthermore, to recover the frequentist solution path in the MAP limit, the Markov Chain must be restarted for each λ value. Its derivation also requires to expand the Laplace prior as an infinite mixture of Gaussians [Andrews and Mallows, 1974, West, 1987] and to define an ad hoc mixing density. Lastly, the obtained Gibbs sampler does not generalize to other priors.

Sparsity with sub -l 1 pseudo-norms The l 0 norm is particularly suited to enforce sparsity because it directly penalizes the non-zero entries. In other words, it is equivalent to best subset selection of the variables. Convex relaxation with the l 1 norm makes the problem tractable, but comes at the cost of encouraging shrinkage also on the retained variables [Hastie et al., 2015]. This motivated interest in subl 1 pseudo-norms, which in the limit reduce to the original l 0 norm formulation. Due to their non-convexity and combinatorial complexity, multiple surrogate objectives have been proposed. Notably, with the proposed approach we do not have to resort to alternative objectives or penalties, as we can directly and exactly enforce subl 1 pseudo-norm regularization with a suitable prior. In the context of linear regression, several algorithms have been proposed. The popular SCAD [Fan and Li,

2001] guarantees unbiasedness, sparsity, and continuity while reducing the overshrinking behaviour. Zhang [2010] proposed the nearly unbiased MC+ method, which consists of a concave penalty and a selection algorithm that bridges the gap between l 1 and l 0 . In the literature, other penalties that approximate the l 0 penalty have been proposed, such as the SICA penalty [Lv and Fan, 2009], the log-penalty [Mazumder et al., 2011], the seamlessl 0 penalty [Dicker et al., 2013] and the atan penalty [Wang and Zhu, 2016]. In the Bayesian framework, Ishwaran and Rao [2005] used a rescaled spike and slab prior to encourage variable selection and drew connections to Ridge regularization.

Normalizing Flows for variational inference Normalizing Flows (NFs) are flexible models that can perform accurate density estimation For this reason, NFs represent a very attractive solution to perform efficient and accurate variational inference [Rezende and Mohamed, 2015, van den Berg et al., 2019]. Suppose X is a continuous d -dimensional random variable with unknown distribution p x , and let Z be any continuous d -dimensional random variable with some known base distribution z ∼ p Z . The key idea of NFs is to construct a diffeomorphism T : supp( X ) ↦→ supp( Z ) , i.e., a differentiable bijection, in order to rewrite p X through the change of variable formula as

where J T is the Jacobian of the transformation T . Assuming that we have successfully learned the transformation T , we can then evaluate the target density through Eq. (4). We can further sample from the target distribution by simply transforming samples of the base distribution through T , namely x = T ( z ) with z being realizations of p Z . In practice, the crucial part of designing NFs is to construct arbitrarily complicated bijections. To do so, NFs exploit that compositions T = T 1 ◦· · ·◦T m of bijections {T i } i m =1 remain bijections. The determinant of the resulting transformation decomposes into the determinants of each T i as

Therefore, by composing computationally tractable non-linear bijections, it is possible to define arbitrarily expressive bijections and hence to transform the base distribution p Z into arbitrarily complicated distributions p X . Among others, Huang et al. [2018] proved that autoregressive flows are universal density approximators of continuous random variables. For a comprehensive review of NFs and their different architectures, see Papamakarios et al. [2021] and Kobyzev et al. [2020]. As first proposed by Atanov et al. [2020], it is also possible to condition the transformation T on some parameter c ∈ R n . The resulting Conditional NF [Kobyzev et al., 2020] models with one flow the family of conditional distributions p X ( x | c ) for continuous values of c .

3 Proposed approach

In this section, we describe how to infer the posterior of Ω in GGMs with conditional NFs and how to do so as a function of the shrinkage parameter λ and of the l q (pseudo-) norm regularization, all within a single model. We illustrate how to define the conditional flow directly over the space of positive definite matrices and argue why training and posterior inference is particularly efficient. Furthermore, we show that NFs provide direct access to the marginal log-likelihood for model selection and show how to recover the frequentist solution path by training through simulated annealing. We term the resulting model Conditional Matrix Flow (CMF).

3.1 Generalized Normal distribution for l q (pseudo-) norms

Unlike previous Bayesian approaches to GGMs, we can flexibly specify any prior (and likelihood) in Eq. (3) without worrying about the existence of a suitable Gibbs sampler. We exploit such flexibility to extend the model beyond the standard l 1 relaxation. Wang [2012] showed that in the MAP limit l 1 regularization is recovered with a Laplace prior on the off-diagonals of Ω . We generalize this idea to any l q (pseudo-) norm with the generalized Normal distribution as prior, which reduces to Laplace and Normal distributions for q = 1 and q = 2 , respectively. Its probability density is defined as:

where Γ( · ) is the Gamma function, α, β > 0 are the scale and shape parameters, respectively. Here, we assumed the distribution to be centered around zero, i.e. µ = 0 . In order to link the prior to the l q norm, we rename its parameters as λ = α -β > 0 and q = β . Our full model can now be conditioned through the prior on both λ and q and is defined as p ( Ω | S , λ, q ) ∝ p ( S | Ω ) p ( Ω | λ, q ) with

If we now consider the log-probability and drop the normalization constant, which does not affect the KL optimization, we recover the l q (pseudo-) norm. For q = 1 the prior on the off-diagonals reduces to the Laplace distribution and we recover the Bayesian Graphical Lasso in Eq. (3). For q = 2 we recover instead the normal distribution, i.e., Ridge regularization. Overall, when employing the distribution in Eq. (7), we can effectively model solutions corresponding to l q (pseudo-) norm regularization for q > 0 , including the non-convex subl 1 pseudo-norms.

3.2 Variational inference with conditional flows for GGMs

As we want to model the posterior p ( Ω | S ) over the precision matrix Ω , we design flows directly over the space of symmetric positive definite matrices. Specifically, we train a NF conditioned on λ ∈ [ λ 1 , λ 2 ] and on q ∈ [ q 1 , q 2 ] with λ 1 , λ 2 , q 1 , q 2 > 0 in order to study the evolution of the posterior as a function of λ and q . From Eq. (4) we see that the resulting flow T θ ( λ,q ) implicitly defines the probability q θ ( λ,q ) ( Ω ) = p base ( T θ ( λ,q ) ( Ω )) | det J T θ ( λ,q ) ( Ω ) | , which we want to train to approximate the posterior p ( Ω | S , λ, q ) ∝ p ( S | Ω ) p ( Ω | λ, q ) . We minimize the KL divergence in Eq. (2) and approximate the expectation with Monte Carlo samples:

Note that this is particularly efficient in NFs because it only requires sampling from the base distribution p base ( T θ ( λ,q ) ( Ω )) , which is computationally cheap. We further need to evaluate the unnormalized posterior p ( Ω | S , λ, q ) ∝ p ( S | Ω ) p ( Ω | λ, q ) , i.e. the product of the likelihood and the prior, which is also trivial. Relevantly, in Eq. (8) we do not need to compute the normalization constant N C ( λ, q ) = log p ( S | λ, q ) because it does not influence the optimization.

Unlike standard Bayesian approaches, the proposed conditional flow provides access directly to the marginal log-likelihood N C ( λ, q ) = log p ( S | λ, q ) as a function of λ and q without further calculations. This is particularly interesting because the marginal log-likelihood is extremely expensive to compute with classical approaches. If we assume that the flow is expressive enough and that the optimization reached the global minimum, then KL ( q θ ( λ,q ) ( Ω ) || p ( Ω | S , λ, q ) ) ≈ 0 . In this case, the loss function at the end of training gives us access directly to the negative marginal log-likelihood for continuous values of λ (and q ). We can then perform model selection by simply choosing λ ∗ = max λ log p ( S | λ, q ) .

3.3 Conditional Matrix Flow

We now describe the proposed Conditional Matrix Flow (CMF) and how we define it over the space of symmetric positive definite matrices by construction. We exploit the well-known Cholesky decomposition [Horn and Johnson, 1985], which states that any symmetric positive definite matrix M ∈ R d × d can be decomposed as M = LL T , where L ∈ R d × d is a lower triangular matrix. The decomposition is unique if L has positive diagonal elements. As a result, the symmetric positive definite M can be fully and uniquely identified by d ( d +1) / 2 numbers, d of which are constrained to be positive. We can then use a d ( d +1) / 2 -dimensional normalizing flow to model probability densities over symmetric-positive matrices. We now illustrate how to transform the initial vector z ∈ R d ( d +1) / 2 into a symmetric positive definite matrix Ω ∈ R d × d . To the best of our knowledge, this is the first attempt to define a flow T ( Ω ) on the space of symmetric positive definite matrices by construction.

...

Sum-of-Sigmoids

Fill-Triangular

Positive-Diagonal

Cholesky-Product

Hyper-Network:

λ

,

...

A high-level visualization of the proposed architecture is shown in Figure 1. The first n layers of the network consist of arbitrary NF transformations and map the initial vector z to z ( n ) λ,q . We propose to use a class of flexible transformations that we called Sum-of-Sigmoids, which worked particularly well for our applications. We provide further insight on the implementation in Appendix A.1. Furthermore, we condition the flow on λ and q via a hypernetwork. The hypernetwork takes λ and q as inputs and returns the parameters of the Matrix Flow. We transform z ( n ) λ,q into a symmetric positive definite matrix in three steps. First, the vector is raveled into a lower triangular matrix L ∈ R d × d , which we call Fill-Triangular . This transformation has a unit Jacobian determinant because it just reshapes the vector into a matrix. In a second step, we bijectively map the diagonal of the resulting lower triangular matrix to positive values with a softplus activation Softplus( x ) = log(1 + exp x ) . This transformation, which we call Positive-Diagonal , acts element-wise and hence admits a cheap Jacobian (log) determinant. Lastly, we compute the Cholesky product Chol( L ) : L ↦→ LL T , which again has an inexpensive Jacobian (log) determinant [Gupta and Nagar, 1999]:

which depends only on the d diagonal elements of L . Note that by enforcing the triangular matrix to be positive, we make sure that the Cholesky decomposition is unique, and hence a bijection.

We open-source the implementation of the conditional bijective layers. 1 We based our library, which was first used in Torres et al. [2023], on the high-level structure provided by the nflows library [Durkan et al., 2020].

3.4 Training through simulated annealing

With our CMF, we can perform posterior inference as a function of λ and q and to perform model selection. These are key advantages of the Bayesian perspective. In addition, we can also recover the frequentist solution path as a function of λ without modifying the model. Specifically, we employ optimization through Simulated Annealing [Kirkpatrick et al., 1983] to approximately sample from the global maxima of the distribution, which for posteriors is the maximum a posteriori estimate (MAP). The idea comes from statistical mechanics, where slowly cooling processes are used to study the ground (optimal) state of the system. When applied to more general optimization tasks, simulated annealing consists in accepting iterative improvements if they lead to a lower cost function. Meanwhile, the temperature is slowly decreased from the initial value T 0 to T n ≈ 0 , where the system is frozen and no further changes occur. In our setting, this corresponds to introducing an artificial temperature T i for the target posterior in Eq. (7):

where T i is the temperature at the i -th iteration. If the initial temperature is high enough, p ( i ) ( Ω ) will likely be very flat, allowing for a better exploration of the support of the distribution. As the temperature decreases, the distribution becomes more peaked. In the limit T n → 0 the distribution

p ( i ) ( Ω ) should concentrate on the global maximum, hence on the MAP solution. Geman and Geman [1984] formally showed that convergence to the set of global minima is guaranteed for logarithmic cooling schedules. Unfortunately, logarithmic cooling schedules are not viable in practice, so alternative schemes have been explored [Abramson et al., 1999]. In this paper, we use the popular geometric cooling schedule [Andrieu and Doucet, 2000, Yuan et al., 2004], which works well in practice: T i = T 0 a i/n for some a > 0 and i ∈ { 0 , . . . , n } . In practice, we train the objective in Eq. (8) while slowly cooling down the system, i.e. decreasing T i in Eq. (10), until we reach a sufficiently low temperature T n . By saving the model at T i = 1 and T n we have access to both the Bayesian and frequentist solutions, respectively.

3.5 Limitations

The proposed approach provides a general framework for posterior inference in GGMs and generalizes to a large family of likelihood and priors, which requires an efficient evaluation of their analytic expression. In some cases, e.g. for posterior predictive distributions, this would require an additional integration step, which could become computationally expensive. The proposed approach is also limited by the expressive power of the bijective layers of the flow. Even though state-of-the-art layers are extremely powerful in modeling high-dimensional distributions, each layer might still be limited in terms of the number of modes that can be modeled [Liao and He, 2021]. Lastly, the proposed model assumes that we can model a family of posterior distributions as a function of the conditioning parameters q and λ , which ultimately depends on the flexibility of the hypernetwork used.

4 Experiments

In this section, we showcase the effectiveness of the proposed CMF first on artificial data and then on a real application. In particular, we study the evolution of the variational posterior as a function of λ and q . We then perform model selection on λ through marginal likelihood maximization. We further illustrate the effect of training through simulated annealing and show that we recover the frequentist solution path through the MAP. Lastly, we show that the proposed method can be readily applied to real data in higher-dimensional settings. Results show that subl 1 pseudo-norms provide sparser solutions and contrast the well-known overshrinking effect of Lasso relaxation. We provide the code to reproduce the experiments at https://github.com/marcello-negri/CMF/ .

4.1 Toy example: synthetic data

We illustrate how the proposed CMF works on artificially generated sparse precision matrices. The data generation process consists of sampling a sparse precision matrix [Pedregosa et al., 2011] with d features and sparsity level α , and then generating n Gaussian samples accordingly. For illustrative purposes, we show results for one precision matrix with α = 90% and n = d = 15 ( n = d ensures the invertibility of the empirical covariance). We trained our model for 10 ′ 000 epochs through simulated annealing with an initial temperature of T 0 = 5 to T n = 0 . 01 and performed 100 geometric cooling steps with T i = T 0 a i/n for a = T n /T 0 .

Figure 2 shows the effect of simulated annealing on the posterior. At temperature T = 1 the conditional flow approximates the true posterior in Eq. (7), which for q = 1 coincides with the Bayesian Graphical Lasso (BGL) model. As we decrease the temperature, the (unnormalized) target posterior in Eq. (10) becomes more peaked and we observe shrinking credible intervals. At the final temperature T n , the distribution converges to the MAP and we recover the frequentist path. As shown in Figure 2 ( right ), our model accurately reconstructs the solution path over λ (MSE = 0 . 052 ). Furthermore, we perform model selection on λ for the model in Eq. (7), which is obtained at temperature T = 1 . In the Appendix A.4 in Figure 7, we show the (approximate) marginal log-likelihood as a function of λ and the resulting optimal λ ∗ CMF = 3 . 52 that maximizes it. We compare the result with the frequentist estimate obtained through MLE with 5-fold cross validation λ ∗ GLasso = 3 . 36 . Again, the model agrees with the frequentist solution.

One of the most significant drawbacks of relaxing the l 0 -norm regularization with l 1 is that sparsity is achieved through shrinkage. This means that the selected relevant features are over-shrunk. With the proposed CMF we show that by exploring subl 1 pseudo-norms we can reduce the shrinkage effect and virtually overcome it in the q → 0 limit. In Figure 3 ( left ) we show 95% posterior

10 0

10 1

0

2

10 0

10 1

0.5

0.0

0.5

1.0

0.3

1.0

3.2

10.0

10

5

0

CMF (q=0.25)

CMF (q=0.50)

CMF (q=0.75)

CMF (q=1.00)

BGL

GLasso

10 0

10 1

0

1

credibility intervals for one entry of the reconstructed precision matrix. We can observe that, as the l q pseudo-norm gets closer to l 1 , the posterior median is progressively shrunk towards zero. With the proposed CMF we can prevent this shrinkage effect by exploring the posterior solution path for subl 1 pseudo-norms. In particular, note that for λ = 0 . 3 the posterior 95% credibility interval does not include the value 0 for q = { 0 . 25 , 0 . 5 , 0 . 75 } , as opposed to the BGL (or q = 1 ). In Figure 3 ( right ) we show the posterior solution path of the CMF for q = 0 . 5 in the MAP limit ( T = 0 . 01 ). Compared to the l 1 solution path, subl 1 solution paths show less shrinkage effect (higher values for the selected features) and steepest decrease towards zero. This behaviour becomes more evident as q decreases, as we show more extensively in the Appendix A.4 in Figure 6 for q = { 1 , 0 . 75 , 0 . 5 , 0 . 25 } . These results support the interest in subl 1 pseudo-norms, and in l 0 norm in the limit.

4.2 Edge recovery with subl 1 pseudo-norms

We now study the behaviour of the proposed CMF in terms of F1 score for edge recovery as a function of the number of samples. We show that the proposed CMF outperforms competing methods, especially in the low sample regime. We compare the CMF with the BGL, which is the only alternative Bayesian model. Then, we compare the CMF in the q → 0 limit against frequentist approaches that use approximate l 0 norm penalties: Atan ('atan') [Wang and Zhu, 2016], Seamless l 0 ('selo') [Dicker et al., 2013], log-penalty ('log') [Mazumder et al., 2011], SICA ('sica') [Lv and Fan, 2009].

We use 10 ground truth precision matrices of dimension d = 30 and generate n Gaussian samples accordingly. We show results for the relevant n < d regime and around n = d , namely for samples

n = { 15 , 25 , 35 , 45 } . For Bayesian approaches (the proposed CMF and the BGL) we draw 1000 samples from the (approximate) posteriors and consider 90% credibility intervals. All results are averaged over the 10 precision matrices. The proposed CMF is trained for 5000 epochs to a final temperature T = 1 , which corresponds to the Bayesian model in Eq. (7). For the BGL we run the Gibbs sampler for 4000 iterations with a burn-in of 1000 and keep every fourth sample. The frequentist approaches were run with the specific hyper-parameters suggested in their original papers.

15

25

35

45

# samples

0.3

0.4

0.5

0.6

0.7

F1

CMF (

p

=0.25)

CMF (

p

=0.5)

CMF (

p

=0.75)

CMF (

p

=1)

BGL

15

25

35

45

# samples

0.3

0.4

0.5

0.6

0.7

F1

CMF (

p

=0.25)

atan

log

sica

As the proposed method is inherently Bayesian we first compare it against the BGL. We show that for q = 1 we obtain results compatible with the BGL. Furthermore, with the proposed method we can additionally infer the posterior for q < 1 . The results in Figure 4 ( left ) show that in the low sample regime ( n < d ) subl 1 pseudo-norms are beneficial and result in a higher F1 score. The effect is stronger as q → 0 , while in the n > d regime subl 1 pseudo-norms do not provide a significant advantage. We also compare the proposed CMF with classical frequentist approaches with surrogate penalties that approximate the l 0 norm. The results in Figure 4 ( right ) show that the proposed CMF with q = 0 . 25 outperforms all competing methods across all sample regimes, but especially for very low sample sizes ( n = 15 ). Note that in the q < 1 regime frequentist algorithms require an ad hoc initialization of the precision matrix, which we provided through the Ledoit-Wolf shrinkage estimator.

4.3 Real data application

We now consider a real-world application to showcase that the proposed CMF can be easily used in high-dimensional settings. We consider a setting in which we are interested in studying only a subset of query variables. In particular, we show that we can avoid inferring the posterior on the full precision matrix, which is expensive, by simply redefining the target posterior, i.e. the prior and likelihood. We consider a colorectal cancer dataset [Sheffer et al., 2009], which contains measurements of 7 clinical variables together with 312 gene measurements from biopsies for 260 cancer patients. As the dataset contains many missing values, we drop the p53 mutation status clinical variable and only consider n = 190 fully measured patients. We study the connections between the s = 6 clinical variables and the t = 312 gene expression measurements. Like Kaufmann et al. [2015], we consider the partition

where s ≪ t . Instead of inferring the expensive ( s + t ) × ( s + t ) precision matrix Ω , we study the Ω 11 and Ω 12 sub-matrices, which overall require only s × ( s + t ) dimensions. Following Kaufmann et al. [2015], we show in Appendix A.2 that we can infer the posterior p ( Ω 11 , Ω 12 | S , λ ) independently of the large Ω 22 . 1 . But in contrast to Kaufmann et al. [2015], we (i) can enforce the correct double-exponential prior on Ω 11 , we (ii) do not need to invert the st × st matrix, which takes O ( st 3 ) operations, and we (iii) do not require a Gibbs sampler, which can still suffer from poor mixing behaviour. In our framework, we only need to define a different posterior as the target

distribution. In practice, we define the flow jointly over the sub-matrices Ω 11 , which must be positive definite, and Ω 12 , which is unconstrained. It is then sufficient to define the bijective layers on the s ( s + 1) / 2 plus s × t dimensions and to perform the Cholesky product only on the dimensions encoding Ω 11 . Note that the determinant of the Jacobian of the full transformation is still the same as in Eq. (9) because the s × t dimensions encoding Ω 12 are just raveled into a matrix.

AGE

SEX

T

N

M

GS

TFG

BIRC5

FZD8

AKT1

LAMA5

CTNNA1

TGFBR1

RALA

FOXO1

CASP9

FGF9

MTOR

FZD3

WNT8B

CTNNA2

WNT5A

LAMA1

PDGFB

CYCS

CDC42

TGFA

SKP2

AR

EGLN1

CDK4

AGE

SEX

T

N

M

GS

TFG

BIRC5

FZD8

CTNNA1

TGFBR1

RALA

CTNNA2

WNT5A

TGFA

AR

EGLN1

CDK4

In Figure 5 we show the inferred network structure with the proposed CMF for q = 1 . 0 ( left ) and for q = 0 . 6 ( right ). Significant edges are obtained by considering 80% credible intervals on the posterior of Ω 11 and Ω 12 . The clinical variables in the data are age, sex, cancer group stage (GS), and TNM classification for colorectal cancer, which measures its size (T), whether it spreads to lymph nodes (N) and if metastases develop (M). Note that, in contrast to [Kaufmann et al., 2015], we can infer the posterior on the Ω 11 block as well, which allows to study the conditional independence structure among clinical variables. For instance, results suggest strong connections between the cancer group stage (GS) and each variable of the TNM classification (T, N, M). More interestingly, with the proposed CMF we can infer the posterior for different subl 1 pseudo-norms. As expected, Figure 5 shows that with lower q values we obtain sparser solutions. Additional quantitative comparisons for q = { 1 . 0 , 0 . 9 , 0 . 8 , 0 . 7 , 0 . 6 } are included in the Appendix A.5 in Figure 8.

5 Conclusions

We propose a very general framework for variational inference in Gaussian Graphical Models through conditional normalizing flows. Our model unifies the benefits of Bayesian and frequentist approaches while avoiding most of their specific problems. Compared to existing approaches, the most important advantage of our method is that it can jointly train a continuum of sparse regression models for all regularization parameters and all l q (pseudo-) norms. All of these models can be analyzed both in a Bayesian fashion (at temperature T = 1 ) or, alternatively, in the frequentist limit (i.e. penalized likelihood) as T approaches 0 . To the best of our knowledge, this is the first Gaussian Graphical Model that can continuously explore all sparsity-inducing priors from the l q -norm family. Moreover, thanks to our variational formalism, we can integrate out the model parameters and use the (approximate) marginal likelihood for model selection without any additional computational costs.

References

Stephen M Smith, Christian F Beckmann, Jesper Andersson, Edward J Auerbach, Janine Bijsterbosch, Gwenaëlle Douaud, Eugene Duff, David A Feinberg, Ludovica Griffanti, Michael P Harms, et al. Resting-state fmri in the human connectome project. Neuroimage , 80:144-168, 2013.

Robert Castelo and Alberto Roverato. A robust procedure for gaussian graphical model search from microarray data with p larger than n. Journal of Machine Learning Research , 7(94):2621-2650, 2006.

Diggle. Analysis of Longitudinal Data . OUP Oxford, 2002. ISBN 9780198524847.

Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical Learning with Sparsity: The Lasso and Generalizations . Chapman &; Hall/CRC, 2015. ISBN 1498712169.

Nicolai Meinshausen and Peter Bühlmann. High-dimensional graphs and variable selection with the Lasso. The Annals of Statistics , 34(3):1436 - 1462, 2006. doi: 10.1214/009053606000000281.

Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Sparse inverse covariance estimation with the graphical lasso. Biostatistics , 9(3):432-441, 12 2007. ISSN 1465-4644. doi: 10.1093/biostatistics/ kxm045.

Ming Yuan and Yi Lin. Model selection and estimation in the gaussian graphical model. Biometrika , 94(1):19-35, 2007. ISSN 00063444, 14643510.

Onureena Banerjee, Laurent El Ghaoui, and Alexandre d'Aspremont. Model selection through sparse maximum likelihood estimation, 2007.

Rahul Mazumder and Trevor Hastie. The graphical lasso: New insights and alternatives. Electronic Journal of Statistics , 6(none):2125 - 2149, 2012. doi: 10.1214/12-EJS740.

Hao Wang. Bayesian Graphical Lasso Models and Efficient Posterior Computation. Bayesian Analysis , 7(4):867 - 886, 2012. doi: 10.1214/12-BA729.

Yunfan Li, Bruce A. Craig, and Anindya Bhadra. The graphical horseshoe estimator for inverse covariance matrices, 2019.

David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association , 112(518):859-877, apr 2017. doi: 10.1080/ 01621459.2017.1285773.

Larissa Alves, Ronaldo Dias, and Helio S. Migon. Variational full bayes lasso: Knots selection in regression splines, 2021.

Derin Babacan, Shinichi Nakajima, and Minh N. Do. Bayesian group-sparse modeling and variational inference. IEEE Transactions on Signal Processing , 62(11):2906-2921, 2014. doi: 10.1109/TSP.2014.2319775.

Trevor Park and George Casella. The bayesian lasso. Journal of the American Statistical Association , 103(482):681-686, 2008. doi: 10.1198/016214508000000337.

Mohammadi and E. C. Wit. Bayesian Structure Learning in Sparse Gaussian Graphical Models. Bayesian Analysis , 10(1):109 - 138, 2015. doi: 10.1214/14-BA889.

F. Andrews and C. L. Mallows. Scale mixtures of normal distributions. Journal of the Royal Statistical Society. Series B (Methodological) , 36(1):99-102, 1974. ISSN 00359246.

Mike West. On scale mixtures of normal distributions. Biometrika , 74(3):646-648, 09 1987. ISSN 0006-3444. doi: 10.1093/biomet/74.3.646.

Jianqing Fan and Runze Li. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association , 96:1348-1360, 02 2001.

Cun-Hui Zhang. Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics , 38(2):894 - 942, 2010. doi: 10.1214/09-AOS729.

Jinchi Lv and Yingying Fan. A unified approach to model selection and sparse recovery using regularized least squares. The Annals of Statistics , 37(6A), dec 2009. doi: 10.1214/09-aos683.

Rahul Mazumder, Jerome H. Friedman, and Trevor Hastie. Sparsenet: Coordinate descent with nonconvex penalties. Journal of the American Statistical Association , 106(495):1125-1138, 2011.

Lee Dicker, Baosheng Huang, and Xihong Lin. Variable selection and estimation with the seamless-l 0 penalty. Statistica Sinica , 23, 04 2013. doi: 10.5705/ss.2011.074.

Yanxin Wang and Li Zhu. Variable selection and parameter estimation with the atan regularization method. Journal of Probability and Statistics , 2016:1-12, 01 2016. doi: 10.1155/2016/6495417.

Hemant Ishwaran and J. Sunil Rao. Spike and slab variable selection: Frequentist and bayesian strategies. The Annals of Statistics , 33(2), apr 2005. doi: 10.1214/009053604000001147.

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37, pages 1530-1538. PMLR, 07-09 Jul 2015.

Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomczak, and Max Welling. Sylvester normalizing flows for variational inference, 2019.

Chin-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows, 2018.

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. J. Mach. Learn. Res. , 22(1), jan 2021. ISSN 1532-4435.

Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence , PP:1-1, 05 2020. doi: 10.1109/TPAMI.2020.2992934.

Andrei Atanov, Alexandra Volokhova, Arsenii Ashukha, Ivan Sosnovik, and Dmitry Vetrov. Semiconditional normalizing flows for semi-supervised learning, 2020.

Roger A. Horn and Charles R. Johnson. Matrix analysis . Cambridge University Press, repr. with corr. edition, 1985.

Arjun K Gupta and Daya K Nagar. Matrix variate distributions , volume 104. CRC Press, 1999.

Fabricio Arend Torres, Marcello Massimo Negri, Marco Inversi, Jonathan Aellen, and Volker Roth. Lagrangian flow networks for conservation laws, 2023.

Conor Durkan, Artur Bekasov, Iain Murray, and George Papamakarios. nflows: normalizing flows in pytorch, 2020.

Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by simulated annealing. Science , 220 (4598):671-680, 1983. doi: 10.1126/science.220.4598.671.

Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence , PAMI-6 (6):721-741, 1984. doi: 10.1109/TPAMI.1984.4767596.

David Abramson, Mohan Krishnamoorthy, and Henry Dang. Simulated annealing cooling schedules for the school timetabling problem. Asia-Pacific Journal of Operational Research , 1999.

Christophe Andrieu and Arnaud Doucet. Simulated annealing for maximum a posteriori parameter estimation of hidden markov models. Information Theory, IEEE Transactions on , 46:994 - 1004, 06 2000. doi: 10.1109/18.841176.

Changhe Yuan, Tsai-Ching Lu, and Marek J. Druzdzel. Annealed map. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence , page 628-635. AUAI Press, 2004.

Huadong Liao and Jiawei He. Jacobian determinant of normalizing flows, 2021.

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Michal Sheffer, Manny D. Bacolod, Or Zuk, Sarah F. Giardina, Hanna Pincas, Francis Barany, Philip B. Paty, William L. Gerald, Daniel A. Notterman, and Eytan Domany. Association of survival and disease progression with chromosomal instability: A genomic exploration of colorectal cancer. Proceedings of the National Academy of Sciences , 106(17):7131-7136, 2009. doi: 10.1073/pnas.0902232106.

Dinu Kaufmann, Sonali Parbhoo, Aleksander Wieczorek, Sebastian Keller, David Adametz, and Volker Roth. Bayesian markov blanket estimation, 2015.

Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems , volume 29. Curran Associates, Inc., 2016.

George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation, 2018.

Fabricio Arend Torres. Sampling and annealing for dependency subnetwork estimation. Master's thesis, University of Basel, 2018.

A Appendix

A.1 Sum-of-Sigmoids layers

We implement the proposed model with a conditional flow architecture that relies on element-wise transformations through monotonic functions, which we term Sum-of-Sigmoids . We further combine these element-wise monotonic transformations in an autoregressive fashion using a MADE-like approach [Kingma et al., 2016, Huang et al., 2018]. The proposed layers are light and very flexible and work particularly well for our purposes already with only 4 layers. Note that, even though it is not needed in our framework, the inverse can be computed numerically, which is relatively cheap since the transformation is element-wise monotonic. Specifically, we implement flexible monotonic transformations by combining shifted and scaled sigmoid activations. Differently from Huang et al. [2018], we also add shifted (and flipped) softplus functions to the element-wise activations. This leads to a linear behaviour outside a specific range [ -s, s ] , s > 0 , alleviating the effect of inputs that lie outside the seen training data. The strictly monotonic element-wise transformation is given by

where σ is the sigmoid function and { v j , w j , b j } k j =1 and a are learnable parameters such that w k , v j , a > 0 and ∑ k j =1 v j = 1 . The advantage of using an element-wise transformation is that the associated Jacobian is diagonal and therefore its log-determinant is straightforward to compute. The proposed Sum-of-Sigmoids layers provide a very flexible transformation, but due to the element-wise nature, different dimensions are not mixed together. We overcome this by predicting the parameters { v j , w j , b j } k j =1 and a of the Sum-of-Sigmoids with masked autoregressive hypernetworks, similar to other masked autoregressive flows. When implemented through masking [Papamakarios et al., 2018], autoregressive flows allow for an elegant extension to conditional settings as well. Relevantly, this autoregressive structure admits a simple log-determinant Jacobian since the Jacobian is lowertriangular by construction.

A.2 Joint posterior for Ω 11 and Ω 12

We show that given the partition in Eq. (11) and the prior in Eq. (7), the joint posterior factorizes as

We further provide the analytic expression for p ( Ω 11 , Ω 12 | S , λ, q ) . The latter is used as target distribution in the training loss in Eq. (8) for the real data application in Section 4. For this proof we followed the approach of Torres [2018].

Let X ∈ R n × ( s + t ) be the design matrix containing independent observations. We are interested in estimating the connections between s query variables with respect to the t remaining ones, typically with s ≪ t . Given the partition of Ω and S in Eq. (11), we define the full posterior p ( Ω , S | λ, q ) = p ( Ω 11 , Ω 12 , Ω 22 , S | λ, q ) as the product of the Wishart likelihood p ( S | Ω ) and a suitable prior p ( Ω | λ, q ) :

Specifically, we select a Wishart prior on Ω to ensure positive definiteness on the full matrix. In order to encourage sparsity we choose a generalized Normal distribution prior over the off-diagonal elements of Ω 11 and over the full Ω 12 :

As a first step we explicitly write down the expression for the likelihood

and for the prior

Overall, the posterior reads as

Following Kaufmann et al. [2015], we now re-write the posterior through the change of variables ( Ω 11 , Ω 12 , Ω 22 ) → ( Ω 11 , Ω 12 , Ω 22 . 1 ) involving the Schur component Ω 22 . 1 = Ω 22 -Ω T 12 Ω -1 11 Ω 12 . Note that the transformation has unit Jacobian: J ( ( Ω 11 , Ω 12 , Ω 22 ) → ( Ω 11 , Ω 12 , Ω 22 . 1 ) ) = 1 . We can now explicitly re-write the posterior as p ( Ω 11 , Ω 12 , Ω 22 . 1 , S | λ, q ) by using the following substitutions, which are straightforward to show:

By re-arranging the terms the posterior simplifies to:

p ( Ω 11 , Ω 12 , Ω 22 . 1 , S | λ, q ) ∝ det( Ω 11 ) n/ 2 det( S ) n -( s + t ) -1 2

If we now condition on S , and specifically on S 22 , we can clearly see that the posterior factorizes as p ( Ω 11 , Ω 12 , Ω 22 . 1 | S , λ, q ) ∝ p ( Ω 11 , Ω 12 | S , λ, q ) p ( Ω 22 . 1 | S , λ, q ) . In particular, we are interested in estimating the joint posterior p ( Ω 11 , Ω 12 | S , λ, q ) , which reads as

A.3 Run-time comparison with Gibbs sampler

The proposed CMF provides a significant speed-up in terms of sampling with respect to standard Gibbs sampling algorithms for posterior inference in Gaussian Graphical Models. For a realistic and fair comparison, we measure run-time on our real data experiment. For this purpose, we trained the Conditional Matrix Flow for q = 1 and compare its sampling speed against the Gibbs sampler introduced in Kaufmann et al. [2015]. The setting presented here is the same one used to obtain the results shown in the Experiment section. The Gibbs sampler takes 89 seconds to generate 500 samples, which translates to about 5 . 6 samples per second. This result was obtained on a Intel(R) Xeon(R) CPU E5-1660 v3 @ 3.00GHz. On the ohter hand, on the consumer-grade GPU NVIDIA TITAN X (12GB VRAM) sampling is extremely efficient: each second we can generate 2000 independent samples from the approximate posterior. Relevantly, in order to retrieve the full posterior path as a function of λ , we would need independent Markov Chains for each λ value, rendering the approach infeasible. In contrast, with the proposed framework we can draw independent samples for different λ values at the same computational cost.

A.4 Artificial data

10 0

10 1

0.5

0.0

0.5

1.0

(a)

q

= 1

.

00

10 0

10 1

0

1

(b)

q

= 0

.

75

10 0

10 1

0

1

(c)

q

= 0

.

50

10 0

10 1

1

0

1

2

(d)

q

= 0

.

25

10 0

10 1

250

200

log

p

(

X

|

)

CMF estimate

A.5 Real data

(M,GS)

(N,GS)

(N,M)

(SEX,CTNNA1)

2

1

0

CMF (q=0.6)

CMF (q=0.7)

CMF (q=0.8)

CMF (q=0.9)

CMF (q=1.0)

Peer reviews:
comments: SUMMARY:
This work proposed a framework for performing inference on Gaussian Graphical Models by approximating the posterior with a normalizing flow over PSD matrices. In this way, the authors can investigate $l_p$-norm regularized GGMs for any value of $p$ in an efficient way.

STRENGTHS:
The idea of using normalizing flows for GGM inference definitely brings in advantages of both Bayesian and frequentist worlds; to me, that's an innovative idea.

WEAKNESSES:
The main weakness that I identified is the lack of comparison between the proposed framework and the well-studied graphical lasso with concave approximations of the $l_0$-norm. More precisely, the authors show that their framework obtains frequentist solution paths through simulated annealing, therefore, it'd be of great interest to see a comparison between these solution paths and those obtained by iterative algorithms such as iterative reweighted l1-norm for graphical lasso.

QUESTIONS:
- in the frequentist case, how does the proposed framework compares against more classical techniques to obtain the solution paths, e.g., iterative reweighted l1-norm?

LIMITATIONS:
The authors adequately addressed the limitations of their proposed framework.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method that can be used to infer conditional independencies in a Gaussian model. These conditional independencies are related to zeros in the precision matrix. Typically, sparse enforcing norms are used to estimate the precision matrix while enforcing zeros in the elements outside of the diagonal. In this paper a Bayesian approach is considered. For this a pseudo-distribution for the data is considered by taking the exponential to the p-norm. The method is trained via variational inference combined with normalizing flows to increase the accuracy of the posterior approximation. The variational distribution is tuned via simulated annealing and a temperature parameter allows to interpolate between the Bayesian and the Map solution.\n\nSTRENGTHS:\n- Well written paper.\n\n        - Illustrative toy experiments.\n\nWEAKNESSES:\n- The proposed method is a combination of already known techniques.\n\n        - The experimental section is weak as only a single real problem is considered.\n\n        - Although the proposed method is a generalization of several known techniques, I have found in the experimental section a lack of comparisons with other related methods.\n\n        My main point of criticism is the weak experimental section which only considers a single real problem and no comparisons with other related methods are carried out in real problems.\n\n        Another point of criticism is that, for some particular values of the p parameter one does not actually observe sparsity in the Bayesian solution. For example, when sampling from the Laplace distribution one never observes zeros in practice. Spike and slab priors (a mix between a Gaussian and a point of mass center at zero) are the ones that actually lead to zeros.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nThe authors have not commented on the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper targets the structure learning problem in Gaussian Graphical Models via (Normalising) Flow-based Variational approximation of the elements of weight metrics that correspond to the Gaussian Bayesian network. \nThey use sub-l1 pseudo norms to penalize dense precision metrics (which correspond to graphs with numerous links) without imposing an extra high penalty for large non-zero values (which typically occurs if $l_{\\geq1}$ is used).\n\nSTRENGTHS:\n1. Up to my knowledge, this is the first time flows are applied to the space of positive definite matrices. \n2. The proposed approach is flexible meaning the class of applicable prior and likelihood functions is quite large.\n3. Using sub-l1 norm is suitable for structure learning. \n4. The proposed algorithm is mathematically sound (as far as I can follow) and is quite interesting. \n5. The paper is well-written, and the relevant work is sufficiently discussed.    \n6. Due to its flexibility, the proposed method has the potential of having a large impact.\n\nWEAKNESSES:\nDue to the factors mentioned in the previous section, I find this work impressive and beautiful. However, unfortunately, the carried out experiments are minimal. Most notably, the algorithm is compared to no alternative work (neither in the main paper nor in the supplementary material). With no quantitative comparisons, it is impossible to evaluate the performance of the proposed algorithm compared to the existing methods. \n\nNOTE: In the Rebuttal, some experiments are carried out (though the code is still not accessible).    \n\nMinor suggestion: \n1. Though it is clear in the context, I suggest that the authors do not use the same letter \"p\" (with the same font) for both probability density and norm parameter.  \n2. Fix minor typos e.g. the end sentence period in line 214.\n\nQUESTIONS:\n* In line 141, what do you mean by \"contradiction\"?\n\nLIMITATIONS:\nThe authors should compare their method with the relevant structure learning lierature and reveal its points of strength as well as its limitations. \n\nThis work is theoretical/methodological and does not have any positive or negative social/ethical impact on its own.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper concerns the estimation of precision matrix under $l_p$ norm sparcity penal. The solution is a variational inference through normalizing flow, which is a function of shrinkage parameter $\lambda$ and non-negative norm parameter $p$. It allows for straightforward computation of solution paths for the intervals of $\lambda$ and $p$, and was empirically evaluated on two relatively small data sets.

STRENGTHS:
Framework for GGM estimation based on conditional normalizing flows, indeed appears novel. Supporting math seems solid. 

Using simulated annealing algorithm to recover a path of solutions for varying $\lambda$ and $p$ is useful, in particular for the case of $p$, as in case of $\lambda$ it was fairly straightforward to perform it with other methods too. I am just wondering how costly and scalable it is under the new framework, an empirical/theoretical analysis would be appreciated.

WEAKNESSES:
Empirical evaluation appears limited. It does not contain comparison with other (e.g. frequentist) approaches to derive the solution paths. Both in terms of estimation accuracy and in terms of computational cost.

QUESTIONS:
In synthetic data example, why did you choose to have more samples than dimensions ( $n>d$ )? Since in that case GGM can be obtained with matrix inverse, and no need for penalized objective.

LIMITATIONS:
Limitations were not discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07