SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Marcello Massimo Negri
University of Basel  
marcellomassimo.negri@unibas.ch
Fabrio Arend Torres
University of Basel fabricio.arendtorres@unibas.ch
Volker Roth
University of Basel volker.roth@unibas.ch
Studying conditional independence among many variables with few observations is a challenging task. Gaussian Graphical Models (GGMs) tackle this problem by encouraging sparsity in the precision matrix through  $l_{q}$  regularization with  $q \leq 1$ . However, most GMMs rely on the  $l_{1}$  norm because the objective is highly non-convex for sub-  $l_{1}$  pseudo-norms. In the frequentist formulation, the  $l_{1}$  norm relaxation provides the solution path as a function of the shrinkage parameter  $\lambda$ . In the Bayesian formulation, sparsity is instead encouraged through a Laplace prior, but posterior inference for different  $\lambda$  requires repeated runs of expensive Gibbs samplers. Here we propose a general framework for variational inference with matrix-variate Normalizing Flow in GGMs, which unifies the benefits of frequentist and Bayesian frameworks. As a key improvement on previous work, we train with one flow a continuum of sparse regression models jointly for all regularization parameters  $\lambda$  and all  $l_{q}$  norms, including non-convex sub-  $l_{1}$  pseudo-norms. Within one model we thus have access to (i) the evolution of the posterior for any  $\lambda$  and any  $l_{q}$  (pseudo-) norm, (ii) the marginal log-likelihood for model selection, and (iii) the frequentist solution paths through simulated annealing in the MAP limit.
Estimating complex relationships between random variables is a central problem in science. When only few observations are available, the problem becomes even more challenging. Examples include functional connectivity in fMRI data [Smith et al., 2013], networks of interactions from microarray data [Castelo and Roverato, 2006], or correlation patterns in longitudinal studies [Diggle, 2002]. In such cases, it is particularly challenging to infer the conditional independence among random variables. Graphical models are commonly used to represent such an independence structure in the form of network graphs. In this work, we focus specifically on Gaussian Graphical Models (GGMs). GGMs assume observations  $\mathbf{X} \in \mathbb{R}^{n \times d}$  to be generated from a multivariate Gaussian distribution  $\mathcal{X} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$  with  $X_{i}$ : being realizations of  $\mathcal{X}$  for  $i \in \{1, \dots, n\}$ . Here  $\boldsymbol{\mu} \in \mathbb{R}^d$  denotes the mean and  $\boldsymbol{\Sigma} \in \mathbb{R}^{d \times d}$  the covariance matrix, which is symmetric positive definite  $\boldsymbol{\Sigma} \succ 0$ . GGMs provide a simple interpretation of conditional independence through the precision matrix  $\boldsymbol{\Omega} = \boldsymbol{\Sigma}^{-1}$ , whenever  $\boldsymbol{\Sigma}$  is non-singular. Specifically,  $\Omega_{i,j} = 0$  implies that the pair of variables  $(i,j)$  is conditionally independent given all remaining variables. That is to say, there is no edge between nodes  $i$  and  $j$  in the underlying undirected graph.
Penalized likelihood formulation Given the centered observations  $\mathbf{X}$  and the associated sample covariance matrix  $S = X^T X$ , we are interested in reconstructing the precision matrix  $\Omega$ , hence the underlying graph structure. This is particularly challenging when  $d > n$  because the sample covariance matrix becomes singular and the precision matrix can no longer be obtained by simply inverting the MLE of the covariance matrix. One way to overcome this problem is to consider likelihood-penalized models that encourage sparsity in the precision matrix. In other words, we trade off the likelihood with the number of zeros in  $\Omega$ :
$$
\hat {\boldsymbol {\Omega}} = \underset {\boldsymbol {\Omega} > 0} {\arg \max } \left\{\log \det  \boldsymbol {\Omega} - \operatorname {T r} \left(\frac {1}{2} \boldsymbol {S} \boldsymbol {\Omega}\right) - \lambda \| \boldsymbol {\Omega} \| _ {0} \right\}, \tag {1}
$$
where  $\log \det \Omega -\operatorname {Tr}\bigl (\frac{1}{2} S\Omega \bigr)$  is the log-likelihood term and  $\| \Omega \| _0 = \sum_{i <   j}1[w_{ij}\neq 0]$  is the  $l_0$  norm, which counts the number of non-zero off-diagonal elements of  $\Omega$ . The trade-off between the two terms is controlled by the parameter  $\lambda \geq 0$ . Note that the optimization must be performed over the space of symmetric positive definite matrices  $\Omega\succ 0$ . In practice, the objective in Eq. (1) is highly non-convex and cannot be optimized easily. Even in the simpler linear regression setting,  $l_0$  regularization can be computed exactly only for few features [Hastie et al., 2015]. Similar problems arise for all non-convex sub- $l_{1}$  pseudo-norms.
Lasso relaxation: Frequentist and Bayesian approaches In order to simplify Eq. (1), most approaches replace the highly non-convex  $l_{0}$  norm with the closest convex norm, the  $l_{1}$  norm:  $\| \Omega \|_{1} = \sum_{i < j} |w_{ij}|$ . Meinshausen and Buhlmann [2006] first proposed using  $l_{1}$  regularization in the Graphical Lasso model, which sparked the development of several likelihood penalized algorithms [Friedman et al., 2007, Yuan and Lin, 2007, Banerjee et al., 2007]. These frequentist approaches make it possible to elegantly compute the solution path as a function of the shrinkage parameter  $\lambda$  [Mazumder and Hastie, 2012]. The  $l_{1}$ -relaxed problem can also be easily extended to a Bayesian formulation. Wang [2012] assumed a Laplace prior on the off-diagonal elements of the precision matrix and showed that the frequentist solution can be recovered as the maximum a posteriori estimate (MAP). In the Bayesian framework, it is possible to explore the full posterior distribution, to formulate the posterior predictive, and to use the marginal likelihood for model selection. But in high-dimensional settings, MCMC samplers suffer from poor mixing behaviour, which becomes increasingly difficult to diagnose in higher dimensions. Furthermore, different hyperparameters values, e.g.,  $\lambda$ , require independent expensive Markov chains. In the literature, alternative priors have also been proposed [Li et al., 2019]. However, the dependence of Gibbs samplers on tractable posterior conditionals typically restricts the choice of priors significantly.
Variational Inference Variational inference approaches [Blei et al., 2017] approximate intractable densities with tractable parameterized distributions. Compared to MCMC samplers, variational inference turns a sampling problem into an optimization one, which generally requires less computational time. Let  $\pmb{x}$  be the observed variables and  $\pmb{z}$  the unobserved ones. Given a family  $\mathcal{Q} = \{q_{\theta}(z) | \theta \in \Theta\}$  of parameterized distributions, variational inference involves finding the distribution  $q_{\theta^*}(z)$  that best approximates the posterior  $p(z|x)$ . The distance between the two distributions is usually measured as the Kullback-Leibler divergence, which defines the following optimization problem:
$$
q _ {\theta^ {*}} (\boldsymbol {z}) = \underset {\theta \in \Theta} {\arg \min } \mathrm {K L} \left(q _ {\theta} (\boldsymbol {z}) | | p (\boldsymbol {z} | \boldsymbol {x})\right) = \underset {\theta \in \Theta} {\arg \min } \mathbb {E} _ {\boldsymbol {z} \sim q _ {\theta}} \left[ \log \frac {q _ {\theta} (\boldsymbol {z})}{p (\boldsymbol {z} | \boldsymbol {x})} \right]. \tag {2}
$$
The goodness of  $q_{\theta^*}(z)$  as an approximation of the posterior  $p(z|x)$  creates a trade-off between the expressive power of the variational family  $\mathcal{Q}$  and the tractability of the optimization in Eq. (2). Common approaches rely on the so-called mean field approximation, which assumes mutual independence among variables:  $q_{\theta}(z) = \prod_{i}q_{\theta_i}(z_i)$ . But this is not viable in GGMs as we intend to model precisely the dependence structure. Several approaches have been proposed for Bayesian Lasso [Alves et al., 2021] and for its group-sparse variant [Babacan et al., 2014] Yet, to the best of our knowledge, variational approaches have not been studied in the context of Bayesian GGMs.
Contribution We present a unified approach to sparse regression for the whole family of all  $l_{q}$  (pseudo-) norms with  $0 < q < \infty$ , which allows for a fully Bayesian and frequentist-type interpretation. Specifically, we propose a very general framework for variational inference in Bayesian GGMs through a Normalizing Flow defined over the space of symmetric positive definite matrices. By conditioning the flow on  $\lambda$  and on  $q$ , we simultaneously train a continuum of sparse regression
models for all choices of shrinkage parameters  $0 < \lambda < \lambda_{\mathrm{max}}$  and all  $l_{q}$  (pseudo-) norms with  $0 < q < q_{\mathrm{max}}$ , including the highly non-convex sub  $l_{1}$  pseudo-norms. We use as prior the generalized Normal distribution, which for  $q = 1$  recovers the Laplace prior and the Lasso regularization in the MAP limit. On the one hand, our approach inherits the advantages of the Bayesian framework while altogether avoiding problems arising from Gibbs sampling strategies. On the other hand, we can still recover the frequentist solution path in the MAP limit by training through simulated annealing.
In summary, our main contributions are the following:
1. We propose a general framework for variational inference in GGMs through a normalizing flow defined directly over the space of symmetric positive definite matrices. We condition such a flow on the shrinkage parameter  $\lambda$  and on  $q > 0$  to model  $l_{q}$  (pseudo-) norms.
2. We combine the advantages of the frequentist and Bayesian frameworks: in a single model we have access to posterior inference and to the marginal likelihood as a function of  $\lambda$  and  $q$ . We can further recover the frequentist solution path as the MAP by annealing the system.
3. To the best of our knowledge, we are the first to propose a unified framework for sub- $l_{1}$  pseudo-norms that does not require surrogate penalties and that enables consistent exploration of the full solution paths in terms of  $\lambda$  and  $q$ .
In this section, we provide an overview of the Bayesian Graphical Lasso and illustrate the limitations of current inference methods. We then outline the limitations of Lasso regularization and briefly review existing approaches for sparsity with sub- $l_{1}$  pseudo-norms. Lastly, as an alternative to MCMC approaches, we briefly review variational inference with normalizing flows.
Bayesian Graphical Lasso Similarly to the Bayesian Lasso [Park and Casella, 2008], Wang [2012] provided a Bayesian interpretation of the Graphical Lasso (BGL) for posterior inference. Specifically, they showed that the  $l_{1}$ -relaxed optimization of Eq. (1) is equivalent to the maximum a posteriori estimate (MAP) of the model defined by  $p(\Omega | S, \lambda) \propto p(S | \Omega) p(\Omega | \lambda)$ , with
$$
\begin{array}{l} p (\boldsymbol {S} | \boldsymbol {\Omega}) \propto \mathcal {W} _ {d} (n, \boldsymbol {\Omega} ^ {- 1}) \\ p (\boldsymbol {\Omega} | \lambda) \propto \mathcal {W} _ {d} (d + 1, \lambda \mathbf {1} _ {d}) \prod_ {i <   j} \mathrm {D E} \left(\omega_ {i j} | \lambda\right) I [ \boldsymbol {\Omega} \succ 0 ], \tag {3} \\ \end{array}
$$
where the indicator function  $I[\Omega \succ 0]$  imposes positive definiteness on  $\Omega$ . The likelihood term reflects that the sample covariance matrix  $\pmb{S} = \pmb{X}^T\pmb{X}$  is distributed according to the Wishart distribution  $\mathcal{W}_d(n,\Omega^{-1})\propto \operatorname *{det}(\Omega)^{n / 2}\operatorname *{det}(\pmb {S})^{(n - d - 1) / 2}\exp \mathrm{Tr}(-\frac{1}{2}\Omega \pmb {S})$ . The prior is instead composed of two terms. The Wishart term  $\mathcal{W}_d(d + 1,\lambda \mathbf{1}_d)$  imposes symmetry and positive definiteness on  $\Omega$ . In addition, the double exponential (or Laplace) prior  $\mathrm{DE}(\omega_{ij}|\lambda) = \frac{\lambda}{2}\exp \{-\lambda |\omega_{ij}|\}$  encourages sparsity on the off-diagonal elements of  $\Omega$ .
Inference in the BGL, and more generally for Bayesian Lassos, is performed through Gibbs samplers. However, these MCMC strategies are computationally expensive and can suffer from high rejection rates, especially in high dimensions [Mohammadi and Wit, 2015]. Furthermore, to recover the frequentist solution path in the MAP limit, the Markov Chain must be restarted for each  $\lambda$  value. Its derivation also requires to expand the Laplace prior as an infinite mixture of Gaussians [Andrews and Mallows, 1974, West, 1987] and to define an ad hoc mixing density. Lastly, the obtained Gibbs sampler does not generalize to other priors.
Sparsity with sub- $l_{1}$  pseudo-norms The  $l_{0}$  norm is particularly suited to enforce sparsity because it directly penalizes the non-zero entries. In other words, it is equivalent to best subset selection of the variables. Convex relaxation with the  $l_{1}$  norm makes the problem tractable, but comes at the cost of encouraging shrinkage also on the retained variables [Hastie et al., 2015]. This motivated interest in sub- $l_{1}$  pseudo-norms, which in the limit reduce to the original  $l_{0}$  norm formulation. Due to their non-convexity and combinatorial complexity, multiple surrogate objectives have been proposed. Notably, with the proposed approach we do not have to resort to alternative objectives or penalties, as we can directly and exactly enforce sub- $l_{1}$  pseudo-norm regularization with a suitable prior. In the context of linear regression, several algorithms have been proposed. The popular SCAD [Fan and Li,
2001] guarantees unbiasedness, sparsity, and continuity while reducing the overshrinking behaviour. Zhang [2010] proposed the nearly unbiased  $\mathrm{MC + }$  method, which consists of a concave penalty and a selection algorithm that bridges the gap between  $l_{1}$  and  $l_{0}$ . In the literature, other penalties that approximate the  $l_{0}$  penalty have been proposed, such as the SICA penalty [Lv and Fan, 2009], the log-penalty [Mazumder et al., 2011], the seamless-  $l_{0}$  penalty [Dicker et al., 2013] and the atan penalty [Wang and Zhu, 2016]. In the Bayesian framework, Ishwaran and Rao [2005] used a rescaled spike and slab prior to encourage variable selection and drew connections to Ridge regularization.
Normalizing Flows for variational inference Normalizing Flows (NFs) are flexible models that can perform accurate density estimation For this reason, NFs represent a very attractive solution to perform efficient and accurate variational inference [Rezende and Mohamed, 2015, van den Berg et al., 2019]. Suppose  $\mathcal{X}$  is a continuous  $d$ -dimensional random variable with unknown distribution  $p_{\mathbf{x}}$ , and let  $\mathcal{Z}$  be any continuous  $d$ -dimensional random variable with some known base distribution  $z \sim p_{\mathcal{Z}}$ . The key idea of NFs is to construct a diffeomorphism  $\mathcal{T} : \operatorname{supp}(\mathcal{X}) \mapsto \operatorname{supp}(\mathcal{Z})$ , i.e., a differentiable bijection, in order to rewrite  $p_{\mathcal{X}}$  through the change of variable formula as
$$
p _ {\mathcal {X}} (\boldsymbol {x}) = p _ {\mathcal {Z}} \left(\mathcal {T} (\boldsymbol {x})\right) | \det  \mathcal {J} _ {\mathcal {T}} (\boldsymbol {x}) |, \tag {4}
$$
where  $\mathcal{I}_{\mathcal{T}}$  is the Jacobian of the transformation  $\mathcal{T}$ . Assuming that we have successfully learned the transformation  $\mathcal{T}$ , we can then evaluate the target density through Eq. (4). We can further sample from the target distribution by simply transforming samples of the base distribution through  $\mathcal{T}$ , namely  $x = \mathcal{T}(z)$  with  $z$  being realizations of  $p_{\mathcal{Z}}$ . In practice, the crucial part of designing NFs is to construct arbitrarily complicated bijections. To do so, NFs exploit that compositions  $\mathcal{T} = \mathcal{T}_1 \circ \dots \circ \mathcal{T}_m$  of bijections  $\{\mathcal{T}_i\}_{i=1}^m$  remain bijections. The determinant of the resulting transformation decomposes into the determinants of each  $\mathcal{T}_i$  as
$$
\det  \mathcal {J} _ {\mathcal {T}} (\boldsymbol {x}) = \prod_ {i = 1} ^ {m} \det  \mathcal {J} _ {\mathcal {T} _ {i}} \left(\boldsymbol {u} _ {i - 1}\right) \quad \text {w i t h} \quad \boldsymbol {u} _ {i - 1} = \mathcal {T} _ {1} (\boldsymbol {x}) \circ \dots \circ \mathcal {T} _ {i - 1} \left(\boldsymbol {u} _ {i - 2}\right). \tag {5}
$$
Therefore, by composing computationally tractable non-linear bijections, it is possible to define arbitrarily expressive bijections and hence to transform the base distribution  $p_{\mathcal{Z}}$  into arbitrarily complicated distributions  $p_{\mathcal{X}}$ . Among others, Huang et al. [2018] proved that autoregressive flows are universal density approximators of continuous random variables. For a comprehensive review of NFs and their different architectures, see Papamakarios et al. [2021] and Kobyzev et al. [2020]. As first proposed by Atanov et al. [2020], it is also possible to condition the transformation  $\mathcal{T}$  on some parameter  $c \in \mathbb{R}^n$ . The resulting Conditional NF [Kobyzev et al., 2020] models with one flow the family of conditional distributions  $p_{\mathcal{X}}(\pmb{x}|c)$  for continuous values of  $c$ .
3 Proposed approach
In this section, we describe how to infer the posterior of  $\Omega$  in GGMs with conditional NFs and how to do so as a function of the shrinkage parameter  $\lambda$  and of the  $l_{q}$  (pseudo-) norm regularization, all within a single model. We illustrate how to define the conditional flow directly over the space of positive definite matrices and argue why training and posterior inference is particularly efficient. Furthermore, we show that NFs provide direct access to the marginal log-likelihood for model selection and show how to recover the frequentist solution path by training through simulated annealing. We term the resulting model Conditional Matrix Flow (CMF).
3.1 Generalized Normal distribution for  $l_{q}$  (pseudo-) norms
Unlike previous Bayesian approaches to GGMs, we can flexibly specify any prior (and likelihood) in Eq. (3) without worrying about the existence of a suitable Gibbs sampler. We exploit such flexibility to extend the model beyond the standard  $l_{1}$  relaxation. Wang [2012] showed that in the MAP limit  $l_{1}$  regularization is recovered with a Laplace prior on the off-diagonals of  $\Omega$ . We generalize this idea to any  $l_{q}$  (pseudo-) norm with the generalized Normal distribution as prior, which reduces to Laplace and Normal distributions for  $q = 1$  and  $q = 2$ , respectively. Its probability density is defined as:
$$
f (x | \alpha , \beta) = \frac {\beta}{2 \alpha \Gamma (1 / \beta)} \exp \left\{- \frac {| x | ^ {\beta}}{\alpha^ {\beta}} \right\}, \tag {6}
$$
where  $\Gamma (\cdot)$  is the Gamma function,  $\alpha ,\beta >0$  are the scale and shape parameters, respectively. Here, we assumed the distribution to be centered around zero, i.e.  $\mu = 0$ . In order to link the prior to the  $l_{q}$  norm, we rename its parameters as  $\lambda = \alpha^{-\beta} > 0$  and  $q = \beta$ . Our full model can now be conditioned through the prior on both  $\lambda$  and  $q$  and is defined as  $p(\Omega |S,\lambda ,q)\propto p(S|\Omega)p(\Omega |\lambda ,q)$  with
$$
p (\boldsymbol {S} | \boldsymbol {\Omega}) \propto \mathcal {W} _ {d} (n, \boldsymbol {\Omega} ^ {- 1})
$$
$$
p (\boldsymbol {\Omega} | \lambda , q) \propto \mathcal {W} _ {d} (d + 1, \lambda \mathbf {1} _ {d}) \prod_ {i <   j} \frac {q \lambda^ {1 / q}}{2 \Gamma (1 / q)} \exp \{- \lambda | \omega_ {i j} | ^ {q} \} I [ \boldsymbol {\Omega} \succ 0 ]. \tag {7}
$$
If we now consider the log-probability and drop the normalization constant, which does not affect the KL optimization, we recover the  $l_{q}$  (pseudo-) norm. For  $q = 1$  the prior on the off-diagonals reduces to the Laplace distribution and we recover the Bayesian Graphical Lasso in Eq. (3). For  $q = 2$  we recover instead the normal distribution, i.e., Ridge regularization. Overall, when employing the distribution in Eq. (7), we can effectively model solutions corresponding to  $l_{q}$  (pseudo-) norm regularization for  $q > 0$ , including the non-convex sub- $l_{1}$  pseudo-norms.
3.2 Variational inference with conditional flows for GGMs
As we want to model the posterior  $p(\Omega |S)$  over the precision matrix  $\Omega$ , we design flows directly over the space of symmetric positive definite matrices. Specifically, we train a NF conditioned on  $\lambda \in [\lambda_1,\lambda_2]$  and on  $q\in [q_{1},q_{2}]$  with  $\lambda_1,\lambda_2,q_1,q_2 > 0$  in order to study the evolution of the posterior as a function of  $\lambda$  and  $q$ . From Eq. (4) we see that the resulting flow  $\mathcal{T}_{\theta (\lambda ,q)}$  implicitly defines the probability  $q_{\theta (\lambda ,q)}(\Omega) = p_{base}(\mathcal{T}_{\theta (\lambda ,q)}(\Omega))|\det \mathcal{J}_{\mathcal{T}_{\theta}(\lambda ,q)}(\Omega)|$ , which we want to train to approximate the posterior  $p(\Omega |S,\lambda ,q)\propto p(S|\Omega)p(\Omega |\lambda ,q)$ . We minimize the KL divergence in Eq. (2) and approximate the expectation with Monte Carlo samples:
$$
\begin{array}{l} \mathcal {L} (\boldsymbol {\theta}; \lambda , q) = \mathrm {K L} \left(q _ {\boldsymbol {\theta} (\lambda , q)} (\boldsymbol {\Omega}) | | p (\boldsymbol {\Omega} | \boldsymbol {S}, \lambda , q)\right) \\ \approx \frac {1}{M} \sum_ {i = 1} ^ {M} \log \frac {q _ {\boldsymbol {\theta} (\lambda , q)} \left(\boldsymbol {\Omega} _ {i}\right)}{p \left(\boldsymbol {S} \mid \boldsymbol {\Omega} _ {i}\right) p \left(\boldsymbol {\Omega} _ {i} \mid \lambda , q\right)} + N _ {C} (\lambda , q). \tag {8} \\ \end{array}
$$
Note that this is particularly efficient in NFs because it only requires sampling from the base distribution  $p_{base}(\mathcal{T}_{\boldsymbol{\theta}(\lambda,q)}(\Omega))$ , which is computationally cheap. We further need to evaluate the unnormalized posterior  $p(\boldsymbol{\Omega}|\boldsymbol{S},\lambda,q) \propto p(\boldsymbol{S}|\boldsymbol{\Omega})p(\boldsymbol{\Omega}|\lambda,q)$ , i.e. the product of the likelihood and the prior, which is also trivial. Relevantly, in Eq. (8) we do not need to compute the normalization constant  $N_C(\lambda,q) = \log p(\boldsymbol{S}|\lambda,q)$  because it does not influence the optimization.
Unlike standard Bayesian approaches, the proposed conditional flow provides access directly to the marginal log-likelihood  $N_{C}(\lambda ,q) = \log p(S|\lambda ,q)$  as a function of  $\lambda$  and  $q$  without further calculations. This is particularly interesting because the marginal log-likelihood is extremely expensive to compute with classical approaches. If we assume that the flow is expressive enough and that the optimization reached the global minimum, then  $\mathrm{KL}\big(q_{\pmb{\theta}(\lambda ,q)}(\pmb {\Omega})||p(\pmb {\Omega}|S,\lambda ,q)\big)\approx 0$ . In this case, the loss function at the end of training gives us access directly to the negative marginal log-likelihood for continuous values of  $\lambda$  (and  $q$ ). We can then perform model selection by simply choosing  $\lambda^{*} = \max_{\lambda}\log p(S|\lambda ,q)$ .
3.3 Conditional Matrix Flow