SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now describe the proposed Conditional Matrix Flow (CMF) and how we define it over the space of symmetric positive definite matrices by construction. We exploit the well-known Cholesky decomposition [Horn and Johnson, 1985], which states that any symmetric positive definite matrix  $M \in \mathbb{R}^{d \times d}$  can be decomposed as  $M = LL^T$ , where  $L \in \mathbb{R}^{d \times d}$  is a lower triangular matrix. The decomposition is unique if  $L$  has positive diagonal elements. As a result, the symmetric positive definite  $M$  can be fully and uniquely identified by  $d(d + 1)/2$  numbers,  $d$  of which are constrained to be positive. We can then use a  $d(d + 1)/2$ -dimensional normalizing flow to model probability densities over symmetric-positive matrices. We now illustrate how to transform the initial vector  $z \in \mathbb{R}^{d(d + 1)/2}$  into a symmetric positive definite matrix  $\Omega \in \mathbb{R}^{d \times d}$ . To the best of our knowledge, this is the first attempt to define a flow  $\mathcal{T}(\Omega)$  on the space of symmetric positive definite matrices by construction.
Figure 1: Architecture of the proposed Conditional Matrix Flow (CMF) model. Light grey is used to denote unconstrained values, dark grey for positive values, and white for zeros.
A high-level visualization of the proposed architecture is shown in Figure 1. The first  $n$  layers of the network consist of arbitrary NF transformations and map the initial vector  $\mathbf{z}$  to  $\boldsymbol{z}_{\lambda,q}^{(n)}$ . We propose to use a class of flexible transformations that we called Sum-of-Sigmoids, which worked particularly well for our applications. We provide further insight on the implementation in Appendix A.1. Furthermore, we condition the flow on  $\lambda$  and  $q$  via a hypernetwork. The hypernetwork takes  $\lambda$  and  $q$  as inputs and returns the parameters of the Matrix Flow. We transform  $\boldsymbol{z}_{\lambda,q}^{(n)}$  into a symmetric positive definite matrix in three steps. First, the vector is raveled into a lower triangular matrix  $\boldsymbol{L} \in \mathbb{R}^{d \times d}$ , which we call Fill-Triangular. This transformation has a unit Jacobian determinant because it just reshapes the vector into a matrix. In a second step, we bijectively map the diagonal of the resulting lower triangular matrix to positive values with a softplus activation  $\operatorname{Softplus}(\boldsymbol{x}) = \log (1 + \exp \boldsymbol{x})$ . This transformation, which we call Positive-Diagonal, acts element-wise and hence admits a cheap Jacobian (log) determinant. Lastly, we compute the Cholesky product  $\mathrm{Chol}(\boldsymbol{L}): \boldsymbol{L} \mapsto \boldsymbol{L}\boldsymbol{L}^T$ , which again has an inexpensive Jacobian (log) determinant [Gupta and Nagar, 1999]:
$$
\det  \mathcal {J} _ {\mathrm {C h o l}} (\boldsymbol {L}) = 2 ^ {d} \prod_ {i = 1} ^ {d} \left(\boldsymbol {L} _ {i i}\right) ^ {d - i + 1}, \tag {9}
$$
which depends only on the  $d$  diagonal elements of  $\pmb{L}$ . Note that by enforcing the triangular matrix to be positive, we make sure that the Cholesky decomposition is unique, and hence a bijection.
We open-source the implementation of the conditional bijective layers. We based our library, which was first used in Torres et al. [2023], on the high-level structure provided by the nflows library [Durkan et al., 2020].
3.4 Training through simulated annealing
With our CMF, we can perform posterior inference as a function of  $\lambda$  and  $q$  and to perform model selection. These are key advantages of the Bayesian perspective. In addition, we can also recover the frequentist solution path as a function of  $\lambda$  without modifying the model. Specifically, we employ optimization through Simulated Annealing [Kirkpatrick et al., 1983] to approximately sample from the global maxima of the distribution, which for posteriors is the maximum a posteriori estimate (MAP). The idea comes from statistical mechanics, where slowly cooling processes are used to study the ground (optimal) state of the system. When applied to more general optimization tasks, simulated annealing consists in accepting iterative improvements if they lead to a lower cost function. Meanwhile, the temperature is slowly decreased from the initial value  $T_{0}$  to  $T_{n} \approx 0$ , where the system is frozen and no further changes occur. In our setting, this corresponds to introducing an artificial temperature  $T_{i}$  for the target posterior in Eq. (7):
$$
p ^ {(i)} (\boldsymbol {\Omega}) = p \left(\boldsymbol {\Omega} \mid \boldsymbol {S}, \lambda , q\right) ^ {1 / T _ {i}} \tag {10}
$$
where  $T_{i}$  is the temperature at the  $i$ -th iteration. If the initial temperature is high enough,  $p^{(i)}(\Omega)$  will likely be very flat, allowing for a better exploration of the support of the distribution. As the temperature decreases, the distribution becomes more peaked. In the limit  $T_{n} \to 0$  the distribution
<sup>1</sup>FlowConductor: (Conditional) Normalizing Flows and bijective Layers for Pytorch https://github.com/FabricioArendTorres/FlowConductor
$p^{(i)}(\Omega)$  should concentrate on the global maximum, hence on the MAP solution. Geman and Geman [1984] formally showed that convergence to the set of global minima is guaranteed for logarithmic cooling schedules. Unfortunately, logarithmic cooling schedules are not viable in practice, so alternative schemes have been explored [Abramson et al., 1999]. In this paper, we use the popular geometric cooling schedule [Andrieu and Doucet, 2000, Yuan et al., 2004], which works well in practice:  $T_{i} = T_{0}a^{i / n}$  for some  $a > 0$  and  $i\in \{0,\dots ,n\}$ . In practice, we train the objective in Eq. (8) while slowly cooling down the system, i.e. decreasing  $T_{i}$  in Eq. (10), until we reach a sufficiently low temperature  $T_{n}$ . By saving the model at  $T_{i} = 1$  and  $T_{n}$  we have access to both the Bayesian and frequentist solutions, respectively.
3.5 Limitations
The proposed approach provides a general framework for posterior inference in GGMs and generalizes to a large family of likelihood and priors, which requires an efficient evaluation of their analytic expression. In some cases, e.g. for posterior predictive distributions, this would require an additional integration step, which could become computationally expensive. The proposed approach is also limited by the expressive power of the bijective layers of the flow. Even though state-of-the-art layers are extremely powerful in modeling high-dimensional distributions, each layer might still be limited in terms of the number of modes that can be modeled [Liao and He, 2021]. Lastly, the proposed model assumes that we can model a family of posterior distributions as a function of the conditioning parameters  $q$  and  $\lambda$ , which ultimately depends on the flexibility of the hypernetwork used.
In this section, we showcase the effectiveness of the proposed CMF first on artificial data and then on a real application. In particular, we study the evolution of the variational posterior as a function of  $\lambda$  and  $q$ . We then perform model selection on  $\lambda$  through marginal likelihood maximization. We further illustrate the effect of training through simulated annealing and show that we recover the frequentist solution path through the MAP. Lastly, we show that the proposed method can be readily applied to real data in higher-dimensional settings. Results show that sub-11 pseudo-norms provide sparser solutions and contrast the well-known overshrinking effect of Lasso relaxation. We provide the code to reproduce the experiments at https://github.com/marcello-negri/CMF/.
4.1 Toy example: synthetic data
We illustrate how the proposed CMF works on artificially generated sparse precision matrices. The data generation process consists of sampling a sparse precision matrix [Pedregosa et al., 2011] with  $d$  features and sparsity level  $\alpha$ , and then generating  $n$  Gaussian samples accordingly. For illustrative purposes, we show results for one precision matrix with  $\alpha = 90\%$  and  $n = d = 15$  ( $n = d$  ensures the invertibility of the empirical covariance). We trained our model for  $10^{\prime}000$  epochs through simulated annealing with an initial temperature of  $T_0 = 5$  to  $T_{n} = 0.01$  and performed 100 geometric cooling steps with  $T_{i} = T_{0}a^{i / n}$  for  $a = T_{n} / T_{0}$ .
Figure 2 shows the effect of simulated annealing on the posterior. At temperature  $T = 1$  the conditional flow approximates the true posterior in Eq. (7), which for  $q = 1$  coincides with the Bayesian Graphical Lasso (BGL) model. As we decrease the temperature, the (unnormized) target posterior in Eq. (10) becomes more peaked and we observe shrinking credible intervals. At the final temperature  $T_{n}$ , the distribution converges to the MAP and we recover the frequentist path. As shown in Figure 2 (right), our model accurately reconstructs the solution path over  $\lambda$  (MSE  $= 0.052$ ). Furthermore, we perform model selection on  $\lambda$  for the model in Eq. (7), which is obtained at temperature  $T = 1$ . In the Appendix A.4 in Figure 7, we show the (approximate) marginal log-likelihood as a function of  $\lambda$  and the resulting optimal  $\lambda_{\mathrm{CMF}}^* = 3.52$  that maximizes it. We compare the result with the frequentist estimate obtained through MLE with 5-fold cross validation  $\lambda_{\mathrm{GLasso}}^* = 3.36$ . Again, the model agrees with the frequentist solution.
One of the most significant drawbacks of relaxing the  $l_{0}$ -norm regularization with  $l_{1}$  is that sparsity is achieved through shrinkage. This means that the selected relevant features are over-shrunk. With the proposed CMF we show that by exploring sub- $l_{1}$  pseudo-norms we can reduce the shrinkage effect and virtually overcome it in the  $q \to 0$  limit. In Figure 3 (left) we show  $95\%$  posterior
Figure 2:  $75\%$  posterior credible intervals as a function of  $\lambda$  for  $q = 1$ . Left: at  $T = 1.0$  the CMF reduces to the BGL. Right: at  $T = 0.01$  the CMF reduces to the frequentist solution path (dashed).
Figure 3: Left:  $95\%$  posterior credibility intervals of the proposed CMF for one entry of the precision matrix as a function of the pseudo-norm  $q = \{1,0.75,0.5,0.25\}$ . Results are compared with the BGL and the frequentist solution (GLasso). Right: MAP estimate of the CMF posterior as a function of  $\lambda$  for  $q = 0.5$  ( $T = 0.01$ ). The dashed line is the frequentist solution path for  $q = 1$ .
credibility intervals for one entry of the reconstructed precision matrix. We can observe that, as the  $l_{q}$  pseudo-norm gets closer to  $l_{1}$ , the posterior median is progressively shrunk towards zero. With the proposed CMF we can prevent this shrinkage effect by exploring the posterior solution path for sub-  $l_{1}$  pseudo-norms. In particular, note that for  $\lambda = 0.3$  the posterior  $95\%$  credibility interval does not include the value 0 for  $q = \{0.25, 0.5, 0.75\}$ , as opposed to the BGL (or  $q = 1$ ). In Figure 3 (right) we show the posterior solution path of the CMF for  $q = 0.5$  in the MAP limit ( $T = 0.01$ ). Compared to the  $l_{1}$  solution path, sub-  $l_{1}$  solution paths show less shrinkage effect (higher values for the selected features) and steepest decrease towards zero. This behaviour becomes more evident as  $q$  decreases, as we show more extensively in the Appendix A.4 in Figure 6 for  $q = \{1, 0.75, 0.5, 0.25\}$ . These results support the interest in sub-  $l_{1}$  pseudo-norms, and in  $l_{0}$  norm in the limit.
4.2 Edge recovery with sub-  $l_{1}$  pseudo-norms
We now study the behaviour of the proposed CMF in terms of F1 score for edge recovery as a function of the number of samples. We show that the proposed CMF outperforms competing methods, especially in the low sample regime. We compare the CMF with the BGL, which is the only alternative Bayesian model. Then, we compare the CMF in the  $q \to 0$  limit against frequentist approaches that use approximate  $l_0$  norm penalties: Atan ("atan") [Wang and Zhu, 2016], Seamless  $l_0$  ("selo") [Dicker et al., 2013], log-penalty ("log") [Mazumder et al., 2011], SICA ("sica") [Lv and Fan, 2009].
We use 10 ground truth precision matrices of dimension  $d = 30$  and generate  $n$  Gaussian samples accordingly. We show results for the relevant  $n < d$  regime and around  $n = d$ , namely for samples
$n = \{15,25,35,45\}$ . For Bayesian approaches (the proposed CMF and the BGL) we draw 1000 samples from the (approximate) posteriors and consider  $90\%$  credibility intervals. All results are averaged over the 10 precision matrices. The proposed CMF is trained for 5000 epochs to a final temperature  $T = 1$ , which corresponds to the Bayesian model in Eq. (7). For the BGL we run the Gibbs sampler for 4000 iterations with a burn-in of 1000 and keep every fourth sample. The frequentist approaches were run with the specific hyper-parameters suggested in their original papers.
Figure 4: F1 score for edge recovery. Left: the proposed CMF for  $q = \{0.25, 0.5, 0.75, 1\}$  against the Bayesian Graphical Lasso (BGL). Right: the proposed CMF for  $q = 0.25$  against various frequentist approaches with penalties that approximate the  $l_0$  norm. Results are averaged over 10 precision matrices with  $d = 30$ .
As the proposed method is inherently Bayesian we first compare it against the BGL. We show that for  $q = 1$  we obtain results compatible with the BGL. Furthermore, with the proposed method we can additionally infer the posterior for  $q < 1$ . The results in Figure 4 (left) show that in the low sample regime ( $n < d$ ) sub- $l_1$  pseudo-norms are beneficial and result in a higher F1 score. The effect is stronger as  $q \to 0$ , while in the  $n > d$  regime sub- $l_1$  pseudo-norms do not provide a significant advantage. We also compare the proposed CMF with classical frequentist approaches with surrogate penalties that approximate the  $l_0$  norm. The results in Figure 4 (right) show that the proposed CMF with  $q = 0.25$  outperforms all competing methods across all sample regimes, but especially for very low sample sizes ( $n = 15$ ). Note that in the  $q < 1$  regime frequentist algorithms require an ad hoc initialization of the precision matrix, which we provided through the Ledoit-Wolf shrinkage estimator.
4.3 Real data application
We now consider a real-world application to showcase that the proposed CMF can be easily used in high-dimensional settings. We consider a setting in which we are interested in studying only a subset of query variables. In particular, we show that we can avoid inferring the posterior on the full precision matrix, which is expensive, by simply redefining the target posterior, i.e. the prior and likelihood. We consider a colorectal cancer dataset [Sheffer et al., 2009], which contains measurements of 7 clinical variables together with 312 gene measurements from biopsies for 260 cancer patients. As the dataset contains many missing values, we drop the  $p53$  mutation status clinical variable and only consider  $n = 190$  fully measured patients. We study the connections between the  $s = 6$  clinical variables and the  $t = 312$  gene expression measurements. Like Kaufmann et al. [2015], we consider the partition
$$
\boldsymbol {\Omega} = \left( \begin{array}{l l} s & t \\ \boldsymbol {\Omega} _ {1 1} ^ {T} & \boldsymbol {\Omega} _ {1 2} \\ \boldsymbol {\Omega} _ {1 2} ^ {T} & \boldsymbol {\Omega} _ {2 2} \end{array} \right) \quad s \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad S = \left( \begin{array}{l l} s & t \\ \boldsymbol {S} _ {1 1} & \boldsymbol {S} _ {1 2} \\ \boldsymbol {S} _ {1 2} ^ {T} & \boldsymbol {S} _ {2 2} \end{array} \right) \quad s \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad (1 1)
$$
where  $s \ll t$ . Instead of inferring the expensive  $(s + t) \times (s + t)$  precision matrix  $\Omega$ , we study the  $\Omega_{11}$  and  $\Omega_{12}$  sub-matrices, which overall require only  $s \times (s + t)$  dimensions. Following Kaufmann et al. [2015], we show in Appendix A.2 that we can infer the posterior  $p(\Omega_{11}, \Omega_{12} | S, \lambda)$  independently of the large  $\Omega_{22,1}$ . But in contrast to Kaufmann et al. [2015], we (i) can enforce the correct double-exponential prior on  $\Omega_{11}$ , we (ii) do not need to invert the  $st \times st$  matrix, which takes  $O(st^3)$  operations, and we (iii) do not require a Gibbs sampler, which can still suffer from poor mixing behaviour. In our framework, we only need to define a different posterior as the target
distribution. In practice, we define the flow jointly over the sub-matrices  $\Omega_{11}$ , which must be positive definite, and  $\Omega_{12}$ , which is unconstrained. It is then sufficient to define the bijective layers on the  $s(s + 1) / 2$  plus  $s \times t$  dimensions and to perform the Cholesky product only on the dimensions encoding  $\Omega_{11}$ . Note that the determinant of the Jacobian of the full transformation is still the same as in Eq. (9) because the  $s \times t$  dimensions encoding  $\Omega_{12}$  are just raveled into a matrix.
Figure 5: Inferred network structure with the proposed CMF for  $q = 1.0$  (left) and  $q = 0.6$  (right). Positive edges are shown in black and negative ones in grey. Clinical variables are highlighted in red.
In Figure 5 we show the inferred network structure with the proposed CMF for  $q = 1.0$  (left) and for  $q = 0.6$  (right). Significant edges are obtained by considering  $80\%$  credible intervals on the posterior of  $\Omega_{11}$  and  $\Omega_{12}$ . The clinical variables in the data are age, sex, cancer group stage (GS), and TNM classification for colorectal cancer, which measures its size (T), whether it spreads to lymph nodes (N) and if metastases develop (M). Note that, in contrast to [Kaufmann et al., 2015], we can infer the posterior on the  $\Omega_{11}$  block as well, which allows to study the conditional independence structure among clinical variables. For instance, results suggest strong connections between the cancer group stage (GS) and each variable of the TNM classification (T, N, M). More interestingly, with the proposed CMF we can infer the posterior for different sub-  $l_{1}$  pseudo-norms. As expected, Figure 5 shows that with lower  $q$  values we obtain sparser solutions. Additional quantitative comparisons for  $q = \{1.0, 0.9, 0.8, 0.7, 0.6\}$  are included in the Appendix A.5 in Figure 8.
We propose a very general framework for variational inference in Gaussian Graphical Models through conditional normalizing flows. Our model unifies the benefits of Bayesian and frequentist approaches while avoiding most of their specific problems. Compared to existing approaches, the most important advantage of our method is that it can jointly train a continuum of sparse regression models for all regularization parameters and all  $l_{q}$  (pseudo-) norms. All of these models can be analyzed both in a Bayesian fashion (at temperature  $T = 1$ ) or, alternatively, in the frequentist limit (i.e. penalized likelihood) as  $T$  approaches 0. To the best of our knowledge, this is the first Gaussian Graphical Model that can continuously explore all sparsity-inducing priors from the  $l_{q}$ -norm family. Moreover, thanks to our variational formalism, we can integrate out the model parameters and use the (approximate) marginal likelihood for model selection without any additional computational costs.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011.
Michal Sheffer, Manny D. Bacolod, Or Zuk, Sarah F. Giardina, Hanna Pincas, Francis Barany, Philip B. Paty, William L. Gerald, Daniel A. Notterman, and Eytan Domany. Association of survival and disease progression with chromosomal instability: A genomic exploration of colorectal cancer. Proceedings of the National Academy of Sciences, 106(17):7131-7136, 2009. doi: 10.1073/pnas.0902232106.
Dinu Kaufmann, Sonali Parbhoo, Aleksander Wieczorek, Sebastian Keller, David Adametz, and Volker Roth. Bayesian markov blanket estimation, 2015.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
George Papamakarios, Theo Pavlakou, and Iain Murray. Masked autoregressive flow for density estimation, 2018.
Fabricio Arend Torres. Sampling and annealing for dependency subnetwork estimation. Master's thesis, University of Basel, 2018.
A.1 Sum-of-Siblings layers
We implement the proposed model with a conditional flow architecture that relies on element-wise transformations through monotonic functions, which we term Sum-of-Sigmoids. We further combine these element-wise monotonic transformations in an autoregressive fashion using a MADE-like approach [Kingma et al., 2016, Huang et al., 2018]. The proposed layers are light and very flexible and work particularly well for our purposes already with only 4 layers. Note that, even though it is not needed in our framework, the inverse can be computed numerically, which is relatively cheap since the transformation is element-wise monotonic. Specifically, we implement flexible monotonic transformations by combining shifted and scaled sigmoid activations. Differently from Huang et al. [2018], we also add shifted (and flipped) softplus functions to the element-wise activations. This leads to a linear behaviour outside a specific range  $[-s, s]$ ,  $s > 0$ , alleviating the effect of inputs that lie outside the seen training data. The strictly monotonic element-wise transformation is given by
$$
\phi_ {s o s} \left(z ^ {(i)}\right) = \underbrace {\left[ a \sum_ {j = 1} ^ {k} v _ {j} \sigma \left(w _ {j} z ^ {(i)} + b _ {j}\right) \right]} _ {\text {s u m o f m o n o t o n i c f u n c t i o n s}} + \underbrace {\left[ \ln \left(1 + e ^ {\left(z ^ {(i)} - s\right)}\right) - \ln \left(1 + e ^ {\left(- z ^ {(i)} - s\right)}\right) \right]} _ {\text {a p p r o x . l i n e a r f o r} | z ^ {(i)} | \gg s , \text {a n d z e r o f o r} | z ^ {(i)} | \ll s}, \tag {12}
$$
where  $\sigma$  is the sigmoid function and  $\{v_{j},w_{j},b_{j}\}_{j = 1}^{k}$  and  $a$  are learnable parameters such that  $w_{k},v_{j},a > 0$  and  $\sum_{j = 1}^{k}v_{j} = 1$ . The advantage of using an element-wise transformation is that the associated Jacobian is diagonal and therefore its log-determinant is straightforward to compute. The proposed Sum-of-Sigmoids layers provide a very flexible transformation, but due to the element-wise nature, different dimensions are not mixed together. We overcome this by predicting the parameters  $\{v_{j},w_{j},b_{j}\}_{j = 1}^{k}$  and  $a$  of the Sum-of-Sigmoids with masked autoregressive hypernetworks, similar to other masked autoregressive flows. When implemented through masking [Papamakarios et al., 2018], autoregressive flows allow for an elegant extension to conditional settings as well. Relevantly, this autoregressive structure admits a simple log-determinant Jacobian since the Jacobian is lower-triangular by construction.
A.2 Joint posterior for  $\Omega_{11}$  and  $\Omega_{12}$
We show that given the partition in Eq. (11) and the prior in Eq. (7), the joint posterior factorizes as
$$
p \left(\boldsymbol {\Omega} _ {1 1}, \boldsymbol {\Omega} _ {1 2}, \boldsymbol {\Omega} _ {2 2. 1} | S, \lambda , q\right) = p \left(\boldsymbol {\Omega} _ {1 1}, \boldsymbol {\Omega} _ {1 2} | S, \lambda , q\right) p \left(\boldsymbol {\Omega} _ {2 2. 1} | S, \lambda , q\right). \tag {13}
$$