SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We further provide the analytic expression for  $p(\Omega_{11}, \Omega_{12} | S, \lambda, q)$ . The latter is used as target distribution in the training loss in Eq. (8) for the real data application in Section 4. For this proof we followed the approach of Torres [2018].
Let  $X \in \mathbb{R}^{n \times (s + t)}$  be the design matrix containing independent observations. We are interested in estimating the connections between  $s$  query variables with respect to the  $t$  remaining ones, typically with  $s \ll t$ . Given the partition of  $\Omega$  and  $S$  in Eq. (11), we define the full posterior  $p(\Omega, S|\lambda, q) = p(\Omega_{11}, \Omega_{12}, \Omega_{22}, S|\lambda, q)$  as the product of the Wishart likelihood  $p(S|\Omega)$  and a suitable prior  $p(\Omega|\lambda, q)$ :
$$
p (\boldsymbol {S} | \boldsymbol {\Omega}) \propto \mathcal {W} _ {s + t} (n, \boldsymbol {\Omega} ^ {- 1}) = \det  (\boldsymbol {\Omega}) ^ {n / 2} \exp \left\{\operatorname {T r} \left(- \frac {1}{2} \boldsymbol {\Omega} \boldsymbol {S}\right) \right\}
$$
$$
p (\boldsymbol {\Omega} | \lambda , q) \propto \mathcal {W} _ {s + t} (s + t + 1, \lambda \mathbf {1} _ {s + t}) p \left(\boldsymbol {\Omega} _ {1 1} | \lambda , q\right) p \left(\boldsymbol {\Omega} _ {1 2} | \lambda , q\right). \tag {14}
$$
Specifically, we select a Wishart prior on  $\Omega$  to ensure positive definiteness on the full matrix. In order to encourage sparsity we choose a generalized Normal distribution prior over the off-diagonal elements of  $\Omega_{11}$  and over the full  $\Omega_{12}$ :
$$
p \left(\boldsymbol {\Omega} _ {1 1} | \lambda , q\right) \propto \prod_ {i <   j} \exp \left(- \lambda \left| \left(\boldsymbol {\Omega} _ {1 1}\right) _ {i j} \right| ^ {q}\right) \quad p \left(\boldsymbol {\Omega} _ {1 2} | \lambda , q\right) \propto \prod_ {i, j} \exp \left(- \lambda \left| \left(\boldsymbol {\Omega} _ {1 2}\right) _ {i j} \right| ^ {q}\right). \tag {15}
$$
As a first step we explicitly write down the expression for the likelihood
$$
\begin{array}{l} p (\boldsymbol {S} | \boldsymbol {\Omega}) = p (\boldsymbol {S} | \boldsymbol {\Omega} _ {1 1}, \boldsymbol {\Omega} _ {1 2}, \boldsymbol {\Omega} _ {2 2. 1}) \\ = \mathcal {W} _ {s + t} \left(n, \boldsymbol {\Omega} ^ {- 1}\right) \tag {16} \\ \propto \det (\boldsymbol {\Omega}) ^ {n / 2} \det (\boldsymbol {S}) ^ {(n - (s + t) - 1) / 2} \exp \big (- \frac {1}{2} \mathrm {T r} [ \boldsymbol {\Omega} \boldsymbol {S} ] \big) \\ \end{array}
$$
and for the prior
$$
\begin{array}{l} p (\boldsymbol {\Omega} | \lambda , q) = p \left(\boldsymbol {\Omega} _ {1 1}, \boldsymbol {\Omega} _ {1 2}, \boldsymbol {\Omega} _ {2 2. 1} | \lambda , q\right) \\ \propto \mathcal {W} _ {s + t} (s + t + 1, \lambda \mathbf {1} _ {s + t}) p (\Omega_ {1 1} | \lambda) p (\Omega_ {1 2} | \lambda) \\ = \exp \left(- \frac {\lambda}{2} \operatorname {T r} [ \boldsymbol {\Omega} ]\right) \prod_ {i <   j} \exp \left(- \lambda | (\boldsymbol {\Omega} _ {1 1}) _ {i j} | ^ {q}\right) \prod_ {i, j} \exp \left(- \lambda | (\boldsymbol {\Omega} _ {1 2}) _ {i j} | ^ {q}\right). \tag {17} \\ \end{array}
$$
Overall, the posterior reads as
$$
\begin{array}{l} p \left(\boldsymbol {\Omega} _ {1 1}, \boldsymbol {\Omega} _ {1 2}, \boldsymbol {\Omega} _ {2 2. 1}, \boldsymbol {S} | \lambda , q\right) \propto \det  (\boldsymbol {\Omega}) ^ {n / 2} \det  (\boldsymbol {S}) ^ {(n - (s + t) - 1) / 2} \\ \times \exp \left(- \frac {1}{2} \operatorname {T r} \left[ \boldsymbol {\Omega} \boldsymbol {S} + \lambda \boldsymbol {\Omega} \right]\right) \tag {18} \\ \times \prod_ {i <   j} \exp \left(- \lambda | (\Omega_ {1 1}) _ {i j} | ^ {q}\right) \prod_ {i, j} \exp \left(- \lambda | (\Omega_ {1 2}) _ {i j} | ^ {q}\right). \\ \end{array}
$$
Following Kaufmann et al. [2015], we now re-write the posterior through the change of variables  $(\Omega_{11},\Omega_{12},\Omega_{22})\to (\Omega_{11},\Omega_{12},\Omega_{22.1})$  involving the Schur component  $\Omega_{22.1} = \Omega_{22} - \Omega_{12}^{T}\Omega_{11}^{-1}\Omega_{12}$ . Note that the transformation has unit Jacobian:  $J\bigl ((\Omega_{11},\Omega_{12},\Omega_{22})\to (\Omega_{11},\Omega_{12},\Omega_{22.1})\bigr) = 1$ . We can now explicitly re-write the posterior as  $p(\Omega_{11},\Omega_{12},\Omega_{22.1},S|\lambda ,q)$  by using the following substitutions, which are straightforward to show:
$$
\det  (\boldsymbol {\Omega}) = \det  (\boldsymbol {\Omega} _ {1 1}) \det  (\boldsymbol {\Omega} _ {2 2} - \boldsymbol {\Omega} _ {1 2} ^ {T} \boldsymbol {\Omega} _ {1 1} ^ {- 1} \boldsymbol {\Omega} _ {1 2}) = \det  (\boldsymbol {\Omega} _ {1 1}) \det  (\boldsymbol {\Omega} _ {2 2. 1}),
$$
$$
\operatorname {T r} (\boldsymbol {\Omega} \boldsymbol {S}) = \operatorname {T r} \left[ \boldsymbol {\Omega} _ {1 1} \boldsymbol {S} _ {1 1} + \boldsymbol {\Omega} _ {1 2} \boldsymbol {S} _ {1 2} ^ {T} + \boldsymbol {\Omega} _ {1 2} ^ {T} \boldsymbol {S} _ {1 2} + \boldsymbol {\Omega} _ {2 2. 1} \boldsymbol {S} _ {2 2} + \boldsymbol {\Omega} _ {1 2} ^ {T} \boldsymbol {\Omega} _ {1 1} ^ {- 1} \boldsymbol {\Omega} _ {1 2} \boldsymbol {S} _ {2 2} \right], \tag {19}
$$
$$
\operatorname {T r} (\lambda \boldsymbol {\Omega}) = \operatorname {T r} \left[ \lambda \boldsymbol {\Omega} _ {1 1} + \lambda \boldsymbol {\Omega} _ {2 2. 1} + \lambda \boldsymbol {\Omega} _ {1 2} ^ {T} \boldsymbol {\Omega} _ {1 1} ^ {- 1} \boldsymbol {\Omega} _ {1 2} \right].
$$
By re-arranging the terms the posterior simplifies to:
$$
\begin{array}{l} p (\boldsymbol {\Omega} _ {1 1}, \boldsymbol {\Omega} _ {1 2}, \boldsymbol {\Omega} _ {2 2. 1}, \boldsymbol {S} | \lambda , q) \propto \det  (\boldsymbol {\Omega} _ {1 1}) ^ {n / 2} \det  (\boldsymbol {S}) ^ {\frac {n - (s + t) - 1}{2}} \\ \times \det  (\boldsymbol {\Omega} _ {2 2. 1}) ^ {n / 2} \exp \left(- \frac {1}{2} \operatorname {T r} \left[ \boldsymbol {\Omega} _ {2 2. 1} (\boldsymbol {S} _ {2 2} + \lambda \mathbf {1} _ {t}) \right]\right) \\ \times \exp \left(- \frac {1}{2} \operatorname {T r} \left[ \boldsymbol {\Omega} _ {1 1} \left(\boldsymbol {S} _ {1 1} + \lambda \mathbf {1} _ {s}\right) + 2 \left(\boldsymbol {\Omega} _ {1 2} ^ {T} \boldsymbol {S} _ {1 2}\right) + \boldsymbol {\Omega} _ {1 2} ^ {T} \boldsymbol {\Omega} _ {1 1} ^ {- 1} \boldsymbol {\Omega} _ {1 2} \left(\boldsymbol {S} _ {2 2} + \lambda \mathbf {1} _ {t}\right) \right]\right) \\ \times \prod_ {i <   j} \exp \left(- \lambda | (\Omega_ {1 1}) _ {i j} | ^ {q}\right) \prod_ {i, j} \exp \left(- \lambda | (\Omega_ {1 2}) _ {i j} | ^ {q}\right). \\ \end{array}
$$
If we now condition on  $S$ , and specifically on  $S_{22}$ , we can clearly see that the posterior factorizes as  $p(\Omega_{11}, \Omega_{12}, \Omega_{22.1}|S, \lambda, q) \propto p(\Omega_{11}, \Omega_{12}|S, \lambda, q) p(\Omega_{22.1}|S, \lambda, q)$ . In particular, we are interested in estimating the joint posterior  $p(\Omega_{11}, \Omega_{12}|S, \lambda, q)$ , which reads as
$$
\begin{array}{l} p (\boldsymbol {\Omega} _ {1 1}, \boldsymbol {\Omega} _ {1 2} | \boldsymbol {S}, \lambda , q) \propto \det  (\boldsymbol {\Omega} _ {1 1}) ^ {n / 2} \\ \times \exp \left(- \frac {1}{2} \operatorname {T r} \left[ \boldsymbol {\Omega} _ {1 1} \left(\boldsymbol {S} _ {1 1} + \lambda \mathbf {1} _ {s}\right) + 2 \left(\boldsymbol {\Omega} _ {1 2} ^ {T} \boldsymbol {S} _ {1 2}\right) + \boldsymbol {\Omega} _ {1 2} ^ {T} \boldsymbol {\Omega} _ {1 1} ^ {- 1} \boldsymbol {\Omega} _ {1 2} \left(\boldsymbol {S} _ {2 2} + \lambda \mathbf {1} _ {t}\right) \right] \right) \\ \times \prod_ {i <   j} \exp \left(- \lambda \left| \left(\Omega_ {1 1}\right) _ {i j} \right| ^ {q}\right) \prod_ {i, j} \exp \left(- \lambda \left| \left(\Omega_ {1 2}\right) _ {i j} \right| ^ {q}\right) \tag {20} \\ \end{array}
$$
A.3 Run-time comparison with Gibbs sampler
The proposed CMF provides a significant speed-up in terms of sampling with respect to standard Gibbs sampling algorithms for posterior inference in Gaussian Graphical Models. For a realistic and fair comparison, we measure run-time on our real data experiment. For this purpose, we trained the Conditional Matrix Flow for  $q = 1$  and compare its sampling speed against the Gibbs sampler introduced in Kaufmann et al. [2015]. The setting presented here is the same one used to obtain the results shown in the Experiment section. The Gibbs sampler takes 89 seconds to generate 500 samples, which translates to about 5.6 samples per second. This result was obtained on a Intel(R) Xeon(R) CPU E5-1660 v3 @ 3.00GHz. On the other hand, on the consumer-grade GPU NVIDIA TITAN X (12GB VRAM) sampling is extremely efficient: each second we can generate 2000 independent samples from the approximate posterior. Relevantly, in order to retrieve the full posterior path as a function of  $\lambda$ , we would need independent Markov Chains for each  $\lambda$  value, rendering the approach infeasible. In contrast, with the proposed framework we can draw independent samples for different  $\lambda$  values at the same computational cost.
A.4 Artificial data
(b)  $q = 0.75$
(a)  $q = 1.00$
(c)  $q = 0.50$
(d)  $q = 0.25$
Figure 6: MAP estimate as a function of  $\lambda$  for different sub- $l_{1}$  pseudo-norms. The dashed line is the frequentist solution path for  $q = 1$ .
Figure 7: Evolution of the (approximate) marginal log-likelihood of the CMF as a function of  $\lambda$  for  $T = 1.00$  and  $q = 1$ . We select the optimal  $\lambda$  as its maximum ( $\lambda_{\mathrm{CMF}} = 3.52$ , in blue) and compare with the frequentist estimate obtained through cross validation ( $\lambda_{\mathrm{Glasso}} = 3.36$ , in red).
A.5 Real data
Figure 8:  $95\%$  posterior credibility intervals of the proposed CMF for the 4 entries with highest absolute median posterior as a function of the pseudo-norm  $q = \{1,0.9,0.8,0.7,0.6\}$ .