SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The second term corresponds to the probability under  $\theta, P$  of observing zero-valued coordinates corresponding to the index set  $S$ , given the positive coordinates  $y_{S^c}$ :
$$
h _ {\theta , \Sigma} (y) = \log \int_ {t \leq 0} \exp \left(- \| P _ {S} ^ {\frac {1}{2}} \left(t - \theta_ {S}\right) + \left(P _ {S}\right) ^ {- 1 / 2} P _ {S S ^ {c}} \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right) \| ^ {2} / 2\right). \tag {22}
$$
The log-likelihood ratio is the difference between  $f_{\theta, \Sigma}$  and  $f_{\theta^{*}, \Sigma^{*}}$ , which we denote by
$$
\gamma_ {\theta , \Sigma} (y) := f _ {\theta , \Sigma} (y) - f _ {\theta^ {*}, \Sigma^ {*}} (y)
$$
Over a dataset  $\{(x_i, y_i)\}_{i \in [n]}$ , the average log-likelihood ratio is given by
$$
\bar {\gamma} _ {W, \Sigma} := \frac {1}{n} \sum_ {i} \gamma_ {W x _ {i}, \Sigma} (y _ {i}).
$$
Remark C.1. For ease of analysis, we will interchange between the precision matrix  $P$  in  $\gamma_{\theta ,P}$  and the covariance matrix  $\Sigma$  in  $\gamma_{\theta ,\Sigma}$ , and it should be understood that  $P = \Sigma^{-1}$ . The same applies to the functions  $g_{\theta ,\Sigma}$  and  $h_{\theta ,\Sigma}$ . Finally, the matrix  $P^{*}$  refers to the ground truth precision matrix  $(= \Sigma^{*-1})$ .
Analogous to Appendix B, we start by showing that the log-likelihood ratio is bounded by the noise in the sample. The proofs of results in this Section are in Subsection C.1.
Lemma C.2. Assume  $P^{*} \coloneqq \Sigma^{*-1}$  satisfies Assumption 4.4.
For all  $y = \phi(\theta^{*} + \eta)$  such that  $S$  denotes the zero-coordinates of  $y$ , and  $\eta$  such that  $\| P_{S}^{*\frac{1}{2}}\eta_{S}\|$ ,  $\| P_{S^c}^{*\frac{1}{2}}\eta_{S^c}\| \leq B$ , if the max eigenvalue  $\lambda_{\max}(P)$  satisfies
$$
\frac {\lambda_ {\operatorname* {m a x}} (P)}{\lambda_ {\operatorname* {m i n}} (P ^ {*})} \leq C,
$$
then for all  $\theta \in \mathbb{R}^d$ , we have
$$
\gamma_ {\theta , P} \leq \frac {d}{2} \log (C) + 3 B ^ {2}.
$$
For the ease of stating the next Lemma, we assume that across the samples of  $y$  in the training data, at least one coordinate has sufficiently many positive samples. The proof of our theorem separately handles cases violating this assumption.
Assumption C.3. Let  $\delta \in (0,1)$  be a parameter corresponding to the failure probability of our algorithm. Then, there exists a coordinate  $j \in [d]$ , such that for at least  $n' = O\left(\log \frac{1}{\delta}\right)$  samples  $y_{i_1}, \ldots, y_{i_{n'}}$  in the dataset, the  $j$ -th coordinate is positive.
This is a very weak assumption: if it is violated, then  $W = 0_{d \times k}, \Sigma = 0$  will achieve a TV distance smaller than  $\frac{2\varepsilon^2}{d}$ .
Appendix B assumed that the variance in  $y$  was 1. Since Section 4.2 considers an unknown  $\Sigma^{*}$ , we need the following Lemma to show that the MLE will select a precision matrix  $P$ , whose eigenvalues are reasonably bounded wrt  $\Sigma^{*-1}$ .
Lemma C.4. Under Assumption 4.4, C.3, consider  $P \in \mathbb{R}_+^{d \times d}$  such that  $\frac{\lambda_{\max}(P)}{\lambda_{\min}(P)} \leq \kappa$  and
$$
\frac {\lambda_ {\mathrm {m a x}} (P)}{\lambda_ {\mathrm {m a x}} (P ^ {*})} \geq O \bigg (\frac {\kappa^ {3} d ^ {2} n ^ {2}}{k ^ {2}} + \frac {B ^ {2} n \kappa}{k} \bigg).
$$
Then, for all  $W \in \mathbb{R}^{d \times k}$ , and for all  $y_i = \phi(W^* x_i + \eta_i)$  with  $\| P_{S^c}^{*\frac{1}{2}} \eta_{S^c} \|, \| P_S^{*\frac{1}{2}} \eta_S \| \leq B$ , we have
$$
\bar {\gamma} _ {W, P} := \frac {1}{n} \sum_ {i \in [ n ]} \gamma_ {W x _ {i}, P} (y _ {i}) <   0.
$$
Lemma C.2 and Lemma C.4 show that the MLE will only select precision matrices  $P$  that have max eigenvalues in a certain range of the true precision matric  $P^{*}$ .
Now, for matrices in the above eigenvalue range, we first construct a geometric net over the max eigenvalue  $\rho$  of the precision matrix, and then cover the matrices whose max eigenvalue is smaller than  $\rho$ .
Lemma C.5 (Σ cover). For  $B > 1$ , and  $0 < L < U$ , let  $A > \max \left\{\sqrt{\log\frac{1}{\varepsilon}}, B^2 U\kappa, \frac{d}{2}\log \left(\frac{\kappa U}{L}\right), 1\right\}$ .
Let  $P^{*} \coloneqq \Sigma^{*-1}$  be the precision matrix of  $\eta$ . Let  $\Omega \subset \mathbb{R}_{+}^{d \times d}$  denote the set of positive definite matrices  $P \in \mathbb{R}_{+}^{d \times d}$  with condition number  $\kappa$  and whose maximum eigenvalue lies in  $[L\lambda_{\min}(P^{*}), U \cdot \lambda_{\max}(P^{*})]$ .
Then, there exists a partition of  $\Omega$  of size
$$
\left(\operatorname {p o l y} \left(A, \frac {1}{\varepsilon}\right)\right) ^ {d ^ {2}}
$$
such that for all  $\theta \in \mathbb{R}^d$  and all  $y = \phi (\theta^{*} + \eta)\in \mathbb{R}^{d}$  with  $\| P_S^{*\frac{1}{2}}\eta_S\|$ ,  $\| P_{S^c}^{*\frac{1}{2}}\eta_{S^c}\| \leq B$ , and each cell  $I$  in the partition, one of the following holds:
for all  $P\in I$ $\gamma_{\theta ,P}(y) <   - A$  or
for all  $P, P' \in I$ , we have  $|\gamma_{\theta, P}(y) - \gamma_{\theta, P'}(y)| \leq \epsilon$ .
Analogous to Appendix B, we now construct a partition over  $W$  for a fixed precision matrix  $P$ , such that each cell in the partition has very small log-likelihood (in which case the MLE will not choose it) or the log-likelihood changes slowly.
Lemma C.6 (W-net). Let  $\eta_{S^c},\eta_S$  be such that
$$
\left\| P _ {S ^ {c}} ^ {* \frac {1}{2}} \eta_ {S ^ {c}} \right\| \leq B _ {1}, \left\| P _ {S} ^ {* \frac {1}{2}} \eta_ {S} \right\| \leq B _ {2},
$$
for  $B_{1},B_{2}\geq 0$
Let  $A > \max \{B_1^2, B_2^2, \mathrm{poly}(C, \kappa)\}$ . Let  $P^{*} = \Sigma^{* - 1}$  be the precision matrix of  $\eta$ . For a fixed matrix  $P \in \mathbb{R}^{d \times d}$  whose condition number satisfies Assumption 4.4 and whose eigenvalues satisfy  $\lambda_{\max}(P) \in [e^{-\frac{2A}{d}}\lambda_{\min}(P^{*}), C\lambda_{\max}(P^{*})]$ , there exists a partition  $\mathcal{I}$  of  $\mathbb{R}^d$  with size
$$
\left(\operatorname {p o l y} \left(A, \frac {1}{\varepsilon}\right)\right) ^ {3 d}
$$
such that for each interval  $I \in \mathcal{I}$ , we have one of the following:
for all  $\theta \in I, \gamma_{\theta, P}(y) < -A$ , or
for all  $\theta, \theta' \in I$ ,  $|\gamma_{\theta, P}(y) - \gamma_{\theta', P}(y)| \leq \epsilon$ .
Using the above lemmas, we can show that the MLE will only pick out  $\widehat{W},\widehat{P}$  such that they have small TV on the dataset of  $\{x_{i}\}$ .
Lemma C.7. Let  $x_{1},\ldots ,x_{n}$  be fixed, and  $y_{i} = \phi (W^{*}x_{i} + \eta_{i})$  for  $\eta_i\sim \mathcal{N}(0,\Sigma^*)$ , and  $W^{*}\in \mathbb{R}^{d\times k}$  with  $\Sigma^{*}\in \mathbb{R}^{d\times d}$  satisfying Assumption 4.4 and Assumption C.3. For a sufficiently large constant  $C > 0$ ,
$$
n = C \cdot \frac {(d ^ {2} + k d)}{\varepsilon^ {2}} \log \frac {k d \kappa}{\varepsilon}
$$
samples suffice to guarantee that with high probability, the MLE  $\widehat{W},\widehat{\Sigma}$  satisfies
$$
\widetilde {d} \Big ((\widehat {W}, \widehat {\Sigma}), (W ^ {*}, \Sigma^ {*}) \Big) \leq \varepsilon .
$$
Lemma C.8. Let  $\{x_i\}_{i=1}^n$  be i.i.d. random variables such that  $x_i \sim \mathcal{D}_x$ .
Let  $P^{*} \coloneqq \Sigma^{*-1}$ . Let  $\lambda_{\min}^{*}, \lambda_{\max}^{*}$  be the minimum and maximum eigenvalues of  $P^{*}$ . For  $0 < L < U$ , let  $\Omega$  denote the following set of precision matrices
$$
\Omega := \left\{P \in \mathbb {R} _ {+} ^ {d \times d}: \frac {\lambda_ {\operatorname* {m a x}} (P)}{\lambda_ {\operatorname* {m i n}} (P)} \leq \kappa \text {a n d} \lambda_ {\operatorname * {m a x}} (P) \in [ L \cdot \lambda_ {\operatorname * {m i n}} ^ {*}, U \cdot \lambda_ {\operatorname * {m a x}} ^ {*} ] \right\}.
$$
Then, for a sufficiently large constant  $C > 0$ , and for
$$
n = C \cdot \left(\frac {k d + d ^ {2}}{\varepsilon^ {2}}\right) \log \left(\frac {k d \kappa}{\varepsilon} \log \left(\frac {U}{L}\right)\right),
$$
$$
\Pr_ {x _ {i} \sim \mathcal {D} _ {x}} \left[ \sup  _ {W \in \mathbb {R} ^ {d \times k}, P \in \Omega} \left| \widetilde {d} ((W, P), (W ^ {*}, P ^ {*})) - d ((W, P), (W ^ {*}, P ^ {*})) \right| > \varepsilon \right] \leq e ^ {- \Omega \left(n \varepsilon^ {2}\right)}.
$$
Proof of Theorem 4.5. First, we consider the cases violating Assumption C.3.
As  $n \propto \frac{d^2}{\varepsilon^2} \log \frac{1}{\delta}$ , if assumption C.3 is violated, then it implies that each coordinate is non-zero in almost a  $\varepsilon^2 / d^2$  fraction of the samples, and a union bound implies that the probability of seeing a non-zero vector is almost  $\varepsilon^2 / d$ . Hence, with high probability over the draws of the data, returning the all-zeros vector always will achieve a TV distance smaller than  $\frac{2\varepsilon^2}{d}$ .
Let  $\widehat{P}, P^{*} = \widehat{\Sigma}^{-1}, \Sigma^{*-1}$ . Now, if Assumption C.3 holds, Lemma C.7 guarantees that the MLE has small TV on the  $x_{i}$  observed in the dataset:
$$
\widetilde {d} \left(\left(\widehat {W}, \widehat {P}\right), \left(W ^ {*}, P ^ {*}\right)\right) \leq \varepsilon .
$$
The above result is over the finite  $x_{i}$  observed in our dataset. To generalize it over  $x \sim \mathcal{D}_x$ , we use Lemma C.8, which gives
$$
d \left(\left(\widehat {W}, \widehat {P}\right), \left(W ^ {*}, P ^ {*}\right)\right) - \widetilde {d} \left(\left(\widehat {W}, \widehat {P}\right), \left(W ^ {*}, P ^ {*}\right)\right) \leq \varepsilon .
$$
Rescaling  $\varepsilon$  gives the conclusion of the Theorem.
C.1 Proofs of Appendix C.
$$
\frac {\lambda_ {\max } (P)}{\lambda_ {\min } (P ^ {*})} \leq C,
$$
$$
\gamma_ {\theta , \Sigma} \leq \frac {1}{2} \log \frac {| \Sigma^ {*} |}{| \Sigma |} + g _ {\theta , \Sigma} - g _ {\theta^ {*}, \Sigma^ {*}} + h _ {\theta , \Sigma} - h _ {\theta^ {*}, \Sigma^ {*}}. \tag {23}
$$
From Lemma C.9, C.10, we have
$$
g _ {\theta , \Sigma} - g _ {\theta^ {*}, \Sigma^ {*}} + h _ {\theta , \Sigma} - h _ {\theta^ {*}, \Sigma^ {*}} \leq g _ {\theta , \Sigma} + \frac {1}{2} \log \frac {| P _ {S} ^ {*} |}{| P _ {S} |} + 3 B ^ {2}.
$$
Substituting in Eqn (23), we get
$$
\gamma_ {\theta , \Sigma} \leq g _ {\theta , \Sigma} + \frac {1}{2} \log \frac {| \Sigma^ {*} |}{| \Sigma |} + \frac {1}{2} \log \frac {| P _ {S} ^ {*} |}{| P _ {S} |} + 3 B ^ {2}.
$$
As  $(P_S^*)^{-1} = \Sigma_S^* -\Sigma_{SS^c}^*\Sigma_{S^c}^{* - 1}\Sigma_{S^c S}^*$ , by the matrix determinant rule, we have
$$
\log | \Sigma^ {*} | + \log | P _ {S} ^ {*} | = \log | \Sigma_ {S ^ {c}} ^ {*} |.
$$
$$
\gamma_ {\theta , \Sigma} \leq g _ {\theta , \Sigma} + \frac {1}{2} \log \frac {| \Sigma_ {S ^ {c}} ^ {*} |}{| \Sigma_ {S ^ {c}} |} + 3 B ^ {2}.
$$
$$
\begin{array}{l} \gamma_ {\theta , \Sigma} \leq g _ {\theta , \Sigma} + \frac {d}{2} \log \frac {\lambda_ {\max } (\Sigma^ {*})}{\lambda_ {\min } (\Sigma)} + 3 B ^ {2}, \\ = g _ {\theta , \Sigma} + \frac {d}{2} \log \frac {\lambda_ {\max } (P)}{\lambda_ {\min } \left(P ^ {*}\right)} + 3 B ^ {2}. \tag {24} \\ \end{array}
$$
As the matrix  $\Sigma_{S^c}^{-1}$  is positive definite, we trivially get
$$
g _ {\theta_ {i}, \Sigma} (y) = - \left(y _ {i, S ^ {c}} - \theta_ {i, S ^ {c}}\right) ^ {T} \left(\Sigma_ {S ^ {c}}\right) ^ {- 1} \left(y _ {i, S ^ {c}} - \theta_ {i, S ^ {c}}\right) / 2 \leq 0.
$$
Substituting in Eqn (24), we get
$$
\gamma_ {\theta , \Sigma} \leq \frac {d}{2} \log \frac {\lambda_ {\mathrm {m a x}} (P)}{\lambda_ {\mathrm {m i n}} (P ^ {*})} + 3 B ^ {2}.
$$
As the Lemma assumes
$$
\lambda_ {\max } (P) \leq C \lambda_ {\min } (P ^ {*}),
$$
$$
\gamma_ {\theta , \Sigma} \leq \frac {d}{2} \log (C) + 3 B ^ {2}.
$$
Lemma C.9. Consider the function  $g$  defined in Eq (21). For the ground truth parameters  $\theta^{*}, \Sigma^{*}$ , the function  $g_{\theta^{*}, \Sigma^{*}}$  satisfies
$$
- g _ {\theta^ {*}, \Sigma^ {*}} \leq \frac {1}{2} \| \eta_ {S ^ {c}} \| _ {\Sigma_ {S ^ {c}}} ^ {2},
$$
which is, with probability  $1 - e^{-\Omega (d)}$
$$
- g _ {\theta^ {*}, \Sigma^ {*}} \leq O (d).
$$
Proof. As  $y_{S^c}$  are the positive valued coordinates in  $y$ , we have
$$
y _ {S ^ {c}} - \theta_ {S ^ {c}} ^ {*} = \eta_ {S ^ {c}},
$$
$$
\begin{array}{l} g _ {\theta^ {*}, \Sigma^ {*}} (y) = - \left(y _ {S ^ {c}} - \theta_ {S ^ {c}} ^ {*}\right) ^ {T} \left(\Sigma_ {S ^ {c}} ^ {*}\right) ^ {- 1} \left(y _ {S ^ {c}} - \theta_ {S ^ {c}} ^ {*}\right) / 2, \\ = - \left\| \eta_ {S ^ {c}} \right\| _ {\Sigma_ {S ^ {c}}} ^ {2} / 2. \\ \end{array}
$$
As  $\eta_{S^c}$  is Gaussian with covariance  $\Sigma_{Sc}^{*}$ , the expected norm is  $\frac{|S^c|}{2}$ , which implies that with probability  $1 - e^{-\Omega (|S^c|)}$ , we have
$$
- g _ {\theta^ {*}, \Sigma^ {*}} (y) \leq O (| S ^ {c} |).
$$
Lemma C.10. Consider  $y$  generated according to Eqn (20) by
$$
y = \phi (\theta^ {*} + \eta), \quad \eta \sim \mathcal {N} (0, \Sigma^ {*}).
$$
For all  $\theta \in \mathbb{R}^d, \Sigma \in \mathbb{R}_+^{d \times d}$ , and the function  $h_{\theta, \Sigma}$  defined in Eqn (22), the difference  $h_{\theta, \Sigma}(y) - h_{\theta^*, \Sigma^*}(y)$  satisfies
$$
h _ {\theta , \Sigma} (y) - h _ {\theta^ {*}, \Sigma^ {*}} (y) \leq \frac {1}{2} \log \frac {\left| P _ {S} ^ {*} \right|}{\left| P _ {S} \right|} + \left\| P _ {S ^ {c}} ^ {* \frac {1}{2}} \eta_ {S ^ {c}} \right\| ^ {2} + 2 \left\| P _ {S} ^ {* \frac {1}{2}} \eta_ {S} \right\| ^ {2} - \left\| \eta_ {S ^ {c}} \right\| _ {\Sigma_ {S ^ {c}}} ^ {2} + O (| S |), \tag {25}
$$
where  $P^{*} = \Sigma^{*-1}$  is the precision matrix of  $\eta$ .
Proof. For  $\theta \in \mathbb{R}^d$ ,  $\Sigma \in \mathbb{R}_+^{d\times d}$ , and  $P = \Sigma^{-1}$ , we have
$$
\begin{array}{l} h _ {\theta , \Sigma} (y) = \log \int_ {t \leq 0} \exp \left(- \| P _ {S} ^ {\frac {1}{2}} (t - \theta_ {S}) + (P _ {S}) ^ {- 1 / 2} P _ {S S ^ {c}} (y _ {S ^ {c}} - \theta_ {S ^ {c}}) \| ^ {2} / 2\right), \\ \leq \log \int_ {t \in \mathbb {R} ^ {| S |}} \exp \Big (- \| P _ {S} ^ {\frac {1}{2}} (t - \theta_ {S}) + (P _ {S}) ^ {- 1 / 2} P _ {S S ^ {c}} (y _ {S ^ {c}} - \theta_ {S ^ {c}}) \| ^ {2} / 2 \Big), \\ \leq \frac {| S |}{2} \log (2 \pi) - \frac {1}{2} \log | P _ {S} |, \tag {26} \\ \end{array}
$$
where the last step follows from the integral of a Gaussian pdf. This gives a sufficient upper bound on  $h_{\theta, \Sigma}(y)$ , and now we will focus on lower bounding  $h_{\theta^{*}, \Sigma^{*}}(y)$ .
For the coordinates of  $y$  in  $S^c$ , we have  $y_{S^c} - \theta_{S^c}^* = \eta_{S^c}$ . Substituting in Eqn (22), we get
$$
\begin{array}{l} h _ {\theta^ {*}, \Sigma^ {*}} (y) = \log \int_ {t \leq 0} \exp \Bigl (- \| P _ {S} ^ {*} ^ {\frac {1}{2}} (t - \theta_ {S} ^ {*}) + (P _ {S} ^ {*}) ^ {- 1 / 2} P _ {S S ^ {c}} ^ {*} \eta_ {S ^ {c}} \| ^ {2} / 2 \Bigr), \\ = \log \int_ {t \leq 0} \exp \left(- \| P _ {S} ^ {* \frac {1}{2}} (t - \theta_ {S} ^ {*}) + (P _ {S} ^ {*}) ^ {- 1 / 2} P _ {S S ^ {c}} ^ {*} \eta_ {S ^ {c}} \| ^ {2} / 2\right). \\ \end{array}
$$
Using  $\| a + b\| ^2\leq 2a^2 +2b^2$  , we get
$$
h _ {\theta^ {*}, \Sigma^ {*}} (y) \geq - \| \left(P _ {S} ^ {*}\right) ^ {- 1 / 2} P _ {S S ^ {c}} ^ {*} \eta_ {S ^ {c}} \| ^ {2} + \log \int_ {t \leq 0} \exp \left(- \| P _ {S} ^ {* \frac {1}{2}} \left(t - \theta_ {S} ^ {*}\right) \| ^ {2}\right).
$$
Set  $u \coloneqq P_S^{*\frac{1}{2}}(t - \theta_S^*)$ , and by the change of variables formula, we get:
$$
\begin{array}{l} h _ {\theta^ {*}, \Sigma^ {*}} (y) \geq - \| \left(P _ {S} ^ {*}\right) ^ {- 1 / 2} P _ {S S ^ {c}} ^ {*} \eta_ {S ^ {c}} \| ^ {2} + \log \int_ {P _ {S} ^ {*} - \frac {1}{2} u + \theta_ {S} ^ {*} \leq 0} \left| \left(P _ {S} ^ {*}\right) ^ {- 1 / 2} \right| \cdot \exp \left(- \| u \| ^ {2}\right), \\ = - \left\| \left(P _ {S} ^ {*}\right) ^ {- 1 / 2} P _ {S S ^ {c}} ^ {*} \eta_ {S ^ {c}} \right\| ^ {2} + \frac {1}{2} \log \left| P _ {S} ^ {* - 1} \right| + \log \int_ {P _ {S} ^ {* - \frac {1}{2}} u + \theta_ {S} ^ {*} \leq 0} \exp \left(- \| u \| ^ {2}\right). \\ \end{array}
$$
For  $i\in S$  , we have  $\theta_i^* +\eta_i\leq 0$  . This gives
$$
P _ {S} ^ {* - \frac {1}{2}} u \leq \eta_ {S} \Rightarrow P _ {S} ^ {* - \frac {1}{2}} u + \theta_ {S} ^ {*} \leq 0 \Rightarrow \log \int_ {P _ {S} ^ {* - \frac {1}{2}} u + \theta_ {S} ^ {*} \leq 0} \exp \bigl (- \| u \| ^ {2} \bigr) \geq \log \int_ {P _ {S} ^ {* - \frac {1}{2}} u \leq \eta_ {S}} \exp \bigl (- \| u \| ^ {2} \bigr),
$$
using which we get
$$
h _ {\theta^ {*}, \Sigma^ {*}} (y) \geq - \| (P _ {S}) ^ {* - 1 / 2} P _ {S S ^ {c}} ^ {*} \eta_ {S ^ {c}} \| ^ {2} - \frac {1}{2} \log | P _ {S} ^ {*} | + \log \int_ {P _ {S} ^ {*} - \frac {1}{2} u \leq \eta_ {S}} \exp \big (- \| u \| ^ {2} \big).
$$
By another change of variables via  $v := P_S^{*\frac{1}{2}}u - \eta_S$ , we get
$$
\begin{array}{l} h _ {\theta^ {*}, \Sigma^ {*}} (y) \geq - \| \left(P _ {S} ^ {*}\right) ^ {- 1 / 2} P _ {S S ^ {c}} ^ {*} \eta_ {S ^ {c}} \| ^ {2} - \frac {1}{2} \log | P _ {S} ^ {*} | + \log \int_ {v \leq 0} \left| P _ {S} ^ {* \frac {1}{2}} \right| \exp \left(- \| P _ {S} ^ {* \frac {1}{2}} (v + \eta_ {S}) \| ^ {2}\right), \\ \geq - \left\| \left(P _ {S} ^ {*}\right) ^ {- 1 / 2} P _ {S S ^ {c}} ^ {*} \eta_ {S ^ {c}} \right\| ^ {2} - \frac {1}{2} \log | P _ {S} ^ {*} | - 2 \| P _ {S} ^ {* \frac {1}{2}} \eta_ {S} \| ^ {2} + \log \int_ {v \leq 0} \left| P _ {S} ^ {* \frac {1}{2}} \right| \exp \left(- 2 \| P _ {S} ^ {* \frac {1}{2}} v \| ^ {2}\right), \\ = - \left\| \left(P _ {S} ^ {*}\right) ^ {- 1 / 2} P _ {S S ^ {c}} ^ {*} \eta_ {S ^ {c}} \right\| ^ {2} - \frac {1}{2} \log | P _ {S} ^ {*} | - 2 \| P _ {S} ^ {* \frac {1}{2}} \eta_ {S} \| ^ {2} + O (| S |). \\ \end{array}
$$
As  $(\Sigma_{S^c}^*)^{-1} = P_{S^c} - P_{S^c S}^*(P_S^*)^{-1}P_{SS^c}$ , we have
$$
- \| (P _ {S}) ^ {* - 1 / 2} P _ {S S ^ {c}} ^ {*} \eta_ {S ^ {c}} \| ^ {2} = \| \eta_ {S ^ {c}} \| _ {\Sigma_ {S ^ {c}} ^ {*}} ^ {2} - \| P _ {S ^ {c}} ^ {* 1 / 2} \eta_ {S ^ {c}} \| ^ {2},
$$
$$
h _ {\theta^ {*}, \Sigma^ {*}} (y) \geq \| \eta_ {S ^ {c}} \| _ {\Sigma_ {S ^ {c}}} ^ {2} - \| P _ {S ^ {c}} ^ {* \frac {1}{2}} \eta_ {S ^ {c}} \| ^ {2} - \frac {1}{2} \log | P _ {S} ^ {*} | - 2 \| P _ {S} ^ {* \frac {1}{2}} \eta_ {S} \| ^ {2} + O (| S |). \tag {27}
$$
From Eqn (26) - Eqn (27), we get
$$
h _ {\theta , \Sigma} (y) - h _ {\theta^ {*}, \Sigma^ {*}} (y) \leq \| P _ {S ^ {c}} ^ {* \frac {1}{2}} \eta_ {S ^ {c}} \| ^ {2} + 2 \| P _ {S} ^ {* \frac {1}{2}} \eta_ {S} \| ^ {2} - \| \eta_ {S ^ {c}} \| _ {\Sigma_ {S ^ {c}} ^ {*}} ^ {2} + \frac {1}{2} \log \frac {| P _ {S} ^ {*} |}{| P _ {S} |} + O (| S |). \tag {28}
$$
$$
\frac {\lambda_ {\operatorname* {m a x}} (P)}{\lambda_ {\operatorname* {m a x}} (P ^ {*})} \geq O \bigg (\frac {\kappa^ {3} d ^ {2} n ^ {2}}{k ^ {2}} + \frac {B ^ {2} n \kappa}{k} \bigg).
$$
Proof of Lemma C.4. For each  $W \in \mathbb{R}^{d \times k}$ , let
$$
\theta_ {i} := W x _ {i}.
$$
From Eqn (24) in Lemma C.2, for each  $i\in [n]$ , we have,
$$
\begin{array}{l} \gamma_ {\theta_ {i}, \Sigma} \leq g _ {\theta_ {i}, \Sigma} + \frac {d}{2} \log \frac {\lambda_ {\max } (P)}{\lambda_ {\min } (P ^ {*})} + 3 B ^ {2}, \\ \leq g _ {\theta_ {i}, \Sigma} + \frac {d}{2} \log \frac {\kappa \lambda_ {\max } (P)}{\lambda_ {\max } (P ^ {*})} + 3 B ^ {2}. \\ \end{array}
$$
Now consider
$$
\begin{array}{l} g _ {\theta_ {i}, \Sigma} (y) = - \frac {1}{2} (y _ {i, S ^ {c}} - \theta_ {i, S ^ {c}}) ^ {T} (\Sigma_ {S ^ {c}}) ^ {- 1} (y _ {i, S ^ {c}} - \theta_ {i, S ^ {c}}), \\ \leq - \frac {1}{2} \| y - \theta \| ^ {2} \lambda_ {\min } \left(\Sigma_ {S ^ {c}} ^ {- 1}\right), \\ \leq - \frac {1}{2} \| y - \theta \| ^ {2} \lambda_ {\min } (\Sigma^ {- 1}) = - \frac {1}{2} \| y - \theta \| ^ {2} \lambda_ {\min } (P), \\ \leq - \frac {1}{2} \| y - \theta \| ^ {2} \frac {\lambda_ {\max } (P)}{\kappa}, \\ \end{array}
$$
where the second inequality comes from the eigenvalue interlacing Theorem, and the last line follows from the condition number assumption on  $\Sigma, P$ .
By Assumption C.3, there exist at least  $\varepsilon^2 n$  samples for a coordinate  $j$  such that  $(y_i)_j > 0$ . Averaging  $g_{\theta_i,\Sigma}$ , by Lemma A.1, we get that with high probability,
$$
\sum_ {i} \left\| y _ {i} - \theta_ {i} \right\| ^ {2} \geq \frac {\sigma_ {j} ^ {* 2} k}{2},
$$
$$
\begin{array}{l} \frac {1}{n} \sum_ {i} g _ {\theta_ {i}, \Sigma} (y _ {i}) \leq - \frac {\sigma_ {j} ^ {* 2} k \lambda_ {\max} (P)}{4 n \kappa}, \\ \leq - \frac {k \lambda_ {\mathrm {m a x}} (P)}{4 n \kappa \lambda_ {\mathrm {m a x}} (P ^ {*})}. \\ \end{array}
$$
$$
\begin{array}{l} \bar {\gamma} _ {W, \Sigma} \leq - \frac {\lambda_ {\max } (P) k}{4 n \kappa \lambda_ {\max } (P ^ {*})} + \frac {d}{2} \log \left(\kappa \cdot \frac {\lambda_ {\max } (P)}{\lambda_ {\max } (P ^ {*})}\right) + 3 B ^ {2}, \\ \leq - \frac {\lambda_ {\operatorname* {m a x}} (P) k}{4 n \kappa \lambda_ {\operatorname* {m a x}} (P ^ {*})} + d \sqrt {\kappa \cdot \frac {\lambda_ {\operatorname* {m a x}} (P)}{\lambda_ {\operatorname* {m a x}} (P ^ {*})}} + 3 B ^ {2}. \\ \end{array}
$$
Completing the squares, we get
$$
\bar {\gamma} _ {W, \Sigma} \leq - \left(\sqrt {\frac {\lambda_ {\max} (P) k}{4 n \kappa \lambda_ {\max} (P ^ {*})}} - \kappa d \sqrt {\frac {n}{k}}\right) ^ {2} + \frac {\kappa^ {2} d ^ {2} n}{k} + 3 B ^ {2}.
$$
$$
\frac {\lambda_ {\operatorname* {m a x}} (P)}{\lambda_ {\operatorname* {m a x}} (P ^ {*})} \geq O \bigg (\frac {\kappa^ {3} d ^ {2} n ^ {2}}{k ^ {2}} + \frac {B ^ {2} n \kappa}{k} \bigg),
$$
the above inequality satisfies
$$
\bar {\gamma} _ {W, \Sigma} \leq 0.
$$
Lemma C.11. Assume  $P^{*} \coloneqq \Sigma^{*-1}$  satisfies Assumption 4.4 with condition number  $\kappa$ .
For all  $y = \phi(\theta^{*} + \eta)$  such that  $S$  denotes the zero-coordinates of  $y$ , and  $\eta$  such that  $\| P_{S}^{*\frac{1}{2}}\eta_{S}\|, \| P_{S^{c}}^{*\frac{1}{2}}\eta_{S^{c}}\| \leq B$ , consider precision matrices  $P$  whose max eigenvalue  $\lambda_{\max}(P)$  satisfies
$$
\frac {\lambda_ {\operatorname* {m a x}} (P)}{\lambda_ {\operatorname* {m i n}} (P ^ {*})} \leq C.
$$
Let  $A \geq 4\max \left\{\frac{d}{2}\log C,3B^2\right\}$ . Then for  $V \coloneqq (P^{-1})_{S^c}$  and  $R_P$  defined as
$$
R _ {P} := 2 B \sqrt {C} + \sqrt {\frac {3}{2} A}, \tag {29}
$$
$$
\left\| \theta_ {S ^ {c}} - \theta_ {S ^ {c}} ^ {*} \right\| _ {V} \geq R _ {P} \Rightarrow \gamma_ {\theta , P} \leq - A.
$$
Proof of Lemma C.11. Consider Eqn (24) in Lemma C.2. We have
$$
\begin{array}{l} \gamma_ {\theta , P} \leq g _ {\theta , P} + \frac {d}{2} \log \frac {\lambda_ {\max } (P)}{\lambda_ {\min } \left(P ^ {*}\right)} + 3 B ^ {2}, \\ \leq g _ {\theta , P} + \frac {d}{2} \log C + 3 B ^ {2}, \\ \end{array}
$$
where the last inequality follows from  $\frac{\lambda_{\max}(P)}{\lambda_{\min}(P^*)} \leq C$  in the statement of the Lemma.
By the definition of  $g_{\theta ,P}$  , we have
$$
g _ {\theta , P} := - \frac {1}{2} \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right) ^ {T} \left(P _ {S ^ {c}} - P _ {S ^ {c} S} P _ {S} ^ {- 1} P _ {S S ^ {c}}\right) \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right).
$$
We can rewrite the matrix  $(P_{S^c} - P_{S^{c_S}}P_S^{-1}P_{SS^c})$  as
$$
\left(P _ {S ^ {c}} - P _ {S ^ {c} S} P _ {S} ^ {- 1} P _ {S S ^ {c}}\right) = \left(\left(P ^ {- 1}\right) _ {S ^ {c}}\right) ^ {- 1}.
$$
By setting
$$
V := (P ^ {- 1}) _ {S ^ {c}},
$$
we can rewrite  $g_{\theta ,P}$  as
$$
g _ {\theta , P} := - \frac {1}{2} \| y _ {S ^ {c}} - \theta_ {S ^ {c}} \| _ {V} ^ {2} = - \frac {1}{2} (y _ {S ^ {c}} - \theta_ {S ^ {c}}) ^ {T} V ^ {- 1} (y _ {S ^ {c}} - \theta_ {S ^ {c}}).
$$
Now, as  $y_{S^c} = \eta_{S^c} + \theta_{S^c}^*$ , we have
$$
\begin{array}{l} g _ {\theta , P} = - \frac {1}{2} \| \eta_ {S ^ {c}} + \theta_ {S ^ {c}} ^ {*} - \theta_ {S ^ {c}} \| _ {V} ^ {2}, \\ = - \frac {1}{2} \| \theta_ {S ^ {c}} ^ {*} - \theta_ {S ^ {c}} \| _ {V} ^ {2} + \| \eta_ {S ^ {c}} \| _ {V} \| \theta_ {S ^ {c}} ^ {*} - \theta_ {S} \| - \frac {1}{2} \| \eta_ {S ^ {c}} \| _ {V} ^ {2}. \\ \end{array}
$$
Ignoring the  $\| \eta_{S^c}\| _V^2$  term, we get
$$
g _ {\theta , P} \leq - \frac {1}{2} \| \theta_ {S ^ {c}} ^ {*} - \theta_ {S ^ {c}} \| _ {V} ^ {2} + \| \eta_ {S ^ {c}} \| _ {V} \| \theta_ {S ^ {c}} ^ {*} - \theta_ {S} \| _ {V}.
$$
By the Cauchy-Schwartz inequality and the eigenvalue interlacing theorem, we have