SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\| \eta_ {S ^ {c}} \| _ {V} \leq \lambda_ {\max} ^ {\frac {1}{2}} (V ^ {- 1}) \cdot \| \eta_ {S ^ {c}} \| _ {2} = \frac {\| \eta_ {S ^ {c}} \| _ {2}}{\lambda_ {\min} ^ {\frac {1}{2}} (V)} = \frac {\| \eta_ {S ^ {c}} \| _ {2}}{\lambda_ {\min} ^ {\frac {1}{2}} (P _ {S ^ {c}} ^ {- 1})} \leq \frac {\| \eta_ {S ^ {c}} \| _ {2}}{\lambda_ {\min} ^ {\frac {1}{2}} (P ^ {- 1})} = \lambda_ {\max} ^ {\frac {1}{2}} (P) \cdot \| \eta_ {S ^ {c}} \| _ {2}
$$
By the statement of the Lemma, we have  $\| P_{S^c}^{*\frac{1}{2}}\eta_{S^c}\| \leq B \Rightarrow \| \eta_{S^c}\|_2 \leq \frac{B}{\lambda_{\min}^{\frac{1}{2}}(P_{S^c}^*)}$ . Substituting in the above inequality, we get
$$
\| \eta_ {S ^ {c}} \| _ {V} \leq \frac {\lambda_ {\max } ^ {\frac {1}{2}} (P) \cdot \| \eta_ {S ^ {c}} \| _ {2}}{\lambda_ {\min } ^ {\frac {1}{2}} (P _ {S ^ {c}} ^ {*})} \leq \sqrt {C} B.
$$
Substituting in the function  $g_{\theta ,P}$  , we get
$$
g _ {\theta , P} \leq - \frac {1}{2} \| \theta_ {S ^ {c}} ^ {*} - \theta_ {S ^ {c}} \| _ {V} ^ {2} + B \sqrt {C} \| \theta_ {S ^ {c}} ^ {*} - \theta_ {S} \| _ {V}.
$$
Hence, for  $\theta$  satisfying
$$
\left\| \theta_ {S ^ {c}} - \theta_ {S ^ {c}} ^ {*} \right\| _ {V} \geq R _ {P} := 2 B \sqrt {C} + 2 \sqrt {A},
$$
$$
\gamma_ {\theta , P} \leq - A.
$$
In order to cover our precision matrices, we will consider a subset of matrices whose entries are quantized by an interval size  $\beta$ :
Definition C.12 (Quantized Precision Matrices). For  $\kappa > 0$ , define  $\Omega \subset \mathbb{R}^{d \times d}$  as the set of positive definite matrices with condition number  $\kappa$ .
For  $\rho > 0$ , define the set  $\Omega_{\rho} \subset \Omega$  as
$$
\Omega_ {\rho} := \left\{P \in \Omega : \lambda_ {\max } (P) \in \left[ \frac {\rho}{2}, \rho \right] \right\}
$$
For a quantization size  $\beta > 0$ , define  $\widetilde{\Omega}_{\rho, \beta} \subset \Omega_{\rho}$  as:
$$
\widetilde {\Omega} _ {\rho , \beta} := \left\{P \in \Omega_ {\rho}: P _ {i j} \in \{- \rho , - \rho (1 - \beta), - \rho (1 - 2 \beta), \dots , \rho (1 - 2 \beta), \rho (1 - \beta), \rho \}. \right\}
$$
such that for all  $\theta \in \mathbb{R}^d$  and all  $y = \phi (\theta^{*} + \eta)\in \mathbb{R}^{d}$  with  $\| P_S^*\frac{1}{2}\eta_S\|$ ,  $\| P_{S^c}^{\frac{1}{2}}\eta_{S^c}\| \leq B$ , and each cell  $I$  in the partition, one of the following holds:
Proof. In order to construct the net over the precision matrices, we will consider geometrically spaced values of  $\rho \in [L\cdot \lambda_{\min}(P^{*}),U\cdot \lambda_{\max}(P^{*})]$ , and for each  $\rho$ , we will construct a net over matrices that have max eigenvalue  $\leq \rho$ .
Now consider  $\rho > 0$  that lies in the following discrete set:
$$
\left. \left\{\lambda_ {\min } (P ^ {*}) 2 ^ {j}, j \in \lceil \log_ {2} (\kappa_ {L} ^ {U}) \rceil \right\} \right.
$$
This set is a geometric partition over the possible max eigenvalues that the MLE can return.
For the current  $\rho$ , let  $\Omega_{\rho}$  follow Definition C.12. Now consider  $P \in \Omega_{\rho}$ .
Constructing the interval for which  $\gamma_{\theta, P} < -A$ . By Lemma C.11, for  $V = (P^{-1})_{S^c}$ , and  $R_P = O\left(B \sqrt{\frac{\rho}{\lambda_{\min}(P^*)}} + \sqrt{A}\right) = O(\sqrt{A})$ , we have
$$
\left\| \theta_ {S ^ {c}} - \theta_ {S ^ {c}} ^ {*} \right\| _ {V} \geq R _ {P} \Rightarrow \gamma_ {\theta , P} <   - A.
$$
For any  $\theta$ , notice that the set of matrices  $P$  satisfying  $\| \theta_{S^c} - \theta_{S^c}^* \|_V \geq R_P$  is connected (as its complement is compact). This forms the set  $I$  for which  $\gamma_{\theta, P} < -A$ .
Constructing intervals for which  $|\gamma_{\theta, P} - \gamma_{\theta, P'}| \leq \varepsilon$ . We will now construct a partition over those  $P$  which satisfy  $\| \theta_{Sc} - \theta_{Sc}^* \|_V < R_P$ , and show that the log-likelihood changes by at most  $\varepsilon$  for each cell in this partition.
If  $P \in \Omega_{\rho}$ , then each of its elements  $P_{ij} \in [-\rho, \rho]$ . For a parameter  $\beta > 0$  that we will specify later, consider the partition  $\widetilde{\Omega}_{\rho,\beta}$  of  $\Omega_{\rho}$ , following Definition C.12. Clearly, the size of  $\widetilde{\Omega}_{\rho,\beta}$  can be upper bounded by
$$
\left| \widetilde {\Omega} _ {\rho , \beta} \right| \leq \left(\frac {2}{\beta}\right) ^ {d ^ {2}}.
$$
We will now analyze the effect of rounding down  $P \in \Omega_{\rho}$  to its nearest element in  $\widetilde{\Omega}_{\rho, \beta}$ .
By Claim C.13, for  $\gamma = 2\kappa \beta d^2$ , we have
$$
(1 - \gamma) \| t - \theta \| _ {\Sigma} ^ {2} \leq \| t - \theta \| _ {\Sigma^ {\prime}} ^ {2} \leq (1 + \gamma) \| t - \theta \| _ {\Sigma} ^ {2}, \tag {30}
$$
Consider the log-likelihood at  $\theta, P'$ :
$$
f _ {\theta , P ^ {\prime}} (y) = \frac {1}{2} \log | P ^ {\prime} | + \log \int_ {t: t _ {S} \leq 0, t _ {S ^ {c}} = y _ {S ^ {c}}} \exp \left(- \| t - \theta \| _ {\Sigma^ {\prime}} ^ {2}\right).
$$
We will use the LHS of Eqn (30) to show that
$$
f _ {\theta , P ^ {\prime}} (y) - \frac {1}{2} \log | P ^ {\prime} | \leq f _ {\theta , P} (y) - \frac {1}{2} \log | P | + \varepsilon ,
$$
and deal with the  $\log |P^{\prime}|$  term later. The lower bound for the log-likelihood at  $P^{\prime}$  can be obtained via analogous proof using the RHS of Eqn (30).
By the LHS of Eqn (30), we get
$$
f _ {\theta , P ^ {\prime}} (y) - \frac {1}{2} \log | P ^ {\prime} | \leq \log \int_ {t: t _ {S} \leq 0, t _ {S ^ {c}} = y _ {S ^ {c}}} \exp \big (- (1 - \gamma) \| t - \theta \| _ {\Sigma} ^ {2} \big).
$$
Rearranging the terms, we get
$$
\begin{array}{l} f _ {\theta , P ^ {\prime}} (y) - \frac {1}{2} \log | P ^ {\prime} | \leq - \frac {(1 - \gamma)}{2} \| y _ {S ^ {c}} - \theta_ {S ^ {c}} \| _ {\Sigma_ {S ^ {c}}} ^ {2} \\ + \log \int_ {t \leq 0} \exp \left(- \frac {(1 - \gamma)}{2} \| P _ {S} ^ {\frac {1}{2}} (t - \theta) _ {S} + P _ {S} ^ {- \frac {1}{2}} P _ {S S ^ {c}} \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right) \| ^ {2}\right) \\ \end{array}
$$
The non-integral term corresponds to  $g_{\theta, P}$  in Eqn (21), while the integral term corresponds to  $h_{\theta, P}$  in Eqn (22).
Handling the non-integral term. As we are only considering  $\theta$  such that  $\| y_{S^c} - \theta_{S^c}\|_{\Sigma_{S^c}}\leq R_P$ , we have that for
$$
\beta = O \left(\frac {\varepsilon}{R _ {P} ^ {2} d ^ {2} \kappa}\right) = O \left(\frac {\varepsilon}{\operatorname {p o l y} (A)}\right),
$$
the non-integral term corresponds to  $g_{\theta ,P} + \varepsilon$  , which gives
$$
\left. f _ {\theta , P ^ {\prime}} (y) - \frac {1}{2} \log | P ^ {\prime} | \leq g _ {\theta , P} + \varepsilon + \log \int_ {t \leq 0} \exp \left(- \frac {(1 - \gamma)}{2} \| P _ {S} ^ {\frac {1}{2}} (t - \theta) _ {S} + P _ {S} ^ {- \frac {1}{2}} P _ {S S ^ {c}} \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right) \| ^ {2}\right) \right. \tag {31}
$$
Handling the integral. Now we consider the integral term. Define the integral
$$
I _ {1} = \log \int_ {t \leq 0} \exp \left(- \frac {(1 - \gamma)}{2} \| P _ {\mathcal {S}} ^ {\frac {1}{2}} (t - \mu) \| ^ {2}\right)
$$
for  $\gamma = 2d^{2}\kappa \beta$  and  $\mu = \theta_{S} - P_{S}^{-1}P_{SS^c}(y_{S^c} - \theta_{S^c})$ .
Define the analogous integral that does not have the  $(1 - \gamma)$  term in the exponential:
$$
I _ {2} = \log \int_ {t \leq 0} \exp \left(- \frac {1}{2} \| P _ {S} ^ {\frac {1}{2}} (t - \mu) \| ^ {2}\right)
$$
Clearly,  $I_1 \geq I_2$ .
We only need to consider  $\mu$  such that  $\| \mu \|_{\infty} \leq O(\sqrt{A}\rho)$ : otherwise the likelihood will be smaller than  $-A$ .
By Lemma C.14, for  $\gamma = O\left(\frac{\varepsilon}{\mathrm{poly}(A)}\right)$ , we have
$$
I _ {1} \leq I _ {2} + \varepsilon .
$$
Handling the log-determinant term. Now consider the  $\log |P|$  term. As we are decreasing each element by atmost  $\beta \rho$ , none of the eigenvalues can increase. Moreover, as
$$
\operatorname {T r} \left(P ^ {\prime}\right) - \operatorname {T r} (P) \geq - d \beta \rho ,
$$
we can conclude that each eigenvalue decreases by at most  $-d\beta \rho$ . Also, as  $\rho \leq \kappa \lambda_j(P) \forall j \in [d]$ , we can conclude that each eigenvalue satisfies
$$
\lambda_ {j} \left(P ^ {\prime}\right) \geq \lambda_ {j} (P) \left(1 - d \beta \kappa\right).
$$
Hence, the log-determinant satisfies
$$
\log | P ^ {\prime} | \geq \log | P | + d \log (1 - \beta d \kappa) \geq \log | P | - \frac {d ^ {2} \beta \kappa}{1 - d \beta \kappa} \geq \log | P | - O (\varepsilon) \mathrm {f o r} \beta \leq \frac {\varepsilon}{\kappa d ^ {2}} \leq \frac {\varepsilon}{\kappa r ^ {2} d ^ {2}}.
$$
This finally gives
$$
\left| \gamma_ {\theta , P} - \gamma_ {\theta , P ^ {\prime}} \right| \leq O (\varepsilon).
$$
Bounding the size of the net As  $\beta = O\left(\frac{\varepsilon}{\mathrm{poly}(A)}\right)$ , and the max radius is also  $O(\mathrm{poly}(A))$ , we have a cover of size  $\left(\frac{\mathrm{poly}(A)}{\varepsilon}\right)$  per entry of the precision matrix (for a fixed  $\Omega_{\rho}$ ).
Intersecting the  $d^2$  nets means that for each  $\Omega_{\rho}$ , we have a net of size
$$
\left(\operatorname {p o l y} \left(A, \frac {1}{\varepsilon}\right)\right) ^ {d ^ {2}}.
$$
As we are considering  $\mathrm{poly}(A)$  many  $\Omega_{\rho}\mathrm{s}$ , the size of the net remains the same as the above.
Claim C.13. In the setting of Lemma C.5, if  $P \in \Omega_{\rho}$  and  $P' \in \widetilde{\Omega}_{\rho,\beta}$  is its nearest neighbor, then for  $\gamma = 2\kappa\beta d^2$ , we have
$$
(1 - \gamma) \| t - \theta \| _ {\Sigma} ^ {2} \leq \| t - \theta \| _ {\Sigma^ {\prime}} ^ {2} \leq (1 + \gamma) \| t - \theta \| _ {\Sigma} ^ {2}, \tag {32}
$$
where  $\Sigma := P^{-1}, \Sigma' := P'^{-1}$ .
Proof. Consider  $P \in \Omega_{\rho}$  and  $P' \in \widetilde{\Omega}_{\rho, \beta}$  such that  $P = P' + \Delta$ . Since  $P'$  is the rounding down of  $P$ , we have  $\Delta_{ij} \in [0, \beta \rho]$ .
As  $\operatorname{Tr}(\Delta) \in [0, \rho \beta d]$ , and  $\| \Delta \|_F \leq \rho \beta d$ , we have
$$
\lambda_ {\max } (\Delta) \leq \rho \beta d \text {a n d} \lambda_ {\min } (\Delta) \geq - \rho \beta d ^ {2}.
$$
This implies that when considering untruncated Gaussians with precision matrices  $P, P'$ , we have that for all  $t, \theta \in \mathbb{R}^d$ ,
$$
\left\| t - \theta \right\| _ {\Sigma} ^ {2} - \rho \beta d ^ {2} \left\| t - \theta \right\| ^ {2} \leq \left\| t - \theta \right\| _ {\Sigma^ {\prime}} ^ {2} \leq \left\| t - \theta \right\| _ {\Sigma} ^ {2} + \rho \beta d \left\| t - \theta \right\| ^ {2}.
$$
Since  $\lambda_{\min}(P) \geq \frac{\rho}{2\kappa}$ , we have
$$
\rho \| t - \theta \| ^ {2} \leq 2 \kappa \| t - \theta \| _ {\Sigma} ^ {2}.
$$
Substituting in the previous inequality, we get
$$
\begin{array}{l} \left\| t - \theta \right\| _ {\Sigma} ^ {2} - \rho \beta d ^ {2} \| t - \theta \| ^ {2} \leq \| t - \theta \| _ {\Sigma^ {\prime}} ^ {2} \leq \| t - \theta \| _ {\Sigma} ^ {2} + \rho \beta d \| t - \theta \| ^ {2}, \\ \Longrightarrow (1 - 2 \kappa \beta d ^ {2}) \| t - \theta \| _ {\Sigma} ^ {2} \leq \| t - \theta \| _ {\Sigma^ {\prime}} ^ {2} \leq (1 + 2 \kappa \beta d) \| t - \theta \| _ {\Sigma} ^ {2}, \\ \end{array}
$$
For the sake of symmetry, we will use the weaker bound of
$$
\left(1 - 2 \kappa \beta d ^ {2}\right) \| t - \theta \| _ {\Sigma} ^ {2} \leq \| t - \theta \| _ {\Sigma^ {\prime}} ^ {2} \leq \left(1 + 2 \kappa \beta d ^ {2}\right) \| t - \theta \| _ {\Sigma} ^ {2},
$$
Setting  $\gamma = 2\kappa \beta d^{2}$  completes the proof.
Lemma C.14. Consider a bounded mean vector  $\mu$  with  $\| \mu \|_{\infty} \leq \alpha$  and precision matrix  $P$  with max eigenvalue  $\rho$  and condition number  $\kappa$ .
For  $\gamma = O\Big(\min \left\{\frac{\varepsilon}{\alpha\rho^{1 / 2}d^{3 / 2}},\frac{\varepsilon}{\alpha^2d^3\rho}\right\} \Big)$  we have
$$
\log \int_ {t \leq 0} \exp \left(- \frac {(1 - \gamma)}{2} \| P ^ {\frac {1}{2}} (t - \mu) \| ^ {2}\right) \leq \varepsilon + \log \int_ {t \leq 0} \exp \left(- \frac {1}{2} \| P ^ {\frac {1}{2}} (t - \mu) \| ^ {2}\right).
$$
Proof of Lemma C.14. Wlog, consider  $\mu \geq 0$ . The case where the entries are possibly negative follows a similar proof.
Define the integral on the LHS and RHS of the Lemma statement by  $I_{1}$  and  $I_{2}$  respectively.
By a change of variables, we set  $t' = \sqrt{1 - \gamma}(t - \mu) + \mu$  in  $I_1$ , to get
$$
I _ {1} = \log \frac {1}{\sqrt {1 - \gamma}} + \log \int_ {t ^ {\prime} \leq (1 - \sqrt {1 - \gamma}) \mu} \exp \left(- \frac {1}{2} \| P ^ {\frac {1}{2}} (t ^ {\prime} - \mu) \| ^ {2}\right).
$$
Since  $\gamma < 1$ , we have  $(1 - \sqrt{1 - \gamma})\mu < \gamma \mu$ . Substituting in  $I_{1}$ , and for  $\gamma = O(\varepsilon)$ , we get
$$
\begin{array}{l} I _ {1} \leq \log \frac {1}{\sqrt {1 - \gamma}} + \log \int_ {t ^ {\prime} \leq \gamma \mu} \exp \left(- \frac {1}{2} \| P ^ {\frac {1}{2}} (t ^ {\prime} - \mu) \| ^ {2}\right), \\ \leq O (\varepsilon) + \log \int_ {t ^ {\prime} \leq \gamma \mu} \exp \left(- \frac {1}{2} \| P ^ {\frac {1}{2}} (t ^ {\prime} - \mu) \| ^ {2}\right). \\ \end{array}
$$
The integrating set in the above inequality can be split into two parts: one over the negative orthant (which is exactly to  $e^{I_2}$ ) and another over the shell
$$
C = \{t ^ {\prime} \leq \gamma \mu \} \setminus \{t ^ {\prime} \leq 0 \}.
$$
$$
I _ {1} \leq O (\varepsilon) + \log \left(e ^ {I _ {2}} + \int_ {t ^ {\prime} \in C} \exp \left(- \frac {1}{2} \| P ^ {\frac {1}{2}} (t ^ {\prime} - \mu) \| ^ {2}\right)\right).
$$
In the above inequality, let  $e^{I_3}$  denote the integral over the shell  $C$ . We will now show that  $I_3$  satisfies
$$
e ^ {I _ {3}} \leq \varepsilon e ^ {I _ {2}}.
$$
Let  $f(x)$  denote the Gaussian density with mean  $\mu$  and precision matrix  $P$ .
For a subset of co-ordinates  $S \subseteq [d], S \neq \emptyset$ , and  $t \in \mathbb{R}^d$ , let  $x_+$ ,  $x_-\in \mathbb{R}^d$  be such that
$$
x _ {+, S} (i) = \left\{ \begin{array}{l l} \gamma \mu_ {i} & \text {i f} i \in S, \\ t _ {i} & \text {i f} i \notin S, \end{array} \right., \quad x _ {-, S} (i) = \left\{ \begin{array}{l l} - \frac {\gamma}{\varepsilon} \mu_ {i} & \text {i f} i \in S, \\ t _ {i} & \text {i f} i \notin S. \end{array} \right.
$$
By the monotonicity of the Gaussian density, the integral over the shell  $C$  can be upper bounded by breaking up into a sum of integrals over lower-dimensional strips, where for a fixed subset  $S \in [d]$ , the variables  $t_{S^c}$  are integrated over  $(-\infty, \gamma \mu_{S^c}]$ , while the variables in  $S$  are fixed to  $\gamma \mu_S$ .
$$
\begin{array}{l} e ^ {I _ {3}} \leq \sum_ {S \subseteq [ d ]} \int_ {t _ {S ^ {c}} \leq \gamma \mu_ {S ^ {c}}} f (x _ {+, S}) \prod_ {i \in S} \gamma \mu_ {i}, \\ \leq \sum_ {S \subseteq [ d ]} \int_ {t _ {S ^ {c}} \leq \gamma \mu_ {S ^ {c}}} f (x _ {+, S}) (\gamma \alpha) ^ {| S |}, \\ \leq \sum_ {k = 1} ^ {d} \binom {d} {k} (\gamma \alpha) ^ {k} \max  _ {S \subseteq [ d ]: | S | = k} \int_ {t _ {S ^ {c}} \leq \gamma \mu_ {S ^ {c}}} f (x _ {+, S}). \\ \end{array}
$$
By Claim C.15, for any  $S$ , and  $\gamma = O\left(\min \left\{\frac{\varepsilon}{\alpha\sqrt{\rho d}},\frac{\varepsilon}{\alpha^2d^2\rho}\right\}\right)$  each summand satisfies
$$
f (x _ {+, S}) \leq 2 f (x _ {-, S})
$$
Furthermore, for any  $S \subseteq [d]$ , we have
$$
\int_ {t _ {S ^ {c}} \leq \gamma \mu_ {S ^ {c}}} f (x _ {-, S}) \left(\frac {\gamma}{\varepsilon} \alpha\right) ^ {| S |} \leq e ^ {I _ {2}}.
$$
$$
e ^ {I _ {3}} \leq \sum_ {k = 1} ^ {d} d ^ {k} 2 \varepsilon^ {k} e ^ {I _ {2}} \leq 3 \varepsilon d e ^ {I _ {2}} \quad \text {i f} \varepsilon d \leq \frac {1}{3}.
$$
Rescaling  $\varepsilon \gets \frac{\varepsilon}{3d}$  completes the proof.
Claim C.15. Let  $f$  be the Gaussian density with mean  $\mu \in [0, \alpha]^d$  and precision matrix  $P \in \mathbb{R}^{d \times d}$  with max eigenvalue  $\rho$  and condition number  $\kappa$ .
Let  $\gamma = O\left(\min \left\{\frac{\varepsilon}{\alpha\sqrt{\rho d}},\frac{\varepsilon}{\alpha^2d^2\rho}\right\}\right)$ . For any subset of co-ordinates  $S\subseteq [d],S\neq \emptyset$  and  $t\in \mathbb{R}^d$ , let  $x_{+},x_{-}\in \mathbb{R}^{d}$  be such that
$$
x _ {+} (i) = \left\{ \begin{array}{l l} \gamma \mu_ {i} & i f i \in S, \\ t _ {i} & i f i \notin S, \end{array} \right., \quad x _ {-} (i) = \left\{ \begin{array}{l l} - \frac {\gamma}{\varepsilon} \mu_ {i} & i f i \in S, \\ t _ {i} & i f i \notin S. \end{array} \right.
$$
$$
f (x _ {+}) \leq 2 f (x _ {-})
$$
Proof. WLOG, let  $S$  be a contiguous set such that we can separate the coordinates of  $x_{+}$  and  $x_{-}$  into disjoint sets. For the coordinates belonging to  $S$ , let  $\mu_S$  denote the coordinates of  $\mu$  belonging to  $\mu$ , and  $\mu_{S^c}$  the coordinates not belonging to  $S$  (similarly for  $t_S$  and  $t_{S^c}$ ).
Taking the logarithm on both sides of the claimed inequality, we want to show that
$$
- \frac {1}{2} \left\| P ^ {\frac {1}{2}} \left[ \begin{array}{c} \gamma \mu_ {S} - \mu_ {S} \\ t _ {S ^ {c}} - \mu_ {S ^ {c}} \end{array} \right] \right\| ^ {2} \leq - \frac {1}{2} \left\| P ^ {\frac {1}{2}} \left[ \begin{array}{c} - \frac {\gamma}{\varepsilon} \mu_ {S} - \mu_ {S} \\ t _ {S ^ {c}} - \mu_ {S ^ {c}} \end{array} \right] \right\| ^ {2} + \log 2
$$
Let  $a$  and  $b$  denote the vectors whose norms correspond to the log-densities in the claimed inequality, and let  $\delta = a - b$ .
$$
b = P ^ {\frac {1}{2}} \left[ \begin{array}{c} - \mu_ {S} (1 - \gamma) \\ t _ {S ^ {c}} - \mu_ {S ^ {c}} \end{array} \right], \qquad a   = P ^ {\frac {1}{2}} \left[ \begin{array}{c} - \mu_ {S} (1 + \frac {\gamma}{\varepsilon}) \\ t _ {S ^ {c}} - \mu_ {S ^ {c}} \end{array} \right], \qquad \delta := a - b = P ^ {\frac {1}{2}} \left[ \begin{array}{c} - \mu_ {S} \gamma \big (\frac {1}{\varepsilon} + 1 \big) \\ 0 _ {S ^ {c}} \end{array} \right]
$$
We want to show that
$$
\begin{array}{l} - \frac {1}{2} \| b \| ^ {2} \leq - \frac {1}{2} \| a \| ^ {2} + \log 2, \\ \Leftrightarrow \langle \delta , b \rangle + \frac {1}{2} \| \delta \| ^ {2} \leq \log 2. \tag {33} \\ \end{array}
$$
As  $\| P\| \leq \rho$  and  $\| \mu \|_{\infty}\leq \alpha$  , we have
$$
\left\| \delta \right\| _ {2} ^ {2} \leq \rho \left(\alpha^ {2} | S |\right) \gamma^ {2} \left(1 + \frac {1}{\varepsilon}\right) ^ {2}.
$$
For  $\gamma = O\left(\frac{\varepsilon}{\alpha\sqrt{\rho d}}\right)$ , we get
$$
\left\| \delta \right\| _ {2} ^ {2} \leq \frac {1}{2} \log 2. \tag {34}
$$
Similarly, consider the inner product  $\langle \delta, b \rangle$  in Eqn (33). By the trace trick, we get
$$
\langle \delta , b \rangle = \operatorname {T r} (\Delta P),
$$
$$
\Delta = \left[ \begin{array}{c} - \mu_ {S} (1 - \gamma) \\ t _ {S ^ {c}} - \mu_ {S ^ {c}} \end{array} \right] \left[ \begin{array}{c} - \mu_ {S} \gamma \big (\frac {1}{\varepsilon} + 1 \big) \\ 0 _ {S ^ {c}} \end{array} \right] ^ {T}
$$
Notice that the diagonal elements of  $\Delta$  are all non-negative. This implies that all singular values are non-negative. The trace of  $\Delta$  is
$$
\operatorname {T r} (\Delta) = \| \mu_ {S} \| _ {2} ^ {2} (1 - \gamma) \gamma \bigg (\frac {1}{\varepsilon} + 1 \bigg).
$$
Hence, by Von Neumann's trace inequality, we get
$$
\langle \delta , b \rangle \leq \operatorname {T r} (\Delta) \operatorname {T r} (P) \leq \| \mu_ {S} \| _ {2} ^ {2} (1 - \gamma) \gamma \left(\frac {1}{\varepsilon} + 1\right) \rho d.
$$
For  $\gamma = O\left(\frac{\varepsilon}{\alpha^2|S|\rho d}\right)$ , this gives
$$
\langle \delta , b \rangle \leq \frac {1}{2} \log 2. \tag {35}
$$
Substituting Eqn (34) and Eqn (35) in Eqn (33) completes the proof.
Lemma C.6 (W-net). Let  $\eta_{S^c}$ ,  $\eta_S$  be such that
for  $B_{1}, B_{2} \geq 0$ .
Let  $A > \max\{B_1^2, B_2^2, \mathrm{poly}(C, \kappa)\}$ . Let  $P^* = \Sigma^{*-1}$  be the precision matrix of  $\eta$ . For a fixed matrix  $P \in \mathbb{R}^{d \times d}$  whose condition number satisfies Assumption 4.4 and whose eigenvalues satisfy  $\lambda_{\max}(P) \in [e^{-\frac{2A}{d}} \lambda_{\min}(P^*), C \lambda_{\max}(P^*)]$ , there exists a partition  $\mathcal{I}$  of  $\mathbb{R}^d$  with size
Proof of Lemma C.6. Recall that the log-likelihood ratio  $\gamma_{\theta}$  can be decomposed into the difference of two terms that depend on  $\theta$ :
$$
\gamma_ {\theta , P} (y) - \frac {1}{2} \log \frac {| P |}{| P ^ {\prime} |} = g _ {\theta , P} (y) - g _ {\theta^ {*}, P ^ {*}} (y) + h _ {\theta , P} (y) - h _ {\theta^ {*}, P ^ {*}} (y).
$$
Without loss of generality, consider the net for the first coordinate  $\theta_{1}$ . The final net will be the intersection of the per-coordinate nets.
We will construct three partitions: the first is  $\mathcal{I}_{h,0}$  for  $h$  when  $y_{1} = 0$ , the second is  $\mathcal{I}_{h,1}$  for  $h$  when  $y_{1} > 0$ , and the last is  $\mathcal{I}_g$  for  $g$  when  $y_{1} > 0$ . The final partition will be the intersection of these partitions.
Case 1: Net over  $h, y_1 = 0$ . As  $y_1 = 0$ , we have  $1 \in S$ . For  $\theta \in \mathbb{R}^d$ , we have
$$
h _ {\theta , P} (y) = \log \int_ {t \leq 0} \exp \Bigl (- \| P _ {S} ^ {\frac {1}{2}} (t - \theta_ {S}) + (P _ {S}) ^ {- 1 / 2} P _ {S S ^ {c}} (y _ {S ^ {c}} - \theta_ {S ^ {c}}) \| ^ {2} / 2 \Bigr),
$$
By Claim C.16, if  $\theta_{1} \geq \Theta\left(\frac{\sqrt{C \kappa A}}{p_{1}}\right)$ , then the log-likelihood is smaller than  $-A$ .
Now, consider  $\theta_{1} < O\left(\frac{\sqrt{C\kappa A}}{p_{1}}\right)$ . Let  $\theta^{\prime} = \theta + \alpha e_{1}$  for  $\alpha > 0$ . As  $h$  is monotonically decreasing per coordinate,  $\theta_{1} < \theta_{1}^{\prime} \Rightarrow h_{\theta, P} \geq h_{\theta^{\prime}, P}$ . We would like to now upper bound  $h_{\theta, P}$  in terms of  $h_{\theta^{\prime}, P}$ . Let
$$
\mu := \theta_ {S} + \left(P _ {S}\right) ^ {- 1} P _ {S S ^ {c}} \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right),
$$
$$
\mu^ {\prime} := \theta_ {S} ^ {\prime} + \left(P _ {S}\right) ^ {- 1} P _ {S S ^ {c}} \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right)
$$
In the function  $h_\theta$ , break the integrating set into two domains: one where  $t - \mu$  is small,
$$
\Omega_ {1} = \left\{t \in \mathbb {R} ^ {| S |}: \| P _ {S} ^ {\frac {1}{2}} (t - \mu) \| \leq r \right\},
$$
and another where it is large:
$$
\Omega_ {2} = \left\{t \in \mathbb {R} ^ {| S |}: \| P _ {S} ^ {\frac {1}{2}} (t - \mu) \| > r \right\},
$$
for some  $r > 0$  that we will specify later.
Let  $I_{1}$  and  $I_{2}$  denote the integrals over  $\Omega_{1}$  and  $\Omega_{2}$  respectively.
$I_{2}$  corresponds to the tail of an unnormalized Gaussian distribution, and hence we have
$$
h _ {\theta , P} (y) = \log \left(I _ {2} + \int_ {t \leq 0, t \in \Omega_ {1}} \exp \left(- \| P _ {S} ^ {\frac {1}{2}} (t - \mu) \| ^ {2} / 2\right)\right),
$$
where  $I_2 \leq (2\pi)^{\frac{|S|}{2}} \left| P_S^{-\frac{1}{2}} \right| e^{-r^2}$ .
We can simplify  $I_{2}$  be comparing  $|P|$  to  $|P^{*}|$ :
$$
\begin{array}{l} I _ {2} \leq (2 \pi) ^ {\frac {| S |}{2}} \frac {\left| P _ {S} ^ {- 1 / 2} \right|}{\left| P _ {S} ^ {* - 1 / 2} \right|} \left| P _ {S} ^ {* - 1 / 2} \right| e ^ {- r ^ {2}} \leq (2 \pi) ^ {\frac {| S |}{2}} \left(\frac {\lambda_ {\max } ^ {*}}{\lambda_ {\min } (P)}\right) ^ {| S |} \left| P _ {S} ^ {* - \frac {1}{2}} \right| e ^ {- r ^ {2}}, \\ \leq \left(2 \pi\right) ^ {\frac {| S |}{2}} \left(\kappa e ^ {\frac {A}{d}}\right) ^ {| S |} \left| P _ {S} ^ {* - \frac {1}{2}} \right| e ^ {- r ^ {2}}. \\ \end{array}
$$
By Lemma C.10, we have
$$
(2 \pi) ^ {\frac {| S |}{2}} \left| P _ {S} ^ {* - \frac {1}{2}} \right| \leq e ^ {h _ {\theta^ {*}, P ^ {*}} (y) + O (d + B _ {2} ^ {2} + B _ {3} ^ {2})}
$$
As we are only consider  $\theta^{\prime}$  such that  $h_{\theta^{*},P^{*}}(y) - A < h_{\theta^{\prime},P}(y)$ , for
$$
r ^ {2} = O (d \log \kappa + A + \log \frac {1}{\varepsilon}) = O (A),
$$
$$
I _ {2} \leq \varepsilon e ^ {h _ {\theta^ {\prime}, P} (y)}
$$
Substituting in  $h_{\theta ,P}(y)$  , we get
$$
h _ {\theta , P} (y) \leq \log \left(\varepsilon e ^ {h _ {\theta^ {\prime}, P} (y)} + \int_ {t \leq 0, t \in \Omega_ {1}} \exp \left(- \| P _ {S} ^ {\frac {1}{2}} (t - \mu) \| ^ {2} / 2\right)\right)
$$
Now consider the integral  $I_{1} = \int_{t\leq 0,t\in \Omega_{1}}\exp \left(-\| P_{S}^{\frac{1}{2}}(t - \mu)\|^{2} / 2\right)$ .