SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By Claim C.17, as  $\Omega_1$  is defined for  $t$  bounded by  $r$ , and  $\mu - \mu' = \alpha e_1$ , we have
$$
I _ {1} \leq \exp \left(2 \alpha p _ {1} r + \alpha^ {2} p _ {1} ^ {2}\right) \cdot I _ {1} ^ {\prime},
$$
$$
\text {w h e r e} I _ {1} ^ {\prime} = \int_ {t \leq 0, t \in \Omega_ {1}} \exp \left(- \| P _ {S} ^ {\frac {1}{2}} (t - \mu^ {\prime}) \| ^ {2} / 2\right) \leq e ^ {h _ {\theta^ {\prime}, P} (y)}.
$$
Substituting in the expression for  $h_{\theta ,P}$  , we get
$$
h _ {\theta , P} (y) \leq \log \left(e ^ {h _ {\theta^ {\prime}, P} (y)} \cdot \left(\varepsilon + \exp \left(2 \alpha p _ {1} r + \alpha^ {2} p _ {1} ^ {2}\right)\right)\right)
$$
As  $\log (\varepsilon +e^{x})\leq \varepsilon +x$  for  $x\geq 0$  , we have
$$
h _ {\theta , P} (y) \leq h _ {\theta^ {\prime}, P} (y) + \varepsilon + 2 \alpha p _ {1} r + \alpha^ {2} p _ {1} ^ {2}.
$$
Setting  $\alpha = O\left(\frac{\varepsilon}{p_1r}\right)$ , we get
$$
h _ {\theta , P} (y) \leq h _ {\theta^ {\prime}, P} (y) + 2 \varepsilon .
$$
This shows that  $h_{\theta, P}$  changes by at most  $\varepsilon$  for the considered net. We need to define the other end point for the net. By a similar argument to the positive end point, if  $\theta_1 = -O\left(\frac{\sqrt{C\kappa\log\left(\frac{1}{\varepsilon}\right)}}{p_1}\right)$ , the log-likelihood ratio changes by at most  $\varepsilon$  until  $\theta_1 = -\infty$ .
As we are only trying to cover  $\theta$  such that  $|\theta_1| \leq O\left(\frac{\sqrt{C\kappa A}}{p_1}\right)$ , this net has size
$$
O \left(\frac {\sqrt {C \kappa A}}{p _ {1} \alpha}\right) = O \left(\frac {\sqrt {C \kappa A}}{p _ {1}} \frac {p _ {1} r}{\varepsilon}\right) = \frac {A}{\varepsilon}.
$$
Case 2: Net over  $h, y > 0$ . A similar argument to Case 1 works here as well.
Case 3: Net over  $g, y_1 > 0$ . By Lemma C.11, if  $|\theta - \theta_*|_{P_{Sc}} > R_1$  for
$$
R _ {1} = O (\sqrt {A}),
$$
$$
g _ {\theta , P} - g _ {\theta^ {*}, P} <   - A.
$$
Now consider  $\theta$  such that
$$
\left| \theta_ {1} - \theta_ {1} ^ {*} \right| \leq R,
$$
and  $\theta, \theta'$  such that  $\theta - \theta' = \alpha e_1$ .
The difference in  $g_{\theta} - g_{\theta^{\prime}}$  is
$$
g _ {\theta} (y) - g _ {\theta^ {\prime}} (y) = \eta_ {S ^ {c}} ^ {T} (\Sigma_ {S ^ {c}}) ^ {- 1} (\theta_ {S ^ {c}} - \theta_ {S ^ {c}} ^ {\prime}) - \frac {1}{2} \| \theta_ {S ^ {c}} ^ {*} - \theta_ {S ^ {c}} \| _ {\Sigma_ {S ^ {c}}} ^ {2} + \frac {1}{2} \| \theta_ {S ^ {c}} ^ {*} - \theta_ {S ^ {c}} ^ {\prime} \| _ {\Sigma_ {S ^ {c}}} ^ {2},
$$
The second and third terms in the RHS can bounded by observing that
$$
\left| 2 \theta^ {*} - \theta^ {\prime} - \theta \right| \leq 2 R _ {1} = O (\sqrt {A}), | \theta^ {\prime} - \theta | \leq \alpha ,
$$
and hence we get
$$
- \frac {1}{2} \| \theta_ {S ^ {c}} ^ {*} - \theta_ {S ^ {c}} \| _ {\Sigma_ {S ^ {c}}} ^ {2} + \frac {1}{2} \| \theta_ {S ^ {c}} ^ {*} - \theta_ {S ^ {c}} ^ {\prime} \| _ {\Sigma_ {S ^ {c}}} ^ {2} \leq O (\sqrt {A}) \alpha .
$$
Now, for the first term in the RHS, we have
$$
\| \eta_ {S ^ {c}} \| _ {P _ {S ^ {c}} ^ {*}} \leq B _ {1} \Rightarrow \| \eta_ {S ^ {c}} \| \leq \frac {B _ {1}}{\lambda_ {\min} ^ {\frac {1}{2}} (P ^ {*})} \leq \frac {B _ {1} \sqrt {\kappa}}{\lambda_ {\max} ^ {\frac {1}{2}} (P ^ {*})}.
$$
This further implies that
$$
\eta_ {S ^ {c}} ^ {T} (\Sigma_ {S ^ {c}}) ^ {- 1} (\theta_ {S ^ {c}} - \theta_ {S ^ {c}} ^ {\prime}) \leq \frac {B _ {1} \sqrt {\kappa}}{\lambda_ {\max} ^ {\frac {1}{2}} (P ^ {*})} \sqrt {p _ {1}} \alpha = \mathrm {p o l y} (A) \alpha .
$$
$$
\alpha = O \bigg (\frac {\epsilon}{\operatorname {p o l y} (A)} \bigg),
$$
$$
\left| g _ {\theta} (y) - g _ {\theta^ {\prime}} (y) \right| \leq \frac {\epsilon}{d}.
$$
As we are covering a set of size  $R_{1}$  using a grid size of  $\alpha$ , the size of this partition is
$$
O \left(\frac {R _ {1}}{\alpha}\right) = \frac {\operatorname {p o l y} (A)}{\varepsilon}.
$$
Claim C.16. In the setting of Lemma C.6, we have  $\lambda_{\max}(P) \leq C \lambda_{\max}(P^{*})$ . Let  $p_1$  denote the first diagonal element of  $P$ .
If  $\theta_{1} \geq \Theta\left(\frac{\sqrt{C \kappa A}}{p_{1}}\right)$ , then the function  $h_{\theta, P}$  is such that
$$
h _ {\theta , P} - h _ {\theta^ {*}, P} <   - A.
$$
Proof of Claim C.16. Recall that the function  $h_{\theta, P}$  is defined as:
$$
h _ {\theta , P} (y) = \log \int_ {t \leq 0} \exp \Bigl (- \| P _ {S} ^ {\frac {1}{2}} (t - \theta_ {S}) + (P _ {S}) ^ {- 1 / 2} P _ {S S ^ {c}} (y _ {S ^ {c}} - \theta_ {S ^ {c}}) \| ^ {2} / 2 \Bigr).
$$
Consider the term  $\| P_S^{\frac{1}{2}}(t - \theta_S) + (P_S)^{-1 / 2}P_{SS^c}(y_{S^c} - \theta_{S^c})\|$ . By the triangle inequality, we have
$$
\begin{array}{l} \| P _ {S} ^ {\frac {1}{2}} (t - \theta_ {S}) + (P _ {S}) ^ {- 1 / 2} P _ {S S ^ {c}} (y _ {S ^ {c}} - \theta_ {S ^ {c}}) \| \\ \geq \| P _ {S} ^ {\frac {1}{2}} (t - \theta_ {S}) + (P _ {S}) ^ {- 1 / 2} P _ {S S ^ {c}} (\theta_ {S ^ {c}} ^ {\prime} - \theta_ {S ^ {c}} ^ {*}) \| - \| P _ {S} ^ {- \frac {1}{2}} P _ {S S ^ {c}} (y _ {S ^ {c}} - \theta_ {S ^ {c}} ^ {*}) \|, \\ \geq \left\| P _ {S} ^ {\frac {1}{2}} \left(t - \theta_ {S}\right) + \left(P _ {S}\right) ^ {- 1 / 2} P _ {S S ^ {c}} \left(\theta_ {S ^ {c}} ^ {\prime} - \theta_ {S ^ {c}} ^ {*}\right) \right\| - \sqrt {C \kappa A}, \\ \end{array}
$$
where the last inequality follows as
$$
\| P _ {S} ^ {- \frac {1}{2}} P _ {S S ^ {c}} (y _ {S ^ {c}} - \theta_ {S ^ {c}} ^ {*}) \| = \| P _ {S} ^ {- \frac {1}{2}} P _ {S S ^ {c}} (\eta_ {S ^ {c}}) \| \leq \| P _ {S ^ {c}} ^ {\frac {1}{2}} \| \| \eta_ {S ^ {c}} \| \leq \frac {\sqrt {C \lambda_ {\max} ^ {*}}}{\sqrt {\lambda_ {\min} ^ {*}}} B \leq \sqrt {C \kappa A}.
$$
Similarly, the function  $g_{\theta ,P}$  only considers  $\theta$  such that  $\| P_{S^c}^{\frac{1}{2}}(\theta_{S^c} - \theta_{S^c}^*)\| \leq \sqrt{C\kappa A}$  (otherwise, the log-likelihood ratio is smaller than  $-A$  by virtue of  $g_{\theta ,P}$ , irrespective of  $h_{\theta ,P}$ ). For these  $\theta$ , we have
$$
\begin{array}{l} \left\| P _ {S} ^ {\frac {1}{2}} \left(t - \theta_ {S}\right) + \left(P _ {S}\right) ^ {- 1 / 2} P _ {S S ^ {c}} \left(\theta_ {S ^ {c}} ^ {*} - \theta_ {S ^ {c}}\right) \right\| \\ \geq \| P _ {S} ^ {\frac {1}{2}} (t - \theta_ {S}) \| - \sqrt {C \kappa A}, \\ \end{array}
$$
$$
\| P _ {S} ^ {\frac {1}{2}} (t - \theta_ {S}) + (P _ {S}) ^ {- 1 / 2} P _ {S S ^ {c}} (y _ {S ^ {c}} - \theta_ {S ^ {c}}) \| \geq \| P _ {S} ^ {\frac {1}{2}} (t - \theta_ {S}) \| - 2 \sqrt {C \kappa A}.
$$
Hence, if  $\theta_{1} \geq O\left(\frac{\sqrt{C_{K}A}}{p_{1}}\right)$ , then we have
$$
\left\| P _ {S} ^ {\frac {1}{2}} \left(t - \theta_ {S}\right) + \left(P _ {S}\right) ^ {- 1 / 2} P _ {S S ^ {c}} \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right) \right\| \geq \Omega \left(\sqrt {C _ {\kappa} A}\right) \forall t \leq 0,
$$
and hence the Gaussian integral is at most  $(2\pi)^{|S| / 2}\Big|P_S^{-\frac{1}{2}}\Big|e^{-\Omega (C_{\kappa A})}.$
By Lemma C.10, we have  $h_{\theta^{*},P^{*}} \geq -\frac{1}{2} \log |P_{S}^{*}| - O(A)$ , which gives
$$
\begin{array}{l} h _ {\theta , P} (y) - h _ {\theta^ {*}, P ^ {*}} (y) <   \frac {1}{2} \log \frac {| P _ {S} ^ {*} |}{| P _ {S} |} - \Omega (C \kappa A) \\ <   \frac {d}{2} \log \frac {\lambda_ {\mathrm {m a x}} ^ {*}}{\lambda_ {\mathrm {m i n}} (P)} - \Omega (C \kappa A) <   O (A \log \kappa) - \Omega (C \kappa A) = - \Omega (C \kappa A). \\ \end{array}
$$
This gives a contiguous interval over  $\theta_{1}$  for which  $\gamma_{\theta ,P} < - A$
Claim C.17. In the setting of Lemma C.6, let  $\mu, \mu'$  be such that  $\mu - \mu' = \alpha e_1$
Then, for all  $t$  such that
$$
\left\| P _ {S} ^ {\frac {1}{2}} (t - \mu) \right\| \leq r,
$$
and  $p_1\coloneqq P_{11}$  ,wehave
$$
\| P _ {S} ^ {\frac {1}{2}} (t - \mu) \| ^ {2} \geq - 2 \alpha p _ {1} r - \alpha^ {2} p _ {1} ^ {2} + \| P _ {S} ^ {\frac {1}{2}} (t - \mu^ {\prime}) \| ^ {2}.
$$
Proof of Claim C.17. Consider the term  $\| P_S^{\frac{1}{2}}(t - \mu)\|^2$ .
Adding and subtracting  $\| P_S^{\frac{1}{2}}(t - \mu')\|^2$ , we get
$$
\begin{array}{l} \left\| P _ {S} ^ {\frac {1}{2}} (t - \mu) \right\| ^ {2} = \left\| P _ {S} ^ {\frac {1}{2}} (t - \mu) \right\| ^ {2} - \left\| P _ {S} ^ {\frac {1}{2}} (t - \mu^ {\prime}) \right\| ^ {2} + \left\| P _ {S} ^ {\frac {1}{2}} (t - \mu^ {\prime}) \right\| ^ {2}, \\ = \left\langle P _ {S} ^ {\frac {1}{2}} \left(2 t - \mu^ {\prime} - \mu\right), P _ {S} ^ {\frac {1}{2}} \left(\mu^ {\prime} - \mu\right) \right\rangle + \left\| P _ {S} ^ {\frac {1}{2}} \left(t - \mu^ {\prime}\right) \right\| ^ {2}, \\ = \langle P _ {S} ^ {\frac {1}{2}} (2 t - 2 \mu - (\mu^ {\prime} - \mu)), P _ {S} ^ {\frac {1}{2}} (\mu^ {\prime} - \mu) \rangle + \| P _ {S} ^ {\frac {1}{2}} (t - \mu^ {\prime}) \| ^ {2}, \\ = 2 \left\langle P _ {S} ^ {\frac {1}{2}} (t - \mu), P _ {S} ^ {\frac {1}{2}} \left(\mu^ {\prime} - \mu\right) \right\rangle - \left\| P _ {S} ^ {\frac {1}{2}} \left(\mu - \mu^ {\prime}\right) \right\| ^ {2} + \left\| P _ {S} ^ {\frac {1}{2}} (t - \mu^ {\prime}) \right\| ^ {2}. \\ \end{array}
$$
As  $\mu -\mu^{\prime} = \alpha e_{1}$  , we have
$$
\left\| P _ {S} ^ {\frac {1}{2}} \left(\mu - \mu^ {\prime}\right) \right\| ^ {2} = \alpha^ {2} p _ {1} ^ {2}.
$$
By the Cauchy-Schwartz inequality, and since  $\mu -\mu^{\prime} = \alpha e_{1}$ , the inner product can be lower bounded as
$$
2 \langle P _ {S} ^ {\frac {1}{2}} (t - \mu), P _ {S} ^ {\frac {1}{2}} (\mu^ {\prime} - \mu) \rangle \geq - 2 \| P _ {S} ^ {\frac {1}{2}} (t - \mu) \| \| P _ {S} ^ {\frac {1}{2}} (\mu^ {\prime} - \mu) \| \geq - 2 \alpha p _ {1} r.
$$
Substituting, we get
$$
\left\| P _ {S} ^ {\frac {1}{2}} (t - \mu) \right\| ^ {2} \geq - 2 \alpha p _ {1} r - \alpha^ {2} p _ {1} ^ {2} + \left\| P _ {S} ^ {\frac {1}{2}} (t - \mu^ {\prime}) \right\| ^ {2}.
$$
Lemma C.18. Following Definition C.12 let  $\Omega_{\rho}$  be the set of precision matrices with condition number  $\kappa$  satisfying  $\lambda_{\max}(P)\in [\frac{\rho}{2},\rho ]$ , and let  $\widetilde{\Omega}_{\rho ,\beta}$  be the quantized net with quantization level  $\beta$ .
For any  $P \in \Omega_{\rho}$ , let  $\widetilde{P} \in \widetilde{\Omega}_{\rho, \beta}$  be its element-wise rounding down. Then, for any  $\mu \in \mathbb{R}^d$ , we have
$$
d _ {T V} \left(\mathcal {N} (\mu ; P), \mathcal {N} (\mu ; \widetilde {P})\right) \leq O \left(d ^ {2} \beta \kappa\right). \tag {36}
$$
Proof of Lemma C.18. Let  $\Sigma = P^{-1}$  and  $\widetilde{\Sigma} = \widetilde{P}^{-1}$ .
By Theorem 1.1 in [16], the TV between two Gaussians with the same mean is
$$
d _ {T V} (\mathcal {N} (\mu ; \Sigma), \mathcal {N} (\mu ; \widetilde {\Sigma})) = \Theta \left(\min  \left\{1, \sqrt {\sum_ {i} \xi_ {i} ^ {2}} \right\}\right),
$$
where  $\xi_{i}$  are the eigenvalues of  $\widetilde{\Sigma}^{-1}\Sigma -I_d$
We can convert the bound on the eigenvalues to the Frobenius norm of  $\widetilde{\Sigma}^{-1}\Sigma -I_d$ :
$$
\sqrt {\sum_ {i} \xi_ {i} ^ {2}} \leq \| \widetilde {\Sigma} ^ {- 1} \Sigma - I _ {d} \| _ {F}.
$$
Recall that  $\widetilde{P}$  is the rounding down per entry of  $P$ . Hence,
$$
\begin{array}{l} \widetilde {\Sigma} ^ {- 1} \Sigma - I _ {d} = \left(\Sigma^ {- 1} - \left[ \nu_ {i j} \right]\right) \Sigma - I _ {d}, \text {w h e r e} 0 \leq \nu_ {i j} <   \beta \rho , \\ = - \nu \Sigma . \\ \end{array}
$$
Taking the Frobenius norm, we get
$$
\begin{array}{l} \left\| \widetilde {\Sigma} ^ {- 1} \Sigma - I _ {d} \right\| _ {F} = \left\| \nu \Sigma \right\| _ {F}, \\ \leq \left(d \beta \rho\right) \left(d \rho_ {\max } (\Sigma)\right) = \left(d \beta \rho\right) \left(\frac {d}{\rho_ {\min } (P)}\right), \\ \leq (d \beta \rho) \left(\frac {d \kappa}{\rho_ {\max } (P)}\right) \leq 2 d ^ {2} \beta \kappa . \\ \end{array}
$$
where the first inequality follows as each element of  $\nu$  is at most  $\beta \rho$ , the second inequality follows as  $P$  has condition number  $\kappa$ , and the third follows as  $P \in \Omega_{\rho} \Longrightarrow \rho_{\max}(P) \geq \frac{\rho}{2}$ .
Lemma C.7. Let  $x_{1},\ldots ,x_{n}$  be fixed, and  $y_{i} = \phi (W^{*}x_{i} + \eta_{i})$  for  $\eta_{i}\sim \mathcal{N}(0,\Sigma^{*})$ , and  $W^{*}\in \mathbb{R}^{d\times k}$  with  $\Sigma^{*}\in \mathbb{R}^{d\times d}$  satisfying Assumption 4.4 and Assumption C.3. For a sufficiently large constant  $C > 0$ ,
Proof of Lemma C.7. For any  $W \in \mathbb{R}^{d \times k}$ ,  $\Sigma \in \mathbb{R}^{d \times d}$  and a sample  $(x_i, y_i)$ , let  $p_{i,W,\Sigma}(y|x_i)$  be the conditional distribution of  $y = \phi(Wx + \eta)$ , and let  $\gamma_{i,W,\Sigma}$  be the log-likelihood ratio between  $(W, \Sigma)$  and  $(W^*, \Sigma^*)$  on this sample:
$$
\gamma_ {i, W, \Sigma} (y) := \log \frac {p _ {i , W , \Sigma} (y \mid x _ {i})}{p _ {i , W ^ {*} , \Sigma^ {*}} (y \mid x _ {i})}.
$$
$$
\underset {y} {\mathbb {E}} \left[ \gamma_ {i, W, \Sigma} (y) \right] = - K L \left(p _ {i, W ^ {*}, \Sigma^ {*}} \left(y \mid x _ {i}\right) \| p _ {i, W, \Sigma} \left(y \mid x _ {i}\right)\right).
$$
Concentration. From Lemma B.1, we see that if  $d_{TV}((W^{*},\Sigma^{*}),(W,\Sigma)) \geq \varepsilon$ , then for  $n \geq O\left(\frac{1}{\varepsilon^2}\log \frac{1}{\delta}\right)$ ,
$$
\bar {\gamma} _ {W, \Sigma} := \frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {i, W, \Sigma} (y _ {i}) <   - \frac {\varepsilon^ {2}}{2}, \tag {37}
$$
Of course, whenever  $\overline{\gamma}_{W,\Sigma} < 0$ , the likelihood under  $W^{*}$ ,  $\Sigma^{*}$  is larger than the likelihood under  $W$ ,  $\Sigma$ . Thus, for each fixed  $W$ ,  $\Sigma$  with  $d_{TV}((W^{*},\Sigma^{*}),(W,\Sigma))\geq \varepsilon$ , maximizing likelihood would prefer  $W^{*}$ ,  $\Sigma^{*}$  to  $W$ ,  $\Sigma$  with probability  $1 - \delta$  if  $n\geq O(\frac{1}{\varepsilon^2}\log \frac{1}{\delta})$ .
Nothing above is specific to our ReLU-based distribution. But to extend to the MLE over all  $W, \Sigma$ , we need to build a net using properties of our distribution.
Building a net. First, for a given sample  $y_{i}$ , let  $S_{i}$  be the set of coordinates of  $y_{i}$  that are zero, and  $S_{i}^{c}$  be its complement. Then, with high probability,
$$
\left\| P _ {S _ {i}} ^ {* \frac {1}{2}} \left(\eta_ {i}\right) _ {S _ {i}} \right\|, \left\| P _ {S _ {i} ^ {c}} ^ {* \frac {1}{2}} \left(\eta_ {i}\right) _ {S _ {i}} \right\| \leq B = O \left(\sqrt {\kappa d \log n}\right) \forall i \in [ n ],
$$
where  $P^{*} = \Sigma^{* - 1}$ , and  $P_{S}^{*}, P_{S^{c}}^{*}$  are the block matrices in  $P^{*}$ .
Supposing the above event happens, we will construct a net over the precision matrices  $P = \Sigma^{-1}$ . Note that as we are only considering matrices with bounded condition number, this is a bijective mapping.
Net over  $\Sigma^{-1}$ . By Lemma C.4, any precision matrix  $P = \Sigma^{-1}$  satisfying Assumption 4.4 and whose max eigenvalue satisfies
$$
\lambda_ {\max } (P) \geq U \cdot \lambda_ {\max } \left(P ^ {*}\right),
$$
for  $U = O\left(\frac{\kappa^3d^2n^2}{k^2} + \frac{\kappa^2dn\log n}{k}\right)$ , will have  $\bar{\gamma}_{W,P^{-1}} < 0$ , irrespective of  $W$ .
Similarly, by Lemma C.2, any precision matrix satisfying Assumption 4.4 and whose max eigenvalue satisfies
$$
\lambda_ {\max } (P) \leq L \cdot \lambda_ {\min } \left(P ^ {*}\right)
$$
for  $L = e^{-O(\kappa \log n)}$  has  $\gamma_{i,W,P^{-1}} \leq 0$  for all  $i \in [n]$ , and hence its average  $\bar{\gamma}_{W,P^{-1}}$  is also  $< 0$ .
This shows that for all precision matrices  $P$  whose max eigenvalue is extremely small / large when compared to the min / max eigenvalues of  $P^*$ , has
$$
\bar {\gamma} _ {W, P ^ {- 1}} <   0,
$$
irrespective of  $W$ , and the MLE, which has non-negative  $\bar{\gamma}$ , will never pick these  $P$ .
Let  $A = \mathrm{poly}(n, d, \kappa, \frac{1}{\varepsilon})$  be large enough such that  $A > n(d \log (U \kappa) + B^2)$ , and such that it meets the requirements of Lemma C.5 and Lemma C.6.
Then, by Lemma C.5 with  $U = \mathrm{poly}(d,\kappa ,n)$  and  $\log (\frac{1}{L}) = \mathrm{poly}(\kappa ,n)$ , there exists a partition  $\mathcal{P}$  of precision matrices whose max-eigenvalue lies in  $[L\cdot \lambda_{\max}(P^{*}),U\cdot \lambda_{\max}(P^{*})]$  into  $\left(\mathrm{poly}\big(d,\kappa ,n,\frac{1}{\varepsilon}\big)\right)^{d^2}$  cells, such that for each cell  $I\in \mathcal{P}$ , and  $P,P^{\prime}\in I$ , the following holds for all  $i\in [n]$  and  $W\in \mathbb{R}^{d\times k}$ :
$$
\left| \gamma_ {i, W, P} \left(y _ {i}\right) - \gamma_ {i, W, P ^ {\prime}} \left(y _ {i}\right) \right| \leq \frac {\varepsilon^ {2}}{1 6} \tag {38}
$$
or  $\gamma_{i,W,P}(y_i) < -A$ .
Using Lemma C.18, we also have that for all  $W$ ,
$$
d _ {T V} ((W, P), (W, P ^ {\prime})) \leq \frac {\varepsilon^ {2}}{1 6}. \tag {39}
$$
We can choose a net  $N$  consisting of precision matrices from each cell in  $\mathcal{P}$ . This net has size
$$
\log | N | \lesssim d ^ {2} \log \left(\frac {d \kappa n}{\varepsilon}\right).
$$
This gives a sufficient net over the precision matrices.
Now we will construct a net over  $W$  for each precision matrix in the net.
$W$ -net. Now, for each  $\widetilde{P} \in N_P$ , by Lemma C.6, for each  $i \in [n]$ , there exists a partition  $\mathcal{P}_{\widetilde{P},i}$  of  $\mathbb{R}^d$  into  $\left(\mathrm{poly}\big(d,k,\kappa,n,\frac{1}{\varepsilon}\big)\right)^d$  cells such that for each cell  $I \in \mathcal{P}_{\widetilde{P},i}$ , and  $W, W' \in I$ , one of the following holds:
$$
\left| \gamma_ {i, W, \widetilde {P}} \left(y _ {i}\right) - \gamma_ {i, W ^ {\prime}, \widetilde {P}} \left(y _ {i}\right) \right| \leq \frac {\varepsilon^ {2}}{1 6} \tag {40}
$$
or  $\gamma_{i,W,\widetilde{P}}(y_i) < -A$ .
Let  $W_{j}$  be the  $j$ -th row of  $W$ . The individual partitions  $\mathcal{P}_{\widetilde{P},i}$  on  $\langle x_i, W_j \rangle$  induce a partition  $\mathcal{P}_{\widetilde{P},i,j}$  on  $\mathbb{R}^k$ , where  $W_{j}, W_{j}'$  lie in the same cell of  $\mathcal{P}_{\widetilde{P},i,j}$  if  $\langle x_i, W_j \rangle$  and  $\langle x_i, W_j' \rangle$  are in the same cell of  $\mathcal{P}_{\widetilde{P},i}$  for all  $i \in [n]$ . Since  $\mathcal{P}_{\widetilde{P},i,j}$  is defined by  $n$  sets of  $(\mathrm{poly}\big(d, k, \kappa, n, \frac{1}{\varepsilon}\big))$  parallel hyperplanes in  $\mathbb{R}^k$ , the number of cells in  $\mathcal{P}_{\widetilde{P},i,j}$  is
$$
\left(\operatorname {p o l y} \left(d, k, \kappa , n, \frac {1}{\varepsilon}\right)\right) ^ {k}.
$$
As there are  $d$  rows in  $W$ , we can intersect  $\mathcal{P}_{\widetilde{P},i,j}$  over  $j \in [d]$ , which induces  $\mathrm{poly}(d,k,\kappa,n,\frac{1}{\varepsilon})^{kd}$  cells in  $\mathbb{R}^k$ . We choose a net  $N_{\widetilde{P}}$  to contain, for each cell in  $\bigcap_{j \in [d]} \mathcal{P}_{\widetilde{P},i,j}$ , the  $W$  in the cell maximizing  $d_{TV}((W^*, P^*), (W, \widetilde{P}))$ . This has size
$$
\log \left| N _ {\widetilde {P}} \right| \lesssim k d \log \left(\frac {\kappa d k n}{\varepsilon}\right).
$$
Proving MLE works. By (37), for our  $n \geq O\left(\frac{(kd + d^2)}{\varepsilon^2} \log \frac{kd\kappa}{\varepsilon}\right)$ , we have with high probability that
$$
\overline {{\gamma}} _ {W, P} \leq - \frac {\varepsilon^ {2}}{2},
$$
for all  $P \in N$  and for all  $W \in N_{\widetilde{P}}$  with  $d_{TV}((W^{*}, P^{*}), (W, P)) \geq \varepsilon$ . Suppose that both this happens, and
$$
\| P _ {S _ {i}} ^ {* \frac {1}{2}} (\eta_ {i}) _ {S _ {i}} \|, \| P _ {S _ {i} ^ {c}} ^ {* \frac {1}{2}} (\eta_ {i}) _ {S _ {i}} \| \leq B = O (\sqrt {\kappa d \log n}) \forall i \in [ n ].
$$
We claim that the MLE  $\widehat{W},\widehat{\Sigma}$  must have  $d_{TV}((W^{*},\Sigma^{*}),(\widehat{W},\widehat{\Sigma})) < \frac{17}{16}\varepsilon$
Consider any  $W \in \mathbb{R}^{d \times k}$  and  $P \in \mathbb{R}^{d \times d}$  with  $d_{TV}((W^*, P^*), (W, P)) \geq \frac{17}{16} \varepsilon$ . Using our net on precision matrices, we can find  $\widetilde{P} \in N$  such that
$$
d _ {T V} ((W ^ {*}, P ^ {*}), (W, \widetilde {P})) \geq d _ {T V} ((W ^ {*}, P ^ {*}), (W, P)) - d _ {T V} ((W, P), (W, \widetilde {P})).
$$
Recall that we are only currently considering  $W, P$  such that  $d_{TV}((W^{*}, P^{*}), (W, P)) \geq \frac{17}{16}\varepsilon$ . By Eqn (39), we have  $d_{TV}((W, P), (W, \widetilde{P})) \leq \frac{\varepsilon^2}{16}$ , which gives
$$
d _ {T V} \left(\left(W ^ {*}, P ^ {*}\right), \left(W, \widetilde {P}\right)\right) \geq \varepsilon \frac {1 7}{1 6} - \frac {\varepsilon^ {2}}{1 6} \geq \varepsilon .
$$
Now, for this  $\widetilde{P}$ , we can find a  $\widetilde{W} \in N_{\widetilde{P}}$ , and by our choice of  $N_{\widetilde{P}}$ , we know that
$$
d _ {T V} ((W ^ {*}, P ^ {*}), (\widetilde {W}, \widetilde {P})) \geq d _ {T V} ((W ^ {*}, P ^ {*}), (W, \widetilde {P})) \geq \varepsilon ,
$$
and by (37), we have  $\bar{\gamma}_{\widetilde{W},\widetilde{P}}\leq -\frac{\varepsilon^2}{2}$
Now we consider two cases. In the first case, there exists  $i$  with  $\gamma_{i,W,P}(y_i) < -A$ . Then
$$
\overline {{\gamma}} _ {W, P} = \frac {1}{n} \sum_ {i} \gamma_ {i, W, P} (y _ {i}) \leq - \frac {A}{n} + B ^ {2} / 2 <   0.
$$
Otherwise, by Eqn (38) and Eqn (40), we have
$$
\begin{array}{l} \overline {{\gamma}} _ {W, P} \leq \overline {{\gamma}} _ {\widetilde {W}, \widetilde {P}} + \left| \overline {{\gamma}} _ {\widetilde {W}, \widetilde {P}} - \overline {{\gamma}} _ {W, \widetilde {P}} \right| + \left| \overline {{\gamma}} _ {W, \widetilde {P}} - \overline {{\gamma}} _ {W, P} \right|, \\ \leq - \frac {\varepsilon^ {2}}{2} + \max _ {i} \Big | \overline {{\gamma}} _ {i, \widetilde {W}, \widetilde {P}} - \overline {{\gamma}} _ {i, W, \widetilde {P}} \Big | + \max _ {i} \Big | \overline {{\gamma}} _ {i, W, \widetilde {P}} - \overline {{\gamma}} _ {i, W, P} \Big |, \\ \leq - \frac {\varepsilon^ {2}}{2} + \frac {\varepsilon^ {2}}{1 6} + \frac {\varepsilon^ {2}}{1 6} <   0. \\ \end{array}
$$
In either case,  $\overline{\gamma}_{W,P} < 0$  and the likelihood under  $w^{*}$  exceeds that under  $w$ . Hence the MLE  $\widehat{w}$  must have  $d_{TV}(w^{*},w)\leq \frac{17}{16}\varepsilon$ . Rescaling  $\varepsilon$  gives the conclusion of the Lemma.
$$
\Omega := \left\{P \in \mathbb {R} _ {+} ^ {d \times d}: \frac {\lambda_ {\max } (P)}{\lambda_ {\min } (P)} \leq \kappa a n d \lambda_ {\max } (P) \in [ L \cdot \lambda_ {\min } ^ {*}, U \cdot \lambda_ {\max } ^ {*} ] \right\}.
$$
$$
\Pr_ {x _ {i} \sim \mathcal {D} _ {x}} \left[ \sup  _ {W \in \mathbb {R} ^ {d \times k}, P \in \Omega} \left| \widetilde {d} ((W, P), (W ^ {*}, P ^ {*})) - d ((W, P), (W ^ {*}, P ^ {*})) \right| > \varepsilon \right] \leq e ^ {- \Omega (n \varepsilon^ {2})}.
$$
Proof of Lemma C.8. For  $P = \Sigma^{-1}$  and  $P^{*} = \Sigma^{*-1}$ , let
$$
f (W, P) := d ((W, \Sigma), (W ^ {*}, \Sigma^ {*}))
$$
$$
f _ {n} (W, P) := \widetilde {d} ((W, \Sigma), (W ^ {*}, \Sigma^ {*})) = \frac {1}{n} \sum_ {i} [ d _ {T V} (p _ {W, P} (y | x _ {i}), p _ {W ^ {*}, P ^ {*}} (y | x _ {i})) ].
$$
Since the function is bounded, for any fixed  $W, P$ , the Chernoff bound gives
$$
\Pr \left[ \left| f _ {n} (W, P) - f (W, P) \right| > \alpha \right] \leq e ^ {- 2 n \alpha^ {2}}. \tag {41}
$$
for any  $\alpha > 0$ . The challenge lies in constructing a net to be able to union bound over  $\mathbb{R}^k$  without assuming any bound on  $W$  or the covariate  $x$ . As before, we do so by constructing a "ghost" sample, symmetrizing, and constructing a net based on these samples.
Ghost sample. First, we construct a "ghost" dataset  $D_x'$  consisting of  $n$  fresh samples IID samples  $\{x_i'\}_{i \in [n]}$  of  $\mathcal{D}_x$ . This gives another metric
$$
f _ {n} ^ {\prime} (W, P) := \widetilde {d} ^ {\prime} ((W, \Sigma), (W ^ {*}, \Sigma^ {*})) = \frac {1}{n} \sum_ {i} [ d _ {T V} (p _ {W, P} (y | x _ {i} ^ {\prime}), p _ {W ^ {*}, P ^ {*}} (y | x _ {i} ^ {\prime})) ].
$$
Similar to the proof in Lemma 4.3, it is sufficient to consider the difference between  $f_{n}(W,P)$  and  $f_{n}^{\prime}(W,P)$  i.e.,
$$
\Pr \left[ \sup  _ {W, P} | f (W, P) - f _ {n} (W, P) | > \varepsilon \right] \leq 2 \Pr \left[ \sup  _ {W, P} | f _ {n} (W, P) - f _ {n} ^ {\prime} (W, P) | > \varepsilon / 2 \right]. \tag {42}
$$