SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| d _ {T V} \left(p _ {w} (y \mid x _ {i}), p _ {w ^ {*}} (y \mid x _ {i})\right) - d _ {T V} \left(p _ {\bar {w}} (y \mid x _ {i}), p _ {w ^ {*}} (y \mid x _ {i})\right) \right| <   \alpha
$$
for all  $i\in [2n]$  .Thus
$$
\left| \left(\widetilde {d} ^ {\prime} (w, w ^ {*}) - \widetilde {d} (w, w ^ {*})\right) - \left(\widetilde {d} ^ {\prime} (\bar {w}, w ^ {*}) - \widetilde {d} (\bar {w}, w ^ {*})\right) \right| \leq \left| \widetilde {d} (w, w ^ {*}) - \widetilde {d} (\bar {w}, w ^ {*}) \right| + \left| \widetilde {d} ^ {\prime} (w, w ^ {*}) - \widetilde {d} ^ {\prime} (\bar {w}, w ^ {*}) \right| \leq 2 \alpha
$$
$$
\Pr \left[ \sup  _ {w \in \mathbb {R} ^ {d}} \left| \widetilde {d} ^ {\prime} (w, w ^ {*}) - \widetilde {d} (w, w ^ {*}) \right| > \varepsilon \right] \leq \Pr \left[ \max  _ {w \in N} \left| \widetilde {d} ^ {\prime} (w, w ^ {*}) - \widetilde {d} (w, w ^ {*}) \right| > \varepsilon - 2 \alpha \right] \leq e ^ {2 k \log \frac {n}{\alpha k} - (\varepsilon - 2 \alpha) ^ {2} n / 2}
$$
Setting  $\alpha = \varepsilon /4$  we have that
$$
n \lesssim \frac {1}{\varepsilon^ {2}} k \log \frac {1}{\varepsilon}
$$
suffices for
$$
\Pr \left[ \max  _ {w \in \mathbb {R} ^ {k}} \widetilde {d} ^ {\prime} (w, w ^ {*}) - \widetilde {d} (w, w ^ {*}) > \varepsilon \right] <   e ^ {- \Omega (\varepsilon^ {2} n)}.
$$
B ReLU Activation with Scalar  $y$
In this section, we consider the model of
$$
y = \phi (w ^ {*} \cdot x + \eta), \quad \eta \sim \mathcal {N} (0, 1),
$$
where  $w^{*},x\in \mathbb{R}^{k},y,\eta \in \mathbb{R}$ . We are given samples  $(x,y)\in \mathbb{R}^k\times \mathbb{R}$ , and want to estimate a  $\widehat{w}$  that estimates the distribution of  $y$  in TV.
The most challenging aspect of the ReLU setting is that we do not have an expression for the TV suffered by the MLE, such as Lemma 4.2 in the linear case. This forces us to directly analyze the log-likelihood.
For a fixed  $x, w$ , the expectation of the log-likelihood ratio over  $y$  is
$$
\mathbb {E} _ {y} \left[ \log \frac {p _ {w} (y \mid x)}{p _ {w ^ {*}} (y \mid x)} \right] = - d _ {K L} \left(w ^ {*} \| w\right) \leq - 2 d _ {T V} ^ {2} \left(w ^ {*}, w\right),
$$
where the last inequality is via Pinsker's inequality. This equation implies that if  $w$  is  $\varepsilon$ -far from  $w^*$ , then the expected log-likelihood ratio(LLR) is  $< -2\varepsilon^2$ . By definition, the MLE has a non-negative LLR. Hence, if the empirical LLR is close to the expectation, this would imply that the MLE has small TV.
However, we only receive a single sample of  $y$  per  $x$ . For a fixed  $w$ , we can prove a Bernstein inequality, showing that given  $1 / \varepsilon^2 \log(1 / \delta)$  samples, the empirical LLR is  $< -\varepsilon^2$  for  $w$  that are  $\varepsilon$ -far.
Lemma B.1. Let  $p_1, \ldots, p_n$  and  $q_1, \ldots, q_n$  be distributions with  $\mathbb{E}_i[d_{TV}(p_i, q_i)] \geq \varepsilon$ , where we use the uniform measure on  $i \in [n]$ . Let  $x_i \sim p_i$  for  $i \in [n]$ . Then w.p.  $1 - \delta$ ,  $\mathbb{E}_i[\log \frac{q_i(x_i)}{p_i(x_i)}] \leq -\frac{\varepsilon^2}{4}$  for  $n \geq O\left(\frac{1}{\varepsilon^2} \log \frac{1}{\delta}\right)$ .
The proof of this Lemma, as well as other Lemmas in this section, can be found in Appendix B.1.
In order to extend this to all  $w \in \mathbb{R}^k$  that are  $\varepsilon$ -far, we will construct a cover over  $\mathbb{R}^k$  depending on the values the log-likelihood ratio can take, and then apply the Bernstein inequality to each element in the cover.
In order to construct the cover, we first show that the log-likelihood ratio is bounded above by the magnitude of noise in  $y$ . For ease of notation, for a fixed  $x \in \mathbb{R}^k$ , and each  $w \in \mathbb{R}^k$ , define
$$
\theta = \langle x, w \rangle \in \mathbb {R},
$$
and let  $\theta^{*} = \langle x, w^{*} \rangle$ . Similar to the notation for  $w$ , for each  $\theta \in \mathbb{R}$ , define  $p_{\theta}$  as the distribution of  $\phi(\theta + \eta)$  for  $\eta \sim N(0, 1)$ . Define the log likelihood ratio
$$
\gamma_ {\theta} (y) := \log \frac {p _ {\theta} (y | x)}{p _ {\theta^ {*}} (y | x)}.
$$
The following Lemma states that for a fixed datapoint  $(x,y)$ , the log-likelihood ratio is bounded by the noise in  $y$ :
Lemma B.2. For any  $y = \phi (\theta +\eta)$
$$
\gamma_ {\theta} (y) = \left\{ \begin{array}{l l} \log \Phi (- \theta) - \log \Phi (- \theta^ {*}) & i f y = 0 \\ \eta (\theta - \theta^ {*}) - \frac {(\theta - \theta^ {*}) ^ {2}}{2} & i f y > 0 \end{array} \right.
$$
and therefore, for all  $y$ ,
$$
\gamma_ {\theta} (y) \leq | \eta | ^ {2} / 2.
$$
Now, as  $\gamma$  is bounded above by  $\frac{|\eta|^2}{2}$ , and it is concave wrt  $\theta$ , the following Lemma shows that we can partition  $\theta$  into  $O\left(\frac{A}{\varepsilon}\right)$  intervals, such that in each interval,  $\gamma$  changes by atmost  $\varepsilon$ , or is very negative, i.e.,  $\gamma < -A$ .
Lemma B.3 (One-dimensional net). Let  $A > B^2 > 1$ . There exists a partition of  $\mathbb{R}$  into  $O(A / \varepsilon)$  intervals such that, for each interval  $I$  in the partition and every  $y = \phi(\theta^{*} + \eta)$  with  $|\eta| \leq B$ , one of the following holds:
- For all  $\theta \in I$ ,  $\gamma_{\theta}(y) \leq -A$
- For all  $\theta, \theta' \in I$ ,  $|\gamma_{\theta}(y) - \gamma_{\theta'}(y)| \leq \varepsilon$
Using Lemma B.2 and Lemma B.3, we can form a uniform bound, such that all  $w$  that are  $\varepsilon$ -far from  $w^{*}$  in distribution will have log-likelihood ratio smaller than  $-\frac{\varepsilon^2}{4}$  on the training set. With some additional arguments, we can now show that as the MLE has positive log-likelihood ratio, it has small empirical TV.
Lemma B.4. Let  $x_{1}, \ldots, x_{n}$  be fixed, and  $y_{i} \sim \phi(x_{i}^{T} w^{*} + \eta_{i})$  for  $\eta_{i} \sim \mathcal{N}(0, 1)$ . For  $n \geq \frac{1}{\varepsilon^{2}} k \log \frac{1}{\varepsilon}$ , the MLE  $\widehat{w}$  satisfies
$$
\tilde {d} (\widehat {w}, w ^ {*}) \leq \varepsilon .
$$
This sample complexity guarantees that the MLE is good for the set of empirical  $x_{i} \sim \mathcal{D}_{x}$ , and we need to extend this to the expectation over  $x \sim \mathcal{D}_x$ , for which we use a Lemma similar to Lemma 4.3 in the linear case, and this completes the proof.
A straight forward combination of Lemma 4.3 and Lemma B.4 gives the following Theorem.
Theorem B.5. Let  $y = \phi(x^T w^* + \eta)$ , for  $w^* \in \mathbb{R}^k$ ,  $x \sim \mathcal{D}_x$ , and  $\eta \sim \mathcal{N}(0,1)$ . Then for a sufficiently large constant  $C > 0$ ,
$$
n = C \cdot \frac {k}{\varepsilon^ {2}} \log \frac {1}{\varepsilon}
$$
samples of  $\{(y_i, x_i)\}_{i=1}^n$  suffices to guarantee that the MLE  $\widehat{w}$  satisfies
B.1 Proofs
Proof. Define  $\gamma_i(x) = \log \frac{q_i(x)}{p_i(x)}$  and  $a_i(x) \coloneqq \max (\gamma_i(x), -2)$ . We have that
$$
\underset {i, x} {\mathbb {E}} [ \gamma_ {i} (x) ] = - \underset {i} {\mathbb {E}} [ d _ {K L} (p _ {i}, q _ {i}) ] \leq - \underset {i} {\mathbb {E}} [ 2 d _ {T V} (p _ {i}, q _ {i}) ^ {2} ] \leq - 2 \varepsilon^ {2}
$$
and want to show that  $\mathbb{E}_i[\gamma_i(x)]\leq -\varepsilon^2 /4$  with high probability. Note that  $a_{i}(x)\geq \gamma_{i}(x)$ , so it suffices to show  $\mathbb{E}_i[a_i(x)]\leq -\varepsilon^2 /4$ . We will do this with Bernstein's inequality, for which we need bounds on the moments of  $a_{i}(x)$ .
To simplify notation, fix a particular  $i$  and consider  $p = p_i, q = q_i, a = a_i$ , and  $x \sim p$ .
For a random variable  $v$ , define  $v_{+}, v_{-}$  to be the positive/negative parts of  $v$ , respectively, so  $v = v_{-} + v_{+}$ . Define  $\Delta(x) = \frac{q(x)}{p(x)} - 1$ . We have that  $\mathbb{E}_{x \sim p}[\Delta(x)] = 0$ , and
$$
\underset {x \sim p} {\mathbb {E}} [ \Delta_ {+} (x) ] = \underset {x \sim p} {\mathbb {E}} [ - \Delta_ {-} (x) ] = d _ {T V} (p, q). \tag {14}
$$
Now, consider the function  $b(z) \coloneqq \max(\log(1 + z), -2) - z$ . This function is nonpositive over  $z \geq -1$ , and  $b(z) \leq -z^2 / 2$  for  $z \leq 0$ . Since
$$
a (x) = b (\Delta (x)) + \Delta (x)
$$
and  $\mathbb{E}_{x\sim p}[\Delta (x)] = 0$ $\mathbb{E}_{x\sim p}[-a(x)] = \mathbb{E}_{x\sim p}[-b(\Delta (x))]$  . This means
$$
\begin{array}{l} \underset {x \sim p} {\mathbb {E}} [ - a (x) ] = \underset {x \sim p} {\mathbb {E}} [ - b (\Delta (x)) ] \geq \underset {x \sim p} {\mathbb {E}} [ - b (\Delta (x)) 1 _ {\Delta (x) <   0} ] \\ \geq \underset {x \sim p} {\mathbb {E}} [ \Delta_ {-} ^ {2} (x) / 2 ] \\ \end{array}
$$
or by (14),
$$
\mathbb {E} _ {x} [ - a (x) ] \geq \mathbb {E} _ {x} [ \Delta_ {-} ^ {2} (x) / 2 ] \geq \frac {1}{2} d _ {T V} (p, q) ^ {2}. \tag {15}
$$
Bounding the positive higher moments. We have that  $p(x)e^{a(x)} = \max (q(x),e^{-2}p(x))$  so
$$
\begin{array}{l} \mathbb {E} [ e ^ {a (x)} ] = \int \max (q (x), e ^ {- 2} p (x)) d x \\ \leq 1 + e ^ {- 2} \Pr [ a (x) = - 2 ]. \\ \end{array}
$$
In the following, we use that  $e^t \geq 1 + t$  for all  $t$ , as well as  $e^t = 1 + t + \sum_{k=2}^{\infty} \frac{1}{k!} t^k$ . Therefore
$$
\begin{array}{l} 1 + e ^ {- 2} \Pr [ a (x) = - 2 ] \\ \geq \mathbb {E} [ e ^ {a (x)} ] = \mathbb {E} [ e ^ {a _ {-} (x)} 1 _ {a (x) \leq 0} + e ^ {a _ {+} (x)} 1 _ {a (x) > 0} ] \\ \geq \mathbb {E} [ 1 + (a _ {-} (x) + a _ {+} (x)) + \sum_ {k = 2} ^ {\infty} \frac {1}{k !} a _ {+} ^ {k} (x) ] \\ = 1 + \mathbb {E} [ a (x) ] + \sum_ {k = 2} ^ {\infty} \frac {1}{k !} \mathbb {E} [ a _ {+} ^ {k} (x) ] \\ \end{array}
$$
$$
\sum_ {k = 2} ^ {\infty} \frac {1}{k !} \mathbb {E} \left[ a _ {+} ^ {k} (x) \right] \leq \mathbb {E} \left[ - a (x) \right] + e ^ {- 2} \Pr [ a (x) = - 2 ].
$$
We now show that the  $\operatorname*{Pr}[a(x) = -2]$  is smaller than the  $\mathbb{E}[-a(x)]$  term, by relating to  $-b$ . When  $a(x) = -2$ ,  $\Delta (x)\leq -1 + 1 / e^2$ , and  $b(\Delta (x)) = -2 - \Delta (x)\leq -1$ . Since  $-b(\Delta (x))$  is non-negative, and at least 1 whenever  $a(x) = -2$ ,
$$
\mathbb {E} [ - a (x) ] = \mathbb {E} [ - b (\Delta (x)) ] \geq \Pr [ a (x) = - 2 ] \cdot 1
$$
$$
\sum_ {k = 2} ^ {\infty} \frac {1}{k !} \mathbb {E} \left[ a _ {+} ^ {k} (x) \right] \leq \left(1 + \frac {1}{e ^ {2}}\right) \mathbb {E} [ - a (x) ]. \tag {16}
$$
In particular,  $\mathbb{E}[a_{+}^{k}(x)]\leq 2k!\mathbb{E}[-a(x)]$  for all  $k\geq 2$
Bounding the second moment of  $a$ . We have that
$$
\mathbb {E} [ a (x) ^ {2} ] = \mathbb {E} [ a _ {+} ^ {2} (x) + a _ {-} ^ {2} (x) ]
$$
and  $\mathbb{E}[a_{+}^{2}(x)]\leq 4\mathbb{E}[-a(x)]$  by (16). We now bound  $\mathbb{E}[a_{-}^{2}(x)]$ . Note that  $|a_{-}(x)|\leq \frac{2}{1 - 1 / e^2} |\Delta_{-}(x)|$  by the construction of  $a$ . Therefore
$$
a _ {-} ^ {2} (x) \leq 6 \Delta_ {-} ^ {2} (x)
$$
and so by (15),
$$
\mathbb {E} \left[ a _ {-} ^ {2} (x) \right] \leq 6 \mathbb {E} \left[ \Delta_ {-} ^ {2} (x) \right] \leq 1 2 \mathbb {E} \left[ - a (x) \right].
$$
$$
\mathbb {E} \left[ a ^ {2} (x) \right] \leq 1 6 \mathbb {E} \left[ - a (x) \right]. \tag {17}
$$
Bernstein Concentration. Now we can apply Bernstein's inequality (Theorem 2.10 of [8]).
We apply the theorem to  $X_{i} \coloneqq a_{i}(x_{i})$ , which are independent. The theorem uses that
$$
\sum_ {i = 1} ^ {n} \mathbb {E} \left[ X _ {i} ^ {2} \right] = n \mathbb {E} _ {i, x} \left[ a _ {i} (x) ^ {2} \right] \leq 1 6 n \mathbb {E} _ {i, x} \left[ - a _ {i} (x) \right] =: v
$$
by (17), and since
$$
\sum_ {i = 1} ^ {n} \mathbb {E} [ (X _ {i}) _ {+} ^ {k} ] = n \mathbb {E} _ {i, x} [ a _ {i, +} (x) ^ {k} ] \leq 2 k! \mathbb {E} _ {i, x} [ - a _ {i} (x) ] \leq \frac {1}{2} v k!
$$
so we can set  $c = 1$ . Applying the theorem, we have that  $S = \sum a_{i}(x_{i}) - \mathbb{E}[a_{i}(x_{i})]$  satisfies
$$
S \leq \sqrt {2 v \log {\frac {1}{\delta}}} + \log {\frac {1}{\delta}}
$$
with probability  $1 - \delta$ . Plugging in  $v$  and rescaling by  $n$ , with probability  $1 - \delta$  we have:
$$
\mathbb {E} _ {i} [ a _ {i} (x _ {i}) ] \leq \mathbb {E} _ {i, x} [ a _ {i} (x) ] + O (1) \cdot \sqrt {\mathbb {E} [ - a (x) ] \frac {1}{n} \log \frac {1}{\delta}} + \frac {1}{n} \log \frac {1}{\delta}
$$
By our assumption on  $n$  for a sufficiently large constant in the big  $O$ , this implies
$$
\mathbb {E} _ {i} \left[ a _ {i} \left(x _ {i}\right) \right] \leq \mathbb {E} _ {i, x} \left[ a _ {i} (x) \right] + \frac {1}{6} \varepsilon \sqrt {\mathbb {E} _ {i , x} \left[ - a _ {i} (x) \right]} + \varepsilon^ {2} / 8
$$
Since by (15),  $\varepsilon \leq \sqrt{\mathbb{E}_i[d_{TV}(p_i,q_i)^2]}\leq \sqrt{\mathbb{E}_{i,x}[-2a_i(x)]}$  , this means
$$
\begin{array}{l} \mathbb {E} _ {i} \left[ \gamma_ {i} \left(x _ {i}\right) \right] \leq \mathbb {E} _ {i} \left[ a _ {i} \left(x _ {i}\right) \right] \leq \left(- 1 + \frac {\sqrt {2}}{6}\right) \mathbb {E} _ {i, x} \left[ - a _ {i} (x) \right] + \varepsilon^ {2} / 8 \\ \leq (- 1 + \frac {\sqrt {2}}{6}) \frac {1}{2} \varepsilon^ {2} + \frac {1}{8} \varepsilon^ {2} \\ \leq - \frac {1}{4} \varepsilon^ {2} \\ \end{array}
$$
Proof. Let  $\Phi(x)$  be the cdf of a standard Gaussian. For  $y > 0$ ,
$$
\begin{array}{l} \gamma_ {\theta} (y) = \frac {1}{2} \left(\left(y - \theta^ {*}\right) ^ {2} - (y - \theta) ^ {2}\right) \\ = \frac {1}{2} \left(\eta^ {2} - \left(\eta + \theta^ {*} - \theta\right) ^ {2}\right) \\ = \eta (\theta - \theta^ {*}) - \frac {(\theta - \theta^ {*}) ^ {2}}{2} \\ \end{array}
$$
$$
\gamma_ {\theta} (y) = \left\{ \begin{array}{l l} \log \Phi (- \theta) - \log \Phi (- \theta^ {*}) & \text {i f} y = 0 \\ \eta (\theta - \theta^ {*}) - \frac {(\theta - \theta^ {*}) ^ {2}}{2} & \text {i f} y > 0 \end{array} \right.
$$
Now suppose  $|\eta |\leq B$ . We can upper bound  $\gamma_{\theta}(y)$  for all  $\theta$
- If  $y = 0$ , then  $-\theta^{*} \geq -B$ , so
$$
\gamma_ {\theta} (0) \leq - \log \Phi (- \theta^ {*}) \leq - \log e ^ {- B ^ {2} / 2} = B ^ {2} / 2.
$$
- If  $y > 0$ , then
$$
\gamma_ {\theta} (y) = (\theta - \theta^ {*}) \eta - \frac {(\theta - \theta^ {*}) ^ {2}}{2} \leq \eta^ {2} / 2 \leq B ^ {2} / 2.
$$
Proof. To define our partition, we actually define two partitions, depending on whether  $y = 0$ , then intersect them for our final partition.
First, consider  $y = 0$ . By Lemma B.2,  $\gamma_{\theta}(0)$  is monotonically decreasing in  $\theta$ , from its maximum of at most  $B^{2}/2$ . We can thus define a partition  $P_{1}$  consisting of intervals of the form  $I_{i} := \{\theta \mid \gamma_{\theta}(0) \in (B^{2}/2 - (i + 1)\varepsilon, B^{2}/2 - i\varepsilon)\}$ , for  $i \in \{0,1,\ldots,(A + B^{2}/2)/\varepsilon\}$ , plus a special interval  $I'$  of  $\{\theta \mid \gamma_{\theta}(0) < -A\}$ . When  $y = 0$ , this partition satisfies the desired conclusion to the lemma:  $|\gamma_{\theta}(0) - \gamma_{\theta'}(0)| \leq \varepsilon$  for all  $\theta, \theta' \in I_{i}$ , while  $\gamma_{\theta}(0) < -A$  for  $\theta \in I'$ . Call this partition  $P_{0}$ , which has size  $O(A/\varepsilon)$ .
Second, consider  $y > 0$ . Define  $R = \sqrt{2A} + B$ . Note that  $R^2 \lesssim A$  and  $(R - B)^2 \geq 2A$ . Therefore for  $|\theta - \theta^*| \geq R$ ,
$$
\gamma_ {\theta} (y) \leq - \frac {1}{2} \max (0, | \theta - \theta^ {*} | - \eta) ^ {2} \leq - A.
$$
Consider any  $\theta, \theta' \in [\theta^* - R, \theta^* + R]$  with  $\alpha \coloneqq |\theta - \theta'|$ . We have
$$
\begin{array}{l} \left| \gamma_ {\theta} (y) - \gamma_ {\theta^ {\prime}} (y) \right| \leq \left| \eta \left(\theta - \theta^ {\prime}\right) \right| + \frac {1}{2} \left| \left(\theta^ {\prime} - \theta^ {*}\right) ^ {2} - \left(\theta - \theta^ {*}\right) ^ {2} \right| \\ \leq B \alpha + \frac {1}{2} | \left(\theta^ {\prime} - \theta\right) \left(- 2 \theta^ {*} + \left(\theta^ {\prime} + \theta\right)\right) | \\ \leq B \alpha + \frac {1}{2} \alpha (2 R) = \alpha (B + R). \\ \end{array}
$$
Thus, for  $\alpha = \frac{\varepsilon}{2R}$ , this is at most  $\varepsilon$ . If we partition  $[\theta^{*} - R,\theta^{*} + R]$  into length- $\alpha$  intervals, we get a size  $O(R^2 /\varepsilon) = O(A / \varepsilon)$  partition  $P_{1}$  of  $\mathbb{R}$  that has the desired property for all  $y > 0$ .
Our final partition is defined by all endpoints in either  $P_0$  and  $P_1$ . This has size  $O(A / \varepsilon)$ , and within each interval the conclusion holds for both  $y = 0$  and  $y > 0$ , as needed.
$$
\tilde {d} (\hat {w}, w ^ {*}) \leq \varepsilon .
$$
Proof. For any  $w \in \mathbb{R}^k$ , and a sample  $(x_i, y_i)$ , let  $p_w(y|x_i)$  be the conditional distribution of  $y = \phi(\langle x_i, w \rangle + \eta)$ , and let  $\gamma_{i,w}$  be the log-likelihood ratio between  $w$  and  $w^*$  on this sample:
$$
\gamma_ {i, w} (y) := \log \frac {p _ {w} (y | x _ {i})}{p _ {w ^ {*}} (y | x _ {i})}.
$$
$$
\underset {y} {\mathbb {E}} [ \gamma_ {i, w} (y) ] = - d _ {K L} (p _ {i, w ^ {*}} (y | x _ {i}) | | p _ {i, w} (y | x _ {i})).
$$
$$
d _ {K L} (w ^ {*}, w) := \frac {1}{n} \sum_ {i = 1} ^ {n} d _ {K L} \left(p _ {i, w ^ {*}} (y | x _ {i}) | | p _ {i, w} (y | x _ {i})\right).
$$
Concentration. From Lemma B.1, we see that if  $\widetilde{d} (w^{*},w)\geq \varepsilon$ , then for  $n\geq O(\frac{1}{\varepsilon^2}\log \frac{1}{\delta})$
$$
\bar {\gamma} _ {w} := \frac {1}{n} \sum_ {i = 1} ^ {n} \gamma_ {i, w} \left(y _ {i}\right) <   - \frac {\varepsilon^ {2}}{4}, \tag {18}
$$
Of course, whenever  $\overline{\gamma}_w < 0$ , the likelihood under  $w^*$  is larger than the likelihood under  $w$ . Thus, for each fixed  $w$  with  $\widetilde{d}(w^*, w) \geq \varepsilon$ , maximizing likelihood would prefer  $w^*$  to  $w$  with probability  $1 - \delta$  if  $n \geq O\left(\frac{1}{\varepsilon^2} \log \frac{1}{\delta}\right)$ .
Nothing above is specific to our ReLU-based distribution. But to extend to the MLE over all  $w$ , we need to build a net using properties of our distribution.
Building a net. First, with high probability,  $|\eta_i| \leq B = O(\sqrt{\log n})$  for all  $i$ . Suppose this happens. For each  $i$ , by an abuse of notation, let  $\gamma_{i,w}(y) = \gamma_{\langle x_i, w \rangle}(y)$  where the value of  $\theta^*$  when considering  $i$  is  $\langle x_i, w^* \rangle$ . By Lemma B.2,
$$
\gamma_ {i, w} (y _ {i}) \leq B ^ {2} / 2
$$
for all  $i$ . Let  $A = O(n\log n) > nB^2$ . By Lemma B.3, for each  $i \in [n]$ , there exists a partition  $P_i$  of  $\mathbb{R}$  into  $O(A / \varepsilon^2)$  intervals, such that for interval  $I \in P_i$ , and any  $w, w'$  with  $x_i^T w, x_i^T w' \in I$ , either
$$
\left| \gamma_ {i, w} \left(y _ {i}\right) - \gamma_ {i, w ^ {\prime}} \left(y _ {i}\right) \right| \leq \varepsilon^ {2} / 2 \tag {19}
$$
or  $\gamma_{i,w}(y_i) < -A$ .
These individual partitions  $P_{i}$  on  $\langle x_i, w \rangle$  induce a partition  $P$  on  $\mathbb{R}^k$ , where  $w, w'$  lie in the same cell of  $P$  if  $\langle x_i, w \rangle$  and  $\langle x_i, w^* \rangle$  are in the same cell of  $P_{i}$  for all  $i \in [n]$ . Since  $P$  is defined by  $n$  sets of  $O\left(\frac{A}{\varepsilon^2}\right)$  parallel hyperplanes in  $\mathbb{R}^k$ , the number of cells in  $P$  is:
$$
2 \left(\frac {2 A e n}{\varepsilon^ {2} k}\right) ^ {k}.
$$
We choose a net  $\mathcal{N}$  to contain, for each cell in  $P$ , the  $w$  in the cell maximizing  $\widetilde{d}(w^*, w)$ . This has size
$$
\log | \mathcal {N} | \lesssim k \log \frac {n}{\varepsilon}.
$$
By (18), for our  $n \geq O\left(\frac{1}{\varepsilon^2} k \log \frac{k}{\varepsilon}\right)$ , we have with high probability that  $\overline{\gamma}_w \leq -\frac{\varepsilon^2}{4}$ , for all  $w \in \mathcal{N}$  with  $\widetilde{d}(w^*, w) \geq \varepsilon$ . Suppose that both this happens, and  $|\eta_i| \leq B$  for all  $i$ . We claim that the MLE  $\widehat{w}$  must have  $\widetilde{d}(w^*, \widehat{w}) < \varepsilon$ .
Consider any  $w \in \mathbb{R}^d$  with  $\widetilde{d}_{TV}(w^*, w) \geq \varepsilon$ . Let  $w' \in \mathcal{N}$  lie in the same cell of  $P$ . By our choice of  $\mathcal{N}$ , we know  $\widetilde{d}_{TV}(w^*, w') \geq \widetilde{d}_{TV}(w^*, w) \geq \varepsilon$ , so  $\overline{\gamma}_{w'} \leq -\varepsilon^2$ . Now we consider two cases. In the first case, there exists  $i$  with  $\gamma_{i,w}(y_i) < -A$ . Then
$$
\overline {{\gamma}} _ {w} = \frac {1}{n} \sum_ {i} \gamma_ {i, w} (y _ {i}) \leq - \frac {A}{n} + B ^ {2} / 2 <   0.
$$
Otherwise, by (19),
$$
\overline {{\gamma}} _ {w} \leq \overline {{\gamma}} _ {w ^ {\prime}} + | \overline {{\gamma}} _ {w} - \overline {{\gamma}} _ {w ^ {\prime}} | \leq - \varepsilon^ {2} + \max  _ {i} | \gamma_ {i, w} (y _ {i}) - \gamma_ {i, w ^ {\prime}} (y _ {i}) | \leq - \varepsilon^ {2} / 2.
$$
In either case,  $\overline{\gamma}_w < 0$  and the likelihood under  $w^*$  exceeds that under  $w$ . Hence the MLE  $\widehat{w}$  must have  $\widetilde{d}(w^*, \widehat{w}) \leq \varepsilon$ .
Proof. Let  $D_x$  denote the dataset  $\{x_i\}_{i \in [n]}$  that is used to find the MLE. Notice that the MLE is found using this finite subset, but we would like to make a claim about  $\mathcal{D}_x$  without making any parametric or simplifying assumptions on the distribution  $\mathcal{D}_x$ .
An application of Lemma 4.3 tells us that with probability  $1 - e^{-\Omega (n\varepsilon^2)}$ , the expectation over the distribution  $\mathcal{D}_x$  and the dataset  $D_{x}$  are within  $\varepsilon /2$  of one another:
$$
d (\widehat {w}, w ^ {*}) \leq \widetilde {d} (\widehat {w}, w ^ {*}) + \varepsilon / 2.
$$
Now, all we need to show is that the MLE has a small TV distance on the finite dataset, and Lemma B.4 tells us that with probability  $1 - e^{-\Omega (n\varepsilon^2)}$ ,
$$
\tilde {d} (\widehat {w}, w ^ {*}) \leq \varepsilon / 2.
$$
Substituting in the above inequality, we get  $d(\widehat{w}, w^{*}) \leq \varepsilon$ .
C ReLU Activations with  $d > 1$ , Unknown Covariance
We recommend the reader review Appendix B, which contains the proof recipe for the case of scalar  $y$ . The proofs in this section generalize those of Appendix B.
Consider a sample  $(x,y)\in \mathbb{R}^{k\times d}$ , with
$$
y = \phi \left(W ^ {*} x + \eta\right), \tag {20}
$$
where  $W^{*} \in \mathbb{R}^{d} \times k$ , and noise  $\eta \sim \mathcal{N}(0, \Sigma^{*})$ . The matrices  $W^{*}$  and  $\Sigma^{*}$  are unknown. For each matrix  $W \in \mathbb{R}^{d \times k}$ , let  $\theta = Wx \in \mathbb{R}^{d}$ , denote a reparametrization of  $W$ , and let  $\theta^{*}$  denote  $\theta^{*} = W^{*}x$ . Let  $S$  denote the co-ordinates of  $y$  that are zero-valued. Then the log-likelihood for each  $\theta, \Sigma$  is given by
$$
f _ {\theta , \Sigma} (y) := \log p _ {W, \Sigma} (y \mid x) = c - \frac {1}{2} \log | \Sigma | + \log \int_ {t: t _ {S} \leq 0, t _ {S ^ {c}} = y _ {S ^ {c}}} \exp \Bigl \{- (t - \theta) ^ {T} \Sigma^ {- 1} (t - \theta) / 2 \Bigr \}.
$$
where  $c$  is a normalization constant which does not depend on  $\theta$  or  $\Sigma$ . Let
$$
P := \Sigma^ {- 1}
$$
and let  $P^{*}$  be the precision matrix of the noise  $\eta$ , and  $P_{S}, P_{SS^{c}}, P_{S^{c}S}, P_{S^{c}}$  be the block matrices of  $P$  corresponding to the index sets  $S$  and its complement  $S^{c}$ .
By some arithmetic involving completion of squares, we can decompose the integral in  $f$  into the sum of two functions  $g, h$ , such that
$$
f _ {\theta , \Sigma} (y) = c - \frac {1}{2} \log | \Sigma | + g _ {\theta , \Sigma} (y) + h _ {\theta , \Sigma} (y).
$$
The first term  $g$  corresponds to the quadratic term involving the observed positive-valued coordinates  $y_{S^{\epsilon}}$ :
$$
g _ {\theta , \Sigma} (y) = - \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right) ^ {T} \left(P _ {S ^ {c}} - P _ {S ^ {c} S} \left(P _ {S}\right) ^ {- 1} P _ {S S ^ {c}}\right) \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right) / 2.
$$
As the matrix  $P_{S^c} - P_{S^c S}(P_S)^{-1}P_{SS^c} = ((P^{-1})_{S^c})^{-1} = \Sigma_{S^c}^{-1}$  is the precision matrix of  $\eta_S$ , if  $\Sigma_S$  were the covariance of  $\eta$ , we can simplify the above equation as
$$
g _ {\theta , \Sigma} (y) = - \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right) ^ {T} \left(\Sigma_ {S ^ {c}}\right) ^ {- 1} \left(y _ {S ^ {c}} - \theta_ {S ^ {c}}\right) / 2. \tag {21}
$$