SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison to [35]. Our result is closely related to [35]. Our ReLU model reduces to their model by setting  $W^{*} \in \mathbb{R}^{d}$  as a vector, and  $x = 1$ . In order to learn the distribution of  $y$ , they first estimate the parameters  $W^{*}, \Sigma^{*}$ , in  $\ell_{2}$  norm and then convert the  $\ell_{2}$  error to a TV error. The parameter recovery is done per coordinate of  $W^{*}, \Sigma^{*}$ , by performing MAP estimation on the positive samples in  $y$ . This crucially assumes that each coordinate has enough positive samples, and to that end, they assume that each entry in  $W^{*}$  is positive—otherwise, their sample complexity scales as  $e^{\|W^{*}\|_{\infty}^{2}}$ . Our results do not make any assumptions on  $W^{*}$  and handles a wider class of matrix valued  $W^{*}$ . Additionally, the objective function (8) does not discard the zero valued samples in  $y$ , making it more sample efficient.
We assumed that the covariance matrix  $\Sigma^{*}$  has condition number  $\kappa$ , and our sample complexity scales as  $\log \kappa$ . Hence, even if  $\kappa = e^{\mathrm{poly}(d,k)}$ , we only pay a  $\mathrm{poly}(d,k)$  penalty. This improves on the result in [35], where the sample complexity scales as  $\kappa^2$ . While our statistical guarantees are strictly better, [35] gives poly-time and poly-space algorithmic guarantees for their estimator. We discuss the empirical limitations of our algorithm in Section 6.
Lower Bounds Ignoring log factors, the complexity factor of  $kd$  is obviously required. Furthermore, learning a Gaussian with unknown covariance matrix in total variation takes  $\hat{\Omega}(d^2)$  samples; see [6]. Our Theorem 4.5 would solve their lower bound instance, the same lower bound applies to our problem.
Extension to Multi-Layer Generative Models. Consider the following  $(L + 1)$ -layered generative model.
$$
x _ {L + 1} = W _ {L} ^ {*} x _ {L} + \eta_ {L}, \quad \text {w h e r e} \quad x _ {\ell} = \phi \left(W _ {\ell - 1} ^ {*} x _ {\ell - 1} + \eta_ {\ell - 1}\right) \in \mathbb {R} ^ {d _ {\ell}} \quad \forall \quad \ell \in [ 1, L ], \tag {9}
$$
$$
x _ {0} \sim \mathcal {D} _ {0} \quad \text {a n d} \quad \eta_ {\ell} \sim \mathcal {N} \left(0, \Sigma_ {\ell} ^ {*}\right) \quad \forall \quad \ell \in [ 0, L ]. \tag {10}
$$
We can compose the guarantees provided by Theorem 4.1 and 4.5 to show that we can learn this model.
Theorem 4.6. Given  $n$  i.i.d. samples of  $(x_0, \ldots, x_{L+1})$ , such that each matrix  $\Sigma_\ell^*$  satisfies Assumption 4.4, let  $\widehat{W}_\ell, \widehat{\Sigma}_\ell$ , be the MLE estimates of  $W_\ell^*, \Sigma_\ell^*$  learned from samples of  $(x_\ell, x_{\ell+1})$ . Define  $m := \max_\ell d_\ell^2$ . Then,
$$
n = O \left(\frac {l ^ {2} m}{\varepsilon^ {2}} \log \left(\frac {l m \kappa}{\varepsilon \delta}\right)\right),
$$
samples suffice to ensure that with probability  $1 - \delta$
$$
d _ {T V} \left(\left\{\widehat {W} _ {\ell}, \widehat {\Sigma} _ {\ell} \right\} _ {\ell = 0} ^ {L}, \left\{\left(W _ {\ell} ^ {*}, \Sigma_ {\ell} ^ {*}\right) \right\} _ {\ell = 0} ^ {L}\right) \leq \varepsilon .
$$
Comparison to [1]. The modelling assumptions in [1] are similar to ours – the authors learn a generative model per layer, using images produced per layer. The key differences of their model are: (i) each layer is deterministic (there is no  $\eta_{\ell}$ ), (ii) their learning algorithm does not require access to the activations of each layer, (iii) their algorithm performs moment-matching by crafting the discriminator strategically.
In order to avoid requiring activations at each layer, [1] imposes a sparsity assumption on the activations: this allows them to leverage tools from existing results in sparse coding [4], such that
(a) Total Variation vs.  $n$
(b) Total Variation vs.  $k$
Figure 2: (a) Plot of TV distance vs.  $n$ .  $\sigma^2 = 1$ ,  $w^* = \mathbb{1}_{k \times 1}$ , for two different values of  $k = 5$  and  $k = 20$ . Plot includes data for two different distributions of  $x$ . Note that distribution has little impact on TV distance, and in both cases, we see the error decreasing with a slope of  $-1/2$  in alignment with our theory. (b) Plot of TV distance vs. input dimension  $k$ . For both  $n = 10^3$  and  $n = 10^4$ , the error grows with a slope of roughly  $1/2$ , in alignment with our theory. In both plots 2000 runs are used to compute the mean. Error bars represent  $95\%$  confidence intervals.
sparse activations at each layer can be recovered using images produced by the layer. This assumption can be somewhat strong, as it implies that the activations are roughly independent of one another, and the sparsity remains constant over layers, despite the layers themselves expanding by a factor of 4, i.e.,  $d_{\ell} \geq 4d_{\ell -1} \forall \ell \leq L - 1$ .
We now numerically verify our theoretical claims and compare against other approaches. A detailed description of simulation methods are included in the appendix. Our code is available at https://github.com/ basics-lab/learningGenerativeModels.git.
5.1 Scaling in  $n$  and  $k$
Figure 2 numerically investigates how TV distance of the MLE scales with the number of samples  $n$  and the input dimension  $k$ . We consider a model with 1-dimensional output and a  $k$ -dimensional input:  $y = \phi (x\cdot w^{*} + \eta)$ , for  $w^{*}\in \mathbb{R}^{k}$  and  $\eta \sim \mathcal{N}(0,\sigma^2)$ . We set  $\sigma^2 = 1$  and  $w^{*} = \mathbb{1}_{k\times 1}$  both unknown to the optimizer, which has samples  $(y_{i},x_{i})_{i = 1}^{n}$ . Figure 2a, which plots the error in TV distance against  $n$  on a log-log plot, has a slope of roughly  $-1 / 2$  as predicted by our theory. Similarly, Figure 2b has a slope of  $1 / 2$ , which is in line with our theory on scaling with respect to input dimension  $k$ . We defer simulations involving scaling in  $d$  to the appendix because computing TV distance becomes increasingly difficult as  $d$  becomes large, and we must resort to using upper bounds.
5.2 Distribution Independence
The fact that our guarantee does not depend on the distribution of  $x$  suggests that the expected TV error of the distribution learned from the MLE may be similar for all distributions over  $x$ . To test this we consider both  $x \sim \mathcal{N}(0, I_k)$  and  $x \sim \bigotimes_{k} \mathrm{Lap}(0, 1)$ , i.e., each element of  $x$  is drawn independently standard Laplace. Figure 2a verifies our hypothesis, showing only very slight differences in our observed empirical average TV error between the two distributions over a wide range of  $n$ , and for  $k = 5$  and  $k = 20$ .
(a) Total Variation vs. Bias
(b) Total Variation vs. Condition Number
Figure 3: (a) Left hand axis shows TV distance vs. bias vector  $b$  with  $y = \phi(\eta + b\mathbb{1}_{d \times 1})$ ,  $d = 3$ ,  $\eta = \mathcal{N}(0, \Sigma)$ , and  $\Sigma = I_d$ . Note that MLE (blue) has error going to zero as bias becomes negative, while the opposite is true for the baseline (red). Right hand axis shows the mean-squared error of the parameters  $\Sigma$  and the mean  $\mu$ , each point was run a total of 2000 times. (b) TV distance vs. condition number,  $d = 3$ . MLE does not exhibit trend with condition number, but baseline does. Error bars are 95% confidence intervals, over 20000 runs.
5.3 Scaling with Bias and Condition Number of Covariance Matrix
A key feature of the MLE is that it makes use of truncated samples. This is in contrast to [35], which leverages results on learning truncated normal distributions [14] where truncated samples are not observed. This leads to a stark difference in performance as the number of truncated samples becomes large. To show this, we consider a model with a  $d$ -dimensional output and 1-dimensional input. We let  $x = 1$  almost surely, and then take  $w^{*} = b\mathbb{1}_{d\times 1}$  for some bias  $b \in \mathbb{R}$  and  $\eta = \mathcal{N}(0,\Sigma)$  with  $\Sigma = I_{d}$ , thus  $y = \phi (\eta + b\mathbb{1}_{d\times 1})$ . As  $b$  becomes more negative, the number of truncated samples increases. Figure 3a shows the differing behavior of MLE and that of [35] as  $b$  becomes negative. For ease of computing the TV distance we set  $d = 3$ , and restrict optimization over diagonal  $\Sigma$ . The solid blue line depicts the performance of the MLE. We observe that the TV error is constant for  $b > 0$  and begins to decrease rapidly for  $b < 0$ . This happens because as  $b$  becomes more negative, the truncation places more probability mass at  $y = 0$ . Indeed, the dashed blue lines indicate that even as the TV error is decreasing rapidly, the mean square estimation error of the covariance and mean increase, however, since most of the probability mass is at zero, this does not significantly impact the TV. In contrast, the method of [35] rapidly deteriorates as  $b < 1$  as the number of untruncated samples decreases. We also point out that even when the bias is large, [35] is still significantly worse. We attribute this to the fact that even when there is no truncated samples, [35] is still minimizing a different MAP objective. More discussion of this is provided in the appendix.
Robustness to Condition Number of  $\Sigma$ . Another concern is how TV error scales as a function of the condition number of  $\Sigma^{*}$ . Poorly conditioned  $\Sigma^{*}$  can put significant probability masses on small sets, and potentially cause large error. We consider a similar environment to the one described above, but fix  $b = 1$  and alter diagonal entries of  $\Sigma^{*}$  such that one entry is  $\sqrt{\kappa}$ , another is  $\sqrt{\kappa^{-1}}$  and the rest are 1, making the condition number  $\kappa$ . Figure 3b shows that the MLE is not measurably impacted by the changing condition number over the range plotted. This is not true of [35], where we observe that TV does grow with condition number.
This work is only a first step to understanding fundamental limits of learning generative models. We showed that our theorems for single-layer networks can be composed to get sample complexity bounds on deep networks, with a critical caveat: we require access to not just the input and output pairs, but also the intermediate activations. This is not practical in many scenarios, and removing this restriction will be an important direction for future research. Beyond this, we assume that the learner has an understanding of the model architecture. In many cases, however, a learner may not be aware
of the number of layers a network has or a vast number of other architectural details. Additionally, we inherit known problems with the MLE, such as exacerbation of biases that exist in the training data (Chapter 24.1.3 in [33]).
Our results place emphasis on sample complexity over computational complexity. Though we show that the MLE problem is concave, this work does not provide a thorough analysis of the optimization problem. It is possible that similar results to [14, 35] can be derived for the MLE problem. Indeed, empirically, we find that a similar projected stochastic gradient ascent performs well in our problem. A careful analysis must consider factors like the distribution over  $x$ , the condition number of  $\Sigma^*$  and the truncation probability, all of which are likely to impact the optimization.
We have studied the problem of learning conditional generative models from a limited number of samples. We have shown that it is possible to learn a 1-layer ReLU conditional generative model in total variation, with no assumption on the distribution of the conditioning variable  $x$  using the MLE. We have also shown that this result can be extended to multi-layer ReLU networks, given access to the internal activations. Our results suggest that MLE is a promising approach for learning feed-forward generative models from limited samples.
Ajil Jalal and Kannan Ramchandran are supported by ARO 051242-002. Justin Kang and Kannan Ramchandran are supported by NSF EAGER Award 2232146. Eric Price is supported by NSF awards CCF-2008868, CCF-1751040 (CAREER), and the NSF AI Institute for Foundations of Machine Learning (IFML).
[1] Zeyuan Allen-Zhu and Yuanzhi Li. Forward super-resolution: How can gans learn hierarchical generative models for real-world distributions. arXiv preprint arXiv:2106.02619, 2021.
[2] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017.
[3] Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In International Conference on Machine Learning, pages 224-232. PMLR, 2017.
[4] Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra. Simple, efficient, and neural algorithms for sparse coding. In Conference on learning theory, pages 113-149. PMLR, 2015.
[5] Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do gans learn the distribution? some theory and empirics. In International Conference on Learning Representations, 2018.
[6] Hassan Ashtiani, Shai Ben-David, Nicholas JA Harvey, Christopher Liaw, Abbas Mehrabian, and Yaniv Plan. Near-optimal sample complexity bounds for robust learning of gaussian mixtures via compression schemes. Journal of the ACM (JACM), 67(6):1-42, 2020.
[7] Francis Bach. Learning theory from first principles. Draft of a book, version of Sept, 6:2021, 2021.
[8] Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymptotic theory of independence. Oxford university press, 2013.
[9] Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[10] Sitan Chen, Jerry Li, and Yuanzhi Li. Learning (very) simple generative models is hard. arXiv preprint arXiv:2205.16003, 2022.
[30] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
[31] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[32] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[33] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
[34] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpaca: A strong, replicable instruction-following model. 2023.
[35] Shanshan Wu, Alexandros G Dimakis, and Sujay Sanghavi. Learning distributions generated by one-layer relu networks. Advances in neural information processing systems, 32, 2019.
A Proofs of Linear Case
Throughout the appendix, for ease of notation, we overload the definition of the function  $d_{TV}(\cdot ,\cdot)$ . When inputs are random variables, it represents the TV distance between the distributions of those random variables.
The proof of this lemma requires Lemma A.1, which characterizes the distribution of the residual error of the MLE.
Lemma A.1. Given  $y \in \mathbb{R}^n$ ,  $X \in \mathbb{R}^{n \times k}$  satisfying  $y = Xw^{*} + \eta$ , where  $\eta \sim \mathcal{N}(0, \sigma^2 I_n)$ , the least square solution  $\widehat{w}$  satisfies
$$
X w ^ {*} - X \widehat {w} \sim \mathcal {N} (0, \sigma^ {2} X (X ^ {T} X) ^ {- 1} X ^ {T}) \Rightarrow \mathbb {E} [ \| X \widehat {w} - X w ^ {*} \| ^ {2} ] = \sigma^ {2} k.
$$
Proof. The least squares solution is given by
$$
\begin{array}{l} \hat {w} = \left(X ^ {T} X\right) ^ {- 1} X ^ {T} y, \\ = \left(X ^ {T} X\right) ^ {- 1} X ^ {T} \left(X w ^ {*} + \eta\right), \\ = w ^ {*} + \left(X ^ {T} X\right) ^ {- 1} X ^ {T} \eta . \\ \end{array}
$$
Multiplying on the left by  $X$ , we have
$$
X \hat {w} = X w ^ {*} + X (X ^ {T} X) ^ {- 1} X ^ {T} \eta .
$$
Since  $\eta$  is i.i.d. Gaussian with variance  $\sigma^2$ , we have,
$$
\begin{array}{l} X \left(X ^ {T} X\right) ^ {- 1} X ^ {T} \eta \sim \mathcal {N} \left(0, \sigma^ {2} X \left(X ^ {T} X\right) ^ {- 1} X ^ {T} X \left(X ^ {T} X\right) ^ {- 1} X ^ {T}\right) \\ \sim \mathcal {N} (0, \sigma^ {2} X (X ^ {T} X) ^ {- 1} X ^ {T}) \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} [ \| X \hat {w} - X w ^ {*} \| ^ {2} ] = \sigma^ {2} \operatorname {T r} \left[ X \left(X ^ {T} X\right) ^ {- 1} X ^ {T} \right], \\ = \sigma^ {2} \operatorname {T r} \left[ \left(X ^ {T} X\right) ^ {- 1} X ^ {T} X \right], \\ = \sigma^ {2} k. \\ \end{array}
$$
Proof of Lemma 4.2. The KL divergence between two Gaussians  $P = \mathcal{N}(\mu_1, \Sigma)$  and  $Q = \mathcal{N}(\mu_2, \Sigma)$  is:
$$
d _ {K L} (P \| Q) = \frac {1}{2} (\mu_ {1} - \mu_ {2}) \Sigma^ {- 1} (\mu_ {1} - \mu_ {2}).
$$
By Pinsker's inequality, this implies
$$
d _ {T V} (P \| Q) \leq \min  \left\{1, \frac {1}{2} \sqrt {\left(\mu_ {1} - \mu_ {2}\right) \Sigma^ {- 1} \left(\mu_ {1} - \mu_ {2}\right)} \right\}.
$$
Hence, the empirical TV on the dataset can be bounded by
$$
\begin{array}{l} \frac {1}{n} \sum_ {i} d _ {T V} \left(p _ {\widehat {w}} \left(y \mid x _ {i}\right), p _ {w ^ {*}} \left(y \mid x _ {i}\right)\right) \leq \frac {1}{n} \sum_ {i} \min  \left\{1, \frac {1}{2} \frac {\left| x _ {i} ^ {T} \left(\widehat {w} - w ^ {*}\right) \right|}{\sigma} \right\}, \\ \leq \sqrt {\frac {1}{n} \sum_ {i} \min  \left\{1 , \frac {1}{2} \frac {\left| x _ {i} ^ {T} (\widehat {w} - w ^ {*}) \right|}{\sigma} \right\} ^ {2}}, \\ \leq \sqrt {\min  \left\{1 , \frac {1}{4 n} \sum_ {i} \frac {\left(x _ {i} ^ {T} \left(\widehat {w} - w ^ {*}\right)\right) ^ {2}}{\sigma^ {2}} \right\}}, \\ = \sqrt {\min \biggl \{1 , \frac {1}{4 n} \frac {1}{\sigma^ {2}} \| X (\widehat {w} - w ^ {*}) \| ^ {2} \biggr \}}. \\ \end{array}
$$
where the second line follows from Jensen's inequality.
$$
\mathbb {E} [ \| X (\widehat {w} - w ^ {*}) \| ^ {2} ] = \sigma^ {2} k.
$$
which implies that with probability  $1 - e^{-\Omega (n)}$  , we have
$$
\left\| X \left(\widehat {w} - w ^ {*}\right) \right\| ^ {2} \leq 2 \sigma^ {2} k.
$$
Substituting in the earlier inequality, we get
$$
\frac {1}{n} \sum_ {i} d _ {T V} \left(p _ {\widehat {w}} \left(y \mid x _ {i}\right), p _ {w ^ {*}} \left(y \mid x _ {i}\right)\right) \leq \sqrt {\min  \left\{1 , \frac {k}{2 n} \right\}} = \sqrt {\frac {k}{2 n}} \text {f o r} n \geq \frac {k}{2}.
$$
Lemma 4.3. Let  $\{x_{i}\}_{i = 1}^{n}$  be i.i.d. random variables such that  $x_{i}\sim \mathcal{D}_{x}$ . For a sufficiently large constant  $C > 0$ , and for  $n = C\frac{k}{\varepsilon^2}\log \frac{1}{\varepsilon}$  with  $n\geq \frac{k}{2}$ , we have:
$$
\operatorname * {P r} _ {x _ {i} \sim \mathcal {D} _ {x}} \left[ \sup  _ {w \in \mathbb {R} ^ {k}} \left| \widetilde {d} (w, w ^ {*}) - d (w, w ^ {*}) \right| > \varepsilon \right] \leq e ^ {- \Omega (n \varepsilon^ {2})}.
$$
Proof. The proof is inspired by Theorem 11.2 in [20], with modifications to our setting.
Let Since  $f_w(x)$  is bounded, for any fixed  $w$ , the Chernoff bound gives
$$
\Pr \left[ \left| \widetilde {d} (w, w ^ {*}) - d (w, w ^ {*}) \right| > \alpha \right] \leq e ^ {- 2 n \alpha^ {2}}. \tag {11}
$$
for any  $\alpha > 0$ . The challenge lies in constructing a "net" to be able to union bound over  $\mathbb{R}^k$  without assuming any bound on  $w$  or the covariate  $x$ . A net is a partitioning of an space, where within each part, points are close together in some way. In this case, we construct a net using what we will refer to as "ghost" samples.
Ghost samples. First, we construct a "ghost" dataset  $D_x'$  consisting of  $n$  new samples, drawn i.i.d.  $\{x_i'\}_{i \in [n]}$  of  $\mathcal{D}_x$ . This gives another metric  $\widetilde{d}'(\cdot, \cdot)$ . Instead of directly considering the distance between  $\widetilde{d}(w, w^*)$  and  $d(w, w^*)$ , it is sufficient to consider the difference between  $\widetilde{d}(w, w^*)$  and  $\widetilde{d}'(w, w^*)$  i.e.,
$$
\Pr \left[ \sup  _ {w} \left| d \left(w, w ^ {*}\right) - \widetilde {d} \left(w, w ^ {*}\right) \right| > \varepsilon \right] \leq 2 \Pr \left[ \sup  _ {w} \left| \widetilde {d} \left(w, w ^ {*}\right) - \widetilde {d} ^ {\prime} \left(w, w ^ {*}\right) \right| > \varepsilon / 2 \right]. \tag {12}
$$
To see this, let  $\bar{w}$  maximize  $\widetilde{d}(w, w^*) - \widetilde{d}'(w, w^*)$ . Since  $\bar{w}$  and  $\{x_i'\}_{i \in [n]}$  are independent, by the Chernoff bound,
$$
\Pr \left[ \left| \widetilde {d} ^ {\prime} (\bar {w}, w ^ {*}) - d (\bar {w}, w ^ {*}) \right| > \varepsilon / 2 | D _ {x} \right] \leq e ^ {- n \varepsilon^ {2} / 2} \leq 1 / 2.
$$
for any  $(D_x,\bar{w})$  and large enough  $n$ . Thus,
$$
\begin{array}{l} \Pr \left[ \left| \widetilde {d} ^ {\prime} (\bar {w}, w ^ {*}) - \widetilde {d} (\bar {w}, w ^ {*}) \right| > \varepsilon / 2 \right] \geq \Pr \left[ \left| d (\bar {w}, w ^ {*}) - \widetilde {d} (\bar {w}, w ^ {*}) \right| > \varepsilon \cap \left| d (\bar {w}, w ^ {*}) - \widetilde {d} ^ {\prime} (\bar {w}, w ^ {*}) \right| <   \varepsilon / 2 \right] \\ = \mathbb {E} _ {D _ {x}} \left[ 1 _ {\{\left| d (\bar {w}, w ^ {*}) - \widetilde {d} (\bar {w}, w ^ {*}) \right| > \varepsilon \}} \Pr \left[ \left| d (\bar {w}, w ^ {*}) - \widetilde {d} ^ {\prime} (\bar {w}, w ^ {*}) \right| <   \varepsilon / 2 | D _ {x} \right] \right] \\ \geq (1 - 1 / 2) \Pr \left[ \left| d (w, w ^ {*}) - \widetilde {d} (w, w ^ {*}) \right| > \varepsilon \right], \\ \end{array}
$$
which implies (12).
Symmetrization. Since  $D_x$  and  $D_x'$  each have  $n$  independent samples, we could instead draw the datasets by first sampling  $2n$  elements  $x_1, \ldots, x_{2n}$  from  $\mathcal{D}_x$ , then randomly partition this sample into two equal datasets. Let  $s_i \in \{\pm 1\}$  so  $s_i = 1$  if  $z_i$  lies in  $D_x'$  and -1 if it lies in  $D_x$ . Then
$$
\widetilde {d} ^ {\prime} (\bar {w}, w ^ {*}) - \widetilde {d} (\bar {w}, w ^ {*}) = \frac {1}{n} \sum_ {i = 1} ^ {2 n} s _ {i} \cdot d _ {T V} (p _ {w} (y | x _ {i}), p _ {w ^ {*}} (y | x _ {i})).
$$
For a fixed  $w$  and  $x_{1},\ldots ,x_{2n}$ , the random variables  $(s_1,\dots ,s_{2n})$  are a permutation distribution, so negatively associated. Then the variables  $s_i\cdot d_{TV}(p_w(y|x_i),p_{w^*}(y|x_i))$  are monotone functions of  $s_i$ , so also negatively associated. They are also bounded in  $[-1,1]$ . Hence we can apply a Chernoff bound:
$$
\Pr \left[ \left| \widetilde {d} ^ {\prime} (\bar {w}, w ^ {*}) - \widetilde {d} (\bar {w}, w ^ {*}) \right| > \varepsilon \right] <   e ^ {- n \varepsilon^ {2} / 2} \tag {13}
$$
for any fixed  $w$ .
Constructing a net. We partition  $\mathbb{R}^k$  the space of  $w$  s.t. if  $w, w'$  are in the same partition then,
$$
\left| d _ {T V} \left(p _ {w} (y | x), p _ {w ^ {*}} (y | x)\right) - d _ {T V} \left(p _ {w ^ {\prime}} (y | x), p _ {w ^ {*}} (y | x)\right) \right| <   \alpha .
$$
for each  $x$  in the dataset  $x_{1},\ldots ,x_{2n}$ . Then take the intersection of all  $2n$  partitions to construct a net over  $\mathbb{R}^k$ .
As the total variation distance is a unimodal function of  $x_{i} \cdot w - x_{i} \cdot w^{*}$ , we partition  $w$  the sets
$$
\{w: d _ {T V} \left(p _ {w} (y | x _ {i}), p _ {w ^ {*}} (y | x _ {i})\right) \in [ j \alpha , (j + 1) \alpha ]
$$
where  $j$  goes from 0 to  $1 / \alpha - 1$ . So the space of  $w$ ,  $\mathbb{R}^k$  is partitioned by  $2n$  sets of  $1 / \alpha$  parallel hyper-planes. Then the total number of cells is at most
$$
\sum_ {i = 0} ^ {k} \binom {2 n} {i} (2 / \alpha) ^ {i} \leq 2 \left(\frac {4 e n}{\alpha k}\right) ^ {k}
$$
We define a net  $N$  by choosing one representative of each cell in the partition, so  $|N| \leq e^{2k\log \frac{n}{\alpha k}}$ . By (13),
$$
\Pr \left[ \max  _ {w \in N} \left| \widetilde {d} ^ {\prime} (\bar {w}, w ^ {*}) - \widetilde {d} (\bar {w}, w ^ {*}) \right| > \varepsilon \right] <   | N | e ^ {- n \varepsilon^ {2} / 2} \leq e ^ {2 k \log \frac {n}{\alpha k} - \varepsilon^ {2} n / 2}.
$$
Finally, for any  $w \in \mathbb{R}^d$  let  $\bar{w} \in N$  be the representative of its cell. By definition of the cells,