SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The situation of ReLU regression, where  $\sigma (\hat{y}) = \max \{\hat{y},0\}$ , is more interesting because  $\sigma$  is not invertible. In order to be able to interpolate, we must have  $y\geq 0$ . If  $y > 0$  with probability 1, then  $\sigma (\hat{y}) = y$  is the same as  $\hat{y} = y$  and we are back to interpolating the square loss  $f(\hat{y},y) = (\hat{y} -y)^2$ . From this observation, we see that  $f(\hat{y},y) = (\sigma (\hat{y}) - y)^2$  cannot be the appropriate loss for consistency even though it's 1 square-root Lipschitz. In contrast, when there is some probability mass at  $y = 0$ , it suffices to output any non-positive value and the minimal norm to interpolate is potentially smaller than requiring  $\hat{y} = 0$ . Similar to the previous section, we can let  $I = \{i\in [n]:y_i > 0\}$  and for any  $(w^{\sharp},b^{\sharp})\in \mathbb{R}^{d + 1}$ , the predictor  $w = w^{\sharp} + w^{\perp}$  satisfies  $\sigma (\langle w,x_i\rangle +b^{\sharp}) = y_{i}$  where
$$
w ^ {\perp} = \quad \arg \min  \quad \| w \| _ {2} \tag {12}
$$
$$
w \in \mathbb {R} ^ {d}:
$$
$$
\forall i \in I, \langle w, x _ {i} \rangle = y _ {i} - \langle w ^ {\sharp}, x _ {i} \rangle - b ^ {\sharp}
$$
$$
\forall i \notin I, \langle w, x _ {i} \rangle = - \sigma (\langle w ^ {\sharp}, x _ {i} \rangle + b ^ {\sharp})
$$
Our analysis will show that the consistent loss for benign overfitting with ReLU regression is
$$
f (\hat {y}, y) = \left\{ \begin{array}{l l} (\hat {y} - y) ^ {2} & \text {i f} \quad y > 0 \\ \sigma (\hat {y}) ^ {2} & \text {i f} \quad y = 0. \end{array} \right.. \tag {13}
$$
We first state the norm bound below.
Theorem 3. Under assumptions (A) and (B), let  $f: \mathbb{R} \times \mathcal{Y} \to \mathbb{R}$  be the loss defined in (13) with  $\mathcal{Y} = \mathbb{R}_{\geq 0}$ . Let  $Q$  be the same as in Theorem 1 and  $\Sigma^{\perp} = Q^{T}\Sigma Q$ . Fix any  $(w^{\sharp}, b^{\sharp}) \in \mathbb{R}^{d + 1}$  such that  $Qw^{\sharp} = 0$  and for some  $\rho \in (0,1)$ , it holds that
$$
\hat {L} _ {f} \left(w ^ {\sharp}, b ^ {\sharp}\right) \leq (1 + \rho) L _ {f} \left(w ^ {\sharp}, b ^ {\sharp}\right). \tag {14}
$$
$$
\min  _ {\substack {(w, b) \in \mathbb {R} ^ {d + 1}: \\ \forall i \in [ n ], \sigma (\langle w, x _ {i} \rangle + b) = y _ {i}}} \| w \| _ {2} \leq \| w ^ {\sharp} \| _ {2} + (1 + \epsilon) \sqrt {\frac {n L _ {f} \left(w ^ {\sharp} , b ^ {\sharp}\right)}{\operatorname{Tr} \left(\Sigma^ {\perp}\right)}}. \tag{15}
$$
Since the loss defined in (13) is also 1-square-root Lipschitz, it can be straightforwardly combined with Theorem 1 to establish benign overfitting in ReLU regression under this loss. The details are exactly identical to the previous section and so we omit it here.
5.3 Benign Overfitting in Matrix Sensing
We now consider the problem of matrix sensing: given random matrices  $A_{1},\ldots ,A_{n}$  (with i.i.d. standard Gaussian entries) and independent linear measurements  $y_{1},\dots,y_{n}$  given by  $y_{i} = \langle A_{i},X^{*}\rangle +$ $\xi_{i}$  where  $\xi_{i}$  is independent of  $A_{i}$ , and  $\mathbb{E}\xi = 0$  and  $\mathbb{E}\xi^2 = \sigma^2$ , we hope to reconstruct the matrix  $X^{*}\in \mathbb{R}^{d_{1}\times d_{2}}$  with sample size  $n\ll d_1d_2$ . To this end, we assume that  $X^{*}$  has rank  $r$ . In this setting, since the measurement matrices have i.i.d. standard Gaussian entries, the test error is closely related to the estimation error:
$$
L (X) = \mathbb {E} \left(\langle A, X \rangle - y\right) ^ {2} = \| X - X ^ {*} \| _ {F} ^ {2} + \sigma^ {2}.
$$
The classical approach to this problem is to find the minimum nuclear norm solution:
$$
\hat {X} = \underset {X \in \mathbb {R} ^ {d _ {1} \times d _ {2}}: \langle A _ {i}, X \rangle = y _ {i}} {\arg \min } \| X \| _ {*}. \tag {16}
$$
Gunasekar et al. (2017) also shows that gradient descent converges to the minimal nuclear norm solution in matrix factorization problems. It is well known that having low nuclear norm can ensure generalization (Foygel and Srebro 2011; Srebro and Shraibman 2005) and minimizing the nuclear norm ensures reconstruction (Candès and Recht 2009; Recht et al. 2010). However, if the noise level  $\sigma$  is high, then even the minimal nuclear norm solution  $\hat{X}$  can have large nuclear norm. Since our result can be adapted to different norms as regularizer, our uniform convergence guarantee can be directly applied. The dual norm of the nuclear norm is the spectral norm, and it is well-known that the spectrum norm of a Gaussian random matrix is approximately  $\sqrt{d_1} +\sqrt{d_2}$  (Vershynin 2018). It remains to analyze the minimal nuclear norm required to interpolate. For simplicity, we assume that  $\xi$  are Gaussian below, but we can extend it to be sub-Gaussian.
Theorem 4. Suppose that  $d_{1}d_{2} > n$ , then there exists some  $\epsilon \lesssim \sqrt{\frac{\log(32 / \delta)}{n}} +\frac{n}{d_1d_2}$  such that with probability at least  $1 - \delta$ , it holds that
$$
\min  _ {\forall i \in [ n ], \langle A _ {i}, X \rangle = y _ {i}} \| X \| _ {*} \leq \sqrt {r} \| X ^ {*} \| _ {F} + (1 + \epsilon) \sqrt {\frac {n \sigma^ {2}}{d _ {1} \vee d _ {2}}}. \tag {17}
$$
Without loss of generality, we will assume  $d_{1} \leq d_{2}$  from now on because otherwise we can take the transpose of  $A$  and  $X$ . Similar to assuming  $n / R(\Sigma^{\perp}) \to 0$  in linear regression, we implicitly assume that  $n / d_{1}d_{2} \to 0$  in matrix sensing. Such scaling is necessary for benign overfitting because of the lower bound on the test error for any interpolant (e.g., Proposition 4.3 of Zhou et al. 2020). Finally, we apply the uniform convergence guarantee.
Theorem 5. Fix any  $\delta \in (0,1)$ . There exist constants  $c_{1}, c_{2}, c_{3} > 0$  such that if  $d_{1}d_{2} > c_{1}n$ ,  $d_{2} > c_{2}d_{1}$ ,  $n > c_{3}r(d_{1} + d_{2})$ , then with probability at least  $1 - \delta$  that
$$
\frac {\left\| \hat {X} - X ^ {*} \right\| _ {F} ^ {2}}{\left\| X ^ {*} \right\| _ {F} ^ {2}} \lesssim \frac {r \left(d _ {1} + d _ {2}\right)}{n} + \sqrt {\frac {r \left(d _ {1} + d _ {2}\right)}{n}} \frac {\sigma}{\left\| X ^ {*} \right\| _ {F}} + \left(\sqrt {\frac {d _ {1}}{d _ {2}}} + \frac {n}{d _ {1} d _ {2}}\right) \frac {\sigma^ {2}}{\left\| X ^ {*} \right\| _ {F} ^ {2}}. \tag {18}
$$
From Theorem 5, we see that when the signal to noise ratio  $\frac{\|X^{*}\|_{F}^{2}}{\sigma^{2}}$  is bounded away from zero, then we obtain consistency  $\frac{\|\hat{X} - X^{*}\|_{F}^{2}}{\|X^{*}\|_{F}^{2}}\to 0$  if (i)  $r(d_1 + d_2) = o(n)$ , (ii)  $d_{1}d_{2} = \omega (n)$ , and (iii)  $d_{1} / d_{2}\rightarrow \{0,\infty \}$ . This can happen for example when  $r = \Theta (1)$ ,  $d_{1} = \Theta (n^{1 / 2})$ ,  $d_{2} = \Theta (n^{2 / 3})$ . As discussed earlier, the second condition is necessary for benign overfitting, and the first consistency condition should be necessary even for regularized estimators. It is possible that the final condition is not necessary and we leave a tighter understanding of matrix sensing as future work.
6 Single-Index Neural Networks
We show that our results extend a even more general setting than (1). Suppose that we have a parameter space  $\Theta \subseteq \mathbb{R}^p$  and a continuous mapping  $w$  from  $\theta \in \Theta$  to a linear predictor  $w(\theta) \in \mathbb{R}^d$ . Given a function  $f: \mathbb{R} \times \mathcal{V} \times \Theta \to \mathbb{R}$  and i.i.d. sample pairs  $(x_i, y_i)$  drawn from distribution  $\mathcal{D}$ , we consider training and test error of the form:
$$
\hat {L} (\theta) = \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(\langle w (\theta), x _ {i} \rangle , y _ {i}, \theta\right) \quad \text {a n d} \quad L (\theta) = \mathbb {E} [ f (\langle w (\theta), x \rangle , y, \theta) ]. \tag {19}
$$
We make the same assumptions (A) and (B) on  $\mathcal{D}$ . Assumptions (C) and (D) can be naturally extended to the following:
(E) there exists a universal constant  $\tau > 0$  such that uniformly over all  $\theta \in \Theta$ , it holds that
$$
\frac {\mathbb {E} [ f (\langle w (\theta) , x \rangle , y , \theta) ^ {4} ] ^ {1 / 4}}{\mathbb {E} [ f (\langle w (\theta) , x \rangle , y , \theta) ]} \leq \tau . \tag {20}
$$
(F) bounded VC dimensions: the class of functions on  $\mathbb{R}^{k + 1}\times \mathcal{V}$  defined by
$$
\{(x, y) \rightarrow \mathbb {1} \{f (\langle w, x \rangle , y, \theta) > t \}: (w, t, \theta) \in \mathbb {R} ^ {k + 1} \times \mathbb {R} \times \Theta \} \tag {21}
$$
has VC dimension at most  $h$ .
Now we can state the extension of Theorem 1.
Theorem 6. Suppose that assumptions (A), (B), (E) and (F) hold. For any  $\delta \in (0,1)$ , let  $C_{\delta}:\mathbb{R}^d\to [0,\infty ]$  be a continuous function such that with probability at least  $1 - \delta /4$  over  $x\sim \mathcal{N}(0,\Sigma)$  uniformly over all  $\theta \in \Theta$ ,
$$
\left\langle w (\theta), Q ^ {T} x \right\rangle \leq C _ {\delta} (w (\theta)). \tag {22}
$$
If for each  $\theta \in \Theta$  and  $y \in \mathcal{Y}$ ,  $f$  is non-negative and  $\sqrt{f}$  is  $\sqrt{H_{\theta}}$ -Lipschitz with respect to the first argument, and  $H_{\theta}$  is continuous in  $\theta$ , then with probability at least  $1 - \delta$ , it holds that uniformly over all  $\theta \in \Theta$ , we have
$$
(1 - \epsilon) L (\theta) \leq \left(\sqrt {\hat {L} (\theta)} + \sqrt {\frac {H _ {\theta} C _ {\delta} (w (\theta)) ^ {2}}{n}}\right) ^ {2}. \tag {23}
$$
Next, we show that the generality of our result allows us to establish uniform convergence bound for two-layer neural networks with weight sharing. In particular, we let  $\sigma(x) = \max(x, 0)$  be the ReLU activation function and  $\theta = (w, a, b) \in \mathbb{R}^{d + 2N}$ , where  $N$  is the number of hidden units. Consider the loss function
$$
f (\hat {y}, y, \theta) = \left(\sum_ {i = 1} ^ {N} a _ {i} \sigma (\hat {y} - b _ {i}) - y\right) ^ {2} \quad \text {o r} \quad f (\hat {y}, y, \theta) = \left(1 - \sum_ {i = 1} ^ {N} a _ {i} \sigma (\hat {y} - b _ {i}) y\right) _ {+} ^ {2}, \tag {24}
$$
then  $L(\theta)$  is the test error of a neural network of the form  $h_{\theta}(x) \coloneqq \sum_{i=1}^{N} a_i \sigma(\langle w, x \rangle - b_i)$ . Since our uniform convergence guarantee holds uniformly over all  $\theta$ , it applies to networks whose first and second layer weights are optimized at the same time. Without loss of generality, we can assume that  $b_1 \leq \ldots \leq b_N$  are sorted, then it is easy to see that  $\sqrt{f}$  is  $\max_{j \in [N]} \left| \sum_{i=1}^{j} a_i \right|$  Lipschitz. Applying Theorem 6, we obtain the following corollary.
Corollary 2. Fix an arbitrary norm  $\| \cdot \|$  and consider  $f$  as defined in (24). Assume that the data distribution  $\mathcal{D}$  satisfy (A), (B), and (E). Then with probability at least  $1 - \delta$ , it holds that uniformly over all  $\theta = (w, a, b) \in \mathbb{R}^{d + 2N}$ , we have
$$
(1 - \epsilon) L (\theta) \leq \left(\sqrt {\hat {L} (\theta)} + \frac {\max  _ {j \in [ N ]} \left| \sum_ {i = 1} ^ {j} a _ {i} \right| \| w \| \left(\mathbb {E} \| Q ^ {T} x \| _ {*} + \epsilon^ {\prime}\right)}{\sqrt {n}}\right) ^ {2} \tag {25}
$$
where  $\epsilon$  is the same as in Theorem 6 with  $h = \tilde{O}(k + N)$  and  $\epsilon' = O\left(\sup_{\|u\| \leq 1} \|u\|_{\Sigma^{\perp}} \sqrt{\log(1/\delta)}\right)$ .
The above theorem says given a network parameter  $\theta$ , after sorting the  $b_{i}$ 's, a good complexity measure to look at is  $\max_{j\in [N]}\left|\sum_{i = 1}^{j}a_{i}\right|\cdot \| w\|$  and equation (25) precisely quantify how the complexity of a network controls generalization.
7 On Gaussian Universality
In this section, we discuss a counterexample to Gaussian universality motivated by Shamir (2022). In particular, we consider a data distribution  $\mathcal{D}$  over  $(x,y)$  given by:
(G)  $x = (x_{|k},x_{|d - k})$  where  $x_{|k}\sim \mathcal{N}(0,\Sigma_{|k})$  and there exists a function  $h:\mathbb{R}^k\to \mathbb{R}$  such that
$$
x _ {| d - k} = h \left(x _ {| k}\right) \cdot z \quad \text {w i t h} \quad z \sim \mathcal {N} \left(0, \Sigma_ {| d - k}\right) \quad \text {i n d e p e n d e n t o f} x _ {| k} \tag {26}
$$
(H) there exists a function  $g:\mathbb{R}^{k + 1}\to \mathbb{R}$  such that
$$
y = g \left(x _ {| k}, \xi\right) \tag {27}
$$
where  $\xi \sim \mathcal{D}_{\xi}$  is independent of  $x$  (but not necessarily Gaussian).
By the independence between  $z$  and  $x_{|k}$ , we can easily see that  $x_{|k}$  and  $x_{|d - k}$  are uncorrelated. Moreover, the covariance matrix of  $x_{|d - k}$  is
$$
\mathbb {E} [ x _ {| d - k} x _ {| d - k} ^ {T} ] = \mathbb {E} [ h (x _ {| k}) ^ {2} ] \cdot \Sigma_ {| d - k}
$$
which is just a re-scaling of  $\Sigma_{|d - k}$  and therefore has the same effective rank as  $R(\Sigma_{|d - k})$ . Even though the feature  $x$  in  $\mathcal{D}$  is non-Gaussian, its tail  $x_{|d - k}$  is Gaussian conditioned on  $x_{|k}$ . Therefore, our proof technique is still applicable after a conditioning step. However, it turns out that the uniform convergence bound in Theorem 1 is no longer valid. Instead, we have to re-weight the loss function. The precise theorem statements can be found in the appendix. We briefly describe our theoretical results below, which follow the same ideas as in section 4 and 5.
Uniform Convergence. Under a similar low-dimensional concentration assumption such as (C), if we let  $C_{\delta}$  be such that  $\langle w, z \rangle \leq C_{\delta}(w)$  for all  $w \in \mathbb{R}^{d - k}$  with high probability, it holds that
$$
\mathbb {E} \left[ \frac {f (\langle w , x \rangle , y)}{h (x _ {| k}) ^ {2}} \right] \leq (1 + o (1)) \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \frac {f (\langle w , x _ {i} \rangle , y _ {i})}{h (x _ {i | k}) ^ {2}} + \frac {C _ {\delta} (w _ {| d - k})}{\sqrt {n}}\right) ^ {2}. \tag {28}
$$
Note that the distribution of  $z$  is specified in (G) and different to the distribution of  $x_{|d - k}$ .
Norm Bound. Next, we focus on linear regression and compute the minimal norm required to interpolate. If we pick  $\Sigma_{|d - k}$  to satisfy the benign overfitting conditions, for any  $w_{|k}^{*} \in \mathbb{R}^{k}$ , we have
$$
\min  _ {\substack {w \in \mathbb {R} ^ {d}: \\ \forall i \in [ n ], \langle w, x _ {i} \rangle = y _ {i}}} \| w \| _ {2} ^ {2} \leq \| w _ {| k} ^ {*} \| _ {2} ^ {2} + (1 + o (1)) \frac {n \mathbb {E} \left[ \left(\frac {y - \langle w _ {| k} ^ {*} , x _ {| k} \rangle}{h (x _ {| k})}\right) ^ {2} \right]}{\operatorname{Tr} (\Sigma_ {| d - k})} \tag{29}
$$
It is easy to see that the  $w^{*}$  that minimizes the population weighted square loss satisfy  $w_{|d - k}^{*} = 0$ , and so we can let  $w_{|k}^{*}$  to be the minimizer.
Benign Overfitting. Let  $\hat{w} = \arg \min_{w\in \mathbb{R}^d:Xw = Y}\| w\| _2$  be the minimal norm interpolant. Plugging in the norm bound (29) into the uniform convergence guarantee (28), we show that
$$
\mathbb {E} \left[ \frac {(\langle \hat {w} , x \rangle - y) ^ {2}}{h (x _ {| k}) ^ {2}} \right] \leq (1 + o (1)) \frac {\| \hat {w} \| _ {2} ^ {2} \operatorname {T r} (\Sigma_ {| d - k})}{n} \rightarrow \mathbb {E} \left[ \frac {(\langle w ^ {*} , x \rangle - y) ^ {2}}{h (x _ {| k}) ^ {2}} \right]. \tag {30}
$$
which recovers the consistency result of Shamir (2022).
**Contradiction.** Suppose that Gaussian universality holds, then our optimistic rate theory would predict that
$$
\mathbb {E} \left[ \left(\langle \hat {w}, x \rangle - y\right) ^ {2} \right] \leq (1 + o (1)) \frac {\| \hat {w} \| _ {2} ^ {2} \cdot \mathbb {E} \left[ h \left(x _ {| k}\right) ^ {2} \right] \operatorname {T r} \left(\Sigma_ {| d - k}\right)}{n}. \tag {31}
$$
Combining (30) with (31), we obtain that
$$
\min  _ {w} \mathbb {E} [ (\langle w, x \rangle - y) ^ {2} ] \leq (1 + o (1)) \min  _ {w} \mathbb {E} [ h (x _ {| k}) ^ {2} ] \cdot \mathbb {E} \left[ \left(\frac {y - \langle w , x \rangle}{h (x _ {| k})}\right) ^ {2} \right] \tag {32}
$$
which cannot always hold. For example, let's consider the case where  $k = 1$ ,  $x_{1} \sim \mathcal{N}(0,1)$ ,  $h(x_{1}) = 1 + |x_{1}|$  and  $y = h(x_{1})^{2}$ . Then it is straightforward to verify that the left hand side of (32) equals  $\mathbb{E}[h(x_1)^4]$  and the right hand side equals  $\mathbb{E}[h(x_1)^2]^2$ , but this is impossible because
$$
\mathbb {E} \left[ h \left(x _ {1}\right) ^ {4} \right] - \mathbb {E} \left[ h \left(x _ {1}\right) ^ {2} \right] ^ {2} = \operatorname {V a r} \left(h \left(x _ {1}\right) ^ {2}\right) > 0.
$$
In the counterexample above, we see that it is possible to introduce strong dependence between the signal and the tail component of  $x$  while ensuring that they are uncorrelated. The dependence will prevent the norm of the tail from concentrating around its mean, no matter how large the sample size is. In contrast, the norm of the tail will concentrate for Gaussian features with a matching covariance — such discrepancy results in an over-optimistic bound for non-Gaussian data.
In this paper, we extend a type of sharp uniform convergence guarantee proven for the square loss in Zhou et al. (2021) to any square-root Lipschitz loss. Uniform convergence with square-root Lipschitz loss is an important tool because the appropriate loss to study interpolation learning is usually square-root Lipschitz instead of Lipschitz. Compared to the prior work of Zhou et al. 2022, our view significantly simplifies the assumptions to establish optimistic rate. Since we don't need to explicitly compute the Moreau envelope for each application, our framework easily leads to many novel benign overfitting results, including low-rank matrix sensing.
In the applications to phase retrieval and ReLU, we identify the appropriate loss function  $f$  and our norm calculation overcomes the challenge that  $f$  is non-convex and so CGMT cannot be directly applied. Furthermore, we explore new extensions of the uniform convergence technique to study single-index neural networks and suggest a promising research direction to understand the generalization of neural networks. Finally, we argue that Gaussian universality cannot always be taken for granted by analyzing a model where only the weighted square loss enjoys an optimistic rate. Our results highlight the importance of tail concentration and shed new lights on the necessary conditions for universality. An important future direction is to extend optimistic rate beyond Gaussian data, possibly through worst-case Rademacher complexity. Understanding the performance of more practical algorithms in phase retrieval and deriving the necessary and sufficient conditions for benign overfitting in matrix sensing are also interesting problems.
Acknowledgements. F.K. was supported in part by NSF award CCF-1704417, NSF award IIS-1908774, and N. Anari's Sloan Research Fellowship.
Andoni, Alexandr, Daniel Hsu, Kevin Shi, and Xiaorui Sun (2017). "Correspondence retrieval." Conference on Learning Theory. PMLR, pp. 105-126.
Anthony, Martin, Peter L. Bartlett, et al. (1999). Neural network learning: Theoretical foundations. Vol. 9. Cambridge university press Cambridge.
Auer, Peter, Mark Herbster, and Manfred KK Warmuth (1995). "Exponentially many local minima for single neurons." Advances in neural information processing systems 8.
Bandeira, Afonso S. (2016). "Ten Lectures and Forty-Two Open Problems in the Mathematics of Data Science." Lecture notes, MIT. URL: https://people.math.ethz.ch/~abandeira/TenLecturesFortyTwoProblems.pdf.
Barbier, Jean, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborova (2019). "Optimal errors and phase transitions in high-dimensional generalized linear models." Proceedings of the National Academy of Sciences 116.12, pp. 5451-5460.
Bartlett, Peter L. (1996). "For valid generalization the size of the weights is more important than the size of the network." Advances in neural information processing systems 9.
Bartlett, Peter L., Nick Harvey, Christopher Liaw, and Abbas Mehrabian (2019). "Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks." The Journal of Machine Learning Research 20.1, pp. 2285-2301.
Bartlett, Peter L., Philip M. Long, Gábor Lugosi, and Alexander Tsigler (2020). "Benign overfitting in linear regression." Proceedings of the National Academy of Sciences 117.48, pp. 30063-30070.
Bauschke, Heinz H., Patrick L Combettes, et al. (2011). Convex analysis and monotone operator theory in Hilbert spaces. Vol. 408. Springer.
Zhou, Lijia, Danica J. Sutherland, and Nathan Srebro (2020). "On Uniform Convergence and Low-Norm Interpolation Learning." Advances in Neural Information Processing Systems. arXiv: 2006. 05942.
A Organization of the Appendices
In the Appendix, we give proofs of all results from the main text. In Appendix B, we study properties of square-root-Lipschitz functions and introduce some technical tools that we use throughout the appendix. In Appendix C, we prove our main uniform convergence guarantee (Theorem 1 and the more general version Theorem 6). In Appendix D, we obtain bounds on the minimal norm required to interpolate in the settings studied in section 5. In Appendix E, we provide details on the counterexample to Gaussian universality described in section 7.
B.1 Properties of Square-root Lipschitz Loss
In this section, we prove that square-root Lipschitzness can be equivalently characterized by a relationship between a function and its Moreau envelope, which can be used to establish uniform convergence results based on the recent work of Zhou et al. 2022. We formally define Lipschitz functions and Moreau envelope below.
Definition 1. A function  $f: \mathbb{R} \to \mathbb{R}$  is  $M$ -Lipschitz if for all  $x, y$  in  $\mathbb{R}$ ,
$$
| f (x) - f (y) | \leq M | x - y |. \tag {33}
$$
Definition 2. The Moreau envelope of a function  $f: \mathbb{R} \to \mathbb{R}$  associated with smoothing parameter  $\lambda \in \mathbb{R}_{+}$  is defined as
$$
f _ {\lambda} (x) := \inf  _ {y \in \mathbb {R}} f (y) + \lambda (y - x) ^ {2}. \tag {34}
$$
Though we define Lipschitz functions and Moreau envelope for univariate functions from  $\mathbb{R}$  to  $\mathbb{R}$  above, we can easily extend definitions 1 and 2 to loss functions  $f:\mathbb{R}\times \mathcal{V}\to \mathbb{R}$  or  $f:\mathbb{R}\times \mathcal{V}\times \Theta \rightarrow \mathbb{R}$ . We say a function  $f:\mathbb{R}\times \mathcal{V}\rightarrow \mathbb{R}$  is  $M$ -Lipschitz if for any  $y\in \mathcal{V}$  and  $\hat{y}_1,\hat{y}_2\in \mathbb{R}$ , we have
$$
\left| f \left(\hat {y} _ {1}, y\right) - f \left(\hat {y} _ {2}, y\right) \right| \leq M \left| \hat {y} _ {1} - \hat {y} _ {2} \right|.
$$
Similarly, we say a function  $f: \mathbb{R} \times \mathcal{Y} \times \Theta \to \mathbb{R}$  is  $M$ -Lipschitz if for any  $y \in \mathcal{Y}, \theta \in \Theta$  and  $\hat{y}_1, \hat{y}_2 \in \mathbb{R}$ , we have
$$
\left| f \left(\hat {y} _ {1}, y, \theta\right) - f \left(\hat {y} _ {2}, y, \theta\right) \right| \leq M \left| \hat {y} _ {1} - \hat {y} _ {2} \right|.
$$
We can also define the Moreau envelope of a function  $f: \mathbb{R} \times \mathcal{V} \to \mathbb{R}$  by
$$
f _ {\lambda} (\hat {y}, y) := \inf  _ {u \in \mathbb {R}} f (u, y) + \lambda (u - \hat {y}) ^ {2},
$$
and the Moreau envelope of a function  $f: \mathbb{R} \times \mathcal{Y} \times \Theta \to \mathbb{R}$  is defined as
$$
f _ {\lambda} (\hat {y}, y, \theta) := \inf  _ {u \in \mathbb {R}} f (u, y, \theta) + \lambda (u - \hat {y}) ^ {2}.
$$
The proof of all results in this section can be straightforwardly extended to these settings. For simplicity, we ignore the additional arguments in  $\mathcal{V}$  and  $\Theta$  in this section.
The Moreau envelope is usually viewed as a smooth approximation to the original function  $f$ ; its minimizer is known as the proximal operator. It plays an important role in convex analysis (see e.g. Boyd et al. 2004; Bauschke, Combettes, et al. 2011; Rockafellar 1970), but is also useful and well-defined when  $f$  is nonconvex. The canonical example of a  $\sqrt{H}$ -square-root-Lipschitz function is  $f(x) = Hx^{2}$ , for which we can easily check
$$
f _ {\lambda} (x) = \frac {\lambda}{\lambda + H} f (x).
$$