SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lijia Zhou
zlj@uchicago.edu
Zhen Dai
zhen9@uchicago.edu
Frederic Koehler
fkoehler@stanford.edu
Toyota Technological Institute at Chicago
Collaboration on the Theoretical Foundations of Deep Learning (deepfoundations.ai)
We establish generic uniform convergence guarantees for Gaussian data in terms of the Rademacher complexity of the hypothesis class and the Lipschitz constant of the square root of the scalar loss function. We show how these guarantees substantially generalize previous results based on smoothness (Lipschitz constant of the derivative), and allow us to handle the broader class of square-root-Lipschitz losses, which includes also non-smooth loss functions appropriate for studying phase retrieval and ReLU regression, as well as rederive and better understand "optimistic rate" and interpolation learning guarantees.
The phenomenon of "interpolation learning", where a learning algorithm perfectly interpolates noisy training labels while still generalizing well to unseen data, has attracted significant interest in recent years. As Shamir (2022) pointed out, one of the difficulties in understanding interpolation learning is that we need to determine which loss function we should use to analyze the test error. In linear regression, interpolating the square loss is equivalent to interpolating many other losses (such as the absolute loss) on the training set. Similarly, in the context of linear classification, many works (Soudry et al. 2018; Ji and Telgarsky 2019; Muthukumar et al. 2021) have shown that optimizing the logistic, exponential, or even square loss would all lead to the maximum margin solution, which interpolates the hinge loss. Even though there are many possible loss functions to choose from to analyze the population error, consistency is often possible with respect to only one loss function because different loss functions generally have different population error minimizers.
In linear regression, it has been shown that minimal norm interpolant can be consistent with respect to the square loss (Bartlett et al. 2020; Muthukumar et al. 2020; Negrea et al. 2020; Koehler et al. 2021), while the appropriate loss for benign overfitting in linear classification is the squared hinge loss (Shamir 2022; Zhou et al. 2022). In this line of work, uniform convergence has emerged as a general and useful technique to understand interpolation learning (Koehler et al. 2021; Zhou et al. 2022). Though the Lipschitz contraction technique has been very useful to establish uniform convergence in classical learning theory, the appropriate loss functions to study interpolation learning (such as the square loss and the squared hinge loss) are usually not Lipschitz. In fact, many papers (Nagarajan and Kolter 2019; Zhou et al. 2020; Negrea et al. 2020) have shown that the traditional notion of uniform convergence implied by Lipschitz contraction fails in the setting of interpolation learning. Instead, we need to find other properties of the loss function to establish a different type of uniform convergence.
Zhou et al. (2022) recently showed that the Moreau envelope of the square loss or the squared hinge loss is proportional to itself and this property is sufficient to establish a type of uniform convergence guarantee known as "optimistic rate" (Panchenko 2002; Srebro et al. 2010). Roughly speaking, if this condition holds, then the difference between the square roots of the population error and the training error can be bounded by the Rademacher complexity of the hypothesis class. Specializing to predictors with zero training error, the test error can then be controlled by the square of the Rademacher complexity, which exactly matches the Bayes error for the minimal norm interpolant when it is consistent (Koehler et al. 2021). However, the Moreau envelope of a loss function is not always easy to solve in closed-form, and it is unclear whether this argument can be applied to problems beyond linear regression and max-margin classification. Moreover, considering the difference of square roots seem like a mysterious choice and does not have a good intuitive explanation.
In this paper, by showing that optimistic rate holds for any square-root Lipschitz loss, we argue that the class of square-root Lipschitz losses is the more natural class of loss functions to consider. In fact, any loss whose Moreau envelope is proportional to itself is square-root Lipschitz. Our result also provides an intuitive explanation for optimistic rate: when the loss is Lipschitz, the difference between the test and training error can be bounded by Rademacher complexity; therefore, when the loss is square-root Lipschitz, the difference between the square roots of the test and training error can be bounded by Rademacher complexity. In addition to avoiding any hidden constant and logarithmic factor, our uniform convergence guarantee substantially generalize previous results based on smoothness (Lipschitz constant of the derivative, such as Srebro et al. 2010). The generality of our results allows us to handle non-differentiable losses. In the problems of phase retrieval and ReLU regression, we identify the consistent loss through a norm calculation, and we apply our theory to prove novel consistency results. In the context of noisy matrix sensing, we also establish benign overfitting for the minimal nuclear norm solution. Finally, we show that our results extend to fully optimized neural networks with weight sharing in the first layer.
Optimistic rates. In the context of generalization theory, our results are related to a phenomena known as "optimistic rates" — generalization bounds which give strengthened guarantees for predictors with smaller training error. Such bounds can be contrasted with "pessimistic" bounds which do not attempt to adapt to the training error of the predictor. See for example Vapnik 1982; Panchenko 2003; Panchenko 2002. The work of Srebro et al. 2010 showed that optimistic rates control the generalization error of function classes learned with smooth losses, and recently the works Zhou et al. 2021; Koehler et al. 2021; Zhou et al. 2020 showed that a much sharper version of optimism can naturally explain the phenomena of benign overfitting in Gaussian linear models. (These works are in turn connected with a celebrated line of work in proportional asymptotics for M-estimation such as Stojnic 2013, see references within.) In the present work, we show that the natural setting for optimistic rates is actually square-root-Lipschitz losses and this allows us to capture new applications such as phase retrieval where the loss is not smooth.
Phase retrieval, ReLU Regression, Matrix sensing. Recent works (Maillard et al. 2020; Barbier et al. 2019; Mondelli and Montanari 2018; Luo et al. 2019) analyzed the statistical limits of phase retrieval in a high-dimensional limit for certain types of sensing designs (e.g. i.i.d. Real or Complex Gaussian), as well as the performance of Approximate Message Passing in this problem (which is used to predict a computational-statistical gap). In the noiseless case, the works Andoni et al. 2017; Song et al. 2021 gave better computationally efficient estimators for phase retrieval based upon the LLL algorithm. Our generalization bound can naturally be applied to any estimator for phase retrieval, including these methods or other ones, such as those based on non-convex optimization (Sun et al. 2018; Netrapalli et al. 2013). For the same reason, they can also naturally be applied in the context of sparse phase retrieval (see e.g. Li and Voroninski 2013; Candes et al. 2015).
Similarly, there have been many works studying the computational tractability of ReLU regression. See for example the work of Auer et al. 1995 where the "matching loss" technique was developed for learning well-specified GLMs including ReLUs. Under misspecification, learning ReLUs can be hard and approximate results have been established, see e.g. Goel et al. 2017; Diakonikolas et al. 2020. In particular, learning a ReLU agnostically in squared loss, even over Gaussian marginals, is known to be hard under standard average-case complexity assumptions (Goel et al. 2019). Again, our
generalization bound has the merit that it can be applied to any estimator. For matrix sensing, we consider nuclear norm minimization (as in e.g. Recht et al. 2010) which is a convex program, so it can always be computed in polynomial time.
Weight-tied neural networks. The work of Bietti et al. 2022 studies a similar two-layer neural network model with weight tying. Their focus is primarily on understanding the non-convex optimization of this model via gradient-based methods. Again, our generalization bound can be straightforwardly combined with the output of their algorithm. The use of norm-based bounds as in our result is common in the generalization theory for neural networks, see e.g. Bartlett (1996) and Anthony, Bartlett, et al. (1999). Compared to existing generalization bounds, the key advantage of our bound is its quantitative sharpness (e.g. small constants and no extra logarithmic factors).
In most of this paper, we consider the setting of generalized linear models. Given any loss function  $f: \mathbb{R} \times \mathcal{Y} \to \mathbb{R}_{\geq 0}$  and independent sample pairs  $(x_i, y_i)$  from some data distribution  $\mathcal{D}$  over  $\mathbb{R}^d \times \mathcal{Y}$ , we can learn a linear model  $(\hat{w}, \hat{b})$  by minimizing the empirical loss  $\hat{L}_f$  with the goal of achieving small population loss  $L_f$ :
$$
\hat {L} _ {f} (w, b) = \frac {1}{n} \sum_ {i = 1} ^ {n} f (\langle w, x _ {i} \rangle + b, y _ {i}), \quad L _ {f} (w, b) = \mathbb {E} _ {(x, y) \sim \mathcal {D}} [ f (\langle w, x \rangle + b, y) ]. \tag {1}
$$
In the above,  $\mathcal{V}$  is an abstract label space. For example,  $\mathcal{V} = \mathbb{R}$  for linear regression and  $\mathcal{V} = \mathbb{R}_{\geq 0}$  for phase retrieval (section 5.1) and ReLU regression (section 5.2). If we view  $x_{i}$  as vectorization of the sensing matrices, then our setting (1) also includes the problem of matrix sensing (section 5.3).
For technical reasons, we assume that the distribution  $\mathcal{D}$  follows a Gaussian multi-index model (Zhou et al. 2022).
(A)  $d$ -dimensional Gaussian features with arbitrary mean and covariance:  $x \sim \mathcal{N}(\mu, \Sigma)$
(B) a generic multi-index model: there exist a low-dimensional projection  $W = [w_1^*, \dots, w_k^*] \in \mathbb{R}^{d \times k}$ , a random variable  $\xi \sim \mathcal{D}_{\xi}$  independent of  $x$  (not necessarily Gaussian), and an unknown link function  $g: \mathbb{R}^{k+1} \to \mathcal{V}$  such that
$$
\eta_ {i} = \left\langle w _ {i} ^ {*}, x \right\rangle , \quad y = g \left(\eta_ {1}, \dots , \eta_ {k}, \xi\right). \tag {2}
$$
We require  $x$  to be Gaussian because the proof of our generalization bound depends on the Gaussian Minimax Theorem (Gordon 1985; Thrampoulidis et al. 2014), as in many other works (Koehler et al. 2021; Zhou et al. 2022; Wang et al. 2021; Donhauser et al. 2022). In settings where the features are not Gaussian, many works (Goldt et al. 2020; Mei and Montanari 2022; Misiakiewicz 2022; Hu and Lu 2023; Han and Shen 2022) have established universality results showing that the test and training error are asymptotically equivalent to the error of a Gaussian model with matching covariance matrix. However, we show in section 7 that Gaussian universality is not always valid and we discuss potential extensions of the Gaussian feature assumption there.
The multi-index assumption is a generalization of a well-specified linear model  $y = \langle w^{*},x\rangle +\xi$  which corresponds to  $k = 1$  and  $g(\eta ,\xi) = \eta +\xi$  in (2). Since  $g$  is not even required to be continuous, the assumption on  $y$  is quite general. It allows nonlinear trend and heteroskedasticity, which pose significant challenges for the analyses using random matrix theory (but not for uniform convergence). In Zhou et al. (2022), the range of  $g$  is taken to be  $\mathcal{V} = \{-1,1\}$  in order to study binary classification. In section 5.2, we allow the conditional distribution of  $y$  to have a point mass at zero, which is crucial for distinguishing ReLU regression from standard linear regression.
4 Optimistic Rate
In order to establish uniform convergence, we need two additional technical assumptions because we have not made any assumption on  $f$  and  $g$ .
(C) hypercontractivity: there exists a universal constant  $\tau > 0$  such that uniformly over all  $(w, b) \in \mathbb{R}^d \times \mathbb{R}$ , it holds that
$$
\frac {\mathbb {E} [ f (\langle w , x \rangle + b , y) ^ {4} ] ^ {1 / 4}}{\mathbb {E} [ f (\langle w , x \rangle + b , y) ]} \leq \tau . \tag {3}
$$
(D) the class of functions on  $\mathbb{R}^{k + 1}\times \mathcal{V}$  defined below has VC dimension at most  $h$  ..
$$
\{(x, y) \rightarrow \mathbb {1} \{f (\langle w, x \rangle , y) > t \}: (w, t) \in \mathbb {R} ^ {k + 1} \times \mathbb {R} \}. \tag {4}
$$
Crucially, the quantities  $\tau$  and  $h$  only depend on the number of indices  $k$  in assumption (B) instead of the feature dimension  $d$ . This is a key distinction because  $k$  can be a small constant even when the feature dimension is very large. For example, recall that  $k = 1$  in a well-specified model and it is completely free of  $d$ . The feature dimension plays no role in equation (3) because conditioned on  $W^{T}x\in \mathbb{R}^{k}$  and  $\xi \in \mathbb{R}$ , the response  $y$  is non-random and the distribution of  $\langle w,x\rangle$  for any  $w\in \mathbb{R}^d$  only depends on  $w^{T}\mu$  and  $w^{T}\Sigma w$  by properties of the Gaussian distribution.
The hypercontractivity (or "norm equivalence") assumption (3) is one of a few different common assumptions made in the statistical learning theory literature to rule out the possibility of a very heavy-tailed loss function. The reason is that if the loss function has a very tiny chance of being extremely large, it is not possible to state any meaningful learning guarantee from a finite number of samples. Hypercontractivity was used as an assumption already in Vapnik 1982. Another common and incomparable assumption made in the literature is boundedness, but this is not suitable for us because, e.g., the squared loss under the Gaussian distribution will not be bounded almost surely. On the other hand, if  $f$  is the squared loss then (3) holds with  $\tau = \sqrt[4]{105}$  by standard properties of the Gaussian distribution. We can also state other versions of our main theorem by using other results in statistical learning theory to handle the low-dimensional concentration; in our proof we simply use a result from Vapnik 1982 in a contained and black-box fashion. (See e.g. Mendelson 2017; Vapnik 1982 for more discussion of the different possible assumptions.)
Similar assumptions are made in Zhou et al. (2022) and more background on these assumptions can be found there, but we note that our assumption (C) and (D) are weaker than theirs because we do not require (3) and (4) to hold uniformly over all Moreau envelopes of  $f$ .
We are now ready to state the uniform convergence results for square-root Lipschitz losses.
Theorem 1. Assume that  $(A)$ ,  $(B)$ ,  $(C)$ , and  $(D)$  holds, and let  $Q = I - W(W^{T}\Sigma W)^{-1}W^{T}\Sigma$ . For any  $\delta \in (0,1)$ , let  $C_{\delta}:\mathbb{R}^d\to [0,\infty]$  be a continuous function such that with probability at least  $1 - \delta /4$  over  $x\sim \mathcal{N}(0,\Sigma)$ , uniformly over all  $w\in \mathbb{R}^d$ ,
$$
\langle w, Q ^ {T} x \rangle \leq C _ {\delta} (w). \tag {5}
$$
If for each  $y \in \mathcal{V}$ ,  $f$  is non-negative and  $\sqrt{f}$  is  $\sqrt{H}$ -Lipschitz with respect to the first argument, then with probability at least  $1 - \delta$ , it holds that uniformly over all  $(w, b) \in \mathbb{R}^d \times \mathbb{R}$ , we have
$$
(1 - \epsilon) L _ {f} (w, b) \leq \left(\sqrt {\hat {L} _ {f} (w , b)} + \sqrt {\frac {H C _ {\delta} (w) ^ {2}}{n}}\right) ^ {2} \tag {6}
$$
where  $\epsilon = O\left(\tau \sqrt{\frac{h\log(n / h) + \log(1 / \delta)}{n}}\right)$ .
Since the label  $y$  only depends on  $x$  through  $W^{T}x$ , the matrix  $Q$  is simply a (potentially oblique) projection such that  $Q^{T}x$  is independent of  $y$ . In the proof, we separate  $x$  into a low-dimensional component related to  $y$  and the independent component  $Q^{T}x$ . We establish a low-dimensional concentration result using VC theory, which is reflected in the  $\epsilon$  term that does not depend on  $d$ , and we control the remaining high-dimensional component using a scale-sensitive measure  $C_{\delta}$ .
The complexity term  $C_{\delta}(w) / \sqrt{n}$  should be thought of as a (localized form of) Rademacher complexity  $\mathcal{R}_n$ . For example, for any norm  $\| \cdot \|$ , we have  $\langle w, Q^T x \rangle \leq \| w \| \cdot \| Q^T x \|_*$  and the Rademacher complexity for linear predictors with norm bounded by  $B$  is  $B \cdot \mathbb{E} \| x \|_* / \sqrt{n}$  (Shalev-Shwartz and Ben-David 2014). More generally, if we only care about linear predictors in a set  $\mathcal{K}$ , then  $C_{\delta} \approx \mathbb{E}\left[\sup_{w \in \mathcal{K}} \langle w, Q^T x \rangle\right]$  is simply the Gaussian width (Bandeira 2016; Gordon 1988; Koehler et al. 2021) of  $\mathcal{K}$  with respect to  $\Sigma^\perp = Q^T\Sigma Q$  and is exactly equivalent to the expected Rademacher complexity of the class of functions (e.g., Proposition 1 of Zhou et al. 2021).
We now go on to analyze several different problems using optimistic rates and sqrt-lipschitz losses. One of the key insights we want to highlight is the identification of the consistent loss for many of these problems — in other words, the test loss which is implicitly being minimized. When we consider interpolators/overfit models, this is not obvious because an interpolator simultaneously minimizes many different training losses (and when we look at the test loss, each different loss functional may correspond to a different minimizer). Nevertheless, we are able to analyze phenomena like benign overfitting by identifying the particular sqrt-Lipschitz losses which the interpolator is consistent in, i.e. for which it approaches the minimum of the test loss in the appropriate limit.
5.1 Benign Overfitting in Phase Retrieval
In this section, we study the problem of phase retrieval where only the magnitude of the label  $y$  can be measured. If the number of observations  $n < d$ , the minimal norm interpolant is
$$
\hat {w} = \underset {w \in \mathbb {R} ^ {d}: \forall i \in [ n ], \langle w, x _ {i} \rangle^ {2} = y _ {i} ^ {2}} {\arg \min } \| w \| _ {2}. \tag {7}
$$
We are particularly interested in the generalization error of (7) when  $y_{i}$  are noisy labels and so  $\hat{w}$  overfits to the training set. It is well-known that gradient descent initialized at zero converges to a KKT point of the problem (7) when it reaches a zero training error solution (see e.g. Gunasekar et al. 2018). While is not always computationally feasible to compute  $\hat{w}$ , we can study it as a theoretical model for benign overfitting in phase retrieval.
Due to our uniform convergence guarantee in Theorem 1, it suffices to analyze the norm of  $\hat{w}$  and we can use the norm calculation to find the appropriate loss for phase retrieval. The key observation is that for any  $w^{\sharp} \in \mathbb{R}^{d}$ , we can let  $I = \{i \in [n] : \langle w^{\sharp}, x_{i} \rangle \geq 0\}$  and the predictor  $w = w^{\sharp} + w^{\perp}$  satisfies  $|\langle w, x_{i} \rangle| = y_{i}$  where
$$
w ^ {\perp} = \quad \underset {w \in \mathbb {R} ^ {d}:} {\arg \min } \quad \| w \| _ {2}. \tag {8}
$$
$$
\forall i \in I, \langle w, x _ {i} \rangle = y _ {i} - | \langle w ^ {\sharp}, x _ {i} \rangle |
$$
$$
\forall i \notin I, \langle w, x _ {i} \rangle = | \langle w ^ {\sharp}, x _ {i} \rangle | - y _ {i}
$$
Then it holds that  $\| \hat{w}\| _2\leq \| w\| _2\leq \| w^\sharp \| _2 + \| w^\perp \| _2$ . By treating  $y_{i} - |\langle w^{\sharp},x_{i}\rangle |$  and  $|\langle w^{\sharp},x_i\rangle | - y_i$  as the residuals, the norm calculation for  $\| w^{\perp}\| _2$  is the same as the case of linear regression (Koehler et al. 2021; Zhou et al. 2022). Going through this analysis yields the following norm bound (here  $R(\Sigma) = \mathrm{Tr}(\Sigma)^2 /\mathrm{Tr}(\Sigma^2)$  is a measure of effective rank, as in (Bartlett et al. 2020)):
Theorem 2. Under assumptions (A) and (B), let  $f: \mathbb{R} \times \mathcal{Y} \to \mathbb{R}$  be given by  $f(\hat{y}, y) \coloneqq (|\hat{y}| - y)^2$  with  $\mathcal{Y} = \mathbb{R}_{\geq 0}$ . Let  $Q$  be the same as in Theorem 1 and  $\Sigma^{\perp} = Q^{T}\Sigma Q$ . Fix any  $w^{\sharp} \in \mathbb{R}^{d}$  such that  $Qw^{\sharp} = 0$  and for some  $\rho \in (0,1)$ , it holds that
$$
\hat {L} _ {f} \left(w ^ {\sharp}\right) \leq (1 + \rho) L _ {f} \left(w ^ {\sharp}\right). \tag {9}
$$
Then with probability at least  $1 - \delta$ , for some  $\epsilon \lesssim \rho + \log \left(\frac{1}{\delta}\right)\left(\frac{1}{\sqrt{n}} + \frac{1}{\sqrt{R(\Sigma^{\perp})}} + \frac{k}{n} + \frac{n}{R(\Sigma^{\perp})}\right)$ , it holds that
$$
\min  _ {\substack {w \in \mathbb {R} ^ {d}: \\ \forall i \in [ n ], \langle w, x _ {i} \rangle^ {2} = y _ {i} ^ {2}}} \| w \| _ {2} \leq \| w ^ {\sharp} \| _ {2} + (1 + \epsilon) \sqrt {\frac {n L _ {f} \left(w ^ {\sharp}\right)}{\operatorname{Tr} \left(\Sigma^ {\perp}\right)}}. \tag{10}
$$
Note that in this result we used the loss  $f(\hat{y}, y) = (|\hat{y}| - y)^2$ , which is 1-square-root Lipschitz and naturally arises in the analysis. We will now show that this is the correct loss in the sense that benign overfitting is consistent with this loss. To establish consistency result, we can pick  $w^{\sharp}$  to be the minimizer of the population error. The population minimizer always satisfies  $Qw^{\sharp} = 0$  because  $L_{f}((I - Q)w) \leq L_{f}(w)$  for all  $w \in \mathbb{R}^{d}$  by Jensen's inequality. Condition (9) can also be easily checked because we just need concentration of the empirical loss at a single non-random parameter. Applying Theorem 1, we obtain the following.
Corollary 1. In the same setting as in Theorem 2, with probability at least  $1 - \delta$ , it holds that for some
$$
\rho \lesssim \tau \sqrt {\frac {k \log (n / k) + \log (1 / \delta)}{n}} + \log (1 / \delta) \left(\frac {1}{\sqrt {n}} + \frac {1}{\sqrt {R (\Sigma^ {\perp})}} + \frac {k}{n} + \frac {n}{R (\Sigma^ {\perp})}\right),
$$
the minimal norm interpolant  $\hat{w}$  given by (7) enjoys the learning guarantee:
$$
L (\hat {w}, \hat {b}) \leq (1 + \rho) \left(\sqrt {L \left(w ^ {\sharp} , b ^ {\sharp}\right)} + \| w ^ {\sharp} \| _ {2} \sqrt {\frac {\operatorname {T r} \left(\Sigma^ {\perp}\right)}{n}}\right) ^ {2}. \tag {11}
$$
Therefore,  $\hat{w}$  is consistent if the same benign overfitting conditions from Bartlett et al. (2020) and Zhou et al. (2022) hold:  $k = o(n), R(\Sigma^{\perp}) = \omega(n)$  and  $\| w^{\sharp} \|_2^2 \operatorname{Tr}(\Sigma^{\perp}) = o(n)$ .
5.2 Benign Overfitting in ReLU Regression
Since we only need  $\sqrt{f}$  to be Lipschitz, for any 1-Lipschitz function  $\sigma : \mathbb{R} \to \mathbb{R}$ , we can consider
(i)  $f(\hat{y},y) = (\sigma (\hat{y}) - y)^2$  when  $\mathcal{V} = \mathbb{R}$
(ii)  $f(\hat{y},y) = (1 - \sigma (\hat{y})y)_+^2$  when  $\mathcal{V} = \{-1,1\}$
We can interpret the above loss function  $f$  as learning a neural network with a single hidden unit. Indeed, Theorem 1 can be straightforwardly applied to these situations. However, we do not always expect benign overfitting under these loss functions for the following simple reason, as pointed out in Shamir (2022): when  $\sigma$  is invertible, interpolating the loss  $f(\hat{y}, y) = (\sigma(\hat{y}) - y)^2$  is the same as interpolating the loss  $f(\hat{y}, y) = (\hat{y} - \sigma^{-1}(y))^2$  and so the learned function would be the minimizer of  $\mathbb{E}[(\langle w, x \rangle + b - \sigma^{-1}(y))^2]$  which is typically different from the minimizer of  $\mathbb{E}[(\sigma(\langle w, x \rangle + b) - y)^2]$ .