Title: Uniform Convergence with Square-Root Lipschitz Loss
Venue: NeurIPS 2023
Accepted: yes
Keywords: Uniform Convergence

Square-Root Lipschitz

Benign Overfitting

Minimal Norm Interpolation

Phase Retrieval

ReLU Regression

Matrix Sensing
Abstract: We establish generic uniform convergence guarantees for Gaussian data in terms of the Radamacher complexity of the hypothesis class and the Lipschitz constant of the square root of the scalar loss function.  We show how these guarantees substantially generalize previous results based on smoothness (Lipschitz constant of the derivative), and allow us to handle the broader class of square-root-Lipschtz losses, which includes also non-smooth loss functions appropriate for studying phase retrieval and ReLU regression, as well as rederive and better understand “optimistic rate” and interpolation learning guarantees.

Uniform Convergence with Square-Root Lipschitz Loss

Abstract

We establish generic uniform convergence guarantees for Gaussian data in terms of the Rademacher complexity of the hypothesis class and the Lipschitz constant of the square root of the scalar loss function. We show how these guarantees substantially generalize previous results based on smoothness (Lipschitz constant of the derivative), and allow us to handle the broader class of square-root-Lipschitz losses, which includes also non-smooth loss functions appropriate for studying phase retrieval and ReLU regression, as well as rederive and better understand 'optimistic rate' and interpolation learning guarantees.

1 Introduction

The phenomenon of "interpolation learning", where a learning algorithm perfectly interpolates noisy training labels while still generalizing well to unseen data, has attracted significant interest in recent years. As Shamir (2022) pointed out, one of the difficulties in understanding interpolation learning is that we need to determine which loss function we should use to analyze the test error. In linear regression, interpolating the square loss is equivalent to interpolating many other losses (such as the absolute loss) on the training set. Similarly, in the context of linear classification, many works (Soudry et al. 2018; Ji and Telgarsky 2019; Muthukumar et al. 2021) have shown that optimizing the logistic, exponential, or even square loss would all lead to the maximum margin solution, which interpolates the hinge loss. Even though there are many possible loss functions to choose from to analyze the population error, consistency is often possible with respect to only one loss function because different loss functions generally have different population error minimizers.

In linear regression, it has been shown that minimal norm interpolant can be consistent with respect to the square loss (Bartlett et al. 2020; Muthukumar et al. 2020; Negrea et al. 2020; Koehler et al. 2021), while the appropriate loss for benign overfitting in linear classification is the squared hinge loss (Shamir 2022; Zhou et al. 2022). In this line of work, uniform convergence has emerged as a general and useful technique to understand interpolation learning (Koehler et al. 2021; Zhou et al. 2022). Though the Lipschitz contraction technique has been very useful to establish uniform convergence in classical learning theory, the appropriate loss functions to study interpolation learning (such as the square loss and the squared hinge loss) are usually not Lipschitz. In fact, many papers (Nagarajan and Kolter 2019; Zhou et al. 2020; Negrea et al. 2020) have shown that the traditional notion of uniform convergence implied by Lipschitz contraction fails in the setting of interpolation learning. Instead, we need to find other properties of the loss function to establish a different type of uniform convergence.

Zhou et al. (2022) recently showed that the Moreau envelope of the square loss or the squared hinge loss is proportional to itself and this property is sufficient to establish a type of uniform convergence guarantee known as "optimistic rate" (Panchenko 2002; Srebro et al. 2010). Roughly speaking, if this condition holds, then the difference between the square roots of the population error and the training error can be bounded by the Rademacher complexity of the hypothesis class. Specializing to predictors with zero training error, the test error can then be controlled by the square of the Rademacher complexity, which exactly matches the Bayes error for the minimal norm interpolant when it is consistent (Koehler et al. 2021). However, the Moreau envelope of a loss function is not always easy to solve in closed-form, and it is unclear whether this argument can be applied to problems beyond linear regression and max-margin classification. Moreover, considering the difference of square roots seem like a mysterious choice and does not have a good intuitive explanation.

In this paper, by showing that optimistic rate holds for any square-root Lipschitz loss, we argue that the class of square-root Lipschitz losses is the more natural class of loss functions to consider. In fact, any loss whose Moreau envelope is proportional to itself is square-root Lipschitz. Our result also provides an intuitive explanation for optimistic rate: when the loss is Lipschitz, the difference between the test and training error can be bounded by Rademacher complexity; therefore, when the loss is square-root Lipschitz , the difference between the square roots of the test and training error can be bounded by Rademacher complexity. In addition to avoiding any hidden constant and logarithmic factor, our uniform convergence guarantee substantially generalize previous results based on smoothness (Lipschitz constant of the derivative, such as Srebro et al. 2010). The generality of our results allows us to handle non-differentiable losses. In the problems of phase retrieval and ReLU regression, we identify the consistent loss through a norm calculation, and we apply our theory to prove novel consistency results. In the context of noisy matrix sensing, we also establish benign overfitting for the minimal nuclear norm solution. Finally, we show that our results extend to fully optimized neural networks with weight sharing in the first layer.

2 Related Work

Optimistic rates. In the context of generalization theory, our results are related to a phenomena known as 'optimistic rates' - generalization bounds which give strengthened guarantees for predictors with smaller training error. Such bounds can be contrasted with 'pessimistic' bounds which do not attempt to adapt to the training error of the predictor. See for example Vapnik 1982; Panchenko 2003; Panchenko 2002. The work of Srebro et al. 2010 showed that optimistic rates control the generalization error of function classes learned with smooth losses, and recently the works Zhou et al. 2021; Koehler et al. 2021; Zhou et al. 2020 showed that a much sharper version of optimism can naturally explain the phenomena of benign overfitting in Gaussian linear models. (These works are in turn connected with a celebrated line of work in proportional asymptotics for M-estimation such as Stojnic 2013, see references within.) In the present work, we show that the natural setting for optimistic rates is actually square-root-Lipschitz losses and this allows us to capture new applications such as phase retrieval where the loss is not smooth.

Phase retrieval, ReLU Regression, Matrix sensing. Recent works (Maillard et al. 2020; Barbier et al. 2019; Mondelli and Montanari 2018; Luo et al. 2019) analyzed the statistical limits of phase retrieval in a high-dimensional limit for certain types of sensing designs (e.g. i.i.d. Real or Complex Gaussian), as well as the performance of Approximate Message Passing in this problem (which is used to predict a computational-statistical gap). In the noiseless case, the works Andoni et al. 2017; Song et al. 2021 gave better computationally efficient estimators for phase retrieval based upon the LLL algorithm. Our generalization bound can naturally be applied to any estimator for phase retrieval, including these methods or other ones, such as those based on non-convex optimization (Sun et al. 2018; Netrapalli et al. 2013). For the same reason, they can also naturally be applied in the context of sparse phase retrieval (see e.g. Li and V oroninski 2013; Candes et al. 2015).

Similarly, there have been many works studying the computational tractability of ReLU regression. See for example the work of Auer et al. 1995 where the 'matching loss' technique was developed for learning well-specified GLMs including ReLUs. Under misspecification, learning ReLUs can be hard and approximate results have been established, see e.g. Goel et al. 2017; Diakonikolas et al. 2020. In particular, learning a ReLU agnostically in squared loss, even over Gaussian marginals, is known to be hard under standard average-case complexity assumptions (Goel et al. 2019). Again, our

generalization bound has the merit that it can be applied to any estimator. For matrix sensing, we consider nuclear norm minimization (as in e.g. Recht et al. 2010) which is a convex program, so it can always be computed in polynomial time.

Weight-tied neural networks. The work of Bietti et al. 2022 studies a similar two-layer neural network model with weight tying. Their focus is primarily on understanding the non-convex optimization of this model via gradient-based methods. Again, our generalization bound can be straightforwardly combined with the output of their algorithm. The use of norm-based bounds as in our result is common in the generalization theory for neural networks, see e.g. Bartlett (1996) and Anthony, Bartlett, et al. (1999). Compared to existing generalization bounds, the key advantage of our bound is its quantitative sharpness (e.g. small constants and no extra logarithmic factors).

3 Problem Formulation

In most of this paper, we consider the setting of generalized linear models. Given any loss function f : R ×Y → R ≥ 0 and independent sample pairs ( x i , y i ) from some data distribution D over R d ×Y , we can learn a linear model ( ˆ w, ˆ b ) by minimizing the empirical loss ˆ L f with the goal of achieving small population loss L f :

In the above, Y is an abstract label space. For example, Y = R for linear regression and Y = R ≥ 0 for phase retrieval (section 5.1) and ReLU regression (section 5.2). If we view x i as vectorization of the sensing matrices, then our setting (1) also includes the problem of matrix sensing (section 5.3).

For technical reasons, we assume that the distribution D follows a Gaussian multi-index model (Zhou et al. 2022).

(A) d -dimensional Gaussian features with arbitrary mean and covariance: x ∼ N ( µ, Σ)

(B) a generic multi-index model: there exist a low-dimensional projection W = [ w ∗ 1 , ..., w ∗ k ] ∈ R d × k , a random variable ξ ∼ D ξ independent of x (not necessarily Gaussian), and an unknown link function g : R k +1 →Y such that

We require x to be Gaussian because the proof of our generalization bound depends on the Gaussian Minimax Theorem (Gordon 1985; Thrampoulidis et al. 2014), as in many other works (Koehler et al. 2021; Zhou et al. 2022; Wang et al. 2021; Donhauser et al. 2022). In settings where the features are not Gaussian, many works (Goldt et al. 2020; Mei and Montanari 2022; Misiakiewicz 2022; Hu and Lu 2023; Han and Shen 2022) have established universality results showing that the test and training error are asymptotically equivalent to the error of a Gaussian model with matching covariance matrix. However, we show in section 7 that Gaussian universality is not always valid and we discuss potential extensions of the Gaussian feature assumption there.

The multi-index assumption is a generalization of a well-specified linear model y = ⟨ w ∗ , x ⟩ + ξ , which corresponds to k = 1 and g ( η, ξ ) = η + ξ in (2). Since g is not even required to be continuous, the assumption on y is quite general. It allows nonlinear trend and heteroskedasticity, which pose significant challenges for the analyses using random matrix theory (but not for uniform convergence). In Zhou et al. (2022), the range of g is taken to be Y = {-1 , 1 } in order to study binary classification. In section 5.2, we allow the conditional distribution of y to have a point mass at zero, which is crucial for distinguishing ReLU regression from standard linear regression.

4 Optimistic Rate

In order to establish uniform convergence, we need two additional technical assumptions because we have not made any assumption on f and g .

(C) hypercontractivity: there exists a universal constant τ > 0 such that uniformly over all ( w,b ) ∈ R d × R , it holds that

(D) the class of functions on R k +1 ×Y defined below has VC dimension at most h :

{ ( x, y ) → ✶ { f ( ⟨ w,x ⟩ , y ) > t } : ( w,t ) ∈ R k +1 × R } . (4) Crucially, the quantities τ and h only depend on the number of indices k in assumption (B) instead of the feature dimension d . This is a key distinction because k can be a small constant even when the feature dimension is very large. For example, recall that k = 1 in a well-specified model and it is completely free of d . The feature dimension plays no role in equation (3) because conditioned on W T x ∈ R k and ξ ∈ R , the response y is non-random and the distribution of ⟨ w,x ⟩ for any w ∈ R d only depends on w T µ and w T Σ w by properties of the Gaussian distribution.

The hypercontractivity (or 'norm equivalence') assumption (3) is one of a few different common assumptions made in the statistical learning theory literature to rule out the possibility of a very heavy-tailed loss function. The reason is that if the loss function has a very tiny chance of being extremely large, it is not possible to state any meaningful learning guarantee from a finite number of samples. Hypercontractivity was used as an assumption already in Vapnik 1982. Another common and incomparable assumption made in the literature is boundedness, but this is not suitable for us because, e.g., the squared loss under the Gaussian distribution will not be bounded almost surely. On the other hand, if f is the squared loss then (3) holds with τ = 4 √ 105 by standard properties of the Gaussian distribution. We can also state other versions of our main theorem by using other results in statistical learning theory to handle the low-dimensional concentration; in our proof we simply use a result from Vapnik 1982 in a contained and black-box fashion. (See e.g. Mendelson 2017; Vapnik 1982 for more discussion of the different possible assumptions.)

Similar assumptions are made in Zhou et al. (2022) and more background on these assumptions can be found there, but we note that our assumption (C) and (D) are weaker than theirs because we do not require (3) and (4) to hold uniformly over all Moreau envelopes of f .

We are now ready to state the uniform convergence results for square-root Lipschitz losses.

Theorem 1. Assume that (A), (B), (C), and (D) holds, and let Q = I -W ( W T Σ W ) -1 W T Σ . For any δ ∈ (0 , 1) , let C δ : R d → [0 , ∞ ] be a continuous function such that with probability at least 1 -δ/ 4 over x ∼ N (0 , Σ) , uniformly over all w ∈ R d ,

If for each y ∈ Y , f is non-negative and √ f is √ H -Lipschitz with respect to the first argument, then with probability at least 1 -δ , it holds that uniformly over all ( w,b ) ∈ R d × R , we have

Since the label y only depends on x through W T x , the matrix Q is simply a (potentially oblique) projection such that Q T x is independent of y . In the proof, we separate x into a low-dimensional component related to y and the independent component Q T x . We establish a low-dimensional concentration result using VC theory, which is reflected in the ϵ term that does not depend on d , and we control the the remaining high-dimensional component using a scale-sensitive measure C δ .

The complexity term C δ ( w ) / √ n should be thought of as a (localized form of) Rademacher complexity R n . For example, for any norm ∥ · ∥ , we have ⟨ w,Q T x ⟩ ≤ ∥ w ∥ · ∥ Q T x ∥ ∗ and the Rademacher complexity for linear predictors with norm bounded by B is B · E ∥ x ∥ ∗ / √ n (Shalev-Shwartz and Ben-David 2014). More generally, if we only care about linear predictors in a set K , then C δ ≈ E [ sup w ∈K 〈 w,Q T x 〉] is simply the Gaussian width (Bandeira 2016; Gordon 1988; Koehler et al. 2021) of K with respect to Σ ⊥ = Q T Σ Q and is exactly equivalent to the expected Rademacher complexity of the class of functions (e.g., Proposition 1 of Zhou et al. 2021).

5 Applications

We now go on to analyze several different problems using optimistic rates and sqrt-lipschitz losses. One of the key insights we want to highlight is the identification of the consistent loss for many of these problems - in other word, the test loss which is implicitly being minimized. When we consider interpolators/overfit models, this is not obvious because an interpolator simultaneously minimizes many different training losses (and when we look at the test loss, each different loss functional may correspond to a different minimizer). Nevertheless, we are able to analyze phenomena like benign overfitting by identifying the particular sqrt-Lipschitz losses which the interpolator is consistent in, i.e. for which it approaches the minimum of the test loss in the appropriate limit.

5.1 Benign Overfitting in Phase Retrieval

In this section, we study the problem of phase retrieval where only the magnitude of the label y can be measured. If the number of observations n < d , the minimal norm interpolant is

We are particularly interested in the generalization error of (7) when y i are noisy labels and so ˆ w overfits to the training set. It is well-known that gradient descent initialized at zero converges to a KKT point of the problem (7) when it reaches a zero training error solution (see e.g. Gunasekar et al. 2018). While is not always computationally feasible to compute ˆ w , we can study it as a theoretical model for benign overfitting in phase retrieval.

Due to our uniform convergence guarantee in Theorem 1, it suffices to analyze the norm of ˆ w and we can use the norm calculation to find the appropriate loss for phase retrieval. The key observation is that for any w ♯ ∈ R d , we can let I = { i ∈ [ n ] : ⟨ w ♯ , x i ⟩ ≥ 0 } and the predictor w = w ♯ + w ⊥ satisfies |⟨ w,x i ⟩| = y i where

Then it holds that ∥ ˆ w ∥ 2 ≤ ∥ w ∥ 2 ≤ ∥ w ♯ ∥ 2 + ∥ w ⊥ ∥ 2 . By treating y i -|⟨ w ♯ , x i ⟩| and |⟨ w ♯ , x i ⟩| -y i as the residuals, the norm calculation for ∥ w ⊥ ∥ 2 is the same as the case of linear regression (Koehler et al. 2021; Zhou et al. 2022). Going through this analysis yields the following norm bound (here R (Σ) = Tr(Σ) 2 / Tr(Σ 2 ) is a measure of effective rank, as in (Bartlett et al. 2020)):

Theorem 2. Under assumptions (A) and (B), let f : R ×Y → R be given by f (ˆ y, y ) := ( | ˆ y | -y ) 2 with Y = R ≥ 0 . Let Q be the same as in Theorem 1 and Σ ⊥ = Q T Σ Q . Fix any w ♯ ∈ R d such that Qw ♯ = 0 and for some ρ ∈ (0 , 1) , it holds that

Then with probability at least 1 -δ , for some ϵ ≲ ρ +log ( 1 δ ) ( 1 √ n + 1 √ R (Σ ⊥ ) + k n + n R (Σ ⊥ ) ) , it holds that

Note that in this result we used the loss f (ˆ y, y ) = ( | ˆ y | -y ) 2 , which is 1-square-root Lipschitz and naturally arises in the analysis. We will now show that this is the correct loss in the sense that benign overfitting is consistent with this loss. To establish consistency result, we can pick w ♯ to be the minimizer of the population error. The population minimizer always satisfies Qw ♯ = 0 because L f (( I -Q ) w ) ≤ L f ( w ) for all w ∈ R d by Jensen's inequality. Condition (9) can also be easily checked because we just need concentration of the empirical loss at a single non-random parameter. Applying Theorem 1, we obtain the following.

Corollary 1. In the same setting as in Theorem 2, with probability at least 1 -δ , it holds that for some

the minimal norm interpolant ˆ w given by (7) enjoys the learning guarantee:

Therefore, ˆ w is consistent if the same benign overfitting conditions from Bartlett et al. (2020) and Zhou et al. (2022) hold: k = o ( n ) , R (Σ ⊥ ) = ω ( n ) and ∥ w ♯ ∥ 2 2 Tr(Σ ⊥ ) = o ( n ) .

5.2 Benign Overfitting in ReLU Regression

Since we only need √ f to be Lipschitz, for any 1-Lipschitz function σ : R → R , we can consider

We can interpret the above loss function f as learning a neural network with a single hidden unit. Indeed, Theorem 1 can be straightforwardly applied to these situations. However, we do not always expect benign overfitting under these loss functions for the following simple reason, as pointed out in Shamir (2022): when σ is invertible, interpolating the loss f (ˆ y, y ) = ( σ (ˆ y ) -y ) 2 is the same as interpolating the loss f (ˆ y, y ) = (ˆ y -σ -1 ( y )) 2 and so the learned function would be the minimizer of E [( ⟨ w,x ⟩ + b -σ -1 ( y )) 2 ] which is typically different from the minimizer of E [( σ ( ⟨ w,x ⟩ + b ) -y ) 2 ] .

The situation of ReLU regression, where σ (ˆ y ) = max { ˆ y, 0 } , is more interesting because σ is not invertible. In order to be able to interpolate, we must have y ≥ 0 . If y > 0 with probability 1, then σ (ˆ y ) = y is the same as ˆ y = y and we are back to interpolating the square loss f (ˆ y, y ) = (ˆ y -y ) 2 . From this observation, we see that f (ˆ y, y ) = ( σ (ˆ y ) -y ) 2 cannot be the appropriate loss for consistency even though it's 1 square-root Lipschitz. In contrast, when there is some probability mass at y = 0 , it suffices to output any non-positive value and the minimal norm to interpolate is potentially smaller than requiring ˆ y = 0 . Similar to the previous section, we can let I = { i ∈ [ n ] : y i > 0 } and for any ( w ♯ , b ♯ ) ∈ R d +1 , the predictor w = w ♯ + w ⊥ satisfies σ ( ⟨ w,x i ⟩ + b ♯ ) = y i where

Our analysis will show that the consistent loss for benign overfitting with ReLU regression is

We first state the norm bound below.

Theorem 3. Under assumptions (A) and (B), let f : R ×Y → R be the loss defined in (13) with Y = R ≥ 0 . Let Q be the same as in Theorem 1 and Σ ⊥ = Q T Σ Q . Fix any ( w ♯ , b ♯ ) ∈ R d +1 such that Qw ♯ = 0 and for some ρ ∈ (0 , 1) , it holds that

Then with probability at least 1 -δ , for some ϵ ≲ ρ +log ( 1 δ ) ( 1 √ n + 1 √ R (Σ ⊥ ) + k n + n R (Σ ⊥ ) ) , it holds that

Since the loss defined in (13) is also 1-square-root Lipschitz, it can be straightforwardly combined with Theorem 1 to establish benign overfitting in ReLU regression under this loss. The details are exactly identical to the previous section and so we omit it here.

5.3 Benign Overfitting in Matrix Sensing

We now consider the problem of matrix sensing: given random matrices A 1 , ..., A n (with i.i.d. standard Gaussian entries) and independent linear measurements y 1 , ..., y n given by y i = ⟨ A i , X ∗ ⟩ + ξ i where ξ i is independent of A i , and E ξ = 0 and E ξ 2 = σ 2 , we hope to reconstruct the matrix X ∗ ∈ R d 1 × d 2 with sample size n ≪ d 1 d 2 . To this end, we assume that X ∗ has rank r . In this setting, since the measurement matrices have i.i.d. standard Gaussian entries, the test error is closely related to the estimation error:

The classical approach to this problem is to find the minimum nuclear norm solution:

Gunasekar et al. (2017) also shows that gradient descent converges to the minimal nuclear norm solution in matrix factorization problems. It is well known that having low nuclear norm can ensure generalization (Foygel and Srebro 2011; Srebro and Shraibman 2005) and minimizing the nuclear norm ensures reconstruction (Candès and Recht 2009; Recht et al. 2010). However, if the noise level σ is high, then even the minimal nuclear norm solution ˆ X can have large nuclear norm. Since our result can be adapted to different norms as regularizer, our uniform convergence guarantee can be directly applied. The dual norm of the nuclear norm is the spectral norm, and it is well-known that the spectrum norm of a Gaussian random matrix is approximately √ d 1 + √ d 2 (Vershynin 2018) . It remains to analyze the minimal nuclear norm required to interpolate. For simplicity, we assume that ξ are Gaussian below, but we can extend it to be sub-Gaussian.

Theorem 4. Suppose that d 1 d 2 > n , then there exists some ϵ ≲ √ log(32 /δ ) n + n d 1 d 2 such that with probability at least 1 -δ , it holds that

Without loss of generality, we will assume d 1 ≤ d 2 from now on because otherwise we can take the transpose of A and X . Similar to assuming n/R (Σ ⊥ ) → 0 in linear regression, we implicitly assume that n/d 1 d 2 → 0 in matrix sensing. Such scaling is necessary for benign overfitting because of the lower bound on the test error for any interpolant (e.g., Proposition 4.3 of Zhou et al. 2020). Finally, we apply the uniform convergence guarantee.

Theorem 5. Fix any δ ∈ (0 , 1) . There exist constants c 1 , c 2 , c 3 > 0 such that if d 1 d 2 > c 1 n , d 2 > c 2 d 1 , n > c 3 r ( d 1 + d 2 ) , then with probability at least 1 -δ that

From Theorem 5, we see that when the signal to noise ratio ∥ X ∗ ∥ 2 F σ 2 is bounded away from zero, then we obtain consistency ∥ ˆ X -X ∗ ∥ 2 F ∥ X ∗ ∥ 2 F → 0 if (i) r ( d 1 + d 2 ) = o ( n ) , (ii) d 1 d 2 = ω ( n ) , and (iii) d 1 /d 2 →{ 0 , ∞} . This can happen for example when r = Θ(1) , d 1 = Θ( n 1 / 2 ) , d 2 = Θ( n 2 / 3 ) . As discussed earlier, the second condition is necessary for benign overfitting, and the first consistency condition should be necessary even for regularized estimators. It is possible that the final condition is not necessary and we leave a tighter understanding of matrix sensing as future work.

6 Single-Index Neural Networks

We show that our results extend a even more general setting than (1). Suppose that we have a parameter space Θ ⊆ R p and a continuous mapping w from θ ∈ Θ to a linear predictor w ( θ ) ∈ R d . Given a function f : R ×Y × Θ → R and i.i.d. sample pairs ( x i , y i ) drawn from distribution D , we consider training and test error of the form:

We make the same assumptions (A) and (B) on D . Assumptions (C) and (D) can be naturally extended to the following:

(E) there exists a universal constant τ > 0 such that uniformly over all θ ∈ Θ , it holds that

(F) bounded VC dimensions: the class of functions on R k +1 ×Y defined by

has VC dimension at most h .

Now we can state the extension of Theorem 1.

Theorem 6. Suppose that assumptions (A), (B), (E) and (F) hold. For any δ ∈ (0 , 1) , let C δ : R d → [0 , ∞ ] be a continuous function such that with probability at least 1 -δ/ 4 over x ∼ N (0 , Σ) , uniformly over all θ ∈ Θ ,

If for each θ ∈ Θ and y ∈ Y , f is non-negative and √ f is √ H θ -Lipschitz with respect to the first argument, and H θ is continuous in θ , then with probability at least 1 -δ , it holds that uniformly over all θ ∈ Θ , we have

Next, we show that the generality of our result allows us to establish uniform convergence bound for two-layer neural networks with weight sharing. In particular, we let σ ( x ) = max( x, 0) be the ReLU activation function and θ = ( w,a, b ) ∈ R d +2 N , where N is the number of hidden units. Consider the loss function

then L ( θ ) is the test error of a neural network of the form h θ ( x ) := ∑ N i =1 a i σ ( ⟨ w,x ⟩ -b i ) . Since our uniform convergence guarantee holds uniformly over all θ , it applies to networks whose first and second layer weights are optimized at the same time. Without loss of generality, we can assume that b 1 ≤ ... ≤ b N are sorted, then it is easy to see that √ f is max j ∈ [ N ] ∣ ∣ ∣ ∑ j i =1 a i ∣ ∣ ∣ Lipschitz. Applying Theorem 6, we obtain the following corollary.

Corollary 2. Fix an arbitrary norm ∥ · ∥ and consider f as defined in (24) . Assume that the data distribution D satisfy (A), (B), and (E). Then with probability at least 1 -δ , it holds that uniformly over all θ = ( w,a, b ) ∈ R d +2 N , we have

where ϵ is the same as in Theorem 6 with h = ˜ O ( k + N ) and ϵ ′ = O ( sup ∥ u ∥≤ 1 ∥ u ∥ Σ ⊥ √ log(1 /δ ) ) .

The above theorem says given a network parameter θ , after sorting the b i 's, a good complexity measure to look at is max j ∈ [ N ] ∣ ∣ ∣ ∑ j i =1 a i ∣ ∣ ∣ · ∥ w ∥ and equation (25) precisely quantify how the complexity of a network controls generalization.

7 On Gaussian Universality

In this section, we discuss a counterexample to Gaussian universality motivated by Shamir (2022). In particular, we consider a data distribution D over ( x, y ) given by:

(H) there exists a function g : R k +1 → R such that

where ξ ∼ D ξ is independent of x (but not necessarily Gaussian).

By the independence between z and x | k , we can easily see that x | k and x | d -k are uncorrelated. Moreover, the covariance matrix of x | d -k is

which is just a re-scaling of Σ | d -k and therefore has the same effective rank as R (Σ | d -k ) . Even though the feature x in D is non-Gaussian, its tail x | d -k is Gaussian conditioned on x | k . Therefore, our proof technique is still applicable after a conditioning step. However, it turns out that the uniform convergence bound in Theorem 1 is no longer valid. Instead, we have to re-weight the loss function. The precise theorem statements can be found in the appendix. We briefly describe our theoretical results below, which follow the same ideas as in section 4 and 5.

Uniform Convergence. Under a similar low-dimensional concentration assumption such as (C), if we let C δ be such that ⟨ w,z ⟩ ≤ C δ ( w ) for all w ∈ R d -k with high probability, it holds that

Note that the distribution of z is specified in (G) and different to the distribution of x | d -k .

Norm Bound. Next, we focus on linear regression and compute the minimal norm required to interpolate. If we pick Σ | d -k to satisfy the benign overfitting conditions, for any w ∗ | k ∈ R k , we have

It is easy to see that the w ∗ that minimizes the population weighted square loss satisfy w ∗ | d -k = 0 , and so we can let w ∗ | k to be the minimizer.

Benign Overfitting. Let ˆ w = arg min w ∈ R d : Xw = Y ∥ w ∥ 2 be the minimal norm interpolant. Plugging in the norm bound (29) into the uniform convergence guarantee (28), we show that

which recovers the consistency result of Shamir (2022).

Contradiction. Suppose that Gaussian universality holds, then our optimistic rate theory would predict that

Combining (30) with (31), we obtain that

which cannot always hold. For example, let's consider the case where k = 1 , x 1 ∼ N (0 , 1) , h ( x 1 ) = 1 + | x 1 | and y = h ( x 1 ) 2 . Then it is straightforward to verify that the left hand side of (32) equals E [ h ( x 1 ) 4 ] and the right hand side equals E [ h ( x 1 ) 2 ] 2 , but this is impossible because

In the counterexample above, we see that it is possible to introduce strong dependence between the signal and the tail component of x while ensuring that they are uncorrelated. The dependence will prevent the norm of the tail from concentrating around its mean, no matter how large the sample size is. In contrast, the norm of the tail will concentrate for Gaussian features with a matching covariance -such discrepancy results in an over-optimistic bound for non-Gaussian data.

8 Conclusion

In this paper, we extend a type of sharp uniform convergence guarantee proven for the square loss in Zhou et al. (2021) to any square-root Lipschitz loss. Uniform convergence with square-root Lipschitz loss is an important tool because the appropriate loss to study interpolation learning is usually square-root Lipschitz instead of Lipschitz. Compared to the prior work of Zhou et al. 2022, our view significantly simplify the assumptions to establish optimistic rate. Since we don't need to explicitly compute the Moreau envelope for each application, our framework easily leads to many novel benign overfitting results, including low-rank matrix sensing.

In the applications to phase retrieval and ReLU, we identify the appropriate loss function f and our norm calculation overcomes the challenge that f is non-convex and so CGMT cannot be directly applied. Furthermore, we explore new extensions of the uniform convergence technique to study single-index neural networks and suggest a promising research direction to understand the generalization of neural networks. Finally, we argue that Gaussian universality cannot always be taken for granted by analyzing a model where only the weighted square loss enjoys an optimistic rate. Our results highlight the importance of tail concentration and shed new lights on the necessary conditions for universality. An important future direction is to extend optimistic rate beyond Gaussian data, possibly through worst-case Rademacher complexity. Understanding the performance of more practical algorithms in phase retrieval and deriving the necessary and sufficient conditions for benign overfitting in matrix sensing are also interesting problems.

Acknowledgements. F.K. was supported in part by NSF award CCF-1704417, NSF award IIS1908774, and N. Anari's Sloan Research Fellowship.

References

Andoni, Alexandr, Daniel Hsu, Kevin Shi, and Xiaorui Sun (2017). 'Correspondence retrieval.' Conference on Learning Theory . PMLR, pp. 105-126.

Anthony, Martin, Peter L. Bartlett, et al. (1999). Neural network learning: Theoretical foundations . Vol. 9. cambridge university press Cambridge.

Auer, Peter, Mark Herbster, and Manfred KK Warmuth (1995). 'Exponentially many local minima for single neurons.' Advances in neural information processing systems 8.

Bandeira, Afonso S. (2016). 'Ten Lectures and Forty-Two Open Problems in the Mathematics of Data Science.' Lecture notes, MIT. URL: https://people.math.ethz.ch/~abandeira/ TenLecturesFortyTwoProblems.pdf .

Barbier, Jean, Florent Krzakala, Nicolas Macris, Léo Miolane, and Lenka Zdeborová (2019). 'Optimal errors and phase transitions in high-dimensional generalized linear models.' Proceedings of the National Academy of Sciences 116.12, pp. 5451-5460.

Bartlett, Peter L. (1996). 'For valid generalization the size of the weights is more important than the size of the network.' Advances in neural information processing systems 9.

Bartlett, Peter L., Nick Harvey, Christopher Liaw, and Abbas Mehrabian (2019). 'Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks.' The Journal of Machine Learning Research 20.1, pp. 2285-2301.

Bartlett, Peter L., Philip M. Long, Gábor Lugosi, and Alexander Tsigler (2020). 'Benign overfitting in linear regression.' Proceedings of the National Academy of Sciences 117.48, pp. 30063-30070.

Bauschke, Heinz H, Patrick L Combettes, et al. (2011). Convex analysis and monotone operator theory in Hilbert spaces . Vol. 408. Springer.

Bietti, Alberto, Joan Bruna, Clayton Sanford, and Min Jae Song (2022). 'Learning single-index models with shallow neural networks.' arXiv preprint arXiv:2210.15651 .

Blumer, Anselm, Andrzej Ehrenfeucht, David Henry Haussler, and Manfred Klaus Warmuth (1989). 'Learnability and the Vapnik-Chervonenkis dimension.' Journal of the ACM 36 (4), pp. 929-965.

Boyd, Stephen, Stephen P Boyd, and Lieven Vandenberghe (2004). Convex optimization . Cambridge University Press.

Candes, Emmanuel J, Xiaodong Li, and Mahdi Soltanolkotabi (2015). 'Phase retrieval via Wirtinger flow: Theory and algorithms.' IEEE Transactions on Information Theory 61.4, pp. 1985-2007.

Candès, Emmanuel J and Benjamin Recht (2009). 'Exact matrix completion via convex optimization.' Foundations of Computational mathematics 9.6, pp. 717-772.

Diakonikolas, Ilias, Surbhi Goel, Sushrut Karmalkar, Adam R Klivans, and Mahdi Soltanolkotabi (2020). 'Approximation schemes for relu regression.' Conference on learning theory . PMLR, pp. 1452-1485.

Donhauser, Konstantin, Nicolo Ruggeri, Stefan Stojanovic, and Fanny Yang (2022). 'Fast rates for noisy interpolation require rethinking the effects of inductive bias.' arXiv: 2203.03597 .

Foygel, Rina and Nathan Srebro (2011). 'Concentration-based guarantees for low-rank matrix reconstruction.' Proceedings of the 24th Annual Conference on Learning Theory . JMLR Workshop and Conference Proceedings, pp. 315-340.

Goel, Surbhi, Varun Kanade, Adam Klivans, and Justin Thaler (2017). 'Reliably learning the relu in polynomial time.' Conference on Learning Theory . PMLR, pp. 1004-1042.

Goel, Surbhi, Sushrut Karmalkar, and Adam Klivans (2019). 'Time/accuracy tradeoffs for learning a relu with respect to gaussian marginals.' Advances in neural information processing systems 32.

Goldt, Sebastian, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mezard, and Lenka Zdeborova (2020). 'The gaussian equivalence of generative models for learning with shallow neural networks.' arXiv:2006.14709 .

Gordon, Yehoram (1985). 'Some inequalities for Gaussian processes and applications.' Israel Journal of Mathematics 50.4, pp. 265-289.

Gordon, Yehoram (1988). 'On Milman's inequality and random subspaces which escape through a mesh in R n .' Geometric Aspects of Functional Analysis . Vol. 1317. Lecture Notes in Mathematics. Springer, pp. 84-106.

Gunasekar, Suriya, Jason Lee, Daniel Soudry, and Nathan Srebro (2018). 'Characterizing implicit bias in terms of optimization geometry.' International Conference on Machine Learning . PMLR, pp. 1832-1841.

Gunasekar, Suriya, Blake Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nathan Srebro (2017). 'Implicit Regularization in Matrix Factorization.' Advances in Neural Information Processing Systems .

Han, Qiyang and Yandi Shen (2022). 'Universality of regularized regression estimators in high dimensions.' arXiv: 2206.07936 .

Hu, Hong and Yue M. Lu (2023). 'Universality Laws for High-Dimensional Learning With Random Features.' IEEE Transactions on Information Theory 69.3, pp. 1932-1964.

Ji, Ziwei and Matus Telgarsky (2019). 'The implicit bias of gradient descent on nonseparable data.' Conference on Learning Theory , pp. 1772-1798.

Koehler, Frederic, Lijia Zhou, Danica J. Sutherland, and Nathan Srebro (2021). 'Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds and Benign Overfitting.' Advances in Neural Information Processing Systems . arXiv: 2106.09276 .

Li, Xiaodong and Vladislav Voroninski (2013). 'Sparse signal recovery from quadratic measurements via convex programming.' SIAM Journal on Mathematical Analysis 45.5, pp. 3019-3033.

Luo, Wangyu, Wael Alghamdi, and Yue M Lu (2019). 'Optimal spectral initialization for signal recovery with applications to phase retrieval.' IEEE Transactions on Signal Processing 67.9, pp. 2347-2356.

Maillard, Antoine, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová (2020). 'Phase retrieval in high dimensions: Statistical and computational phase transitions.' Advances in Neural Information Processing Systems 33, pp. 11071-11082.

Mei, Song and Andrea Montanari (2022). 'The generalization error of random features regression: Precise asymptotics and the double descent curve.' Communications on Pure and Applied Mathematics 75.4, pp. 667-766.

Mendelson, Shahar (2017). 'Extending the scope of the small-ball method.' arXiv: 1709.00843 .

Misiakiewicz, Theodor (2022). 'Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression.' arXiv:2204.10425 .

Mondelli, Marco and Andrea Montanari (2018). 'Fundamental limits of weak recovery with applications to phase retrieval.' Conference On Learning Theory . PMLR, pp. 1445-1450.

Muthukumar, Vidya, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai (2021). 'Classification vs regression in overparameterized regimes: Does the loss function matter?' Journal of Machine Learning Research 22, pp. 1-69. arXiv: 2005.08054 .

Muthukumar, Vidya, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai (2020). 'Harmless interpolation of noisy data in regression.' IEEE Journal on Selected Areas in Information Theory . arXiv: 1903.09139 .

Nagarajan, Vaishnavh and J. Zico Kolter (2019). 'Uniform convergence may be unable to explain generalization in deep learning.' Advances in Neural Information Processing Systems . arXiv: 1902.04742 .

Negrea, Jeffrey, Gintare Karolina Dziugaite, and Daniel M. Roy (2020). 'In Defense of Uniform Convergence: Generalization via derandomization with an application to interpolating predictors.' International Conference on Machine Learning . arXiv: 1912.04265 .

Netrapalli, Praneeth, Prateek Jain, and Sujay Sanghavi (2013). 'Phase retrieval using alternating minimization.' Advances in Neural Information Processing Systems 26.

Panchenko, Dmitry (2002). 'Some Extensions of an Inequality of Vapnik and Chervonenkis.' Electronic Communications in Probability 7, pp. 55-65. arXiv: 0405342 .

Panchenko, Dmitry (2003). 'Symmetrization approach to concentration inequalities for empirical processes.' The Annals of Probability 31.4, pp. 2068-2081.

Recht, Benjamin, Maryam Fazel, and Pablo A Parrilo (2010). 'Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization.' SIAM review 52.3, pp. 471-501.

Rockafellar, R Tyrrell (1970). Convex analysis . Vol. 18. Princeton university press.

Shalev-Shwartz, Shai and Shai Ben-David (2014). Understanding machine learning: From theory to algorithms . Cambridge University Press.

Shamir, Ohad (2022). 'The Implicit Bias of Benign Overfitting.' arXiv: 2201.11489 .

Song, Min Jae, Ilias Zadik, and Joan Bruna (2021). 'On the cryptographic hardness of learning single periodic neurons.' Advances in neural information processing systems 34, pp. 29602-29615.

Soudry, Daniel, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro (2018). 'The implicit bias of gradient descent on separable data.' Journal of Machine Learning Research 19, pp. 2822-2878.

Srebro, Nathan and Adi Shraibman (2005). 'Rank, trace-norm and max-norm.' International Conference on Computational Learning Theory , pp. 545-560.

Srebro, Nathan, Karthik Sridharan, and Ambuj Tewari (2010). 'Optimistic Rates for Learning with a Smooth Loss.' arXiv: 1009.3896 .

Stojnic, Mihailo (2013). 'A framework to characterize performance of LASSO algorithms.' arXiv: 1303.7291 .

Sun, Ju, Qing Qu, and John Wright (2018). 'A geometric analysis of phase retrieval.' Foundations of Computational Mathematics 18, pp. 1131-1198.

Thrampoulidis, Christos, Samet Oymak, and Babak Hassibi (2014). 'The Gaussian min-max theorem in the presence of convexity.' arXiv: 1408.4837 .

Thrampoulidis, Christos, Samet Oymak, and Babak Hassibi (2015). 'Regularized linear regression: A precise analysis of the estimation error.' Conference on Learning Theory .

Vapnik, Vladimir (1982). Estimation of dependences based on empirical data . Springer Science & Business Media.

Vershynin, Roman (2010). 'Introduction to the non-asymptotic analysis of random matrices.' arXiv preprint arXiv:1011.3027 .

Vershynin, Roman (2018). High-dimensional probability: An introduction with applications in data science . Vol. 47. Cambridge University Press.

Wang, Guillaume, Konstantin Donhauser, and Fanny Yang (2021). 'Tight bounds for minimum l1-norm interpolation of noisy data.' arXiv: 2111.05987 .

Zhou, Lijia, Frederic Koehler, Pragya Sur, Danica J. Sutherland, and Nathan Srebro (2022). 'A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Models.' Advances in Neural Information Processing Systems . arXiv: 2210.12082 .

Zhou, Lijia, Frederic Koehler, Danica J. Sutherland, and Nathan Srebro (2021). 'Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regression.' arXiv: 2112.04470 .

Zhou, Lijia, Danica J. Sutherland, and Nathan Srebro (2020). 'On Uniform Convergence and LowNorm Interpolation Learning.' Advances in Neural Information Processing Systems . arXiv: 2006. 05942 .

A Organization of the Appendices

In the Appendix, we give proofs of all results from the main text. In Appendix B, we study properties of square-root-Lipschitz functions and introduce some technical tools that we use throughout the appendix. In Appendix C, we prove our main uniform convergence guarantee (Theorem 1 and the more general version Theorem 6). In Appendix D, we obtain bounds on the minimal norm required to interpolate in the settings studied in section 5. In Appendix E, we provide details on the counterexample to Gaussian universality described in section 7.

B Preliminaries

B.1 Properties of Square-root Lipschitz Loss

In this section, we prove that square-root Lipschitzness can be equivalently characterized by a relationship between a function and its Moreau envelope, which can be used to establish uniform convergence results based on the recent work of Zhou et al. 2022. We formally define Lipschitz functions and Moreau envelope below.

Definition 1. A function f : R → R is M -Lipschitz if for all x, y in R ,

Definition 2. The Moreau envelope of a function f : R → R associated with smoothing parameter λ ∈ R + is defined as

Though we define Lipschitz functions and Moreau envelope for univariate functions from R to R above, we can easily extend definitions 1 and 2 to loss functions f : R ×Y → R or f : R ×Y× Θ → R . We say a function f : R ×Y → R is M -Lipschitz if for any y ∈ Y and ˆ y 1 , ˆ y 2 ∈ R , we have

Similarly, we say a function f : R × Y × Θ → R is M -Lipschitz if for any y ∈ Y , θ ∈ Θ and ˆ y 1 , ˆ y 2 ∈ R , we have

We can also define the Moreau envelope of a function f : R ×Y → R by

and the Moreau envelope of a function f : R ×Y × Θ → R is defined as

The proof of all results in this section can be straightforwardly extended to these settings. For simplicity, we ignore the additional arguments in Y and Θ in this section.

The Moreau envelope is usually viewed as a smooth approximation to the original function f ; its minimizer is known as the proximal operator. It plays an important role in convex analysis (see e.g. Boyd et al. 2004; Bauschke, Combettes, et al. 2011; Rockafellar 1970), but is also useful and well-defined when f is nonconvex. The canonical example of a √ H -square-root-Lipschitz function is f ( x ) = Hx 2 , for which we can easily check

In proposition 1 below, we show that the condition f λ ≥ λ λ + H f is exactly equivalent to √ H -squareroot-Lipschitzness.

Proposition 1. A function f : R → R is non-negative and √ H -square-root-Lipschitz if and only if for any x ∈ R and λ ≥ 0 , it holds that

Proof. Suppose that equation (35) holds, then by taking λ = 0 and the definition in equation (2), we see that f must be non-negative. For an non-negative function f , we observe for any x ∈ R , it holds that

Therefore, f must be H -square-root-Lipschitz as well. Conversely, if f is non-negative and √ H -square-root-Lipschitz, then the above implies that (2) must hold and we are done.

Interestingly, there is a similar equivalent characterization for Lipschitz functions as well.

Proposition 2. A function f : R → R is M -Lipschitz if and only if for any x ∈ R and λ > 0 , it holds that

Proof. Observe that for any x ∈ R , it holds that

and we are done.

Finally, we show that any smooth loss is square-root-Lipschitz. Therefore, the class of square-rootLipschitz losses is more general than the class of smooth losses studied in Srebro et al. 2010.

Definition 3. A twice differentiable 1 function f : R → R is H -smooth if for all x in R

The following result is similar to to Lemma 2.1 in Srebro et al. 2010:

Proposition 3. Let f : R → R be a H -smooth and non-negative function. Then for any x ∈ R , it holds that

Proof. Since f is H -smooth and non-negative, by Taylor's theorem, for any x, y ∈ R , we have

where a ∈ [min( x, y ) , max( x, y )] . Setting y = x -f ′ ( x ) H yields the desired bound. To show that √ f is Lipschitz, we observe that for any x ∈ R

and so we apply Taylor's theorem again to show that

which is the desired definition.

B.2 Properties of Gaussian Distribution

We will make use of the following results without proof.

Gaussian Minimax Theorem. Our proof of Theorem 1 and 6 will closely follow prior works that apply Gaussian Minimax Theorem (GMT) to uniform convergence (Koehler et al. 2021; Zhou et al. 2021; Zhou et al. 2022; Wang et al. 2021; Donhauser et al. 2022). The following result is Theorem 3 of Thrampoulidis et al. 2015 (see also Theorem 1 in the same reference). As explained there, it is a consequence of the main result of Gordon (1985), known as Gordon's Theorem.

Theorem 7 (Thrampoulidis et al. 2015; Gordon 1985) . Let Z : n × d be a matrix with i.i.d. N (0 , 1) entries and suppose G ∼ N (0 , I n ) and H ∼ N (0 , I d ) are independent of Z and each other. Let S w , S u be compact sets and ψ : S w × S u → R be an arbitrary continuous function. Define the Primary Optimization (PO) problem

and the Auxiliary Optimization (AO) problem

Under these assumptions, Pr(Φ( Z ) < c ) ≤ 2 Pr( ϕ ( G,H ) ≤ c ) for any c ∈ R .

Furthermore, if we suppose that S w , S u are convex sets and ψ ( w,u ) is convex in w and concave in u , then Pr(Φ( Z ) > c ) ≤ 2 Pr( ϕ ( G,H ) ≥ c ) .

GMT is an extremely useful tool because it allows us to convert a problem involving a random matrix into a problem involving only two random vectors. In our analysis, we will make use of a slightly more general version of Theorem 7, introduced by Koehler et al. (2021), to include additional variables which only affect the deterministic term in the minmax problem.

Theorem 8 (Variant of GMT) . Let Z : n × d be a matrix with i.i.d. N (0 , 1) entries and suppose G ∼ N (0 , I n ) and H ∼ N (0 , I d ) are independent of Z and each other. Let S W , S U be compact sets in R d × R d ′ and R n × R n ′ respectively, and let ψ : S W × S U → R be an arbitrary continuous function. Define the Primary Optimization (PO) problem

and the Auxiliary Optimization (AO) problem

Under these assumptions, Pr(Φ( Z ) < c ) ≤ 2 Pr( ϕ ( G,H ) ≤ c ) for any c ∈ R .

Theorem 8 requires S W and S U to be compact. However, we can usually get around the compactness requirement by a truncation argument.

Lemma 1 (Zhou et al. 2022, Lemma 6) . Let f : R d → R be an arbitrary function and S d r = { x ∈ R d : ∥ x ∥ 2 ≤ r } , then for any set K , it holds that

If f is a random function, then for any t ∈ R

Lemma 2 (Zhou et al. 2022, Lemma 7) . Let K be a compact set and f, g be continuous real-valued functions on R d . Then it holds that

If f and g are random functions, then for any t ∈ R

Concentration inequalities. Let σ min ( A ) denote the minimum singular value of an arbitrary matrix A , and σ max the maximum singular value. We use ∥ A ∥ op = σ max ( A ) to denote the operator norm of matrix A . The following concentration results for Gaussian vector and matrix are standard.

Lemma 3 (Special case of Theorem 3.1.1 of Vershynin 2018) . Suppose that Z ∼ N (0 , I n ) . Then

Lemma 4 (Koehler et al. 2021, Lemma 10) . For any covariance matrix Σ and H ∼ N (0 , I d ) , with probability at least 1 -δ , it holds that

Therefore, provided that R (Σ) ≳ log(4 /δ ) 2 , it holds that

Theorem 9 (Vershynin 2010, Corollary 5.35) . Let n, N ∈ N . Let A ∈ R N × n be a random matrix with entries i.i.d. N (0 , 1) . Then for any t > 0 , it holds with probability at least 1 -2 exp( -t 2 / 2) that √ √ √

Conditional Distribution of Gaussian. To handle arbitrary multi-index conditional distributions of y given by assumption (B), we will apply a conditioning argument. After conditioning on W T x and ξ , the response y is no longer random. Importantly, the conditional distribution of x remains Gaussian (though with a different mean and covariance) and so we can still apply GMT. In the lemma below, Z ∈ R n × d is a random matrix with i.i.d. N (0 , 1) entries and X = Z Σ 1 / 2 .

Lemma 5 (Zhou et al. 2022, Lemma 4) . Fix any integer k < d and any k vectors w ∗ 1 , ..., w ∗ k in R d such that Σ 1 / 2 w ∗ 1 , ..., Σ 1 / 2 w ∗ k are orthonormal. Denoting

the distribution of X conditional on Xw ∗ 1 = η 1 , ..., Xw ∗ k = η k is the same as that of

and

B.3 Vapnik-Chervonenkis (VC) theory

By the conditioning step mentioned above, we will separate x into a low-dimensional component W T x and the independent component Q T x . Concentration results for the low-dimensional component can be easily established using VC theory. As mentioned in Zhou et al. 2022, low-dimensional concentration can be established using alternative results (e.g., Vapnik 1982; Panchenko 2002; Panchenko 2003; Mendelson 2017).

Recall the following definition of VC-dimension from Shalev-Shwartz and Ben-David (2014).

Definition 4. Let H be a class of functions from X to { 0 , 1 } and let C = { c 1 , ..., c m } ⊂ X . The restriction of H to C is

A hypothesis class H shatters a finite set C ⊂ X if |H C | = 2 | C | . The VC-dimension of H is the maximal size of a set that can be shattered by H . If H can shatter sets of arbitrary large size, we say H has infinite VC-dimension.

Also, we have the following well-known result for the class of nonhomogenous halfspaces in R d (Theorem 9.3 of Shalev-Shwartz and Ben-David (2014)), and the result on VC-dimension of the union of two hypothesis classes (Lemma 3.2.3 of Blumer et al. (1989)):

Theorem 10. The class { x ↦→ sign ( ⟨ w,x ⟩ + b ) : w ∈ R d , b ∈ R } has VC-dimension d +1 .

Theorem 11. Let H a hypothesis classes of finite VC-dimension d ≥ 1 . Let H 2 := { max( h 1 , h 2 ) : h 1 , h 2 ∈ H} and H 3 := { min( h 1 , h 2 ) : h 1 , h 2 ∈ H} . Then, both the VC-dimension of H 2 and the VC-dimension of H 3 are O ( d ) .

By combining Theorem 10 and 11, we can easily verify the VC assumption in Corollary 1 for the phase retrieval loss f (ˆ y, y ) = ( | ˆ y | -y ) 2 . Similar results can be proven for ReLU regression. To verify the VC assumption for single-index neural nets in Corollary 2, we can use the following result (equation 2 of Bartlett et al. (2019)):

Theorem 12. The VC-dimension of a neural network with piecewise linear activation function, W parameters, and L layers has VC-dimension O ( WL log W ) .

We can easily establish low-dimensional concentration due to the following result:

Theorem 13 (Vapnik 1982, Special case of Assertion 4 in Chapter 7.8; see also Theorem 7.6) . Suppose that the loss function l : Z × Θ → R ≥ 0 satisfies

(i) for every θ ∈ Θ , the function l ( · , θ ) is measurable with respect to the first argument

(ii) the class of functions { z ↦→ ✶ { l ( z, θ ) > t } : ( θ, t ) ∈ Θ × R } has VC-dimension at most h and the distribution D over Z satisfies for every θ ∈ Θ

then for any n > h , with probability at least 1 -δ over the choice of ( z 1 , . . . , z n ) ∼ D n , it holds uniformly over all θ ∈ Θ that

C Proof of Theorem 6

It is clear that Theorem 1 is a special case of Theorem 6. Therefore, we will prove the more general result here.

Notation. Following the tradition in statistics, we denote X = ( x 1 , ..., x n ) T ∈ R n × d as the design matrix. In the proof section, we slightly abuse the notation of η i to mean Xw ∗ i and ξ to mean the n -dimensional random vector whose i -th component satisfies y i = g ( η 1 ,i , ..., η k,i , ξ i ) . We will write X = Z Σ 1 / 2 where Z is a random matrix with i.i.d. standard normal entries if µ = 0 .

Throughout this section, we can first assume µ = 0 in Assumption (A) without loss of generality because if we define ˜ f : R ×Y × Θ → R by

then by definition, it holds that

and so we can apply the theory on ˜ f first and then translate to the problem on f . Similarly, we can also assume Σ 1 / 2 w ∗ 1 , ..., Σ 1 / 2 w ∗ k are orthonormal without loss of generality. This is because we can denote W ∈ R d × k by W = [ w ∗ 1 , ..., w ∗ k ] and let ˜ W = W ( W T Σ W ) -1 / 2 . By definition, it holds that ˜ W T Σ ˜ W = I and so the columns of ˜ W = [ ˜ w ∗ 1 , ..., ˜ w ∗ k ] satisfy Σ 1 / 2 ˜ w ∗ 1 , ..., Σ 1 / 2 ˜ w ∗ k are orthonormal. If we define ˜ g : R k +1 → R by

then y = ˜ g ( x T ˜ W,ξ ) and so we can apply the theory on ˜ g .

We will write the generalization problem as a Primary Optimization problem in Theorem 8. For generality, we will let F be any deterministic function and then choose it in the end.

Lemma 6. Fix an arbitrary set Θ ⊆ R p and let F : Θ → R be any deterministic and continuous function. Consider dataset ( X,Y ) drawn i.i.d. from the data distribution D according to (A) and (B) with µ = 0 and orthonormal Σ 1 / 2 w ∗ 1 , ..., Σ 1 / 2 w ∗ k . Then conditioned on Xw ∗ 1 = η 1 , ..., Xw ∗ k = η k and ξ , if we define

where P is defined in (50) and ψ is a deterministic and continuous function given by

then it holds that for any t ∈ R , we have

Proof. By introducing a variable u = Xw ( θ ) , we have

Conditioned on Xw ∗ 1 = η 1 , ..., Xw ∗ k = η k and ξ , the above is only random in X by our multi-index model assumption on y . By Lemma 5, the above is equal in law to

The function ψ is continuous because we require F, f and w to be continuous in the definitions.

Next, we are ready to apply Gaussian Minimax Theorem. Although the domains in (56) are not compact, we can use the truncation lemmas 1 and 2 in Appendix B.

Lemma 7. In the same setting as Lemma 6, define the auxiliary problem as

then for any t ∈ R , it holds that

where the randomness in the second probability is taken over G,H,η 1 , ..., η k and ξ .

Proof. Denote S r = { ( w,u,θ ) ∈ R d × R n × Θ : w = P Σ 1 / 2 w ( θ ) and ∥ w ∥ 2 + ∥ u ∥ 2 + ∥ θ ∥ 2 ≤ r } . The set S r is bounded by definition and closed by the continuity of w . Hence, it is compact. Next, we denote the truncated problems:

By definition, we have Φ r ≤ Φ r,s and so

The corresponding auxiliary problems are

and the limit of s →∞ :

By definition, it holds that Ψ r ≤ Ψ and so

Thus, it holds that

The proof concludes by applying Lemma 6 and the tower law.

The following two simple lemmas will be useful to analyze the auxiliary problem. Lemma 8. For a, b, H > 0 , we have

Define f ( λ ) = λa + H H + λ b , then

Since we require λ ≥ 0 , we only need to consider whether √ Hb a -H ≥ 0 ⇐⇒ b ≥ Ha . If b < Ha , the infimum is attained at λ = 0 . Otherwise, the infimum is attained at λ ∗ = √ Hb a -H , at which point √

Plugging in, we see that the expression is equivalent to ( √ b -√ Ha ) 2 + in both cases.

Lemma 9. For a, b ≥ 0 , we have

Proof. Observe that

Proof. Define f ( λ ) = -λa -b λ , then

and so in the domain λ ≥ 0 , the optimum is attained at λ ∗ = √ b/a at which point f ( λ ∗ ) = -2 √ ab .

We are now ready to analyze the auxiliary problem.

Lemma 10. In the same setting as in Lemma 6, assume that for every δ > 0

(A) C δ : R d → [0 , ∞ ] is a continuous function such that with probability at least 1 -δ/ 4 over H ∼ N (0 , I d ) , uniformly over all w ∈ R d , we have that

(B) ϵ δ is a positive real number such that with probability at least 1 -δ/ 4 over { (˜ x i , ˜ y i ) } n i =1 drawn i.i.d. from ˜ D , it holds uniformly over all θ ∈ Θ that

where the distribution ˜ D over (˜ x, ˜ y ) is given by

and the mapping ϕ : R d → R k +1 is defined as

Then the following is true:

(i) suppose for some choice of M θ that is continuous in θ , it holds for every y ∈ Y and θ ∈ Θ , f is M θ -Lipschitz with respect to the first argument, then with probability at least 1 -δ , uniformly over all θ ∈ Θ , we have

(ii) suppose for some choice of H θ that is continuous in θ , it holds for every y ∈ Y and θ ∈ Θ , f is non-negative and √ f is √ H θ -Lipschitz with respect to the first argument, then with probability at least 1 -δ , uniformly over all θ ∈ Θ , we have

Proof. First, let's simplify the auxiliary problem (59). Changing variables to subtract the quantity G i ∥ ∥ P Σ 1 / 2 w ( θ ) ∥ ∥ 2 + ∑ k l =1 ⟨ w ( θ ) , Σ w ∗ l ⟩ η l,i from each of the former u i , we have that

and separating the optimization problem in u and θ , we obtain

Ψ = sup

θ

∈

Θ

Next, we will lower bound the infimum term by weak duality to obtain upper bound on Ψ :

Suppose that for every y ∈ Y and θ ∈ Θ , f is M θ -Lipschitz with respect to the first argument, then by Proposition 2, the above can be further lower bounded by the following quantity:

F

(

θ

)

On the other hand, suppose that for every y ∈ Y and θ ∈ Θ , f is non-negative and √ f is √ H θ -Lipschitz with respect to the first argument, then by Proposition 1, the above can be further lower bounded by:

Notice that if we write ˜ x i = ( η 1 ,i , ..., η k,i , G i ) , then (˜ x i , y i ) are independent with distribution exactly equal to ˜ D . Moreover, we have

and it is easy to see that the joint distribution of ( ⟨ ϕ ( w ( θ )) , ˜ x ⟩ , y ) with (˜ x, y ) ∼ ˜ D is exactly the same as ( ⟨ w ( θ ) , x ⟩ , y ) with ( x, y ) ∼ D . As a result, we have that

By our assumption (63), (64) and a union bound, we have with probability at least 1 -δ/ 2

|⟨

Σ

1

/

2

PH,w

(

θ

)

⟩| ≤

C

δ

(

w

(

θ

))

Therefore, if f is M θ -Lipschitz, then by by Lemma 9, we have

Consequently, by taking F ( θ ) = 1 1+ ϵ δ L ( θ ) -M θ √ C δ ( w ( θ )) 2 n and Lemma 7, we have shown that with probability at least 1 -δ , we have

If √ f is √ H θ -Lipschitz, then by Lemma 8

Consequently, by taking F ( θ ) = (√ L ( θ ) 1+ ϵ δ -√ H θ C δ ( w ( θ )) 2 n ) 2 + and Lemma 7, we have shown that with probability at least 1 -δ , we have

Rearranging, either we have

or we have

In either case, the desired bound holds.

Finally, we are ready to prove Theorem 6. In the version below, we also provide uniform convergence guarantee (with sharp constant) for Lipschitz loss.

Theorem 14. Suppose that assumptions (A), (B), (E) and (F) hold. For any δ ∈ (0 , 1) , let C δ : R d → [0 , ∞ ] be a continuous function such that with probability at least 1 -δ/ 4 over x ∼ N (0 , Σ) , uniformly over all θ ∈ Θ ,

Then it holds that

(i) if for each θ ∈ Θ and y ∈ Y , f is M θ -Lipschitz with respect to the first argument and M θ is continuous in θ , then with probability at least 1 -δ , it holds that uniformly over all θ ∈ Θ , we have

(ii) if for each θ ∈ Θ and y ∈ Y , f is non-negative and √ f is √ H θ -Lipschitz with respect to the first argument, and H θ is continuous in θ , then with probability at least 1 -δ , it holds that uniformly over all θ ∈ Θ , we have

Proof. We apply the reduction argument at the beginning of the appendix. Given D that satisfies assumptions (A) and (B), we define [ ˜ w ∗ 1 , ..., ˜ w ∗ k ] = ˜ W = W ( W T Σ W ) -1 / 2 and ˜ f, ˜ g as in (54) and (55). For { ( x i , y i ) } n i =1 sampled independently from D , we observe that the joint distribution of ( x i -µ, y i ) can also be described by D ′ as follows:

Indeed, we can check that

Moreover, by construction, we have

and D ′ satisfies assumptions (A) and (B) with µ = 0 and orthonormal Σ 1 / 2 ˜ w ∗ 1 , ..., Σ 1 / 2 ˜ w ∗ 1 and falls into the setting in Lemma 6. We see that f being Lipschitz or square-root Lipschitz is equivalent to

˜ f being Lipschitz or square-root Lipschitz. It remains to check assumptions (63) and (64) and then apply Lemma 10. Observe that

To check that (63) holds, observe that ⟨ Σ 1 / 2 PH,w ⟩ has the same distribution as ⟨ Qw,x ⟩ . To check that (64) holds, we will apply Theorem 13. Note that the joint distribution of ( ⟨ ϕ ( w ( θ )) , ˜ x ⟩ , ˜ y ) with (˜ x, ˜ y ) ∼ ˜ D is exactly the same as ( ⟨ w ( θ ) , x ⟩ , y ) with ( x, y ) ∼ D ′ and so

Therefore, the assumption (E) is equivalent to the hypercontractivity condition in Theorem 13. Note that { ( x, y ) ↦→ ✶ { ˜ f ( ⟨ ϕ ( w ( θ )) , x ⟩ , y, θ ) > t } : ( θ, t ) ∈ Θ × R } is a subclass of { ( x, y ) ↦→ ✶ { f ( ⟨ w,x ⟩ + b, y, θ ) > t } : ( w,b, t, θ ) ∈ R k +1 × R × R × Θ } . Therefore, by assumption (F), we can apply Theorem 13 and (64) holds.

D Norm Bounds

The following lemma is a version of Lemma 7 of Koehler et al. (2021) and follows straightforwardly from CGMT (Theorem 7), though it requires a slightly different truncation argument compared to the proof Theorem 6. For simplicity, we won't repeat the proof here and simply use it for our applications.

Lemma 11 (Koehler et al. 2021, Lemma 7) . Let Z : n × d be a matrix with i.i.d. N (0 , 1) entries and suppose G ∼ N (0 , I n ) and H ∼ N (0 , I d ) are independent of Z and each other. Fix an arbitrary norm ∥ · ∥ , any covariance matrix Σ , and any non-random vector ξ ∈ R n , consider the Primary Optimization (PO) problem:

and the Auxiliary Optimization (AO) problem:

Then for any t ∈ R , it holds that

The next lemma analyzes the AO in Lemma 11. Our proof closely follows Lemma 8 of Koehler et al. 2021, but we don't make assumptions on ξ yet to allow more applications.

Lemma 12. Let Z : n × d be a matrix with i.i.d. N (0 , 1) entries. Fix any δ > 0 , covariance matrix Σ and non-random vector ξ ∈ R n , then there exists ϵ ≲ log(1 /δ ) ( 1 n + 1 √ R (Σ) + n R (Σ) ) such that with probability at least 1 -δ , it holds that

Proof. By a union bound, there exists a constant C > 0 such that the following events occur together with probability at least 1 -δ/ 2 :

Since ⟨ G,ξ ⟩ ∼ N (0 , ∥ ξ ∥ 2 2 ) , by the standard Gaussian tail bound Pr( | Z | ≥ t ) ≤ 2 e -t 2 / 2 , we have

Using subexponential Bernstein's inequality (Theorem 2.8.1 of Vershynin (2018)), requiring n = Ω(log(1 /δ )) , we have

Using the first part of Lemma 4, we have

Using the last part of Lemma 4, requiring R (Σ) ≳ log(32 /δ ) 2

Therefore, by the AM-GM inequality, it holds that

To apply lemma 11, we will consider w of the form w = α Σ 1 / 2 H ∥ Σ 1 / 2 H ∥ 2 for some α > 0 . Then we have

and

So it suffices to choose α such that

and we are done.

A challenge for analyzing the minimal norm to interpolate is that the projection matrix Q is not necessarily an orthogonal projection. However, the following lemma suggests that if Σ ⊥ = Q T Σ Q has high effective rank, then we can let R be the orthogonal projection matrix onto the image of Q and R Σ R is approximately the same as Σ ⊥ in terms of the quantities that are relevant to the norm analysis.

Lemma 13. Consider Q = I -∑ k i =1 w ∗ i ( w ∗ i ) T Σ where Σ 1 / 2 w ∗ 1 , ..., Σ 1 / 2 w ∗ k are orthonormal and we let R be the orthogonal projection matrix onto the image of Q . Then it holds that rank( R ) = d -k and

Moreover, we have QR = R and RQ = Q , and so

Proof. It is obvious that rank( R ) = rank( Q ) and by the rank-nullity theorem, it suffices to show the nullity of Q is k . To this end, we observe that

It is also straightforward to verify that Q 2 = Q and Q T Σ w ∗ i = 0 for i = 1 , ..., k . For any v ∈ R d , Rv lies in the image of Q and so there exists w such that Rv = Qw . Then we can check that

Since the choice of v is arbitrary, it must be the case that R Σ w ∗ i = 0 and QR = R . For any v ∈ R d , we can check

by the definition of orthogonal projection. Therefore, it must be the case that RQ = Q . Finally, we use R = QR = RQ T to show that

and

Rearranging concludes the proof.

D.1 Phase Retrieval

Theorem 2. Under assumptions (A) and (B), let f : R ×Y → R be given by f (ˆ y, y ) := ( | ˆ y | -y ) 2 with Y = R ≥ 0 . Let Q be the same as in Theorem 1 and Σ ⊥ = Q T Σ Q . Fix any w ♯ ∈ R d such that Qw ♯ = 0 and for some ρ ∈ (0 , 1) , it holds that

and

Then with probability at least 1 -δ , for some ϵ ≲ ρ +log ( 1 δ ) ( 1 √ n + 1 √ R (Σ ⊥ ) + k n + n R (Σ ⊥ ) ) , it holds that

Proof. Without loss of generality, we assume that µ lies in the span of { Σ w ∗ 1 , ..., Σ w ∗ k } because otherwise we can simply increase k by one. Moreover, we can assume that { Σ 1 / 2 w ∗ 1 , ..., Σ 1 / 2 w ∗ k } are orthonormal because otherwise we let ˜ W = W ( W T Σ W ) -1 and conditioning on W T ( x -µ ) is the same as conditioning on ˜ W T ( x -µ ) . By Lemma 5, conditioned on

the distribution of X is the same as

where Z has i.i.d. standard normal entries. Furthermore, conditioned on W T ( x -µ ) and the noise of variable in y (which is independent of x ), by the multi-index assumption (B), the label y is non-random. Since Qw ♯ = 0 , we have w ♯ = ∑ k i =1 ⟨ w ∗ i , Σ w ♯ ⟩ w ∗ i and so

Therefore, ⟨ w ♯ , x ⟩ also becomes non-random after conditioning. We can let I = { i ∈ [ n ] : ⟨ w ♯ , x i ⟩ ≥ 0 } and define ξ ∈ R n by

and ξ is non-random after conditioning. Following the construction discussed in the main text, for any w ♯ ∈ R d , the predictor w = w ♯ + w ⊥ satisfies |⟨ w,x i ⟩| = y i where

by the definition of ξ . Hence, we have

and it suffices to control ∥ w ⊥ ∥ 2 .

Let R be the orthogonal projection matrix onto the image of Q and we consider w of the form Rw to upper bound ∥ w ⊥ ∥ 2 . By Lemma 13, we know QR = R and R Σ w ∗ i = 0 . By the assumption that µ lies in the span of { Σ w ∗ 1 , ..., Σ w ∗ k } , we have

Since R is an orthogonal projection, it holds that ∥ Rw ∥ 2 ≤ ∥ w ∥ 2 . Finally, we observe that the distribution of Z Σ 1 / 2 R is the same as Z ( R Σ R ) 1 / 2 and so

We are now ready to apply Lemma 12 to the covariance R Σ R . We are allowed to replace the dependence on R Σ R by the dependence on Σ ⊥ by the last two inequalities of Lemma 13. The desired conclusion follows by the observation that ∥ ξ ∥ 2 2 = n ˆ L f ( w ♯ ) and the assumption that ˆ L f ( w ♯ ) ≤ (1 + ρ ) L f ( w ♯ ) .

D.2 ReLU Regression

The proof of Theorem 3 will closely follow the proof of Theorem 2.

Theorem 3. Under assumptions (A) and (B), let f : R ×Y → R be the loss defined in (13) with Y = R ≥ 0 . Let Q be the same as in Theorem 1 and Σ ⊥ = Q T Σ Q . Fix any ( w ♯ , b ♯ ) ∈ R d +1 such that Qw ♯ = 0 and for some ρ ∈ (0 , 1) , it holds that

Then with probability at least 1 -δ , for some ϵ ≲ ρ +log ( 1 δ ) ( 1 √ n + 1 √ R (Σ ⊥ ) + k n + n R (Σ ⊥ ) ) , it holds that

Proof. We let I = { i ∈ [ n ] : y i > 0 } and for any ( w ♯ , b ♯ ) ∈ R d +1 , we define ξ ∈ R n by

By the definition of ξ , the predictor ( w,b ) = ( w ♯ + w ⊥ , b ♯ ) satisfies σ ( ⟨ w,x i ⟩ + b ) = y i where

and it suffices to control ∥ w ⊥ ∥ 2 .

Similar to the proof of Theorem 2, we make the simplifying assumption that µ lies in the span of { Σ w ∗ 1 , ..., Σ w ∗ k } and { Σ 1 / 2 w ∗ 1 , ..., Σ 1 / 2 w ∗ k } are orthonormal. Conditioned on W T ( x i -µ ) and the noise variable in y i , both y i and ⟨ w ♯ , x i ⟩ are non-random, and so ξ is also non-random. The distribution of X is the same as

If we consider w of the form Rw , then we have

We are now ready to apply Lemma 12 to the covariance R Σ R . We are allowed to replace the dependence on R Σ R by the dependence on Σ ⊥ by the last two inequalities of Lemma 13. The desired conclusion follows by the observation that ∥ ξ ∥ 2 2 = n ˆ L f ( w ♯ , b ♯ ) due to the definition (13) and the assumption that ˆ L f ( w ♯ ) ≤ (1 + ρ ) L f ( w ♯ , b ♯ ) .

D.3 Low-rank Matrix Sensing

Theorem 4. Suppose that d 1 d 2 > n , then there exists some ϵ ≲ √ log(32 /δ ) n + n d 1 d 2 such that with probability at least 1 -δ , it holds that

Hence, we have

Proof. Without loss of generality, we will assume that d 1 ≤ d 2 . We will vectorize the measurement matrices and estimator A 1 , ..., A n , X ∈ R d 1 × d 2 as a 1 , ..., a n , x ∈ R d 1 d 2 and define ∥ x ∥ ∗ = ∥ X ∥ ∗ . Denote A = [ a 1 , ..., a n ] T ∈ R n × d 1 d 2 . We define the primary problem Φ by

By Lemma 11, it suffices to consider the auxiliary problem

We will pick x of the form x = -αH for some α ≥ 0 , which needs to satisfy α ∥ H ∥ 2 2 ≥ ∥ αG ∥ H ∥ 2 -ξ ∥ 2 . By a union bound, the following events occur simultaneously with probability at least 1 -δ/ 2 :

by Lemma 3, it holds that

Condition on ξ , we have 1 ∥ ξ ∥ ⟨ G,ξ ⟩ ∼ N (0 , 1) and so by standard Gaussian tail bound -t 2 / 2

Then we can use AM-GM inequality to show for sufficiently large n

and it suffices to let

Rearranging the above inequality, we can choose

and since H as a matrix can have at most rank d 1 , by Cauchy-Schwarz inequality on the singular values of H , we have ∥ H ∥ ∗ ≤ √ d 1 ∥ H ∥ 2 and

for some ϵ ≲ √ log(32 /δ ) n + n d 1 d 2 . The desired conclusion follows by the observation that ∥ X ∗ ∥ ∗ ≤ √ r ∥ X ∗ ∥ F because X ∗ has rank r .

Theorem 5. Fix any δ ∈ (0 , 1) . There exist constants c 1 , c 2 , c 3 > 0 such that if d 1 d 2 > c 1 n , d 2 > c 2 d 1 , n > c 3 r ( d 1 + d 2 ) , then with probability at least 1 -δ that

Proof. Note that ⟨ A,X ∗ ⟩ ∼ N (0 , ∥ X ∗ ∥ 2 F ) and so by the standard Gaussian tail bound Pr( | Z | ≥ t ) ≤ 2 e -t 2 / 2 , Theorem 9 and a union bound, it holds with probability at least 1 -δ/ 8 that

Then it holds that

Therefore, we can choose C δ in Theorem 1 by

and applying Theorem 1 and Theorem 4, we have

where ϵ is the maximum of the two ϵ in Theorem 1 and Theorem 4. Finally, recall that

Assuming that d 1 ≤ d 2 , then the above implies that

and we are done.

E Counterexample to Gaussian Universality

By assumption (G), we can write x i | d -k = h ( x i | k ) · Σ 1 / 2 | d -k z i where z i ∼ N (0 , I d -k ) . We will denote the matrix Z = [ z 1 , ..., z n ] T ∈ R n × ( d -k ) . Following the notation in section 7, we will also write X = [ X | k , X | d -k ] where X | k ∈ R n × k and X | d -k ∈ R n × ( d -k ) . The proofs in this section closely follows the proof of Theorem 6.

Theorem 15. Consider dataset ( X,Y ) drawn i.i.d. from the data distribution D according to (G) and (H), and fix any f : R ×Y → R ≥ 0 such that √ f is 1-Lipschitz for any y ∈ Y . Fix any δ > 0 and suppose there exists ϵ δ < 1 and C δ : R d -k → [0 , ∞ ] such that

(i) with probability at least 1 -δ/ 2 over ( X,Y ) and G ∼ N (0 , I n ) , it holds uniformly over all w | k ∈ R k and ∥ w | d -k ∥ Σ | d -k ∈ R ≥ 0 that

(ii) with probability at least 1 -δ/ 2 over z | d -k ∼ N (0 , Σ | d -k ) , it holds uniformly over all w | d -k ∈ R d -k that

then with probability at least 1 -δ , it holds uniformly over all w ∈ R d that

Proof. Note that

and so for any f : R ×Y × R k → R , we can write

By the same truncation argument used in Lemma 7, it suffices to consider the auxiliary problem:

Therefore, it holds that

Next, we analyze the infimum term:

Now suppose that f takes the form f (ˆ y, y, x | k ) = 1 h ( x | k ) 2 ˜ f (ˆ y, y ) for some 1 square-root Lipschitz ˜ f and by a union bound, it holds with probability at least 1 -δ that

where we apply Lemma 8 in the last step. Then if we take

then we have Ψ ≤ 0 . To summarize, we have shown

Theorem 16. Under assumptions (G) and (H), fix any w ∗ | k ∈ R k and suppose for some ρ ∈ (0 , 1) , it holds with probability at least 1 -δ/ 8

Then with probability at least 1 -δ , for some ϵ ≲ ρ +log ( 1 δ ) ( 1 √ n + 1 √ R (Σ | d -k ) + n R (Σ | d -k ) ) , it holds that

Proof. Fix any w ∗ | k ∈ R k , we observe that

Therefore, it is enough analyze

By introducing the Lagrangian, we have

Similarly, the above is only random in Z after conditioning on X | k w ∗ | k and ξ and the distribution of Z remains unchanged after conditioning because of the independence. By the same truncation argument as before and CGMT, it suffices to consider the auxiliary problem:

and so we can define

To upper bound Ψ , we consider w | d -k of the form -α Σ 1 / 2 | d -k H ∥ Σ 1 / 2 | d -k H ∥ 2 , then we just need

By a union bound, the following occur together with probability at least 1 -δ/ 2 for some absolute constant C > 0 :

Using the first part of Lemma 4, we have

Using the last part of Lemma 4, requiring R (Σ | d -k ) ≳ log(32 /δ ) 2

Using subexponential Bernstein's inequality (Theorem 2.8.1 of Vershynin (2018)), requiring n = Ω(log(1 /δ )) ,

Using standard Gaussian tail bound Pr( | Z | ≥ t ) ≤ 2 e -t 2 / 2 , we have

By assumption, it holds that

Then we use the above and the AM-GM inequality to show that

After some rearrangements, it is easy to see that we can choose

and the proof is complete.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops uniform convergence result for empirical risk minimization with a loss function whose square root is Lipschitz. It is assumed that the covariate vector $x$ is drawn from a $d$-dimensional multivariate normal and the response $y$ is generated by a structural equation that only depends on $x$ through its $k$-dimensional projection. The result bounds the root of population risk in terms of the root of the empirical risk and another term concerning the Lipschitz constant, a complexity measure and the sample size. The bound holds uniformly (in high probability) over all possible affine coefficients in the empirical risk minimization. This key result is then applied to study \"benign overfitting\" in several problems, including phase retrieval, ReLU regression and matrix sensing. In addition, it is also shown that Gaussian universality does not hold for the setting considered.\n\nSTRENGTHS:\n1. The paper builds upon and generalizes related results in the literature. \n2. It establishes a key generic result and then illustrates its strength through application to several topical problems. \n3. The presentation is rigorous and clear.\n\nWEAKNESSES:\nAs this paper falls out of my area of expertise, I am only listing below a few points that might help me (or someone from outside the field) to better understand the paper. \n\n1. The notion of \"consistency\" or a \"consistent loss\" is mentioned in the Introduction and throughout the applications. It is worth explaining what that means and how it relates the usual notion of \"consistency\" in statistics. \n2. Assumption (C) perhaps deserves more explanation: why is it necessary and when is it expected to hold?\n\nQUESTIONS:\n1. Page 4, line 135: I do not see why the distribution of $\\langle w, x \\rangle$ only depends on $w^{T} \\Sigma W$. By $x \\sim \\mathcal{N}(\\mu, \\Sigma)$ as in Assumption (A), wouldn't it depend on $w^{T} \\mu$ and $w^T \\Sigma w$?\n2. Page 2, line 76: missing \"be\" before \"applied\".\n\nLIMITATIONS:\nI do not foresee any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper introduces sharp uniform convergence guarantees for generalized linear models in Gaussian space  for square-root Lipschitz losses. These results extend the scope of previous findings and open up possibilities for applying the derived loss bounds in new contexts.

STRENGTHS:
1.The paper is well-written, providing clear explanations of its contributions in relation to previous work.
2.The paper achieves fast convergence rates for a broader class of losses, surpassing previous research in this area.
3.By simplifying the assumptions compared to previous work, the paper is able to derive new bounds applicable to scenarios where solving the Moreau envelope of the loss function is challenging or not possible in closed-form.

WEAKNESSES:
While the authors discuss potential extensions of the Gaussian feature assumption,  the paper still focuses on the setting of Gaussian data, similar to previous work. Additionally, the proof technique employed in the paper is not entirely novel, as it follows prior works that utilize the Gaussian Minimax Theorem for establishing uniform convergence.

QUESTIONS:
See weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates optimistic rates under square-root Lipschitz losses and Gaussian data. Applications to phase retrieval, ReLu regression, matrix sensing, and single-index NNs are given.\n\nSTRENGTHS:\nThe paper extends the analysis of optimistic rates to square-root Lipschitz losses, which goes beyond the classical smoothness assumption, which is interesting.\n\nWEAKNESSES:\nThe authors claim that the obtained results provide a better understanding of benign overfitting, but the paper needs more explanations and comments on the main results.\n\nAs an example, the bound of eq. (10) features ten distinct terms. It would be nice to discuss which terms refer to what and how they compare with existing terms.\n\nAnother aspect I need clarification on is the notion of uniform convergence. In traditional statical learning, uniform convergence controls the rate of decay of |R(f) - \\hat{R}_n(f)| for all f \\in F and every distribution in a given class. On the other hand, the rate in Th.1 feature (1-\\varepsilon) R(f) on the LHS. I am unfamiliar with benign overfitting literature (which could be standard). However, I invite the authors to add a discussion on this to broaden the paper's audience.\n\nAs for the assumption, I think the Gaussian data is quite restrictive. In modern applications, it is not uncommon to have abundant data but poor quality. In these cases, the data distribution feature tails fatter than any sub-Gaussian (and even sub-Exponential). \n\nAssumption (C) is related to the L_4 norm of the loss. It looks formidable to verify in practice. Even for sub-Gaussian losses, the L_4 would be proportional to \\sqrt{4}*\\sqrt{VAR(loss)}, while the condition requires proportionality to the E[loss].\n\nPlease provide a discussion on these aspects. I may raise my score accordingly.\n\nQUESTIONS:\nPlease provide more discussion on the highlighted weaknesses.\n\nLIMITATIONS:\nAssuming the data are Gaussian is limiting.\n\nPlease provide examples when Assumption (C) is satisfied.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper, the authors extended a type of sharp uniform convergence guarantee for the square loss to any square-root Lipschitz loss. The proof is considered to be simplified compared to ... \n\nIn the paper, the authors extend the theory of optimistic rates, a type of sharp uniform convergence guarantee, to square-root-Lipschitz losses, enabling new applications like phase retrieval. The authors have shown the uniform convergence for the multi-index model with Gaussian feature when the loss function is non-negative, square-root Lipshichtz and satisfies hypercontractivity. The convergence is controlled by the Radamacher complexity of the hypothesis class and the square-root Lipshichitz constant. The authors then show how this result can be applied to applications including phase retrieval, ReLU regression, matrix sensing, and single-index neural networks. \n\nCompared to other loss function classes being considered in the literature for uniform convergence, square-root Lipshichtz loss is more general that it includes certain nonsmooth non-convex functions, and provides a better intuition of where the square root comparison between loss functions shows up. \n\nThe author also provides a counter-example to argue that Gaussian universality cannot always be taken for granted, and why the result can be over-optimistic for non-Gaussian data.\n\nSTRENGTHS:\n1. This paper goes beyond Lipschitz and smooth loss functions and extends the analysis to square-root-Lipschitz losses. This extension is valuable as it includes square loss and can address nonsmoothness. Also, it is easier to verify compared to the Moreau envelope condition used in [Zhou et al. 2022]. As the authors have shown in section 5, their results can be applied to show the benign overfitting for a wide range of real-world scenarios, yielding consistent results compared to past literature. \n\n2. The concept of square-root Lipschitz losses is clean to state, and it indeed provides good intuition for the square-root relationship between losses shown up in previous optimistic rates analysis ([Zhou et al. 2020]). As a result, this new loss function class does seem to capture some intrinsic principle of the problem of optimistic rate and does contribute intuitions for the community. This expansion of applicable loss functions enhances the paper’s relevance and potential impact. \n\n3. Application in Neural Networks: The paper’s extension of norm-based bounds and optimistic rates to weight-tied neural networks contributes to the field of neural network research. By providing a generalization bound that can be combined with the algorithmic outputs of Bietti et al., the authors offer practical insights for optimizing and understanding the non-convex optimization of these network architectures.\n\nWEAKNESSES:\n1. The major concern of the paper is that the technical contribution of the paper is limited. It is mostly built upon previous optimistic rate results ([Zhou et al. 2021, Zhou et al. 2022]). Upon reading the appendix and related literature, the reviewer is on the side that the paper serves as a simplification and rewritten of some past results (except the fact that there is a generalization of previous results by including an extra (network) parameter $\\theta$). Still, the generalization results over general square-root Lipschitz loss is interesting, but need to note that there might not be enough technical contribution in the paper for the main theorem. The norm bounds in the applications are novel. \n\n2. The paper itself is not presented in a way to highlight its contribution either. The paper has done a good job for the literature review, and it has spent more space introducing the results instead of providing intuitions for why the results hold true. This can be understandable as the space for the main submission is rather limited and there are many theorems to be stated in a single paper. Still, the reviewer thinks it can be beneficial in spending a bit more space on proof outlines. \n\n3. There are some other limitations of the work, including the Gaussian feature assumption and the multi-index model for neural networks. Gaussian feature assumption can be served as a good starting assumption for the theory and the authors have addressed the limitation of this assumption appropriately. For neural networks, the multi-index model is a limited shallow neural network. In general, the Rademacher complexity for deep neural networks is normally only a loose estimation and thus results in unrealistic generalization bounds. Still, the multi-index model has been studied extensively recently, so the concern for the limitation here is just a minor one.\n\nQUESTIONS:\n1. How should I think about the technical contribution of the paper? Specifically, how should I consider the difference between the proof here for the square-root Lipschitz functions and the proof for a general Moreau envelope? \n\n2. The generalization results over single-index model is interesting, where you have shown that $\\max_{j \\in [N]} |\\sum_{i = 1}^j a_i| || w ||$ is a good complexity measure. Potentially combined with known results in the single index model, how large should this quantity be? In some way, I think this can be considered as future work to achieve end-to-end results for the weight-tied neural networks so should be fine if there is no clear answer here. Just trying to understand if the generalization bound we achieve here is tight in some sense.\n\nLIMITATIONS:\nThe authors have addressed the limitation of the paper appropriately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, generic uniform convergence guarantees are provided for Gaussian data in terms of the Rademacher complexity of the hypothesis class and the Lipschitz constant of the square root of the scalar loss function. Square-Root Lipschitz Loss is an important class of loss function to study because it is a suitable loss to study interpolation learning in cases like phase retrieval and matrix sensing. The authors obtain an optimistic rate in theorem 1 of the paper and the analysis is mostly inspired by the previous work of  Zhou et al. 2022. Then the authors discuss multiple use cases and provide generalization error bound for over-parametrized phase retrieval, matrix sensing, and single index weight tied neural network.\n\nSTRENGTHS:\nThe paper is technically a good paper. Though I am not an expert in this area but it looks to me like, this paper modified the analysis in the previous recent work by Zhou et al. 2022 to obtain new overfitting results.\n\nWEAKNESSES:\nI have a few questions and some might seem very trivial and please correct me if I am wrong.\n1. I can see that the paper utilizes the Gaussian minimax theorem that is explained in Appendix B.2. However, this still seems like a very restricted setting to me. What happens when one assumes that data is coming from a more general distribution?  Previous results (though not applicable for square roor Lipschitz loss) have no major distributional assumption. \n2. I am also curious to know how realistic is overparametrization setting in phase retrieval.\n3. I am also curious to understand the results of the single index neural network model as the paper cited in this work Biettie et al and other related papers in the domain seem to have the stronger guarantee of recovering the direction using spherical SGD or gf. In this paper, the guarantees are for generalization errors. Is there a way to compare these two results as the results in the other papers where there recover the optimal direction sounds stronger to me? \n\nAs a minor comment, it would also be great if the authors can discuss and compare the results in section 5 and 6 with previously known results. A discussion would be very helpful.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}