Title: Precise asymptotic generalization for multiclass classification with overparameterized linear models
Venue: NeurIPS 2023
Accepted: yes
Keywords: overparameterized

multiclass

classification

theory

generalization

interpolation

bi-level

Gaussian model
Abstract: We study the asymptotic generalization of an overparameterized linear model for multiclass classification under the Gaussian covariates bi-level model introduced in Subramanian et al. (NeurIPS'22), where the number of data points, features, and classes all grow together. We fully resolve the conjecture posed in Subramanian et al. '22, matching the predicted regimes for which the model does and does not generalize. Furthermore, our new lower bounds are akin to an information-theoretic strong converse: they establish that the misclassification rate goes to 0 or 1 asymptotically. One surprising consequence of our tight results is that the min-norm interpolating classifier can be asymptotically suboptimal relative to noninterpolating classifiers in the regime where the min-norm interpolating regressor is known to be optimal. 

The key to our tight analysis is a new variant of the Hanson-Wright inequality which is broadly useful for multiclass problems with sparse labels. As an application, we show that the same type of analysis can be used to analyze the related multi-label classification problem under the same bi-level ensemble.

Precise Asymptotic Generalization for Multiclass Classification with Overparameterized Linear Models

Abstract

We study the asymptotic generalization of an overparameterized linear model for multiclass classification under the Gaussian covariates bi-level model introduced in Subramanian et al. (2022), where the number of data points, features, and classes all grow together. We fully resolve the conjecture posed in Subramanian et al. (2022), matching the predicted regimes for generalization. Furthermore, our new lower bounds are akin to an information-theoretic strong converse: they establish that the misclassification rate goes to 0 or 1 asymptotically. One surprising consequence of our tight results is that the min-norm interpolating classifier can be asymptotically suboptimal relative to noninterpolating classifiers in the regime where the min-norm interpolating regressor is known to be optimal.

The key to our tight analysis is a new variant of the Hanson-Wright inequality which is broadly useful for multiclass problems with sparse labels. As an application, we show that the same type of analysis can be used to analyze the related multilabel classification problem under the same bi-level ensemble.

1 Introduction

In this paper, we directly follow up on a specific line of work initiated by Subramanian et al. (2022); Wu and Sahai (2023). For the sake of self-containedness, we briefly reiterate the context, directing the reader to Subramanian et al. (2022) and the references cited therein for more. A broader story can be found in Bartlett et al. (2021); Belkin (2021); Dar et al. (2021); Oneto et al. (2023).

Classical statistical learning theory intuition predicts that highly expressive models, which can interpolate random labels (Zhang et al., 2016; 2021), ought not to generalize well. However, deep learning practice has seen such models performing well when trained with good labels. Resolving this apparent contradiction has recently been the focus of a multitude of works, and this paper builds on one particular thread of investigation that can be rooted in Bartlett et al. (2020); Muthukumar et al. (2020) where the concept of benign/harmless interpolation was crystallized in the context of overparameterized linear regression problems and conditions given for when this can happen. In Muthukumar et al. (2021), a specific toy "bi-level model" with Gaussian features was introduced to study overparameterized binary classification and show that successful generalization could happen even beyond the conditions for benign interpolation for regression. Following the introduction of the corresponding multi-class problem in Wang et al. (2021) with a constant number of classes, an asymptotic setting where the number of classes can grow with the number of training examples was introduced in Subramanian et al. (2022) where a conjecture was presented for when minimum-norm interpolating classifiers will generalize. We are now in a position to state our main contributions; afterwards, we expand on the related works.

Our contributions

Our main contribution is crisply identifying the asymptotic regimes where an overparameterized linear model which performs minimum-norm interpolation does and does not generalize for multiclass classification under a Gaussian features assumption, thus resolving the main conjecture posed by (Subramanian et al., 2022). We improve on the analysis of Subramanian et al. (2022); Wu and Sahai (2023), covering all regimes with the asymptotically optimal misclassification rate. When the model generalizes, it does so with a misclassification rate o (1) , and we show a matching "strong converse" establishing when it misclassifies, it does so with rate 1 -o (1) , where the explicit rate is nearly identical to that of random guessing. The critical component of our analysis is a new variant of the Hanson-Wright inequality, which applies to bilinear forms between a vector with subgaussian entries and a vector that is bounded and has soft sparsity , a notion we will define in Section 4.2. We show how this tool can be used to analyze other multiclass problems, such as multilabel classification.

1.1 Brief treatment of related work

Our thread begins with a recent line of work that analyzes the generalization behavior of overparameterized linear models for regression (Hastie et al., 2022; Mei and Montanari, 2022; Bartlett et al., 2020; Belkin et al., 2020; Muthukumar et al., 2020). These simple models demonstrate how the capacity to interpolate noise can actually aid in generalization: training noise can be harmlessly absorbed by the overparameterized model without contaminating predictions on test points. In effect, extra features can be regularizing (in the context of descent algorithms' implicit regularization (Soudry et al., 2018; Ji and Telgarsky, 2019; Engl et al., 1996; Gunasekar et al., 2018)), but an excessive amount of such regularization causes regression to fail because even the true signal will not survive the training process. Although works in this thread focus on very shallow networks, Chatterji and Long (2023) established that deeper networks can behave similarly. Note that recently, Mallinar et al. (2022) called-out an alternative regime (behaving like 1-nearest-neighbor learning) called "tempered" overfitting in which training noise is not completely absorbed but the true signal does survive training.

The thread continues in a line of work that studies binary classification (Muthukumar et al., 2021; Chatterji and Long, 2021; Wang and Thrampoulidis, 2021) in similar overparameterized linear models. While confirming that the basic story is similar to regression, these works identify a further surprise: binary classification can work in some regimes where the corresponding regression problem would not work 1 due to the regularizing effect of overparameterization being too strong. Just as in the regression case, the results here are sharp in toy models: we can exactly characterize where binary classification using an interpolating classifier asymptotically generalizes.

With binary classification better understood, the thread continues to multiclass classification. After all, the current wave of deep learning enthusiasm originated in breakthrough performance in multiclass classification, and we have seen a decade of ever larger networks trained on ever larger datasets with ever more classes Kaplan et al. (2020). Using similar toy models (Muthukumar et al., 2020; 2021; Wang et al., 2022), the constant number of classes case was studied in Wang et al. (2021) to recover results similar to binary classification and subsequently generalized to general convex losses with regularization in Loureiro et al. (2021) and student-teacher networks in Cornacchia et al. (2023).

Subramanian et al. (2022) further introduced a model where the number of classes grows with the number of training points and proved an achievability result on how fast the number of classes can grow while still allowing the interpolating classifier to asymptotically generalize. While Subramanian et al. (2022) gave a conjecture for what the full region should be, there was no converse proof, and they could not show generalization in entire conjectured region. Wu and Sahai (2023) proved a partial weak converse; they showed that the misclassification rate is bounded away from 0 -rather than tending to 1 -in some of the predicted regimes.

The model, formally defined in the following section, is a stylized version of the well-known spiked covariance model (Johnstone, 2001; Donoho et al., 2018). On the theoretical front, it is related to several problems such as PCA variants (Montanari and Richard, 2015; Richard and Montanari, 2014)

and community detection in the stochastic block model (Abbe, 2017). These models have also been applied in practice for climate studies and functional data analysis (Johnstone, 2001). At a high level, spiked covariance models can be interpreted as a linearized version of the manifold hypothesis.

2 Problem setup

The following exposition is lifted from Subramanian et al. (2022), which we include for the sake of staying consistent and self-contained. In Appendix K, we include an alternative high level framing of the problem which readers may find helpful for intuition.

We consider the multiclass classification problem with k classes. The training data consists of n pairs { x i , ℓ i } n i =1 where x i ∈ R d are i.i.d standard Gaussian vectors 2 . We assume that the labels ℓ i ∈ [ k ] are generated as follows.

Assumption 1 (1-sparse noiseless model) . The class labels ℓ i are generated based on which of the first k dimensions of a point x i has the largest value,

Let us emphasize at this point that the classifier that we analyze only observes the training data, and does not use any of the data-generating assumptions.

For a vector x , we index its j th entry with x [ j ] . Hence, under Assumption 1, x i [ m ] can be interpreted as how representative of class m the i th training point is.

For clarity of exposition in the analysis, we make explicit a feature weighting that transforms the training points:

Here λ ∈ R d contains the squared feature weights. The feature weighting serves the role of favoring the true pattern, something that is essential for good generalization. Again, we emphasize that the classifier does not do any reweighting of features; this explicit step is purely syntactic. 3

The weighted feature matrix X w ∈ R n × d is given by

where we introduce the notation z j ∈ R n to contain the j th feature from the n training points. Note that z j ∼ N (0 , I n ) are i.i.d Gaussians. We use a one-hot encoding for representing the labels as the matrix Y oh ∈ R n × k

Since we consider linear models, we center the one-hot encodings and define

Our classifier consists of k coefficient vectors ̂ f m for m ∈ [ k ] that are learned by minimum-norm interpolation (MNI) of the zero-mean one-hot variants using the weighted features: 4

We can express these coefficients in closed form as

On a test point x test ∼ N (0 , I d ) we predict a label as follows: First, we transform the test point into the weighted feature space to obtain x w test where x w test [ j ] = √ λ j x test [ j ] for j ∈ [ d ] . Then we compute k scalar 'scores' and assign the class based on the largest score as follows:

By assumption, a misclassification event E err occurs whenever

̸

We study where the MNI generalizes in an asymptotic regime where the number of training points, features, classes, and feature weights all scale according to the bi-level ensemble model 5 :

Definition 1 (Bi-level ensemble) . The bi-level ensemble is parameterized by p, q, r and t where p > 1 , 0 ≤ r < 1 , 0 < q < ( p -r ) and 0 ≤ t < r . Here, parameter p controls the extent of overparameterization, r determines the number of favored features, q controls the weights on favored features and t controls the number of classes. The number of features ( d ), number of favored features ( s ), and number of classes ( k ) all scale with the number of training points ( n ) as follows:

where c k is a positive integer. Define the feature weights by

We introduce the notation λ F ≜ ad s and λ U ≜ (1 -a ) d d -s to distinguish between the (squared) favored and unfavored weights, respectively.

We visualize the bi-level model in Fig. 1, reproduced from Subramanian et al. (2022). Intuitively, the bi-level ensemble captures a simple family of overparameterized problems where learning can succeed. Although there are d = n p features where d ≫ n , there is a low dimensional subspace of favored, higher weight features of dimension s = n r , and s ≪ n . From this perspective, the bi-level model can be viewed as a parameterized version of an approximate linear manifold hypothesis. Depending on the signal strength, the noise added from the d -s unfavored features can either help generalization ('benign overfitting') or overwhelm the true signal and cause the classifier to fail.

3 Main results

In this section we state our main results and compare them to what was known and conjectured previously. Subramanian et al. (2022) use heuristic calculations to conjecture necessary and sufficient conditions for the bi-level model to generalize; we restate the conjecture here for reference.

ad

p-q-r

n

2

Feature

S

weight,√入j

(1-a)d

d-s

d = np

s=nr

Feature index, j

Conjecture 3.1 (Conjectured bi-level regions) . Under the bi-level ensemble model (Definition 1), when the true data generating process is 1-sparse (Assumption 1), as n → ∞ , the probability of misclassification Pr[ E err ] for MNI as described in Eq. (6) satisfies

Our main theorem establishes that Conjecture 3.1 indeed captures the correct generalization behavior of the overparameterized linear model.

Theorem 3.2 (Generalization for bi-level-model) . Under the bi-level ensemble model (Definition 1), when the true data generating process is 1-sparse (Assumption 1), Conjecture 3.1 holds.

For comparison, we quote the best known previous positive and negative results for the bi-level model, which only hold in the restricted regime where regression fails ( q + r > 1 ).

Theorem 3.3 (Generalization for bi-level model (Subramanian et al., 2022)) . In the same setting as Conjecture 3.1, in the regime where regression fails ( q + r > 1 ), as n →∞ we have Pr[ E err ] → 0 if

Theorem 3.4 (Misclassification in bi-level model (Wu and Sahai, 2023)) . In the same setting as Conjecture 3.1, in the regime where regression fails ( q + r > 1 ), as n →∞ we have Pr[ E err ] ≥ 1 2 if

Let us interpret the different conditions in Conjecture 3.1. To interpret the condition t < 1 -r , first rearrange it to t + r < 1 . Recall that the parameter r controls the number of favored features, and hence is a proxy for the 'effective dimension' of the problem. On the other hand, the parameter t controls the number of classes, so in a loose sense there are n t + r parameters being learned. From this perspective, the condition t + r < 1 says that the problem is 'effectively underparameterized'.

The other condition on p +1 -2 max { 1 , q + r } comes from looking at the noise from the unfavored features. To see why, recall that the squared favored feature weighting is λ F = n p -q -r . So for fixed p , the quantity q + r controls the level of favored feature weighting. When q + r > 1 , the favored feature weighting is small enough that regression fails, and the empirical covariance becomes flat. In this case, the condition becomes t < p +1 -2( q + r ) = ( p -q -r ) + 1 -( q + r ) . As q + r increases, the amount of favoring decreases, making it harder to generalize.

For ease of comparison between our main result and Theorems 3.3 and 3.4, we visualize the regimes in Fig. 2, as in Subramanian et al. (2022); Wu and Sahai (2023). In particular, the blue starred and dashed regions in Fig. 2 indicate how Theorem 3.3 only applies where regression fails. In contrast, our new result holds regardless of whether regression fails or not, as in the the green diamond region and light blue triangle regions. The regions are also completely tight; the looseness between the prior Theorem 3.3 and our result can be seen in the light blue square region.

The weak converse in the prior Theorem 3.4 captures some of the correct conditions for misclassification, but again only when q + r > 1 . As depicted in the maroon X region for r < 0 . 25 in Fig. 2b, our main theorem gives a strong converse, whereas Theorem 3.4 has nothing to say because q + r < 1 . Theorem 3.4 also only proves that the misclassification rate is asymptotically at least 1 2 . In the red circle and maroon X regions, we illustrate how our result pushes the misclassification rate to 1 -o (1) , which requires a more refined analysis. We elaborate on this further in Section 4.

We remark that it is simpler to analyze the case where regression fails, as the random matrices that arise in the analysis are flat , i.e. approximately equal to a scaled identity matrix. However, in the regime where regression works, the same matrices have a spiked spectrum, which complicates the analysis. To smoothly handle both cases, we leverage a new variant of the Hanson-Wright inequality to show concentration of certain sparse bilinear forms; see Section 4.1 for more details.

(2a)q=0.75,t=0.00

(2b) q=0.75, p=1.1

(2c) q=0.75,p=2.3

1.0

1.0

0.8

0.8

4

0.6

0.6

Q3

0.4

0.4

2

0.2

0.2

1

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.0

AAA

0.2

0.4

0.6

0.8

1.0

0.0

0.0

A

0.2

0.4

0.6

0.8

1.0

r

r

r

Ereg →1, &binary →0,&multi→0 (Subramanian et al.'22)

Emulti→0 (Theorem3.2)

Ereg →0, Ebinary →0, &multi →0 (Theorem 3.2)

Emulti →0 (Subramanian et al.'22)

&reg →1, &binary →0, &muti →0 (Theorem 3.2)

Emulti →1 (Theorem3.2)

Ereg →1,&binary →1, &multi→1 (Theorem 3.2)

4 Technical overview

We now sketch out the proof for our main theorem. As in Subramanian et al. (2022); Wu and Sahai (2023), the starting point is writing out the necessary and sufficient conditions for misclassification.

̸

Assume without loss of generality that the test point x test ∼ N (0 , I d ) has true label α for some α ∈ [ k ] . Let x w test be the weighted version of this test point. From (10), an equivalent condition for misclassification is that for some β = α, β ∈ [ k ] , we have ̂ f ⊤ α x w test < ̂ f ⊤ β x w test , i.e. the score for β outcompetes the score for α . Define the Gram matrix A ≜ X w ( X w ) ⊤ , the relative label vector ∆ y ≜ y α -y β ∈ {-1 , 0 , 1 } n , and the relative survival vector ̂ h α,β ∈ R d which compares the signal from α and β :

where to obtain the last line we have used (8). By converting the misclassification condition into the unweighted feature space we see that we will have errors when

Define the contamination term CN α,β :

Note that CN α,β normalizes the RHS of (18) into a standard Gaussian. Indeed, define

̸

Since α, β ∈ [ k ] are favored, we have λ α = λ β = λ F . Hence, an equivalent condition for misclassification is that there exists some β = α , β ∈ [ k ] such that

We now translate the above criterion into sufficient conditions for correct classification and misclassification and analyze these two cases separately.

̸

Correct classification: For correct classification, it suffices for the minimum value of the LHS of Eq. (21) to outcompete the maximum value of the RHS, where the max is taken over β ∈ [ k ] , β = α . Some algebra, as in Subramanian et al. (2022), shows that we correctly classify if

We will show that under the conditions specified in Conjecture 3.1, with high probability, the relevant survival to contamination ratio SU / CN grows at a polynomial rate n v for some v > 0 , whereas the term in the parentheses shrinks at a subpolynomial rate ω ( n -δ ) for any δ > 0 . Further, by standard subgaussian maximal inequalities, the magnitudes of the normalized contamination is no more than O ( √ log( nk )) with high probability. Thus, with high probability the LHS outcompetes the RHS, leading to correct classification. See Section 4.1 for more discussion on how we prove tight bounds on the survival-to-contamination ratios.

Misclassification: On the other hand, for misclassification it suffices for the maximum absolute value of the LHS of Eq. (21) to be outcompeted by the maximum value of the RHS. Some manipulations yield the following sufficient condition for misclassification:

̸

We show that within the misclassification regimes in Conjecture 3.1, the survival-to-contamination ratio SU / CN shrinks at a polynomial rate n -w for some w > 0 . By standard subgaussian maximal inequalities, the largest label-defining feature is O ( √ log( nk )) with high probability. Gaussian anticoncentration implies that for some β = α, β ∈ [ k ] , Z ( β ) outcompetes the LHS with probability at least 1 2 -o (1) . Hence, we conclude that the model will misclassify with rate at least 1 2 asymptotically.

Let us now describe how to boost the misclassification rate to 1 -o (1) . Notice that the above argument only considered the competition between the LHS of Eq. (23) and one of the Z ( β ) 's on the RHS instead of the maximum Z ( β ) . It's not hard to see from the definition of Z ( β ) in Eq. (20) that the Z ( β ) are jointly Gaussian. For intuition's sake, assuming the Z ( β ) were independent , then max β Z ( β ) would outcompete with probability ( 1 2 -o (1)) k -1 .

In reality, the Z ( β ) are correlated, but we are able to show that the maximum correlation between the Z ( β ) is 1 2 + o (1) with high probability. An application of Slepian's lemma (Slepian (1962)) and some explicit bounds on orthant probabilities (Pinasco et al. (2021)) implies that max β Z ( β ) > 0 with probability at least 1 -1 k 1+ o (1) . Another application of anticoncentration implies that max β Z ( β ) > n -w with probability 1 -o (1) , which finishes off the proof.

4.1 Bounding the survival-to-contamination ratio

Note that the critical survival-to-contamination ratio appears in both Eqs. (22) and (23). The most involved part of the proof is nailing down the correct order of growth of the survival to contamination ratio; a similar analysis tightly bounds the survival variation and the correlation structure of the Z ( β ) .

To understand the relative survival and contamination, we must analyze the bilinear forms ̂ h α,β [ j ] = z ⊤ j A -1 ∆ y . Similarly, to control the correlation of the Z ( β ) , we must understand the correlation between the ̂ h α,β vectors, which reduces to understanding the bilinear forms z ⊤ j A -1 y α for j ∈ [ d ] , α ∈ [ k ] . The main source of inspiration for bounding these bilinear forms is the heuristic style of calculation carried out in Appendix K of Subramanian et al. (2022) that leads to Conjecture 3.1.

To simplify the discussion, we temporarily restrict to the regime where regression fails ( q + r > 1 ). However, our main technical tool seamlessly generalizes to the regime where regression works ( q + r < 1 ). In the regime where regression fails, A -1 turns out to have a flat spectrum: A -1 ≈ α I for some constant α > 0 . Assume for now that A -1 is exactly equal to a scaled identity matrix. Then the survival is proportional to z ⊤ α ∆ y , which is a random inner product. Similarly, to bound the contamination terms we must control the random inner product z ⊤ j ∆ y for j ̸∈ { α, β } .

Since ∆ y is a sparse vector - it only has 2 n k nonzero entries in expectation - a quick computation reveals that E [ z ⊤ α ∆ y ] = ˜ O ( n k ) and E [ z ⊤ j ∆ y ] = 0 . The deciding factor, then, is how tightly these quantities concentrate around their means. A naïve application of Hoeffding implies a concentration radius of order ˜ O ( √ n ) , which would lead to looseness in the overall result. The hope is to exploit sparsity to get a concentration radius of order ˜ O ( √ n/k ) . This is where our new technical tool Theorem 4.1 comes in, which may be of independent interest; we present it in the following section.

4.2 A new variant of the Hanson-Wright inequality

In reality, even in the regime where regression fails, A -1 is not actually perfectly flat. Even worse, in the regime where regression works, A -1 is actually spiked. Thus, we cannot simply reduce the bilinear form z ⊤ j A -1 ∆ y to an inner product. Instead, we turn to the well-known Hanson-Wright inequality (Rudelson and Vershynin, 2013), which tells us that quadratic forms of random vectors with independent, mean zero, subgaussian entries concentrate around their mean. It was used extensively to study binary classification (Muthukumar et al., 2021), and multiclass classification (Subramanian et al., 2022; Wu and Sahai, 2023).

However, just as Hoeffding is loose, so too is the standard form of Hanson-Wright, because it also does not exploit sparsity. This motivates a new variant of Hanson-Wright which fully leverages the (soft) sparsity inherent to multiclass problems with an increasing number of classes. We now formally define the notions of soft and hard sparsity.

Definition 2 (Soft and hard sparsity) . For π ≤ 1 , we say that random vector y = ( Y i ) n i =1 has soft sparsity at level π if | Y i | ≤ 1 almost surely and Var( Y i ) ≤ π for all i . On the other hand, we say that y has hard sparsity at level π if at most a π fraction of the Y i are nonzero.

In particular, our variant Theorem 4.1 below requires that one of the vectors in the bilinear form has soft sparsity at level π . Throughout, one should think of π = o (1) , and for us indeed π = O ( 1 k ) . One can check that a bounded random vector y with hard sparsity level π must also have soft sparsity at level O ( π ) , so soft sparsity is more general for bounded random vectors. In Table 1 we compare our variant with several variants of Hanson-Wright which have appeared in the literature, some of which involve hard sparsity.

Define the subgaussian norm ∥ ξ ∥ ψ 2 (Vershynin, 2018) as

Theorem 4.1 (Hanson-Wright for bilinear forms with soft sparsity) . Let x = ( X 1 , . . . , X n ) ∈ R n and y ∈ ( Y 1 , . . . , Y n ) ∈ R n be random vectors such that ( X i , Y i ) are independent pairs of (possibly correlated) centered random variables such that ∥ X i ∥ ψ 2 ≤ K and Y i has soft sparsity at level π , i.e. | Y i | ≤ 1 almost surely, and E [ Y 2 i ] ≤ π . Assume that conditioned on Y j , ∥ X j ∥ ψ 2 ≤ K . Then there exists an absolute constant c > 0 such that for all M ∈ R n × n and ϵ ≥ 0 we have

The full proof of Theorem 4.1 is deferred to Appendix G. The main proof techniques are heavily inspired by those of Rudelson and Vershynin (2013); Zhou (2019); Park et al. (2022). However, the

Table 1: Comparison of different variants of the Hanson-Wright inequality. In all variants, we assume that ( x , y ) = ( X i , Y i ) n i =1 are subgaussian, centered, and the pairs ( X i , Y i ) are independent across i . We use ◦ to denote elementwise multiplication, which allows us to express hard sparsity with the sparsity mask γ ∈ { 0 , 1 } n . The concentration radius corresponds to the size of typical fluctuations guaranteed by the concentration inequality, i.e. the ϵ needed for high probability guarantees.

a (Rudelson and Vershynin, 2013, Theorem 1.1); b (Park et al., 2022, Theorem 1); c (Wu and Sahai, 2023, Theorem 4)

proof of Theorem 4.1 is actually simpler than in Park et al. (2022); Wu and Sahai (2023), as bounded with soft sparsity turns out to be easier to work with than subgaussian with hard sparsity. We refer readers to Wu and Sahai (2023) for a more in-depth discussion of how these new 'sparse' variants overcome the limitations of previous proof techniques used to study classification problems.

We briefly illustrate how Theorem 4.1 can be used to get tighter results throughout our analysis. A quick calculation reveals that the label vectors ∆ y and y α both have soft sparsity at level π = O (1 /k ) . However, y α does not have hard sparsity as required by the variants in Park et al. (2022); Wu and Sahai (2023). Since ∥ M ∥ 2 F ≤ n ∥ M ∥ 2 2 , we obtain a concentration radius ϵ which scales like √ n/k rather than √ n (obtained via vanilla Hanson-Wright) or n/ √ k (obtained via Cauchy-Schwarz). This gain is crucial to tightly analyzing the survival, contamination, and correlation structure.

4.3 Completing the proof sketch

Theorem 4.1 and the above insights about sparsity and independence allow us to prove the following bounds on the relative survival and contamination terms which are tight up to log factors; see the Appendix for more details. For brevity's sake, we introduce the notation µ ≜ n q + r -1 .

Proposition 4.2 (Bounds on relative survival) . Suppose we are in the bi-level model. With probability at least 1 -O (1 /n ) ,

Next, we state our bounds on contamination.

Proposition 4.3 (Bounds on contamination) . Suppose we are in the bi-level model. Then with probability at least 1 -O (1 /n ) , the contamination satisfies

Translating the parameters in Propositions 4.2 and 4.3 we see that (i) the relative survival is diminished by a factor 1 /k as long as k = o ( √ n ) , and a factor 1 / √ n for k = Ω( √ n ) (this looseness ends up being negligible for the final result) and (ii) the contamination is diminished by a factor of 1 / √ k . This essentially matches the expected behavior from the heuristic calculation in Subramanian et al. (2022). Together with some straightforward algebra, Propositions 4.2 and 4.3 allow us to compute the regimes where the survival-to-contamination ratio SU / CN grows or decays polynomially. This yields the stated regimes in Conjecture 3.1; see Appendix A.1 for more details.

For technical reasons, the analogous bounds in Subramanian et al. (2022) are loose, giving rise to unnecessary conditions for good generalization such as t < p -2 and t < 2 q + r -2 . Moreover, we are able to give both upper and lower bounds on the survival and contamination terms, whereas they only give one-sided inequalities for each quantity.

5 Discussion

In this paper we resolve the main conjecture of Subramanian et al. (2022), identifying the exact regimes where an overparameterized linear model succeeds at multiclass classification. Our techniques also lay the foundation for investigating related generalization for other multiclass tasks and nonlinear algorithms. We hope that by bringing the rigorous proofs closer to the heuristic style of calculation, we open the path for analyzing more complicated and realistic models.

An important next step is to extend our results to more realistic spiked covariance models. For example, one typically observes power-law decay for the extreme eigenvalues in applications. We expect that the bi-level model can be relaxed to allow for constant deviations in the weightings for the favored, non-label defining features and power law decay for the unfavored features. The former change would likely only affect constants in certain areas of the argument that do not crucially depend on the exact constants involved, whereas the latter would likely just change the effective degree of overparameterization (Bartlett et al., 2020). However, even constant fluctuations in weighting for the label-defining features can lead to significant subtleties, as these constant deviations manifest as polynomially large variations in the number of examples of each class. Such heterogeneity between label-defining directions would likely lead to significantly messier conditions for generalization.

Another future direction is to move beyond Gaussian features. It is plausible that similar results would hold for vector subgaussian features which are rotationally invariant, allowing us to rotate into the basis where the features have diagonal covariance. One place where Gaussianity is crucially used is to obtain an explicit lower bound on the margin between the features.

As an example application, we sketch out how our proof techniques imply precise conditions for a variant of the learning task called multilabel classification. In a simple model for multilabel classification, each datapoint can have several of k possible labels - corresponding to the positive valued features - but in the training set only one such correct label is provided at random for each datapoint. We deem that the model generalizes if for any queried label it successfully labels test inputs as positive or negative. We can use the MNI approach here to learn classifiers.

Some thought reveals that the main difference between multilabel classification and multiclass classification from a survival and contamination perspective is that positive features no longer need to outcompete other features. Thus, the main object of study would be the bilinear forms z ⊤ j A -1 y α , which is possible thanks to Theorem 4.1. The survival and contamination terms are only affected by the expected values of these bilinear forms, but the expected values match the multiclass behavior up to log factors, which do not affect the regimes where SU / CN will grow or shrink polynomially. A similar analysis thus reveals that MNI will generalize in exactly the same regimes as in Conjecture 3.1. Here, the model generalizes in the sense that with high probability over the labels the model will correctly classify, and failure to generalize means that the model will do no better than a coin toss.

Perhaps surprisingly, resolving Conjecture 3.1 also implies that MNI is asymptotically suboptimal compared to a natural non-interpolative approach: simply make ̂ f m equal to the average 6 of all positive training examples of class m . A straightforward analysis, detailed in the supplementary material, reveals this scheme fails to generalize exactly when t < min { 1 -r, p +1 -2( q + r ) } , even in the regime where regression succeeds ( q + r < 1 ). This is particularly interesting because we have shown that in the regime where regression succeeds, MNI generalizes only when t < min { 1 -r, p -1 } , which is a smaller region. In light of this gap, it would be interesting to identify the information-theoretic barrier for multiclass classification, especially within the broader context of statistical-computation gaps (see e.g. Wu and Xu (2021); Brennan and Bresler (2020)).

Acknowledgments and Disclosure of Funding

DWacknowledges support from NSF Graduate Research Fellowship DGE-2146752. We acknowledge funding support from NSF AST-2132700 for this work. DW appreciates helpful discussions with Sidhanth Mohanty and Prasad Raghavendra.

References

Emmanuel Abbe. Community detection and stochastic block models: recent developments. The Journal of Machine Learning Research , 18(1):6446-6531, 2017.

Robert J Adler, Jonathan E Taylor, et al. Random fields and geometry , volume 80. Springer, 2007.

Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica , 30:87-201, 2021.

Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica , 30:203-248, 2021.

Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science , 2(4):1167-1180, 2020.

Matthew Brennan and Guy Bresler. Reducibility and statistical-computational gaps from secret leakage. In Conference on Learning Theory , pages 648-847. PMLR, 2020.

Niladri S Chatterji and Philip M Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. Journal of Machine Learning Research , 22(129):1-30, 2021.

Niladri S Chatterji and Philip M Long. Deep linear networks can benignly overfit when shallow ones do. Journal of Machine Learning Research , 24(117):1-39, 2023.

Victor Chernozhukov, Denis Chetverikov, and Kengo Kato. Comparison and anti-concentration bounds for maxima of gaussian random vectors. Probability Theory and Related Fields , 162(1): 47-70, 2015.

Elisabetta Cornacchia, Francesca Mignacco, Rodrigo Veiga, Cédric Gerbelot, Bruno Loureiro, and Lenka Zdeborová. Learning curves for the multi-class teacher-student perceptron. Machine Learning: Science and Technology , 4(1):015019, 2023.

Yehuda Dar, Vidya Muthukumar, and Richard G Baraniuk. A farewell to the bias-variance tradeoff? an overview of the theory of overparameterized machine learning. arXiv preprint arXiv:2109.02355 , 2021.

David L Donoho, Matan Gavish, and Iain M Johnstone. Optimal shrinkage of eigenvalues in the spiked covariance model. Annals of statistics , 46(4):1742, 2018.

Heinz Werner Engl, Martin Hanke, and Andreas Neubauer. Regularization of inverse problems , volume 375. Springer Science & Business Media, 1996.

Spencer Frei, Gal Vardi, Peter L Bartlett, and Nathan Srebro. Benign overfitting in linear classifiers and leaky relu networks from kkt conditions for margin maximization. arXiv preprint arXiv:2303.01462 , 2023.

Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. The Gaussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning , pages 426-471. PMLR, 2022.

Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning , pages 1832-1841, 2018.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949-986, 2022.

Roger A Horn and Charles R Johnson. Matrix analysis . Cambridge university press, 2012.

Daniel Hsu, Vidya Muthukumar, and Ji Xu. On the proliferation of support vectors in high dimensions. In International Conference on Artificial Intelligence and Statistics , pages 91-99. PMLR, 2021.

Hong Hu and Yue M Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory , 2022.

Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference on Learning Theory , pages 1772-1798, 2019.

Ziwei Ji and Matus Telgarsky. Characterizing the implicit bias via a primal-dual analysis. In Algorithmic Learning Theory , pages 772-804. PMLR, 2021.

Iain M Johnstone. On the distribution of the largest eigenvalue in principal components analysis. The Annals of statistics , 29(2):295-327, 2001.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Chiraag Kaushik, Andrew D McRae, Mark A Davenport, and Vidya Muthukumar. New equivalences between interpolation and svms: Kernels and structured features. arXiv preprint arXiv:2305.02304 , 2023.

Kuo-Wei Lai and Vidya Muthukumar. General loss functions lead to (approximate) interpolation in high dimensions. arXiv preprint arXiv:2303.07475 , 2023.

Bruno Loureiro, Gabriele Sicuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. Learning gaussian mixtures with generalized linear models: Precise asymptotics in high-dimensions. Advances in Neural Information Processing Systems , 34:10144-10157, 2021.

Yue M Lu and Horng-Tzer Yau. An equivalence principle for the spectrum of random inner-product kernel matrices. arXiv preprint arXiv:2205.06308 , 2022.

Neil Mallinar, James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. Advances in Neural Information Processing Systems , 35:1182-1195, 2022.

Andrew D McRae, Santhosh Karnik, Mark Davenport, and Vidya K Muthukumar. Harmless interpolation in regression and classification with structured features. In International Conference on Artificial Intelligence and Statistics , pages 5853-5875. PMLR, 2022.

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75 (4):667-766, 2022.

Theodor Misiakiewicz. Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression. arXiv preprint arXiv:2204.10425 , 2022.

Andrea Montanari and Emile Richard. Non-negative principal component analysis: Message passing algorithms and sharp asymptotics. IEEE Transactions on Information Theory , 62(3):1458-1484, 2015.

Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpolation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory , 1(1): 67-83, 2020.

Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel J. Hsu, and Anant Sahai. Classification vs regression in overparameterized regimes: Does the loss function matter? Journal of Machine Learning Research , 22:222:1-222:69, 2021.

Luca Oneto, Sandro Ridella, and Davide Anguita. Do we really need a new theory to understand over-parameterization? Neurocomputing , page 126227, 2023.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40): 24652-24663, 2020.

Seongoh Park, Xinlei Wang, and Johan Lim. Estimating high-dimensional covariance and precision matrices under general missing dependence. Electronic Journal of Statistics , 15(2):4868-4915, 2021.

Seongoh Park, Xinlei Wang, and Johan Lim. Sparse Hanson-Wright inequality for a Bilinear Form of Sub-Gaussian variables. arXiv preprint arXiv:2209.05685 , 2022.

Luca Pesce, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Are gaussian data all you need? extents and limits of universality in high-dimensional generalized linear estimation. arXiv preprint arXiv:2302.08923 , 2023.

Damián Pinasco, Ezequiel Smucler, and Ignacio Zalduendo. Orthant probabilities and the attainment of maxima on a vertex of a simplex. Linear Algebra and its Applications , 610:785-803, 2021.

Emile Richard and Andrea Montanari. A statistical model for tensor pca. Advances in neural information processing systems , 27, 2014.

Mark Rudelson and Roman Vershynin. Non-asymptotic theory of random matrices: extreme singular values. In Proceedings of the International Congress of Mathematicians 2010 (ICM 2010) (In 4 Volumes) Vol. I: Plenary Lectures and Ceremonies Vols. II-IV: Invited Lectures , pages 1576-1602. World Scientific, 2010.

Mark Rudelson and Roman Vershynin. Hanson-Wright inequality and sub-Gaussian concentration. Electronic Communications in Probability , 18:1-9, 2013.

David S. Slepian. The one-sided barrier problem for gaussian noise. Bell System Technical Journal , 41:463-501, 1962.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research , 19(1): 2822-2878, 2018.

Vignesh Subramanian, Rahul Arya, and Anant Sahai. Generalization for multiclass classification with overparameterized linear models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=ikWvMRVQBWW .

Jasper Tan, Daniel LeJeune, Blake Mason, Hamid Javadi, and Richard G Baraniuk. A blessing of dimensionality in membership inference through regularization. In International Conference on Artificial Intelligence and Statistics , pages 10968-10993. PMLR, 2023.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Guillaume Wang, Konstantin Donhauser, and Fanny Yang. Tight bounds for minimum ℓ 1 -norm interpolation of noisy data. In International Conference on Artificial Intelligence and Statistics , pages 10572-10602. PMLR, 2022.

Ke Wang and Christos Thrampoulidis. Benign overfitting in binary classification of Gaussian mixtures. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4030-4034. IEEE, 2021.

Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis. Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation. arXiv e-prints , art. arXiv:2106.10865, June 2021.

Weichen Wang and Jianqing Fan. Asymptotics of empirical eigenstructure for high dimensional spiked covariance. Annals of statistics , 45(3):1342, 2017.

Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 23549-23588. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/ wei22a.html .

David X Wu and Anant Sahai. Lower bounds for multiclass classification with overparameterized linear models. In International Symposium on Information Theory , 2023.

Yihong Wu and Jiaming Xu. Statistical problems with planted structures: Information-theoretical and computational limits. Information-Theoretic Methods in Data Science , 383:13, 2021.

Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. Research , 6:0024, 2023.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 , 2016.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Shuheng Zhou. Sparse Hanson-Wright inequalities for subgaussian quadratic forms. Bernoulli , 25 (3):1603-1639, 2019.

Contents

K Alternative framing of bi-level ensemble

47

A Preliminaries and notation

For positive integers n , we use the shorthand [ n ] ≜ { 1 , . . . , n } . For a vector v ∈ R n , ∥ v ∥ 2 always denotes the Euclidean norm. We index entries by using square brackets, so v [ j ] denotes the j th entry of v . For any matrix M ∈ R m × n , we denote its ij th entry by m ij , ∥ M ∥ 2 denotes the spectral norm, and ∥ M ∥ F = Tr ( M ⊤ M ) denotes the Frobenius norm. We use σ max ( M ) and σ min ( M ) to denote the maximum and minimum singular values of M , respectively. If M ∈ R n × n is symmetric, we write µ 1 ( M ) ≥ µ 2 ( M ) ≥ . . . ≥ µ n ( M ) to denote the ordered eigenvalues of M . Given two vectors v , u ∈ R n , we write v ◦ u ∈ R n to denote the entrywise product of v and u .

We make extensive use of bigO notation. In this paragraph, c refers to a positive constant which does not depend on n , and all statements hold for sufficiently large n . If f ( n ) = O ( g ( n )) , then f ( n ) ≤ cg ( n ) for some c . If f ( n ) = ˜ O ( g ( n )) , then f ( n ) ≤ cg ( n ) poly log( n ) for some c . If f ( n ) = o ( g ( n )) , then for all c > 0 we have f ( n ) ≤ cg ( n ) . We write f ( n ) = Ω( g ( n )) if f ( n ) ≥ cg ( n ) for some c . Finally, we write f ( n ) = Θ( g ( n )) if there exists positive constants c 1 and c 2 such that c 1 g ( n ) ≤ f ( n ) ≤ c 2 g ( n ) .

Let us now describe the organization of the appendix. In Appendix A.1, we give a more detailed proof sketch and introduce the main propositions that complete the proof of Theorem 3.2. In Appendix B, we introduce the main tools that allow us to prove that the critical bilinear forms z ⊤ j A -1 ∆ y concentrate: our new variant of the Hanson-Wright inequality, the Woodbury inversion formula, and Wishart concentration to bound the spectra of the relevant random matrices that appear. In Appendix C we apply these tools to bound some useful quantities that repeatedly appear in the rest of the proofs. After that, we proceed to bound the survival, contamination, and correlation structure in Appendices D to F. In Appendix J, we present the analysis for the averaging scheme described in Section 5. Finally, we prove our new variant of Hanson-Wright (Theorem 4.1) in Appendix G.

A.1 Proof of Theorem 3.2

In this section, we fill in some of the details of the proof sketch of Theorem 3.2. After recalling the beginning of the proof, we will split up the proof into two subtheorems: one for the positive result where MNI generalizes (Theorem A.2), and another for the negative result where MNI misclassifies (Theorem A.4).

̸

Assume without loss of generality that the test point x test ∼ N (0 , I d ) has true label α for some α ∈ [ k ] . Let x w test be the weighted version of this test point. From (10), an equivalent condition for misclassification is that for some β = α, β ∈ [ k ] , we have ̂ f ⊤ α x w test < ̂ f ⊤ β x w test , i.e. the score for β outcompetes the score for α . Define the Gram matrix A ≜ X w ( X w ) ⊤ , the relative label vector ∆ y ≜ y α -y β ∈ {-1 , 0 , 1 } n , and the relative survival vector ̂ h α,β ∈ R d which compares the signal from α and β :

where to obtain the last line we have used the explicit formula for the MNI classifiers (8). By converting the misclassification condition into the unweighted feature space we see that we will have errors when

Define the contamination term CN α,β :

Note that CN α,β normalizes the RHS of (29) into a standard Gaussian. Indeed, define

̸

Since α, β ∈ [ k ] are favored, we have λ α = λ β = λ F . Hence an equivalent condition for misclassification is that there exists some β = α , β ∈ [ k ] such that

We will translate the above criterion into sufficient conditions for correct classification and misclassification and analyze these two cases separately.

First, let us present our tight characterization of the survival and contamination terms, which will be useful for both sides of the theorem. Recall our definition of µ ≜ n q + r -1 ; whether this quantity polynomially shrinks or decays directly determines if regression works or fails.

Proposition 4.2 (Bounds on relative survival) . Suppose we are in the bi-level model. With probability at least 1 -O (1 /n ) ,

Proposition 4.3 (Bounds on contamination) . Suppose we are in the bi-level model. Then with probability at least 1 -O (1 /n ) , the contamination satisfies

We defer the proof of Proposition 4.2 to Appendix D and the proof of Proposition 4.3 to Appendix E. Combining Propositions 4.2 and 4.3 yields the following sufficient conditions for when the SU / CN ratio grows or shrinks polynomially.

Proposition A.1 (Regimes for survival-to-contamination) . Under the bi-level ensemble model (Definition 1), when the true data generating process is 1-sparse (Assumption 1), as n → ∞ , with probability at least 1 -O (1 /n ) , the survival-to-contamination ratio satisfies

̸

Here, the max and min are being taken over β = α, β ∈ [ k ] .

Proof. We do casework on whether we want to prove an upper bound or lower bound on SU / CN . First, suppose we want to prove the lower bound, so assume t < min { 1 -r, p +1 -2 max { 1 , q + r }} . Since t < r by the definition of the bi-level ensemble (Definition 1), we have that t < 1 2 . So by union bounding over β , Proposition 4.2 implies that with probability 1 -O (1 /n )

Then from Proposition 4.3, by union bounding over β we see that with probability 1 -O (1 /n ) ,

Let us combine these two bounds. If we compare the survival to the contamination coming from favored features, we obtain

where we have included the explicit poly log factors for precision. Hence, if t < 1 -r , the numerator grows polynomially and dominates the denominator. Now let's compare the survival to the contamination coming from unfavored features. This yields

Hence, by union bounding, we see that with probability 1 -O (1 /n ) ,

where v ≜ 1 4 (min { 1 -r, p +1 -2 max { 1 , q + r }} -t ) > 0 by assumption.

For the upper bound, suppose t > min { 1 -r, p +1 -2 max { 1 , q + r }} . Hence t > 0 , and by union bounding we conclude that with probability at least 1 -O (1 /n ) ,

and

Combining these and union bounding yields that with probability 1 -O (1 /n ) ,

where w ≜ 1 4 ( t -min { 1 -r, p +1 -2 max { 1 , q + r }} ) > 0 by asssumption.

We now sketch out a proof of both the positive and negative sides of Theorem 3.2. We point out that the regimes for generalization and misclassification exactly match the regimes above for where the SU / CN ratio grows or shrinks polynomially.

Theorem A.2 (Positive side of Theorem 3.2) . Under the bi-level ensemble model (Definition 1), when the true data generating process is 1-sparse (Assumption 1), as n → ∞ , the probability of misclassification for MNI satisfies Pr[ E err ] → 0 if

Proof sketch. For correct classification, it suffices for the minimum value of the LHS of Eq. (32) to outcompete the maximum value of the RHS, where the max is taken over β ∈ [ k ] , β = α . Some algebra, as in Subramanian et al. (2022), shows that we correctly classify if

̸

By our lower bound on the survival to contamination ratio (Proposition A.1), assuming t < min { 1 -r, p +1 -2( q + r ) } , then with probability at least 1 -O (1 /n ) we have that λ F ̂ h α,β [ α ] CN α,β ≥ n u for some constant u > 0 . By Lemmas B.2 and B.3 in Subramanian et al. (2022) for every ϵ > 0 , with probability at least 1 -ϵ , we have min β x test [ α ] -x test [ β ] ≥ Ω( 1 √ log k ) .

Next, by standard subgaussian maxima tail bounds we have that | x test [ β ] | ≤ 2 √ log( nk ) and Z ( β ) ≤ 2 √ log( nk ) with probability at least 1 -O (1 /nk ) . Finally, applying our upper bound on the relative survival variance (Proposition A.3, which we prove below), the survival variation is at most a polynomially decaying n -w with probability at least 1 -O (1 /nk ) .

By union bounding, we see that with probability at least 1 -O (1 /n ) -ϵ , the LHS outcompetes the RHS, implying that the model correctly classifies.

In fact, given Proposition 4.2, it is straightforward to bound the survival variation.

Proposition A.3 (Upper bound on the survival variation) . Suppose that t < 1 -r . With probability at least 1 -2 /n , we have

where c 1 and w are both positive constants.

Proof. Since we have ̂ h α,β [ α ] = z ⊤ α A -1 ∆ y , the survival variation is

Since t < 1 -r and t < r by definition, we know that t < 1 2 ,and we can apply Proposition 4.2 to see that with probability at least 1 -2 /n we have

Hence we have

where c 1 is an appropriately defined positive constant.

Theorem A.4 (Negative side of Theorem 3.2) . Under the bi-level ensemble model (Definition 1), when the true data generating process is 1-sparse (Assumption 1), as n → ∞ , the probability of misclassification for MNI satisfies Pr[ E err ] → 1 if

Proof sketch. On the other hand, for misclassification it suffices for the maximum absolute value of the LHS of Eq. (32) to be outcompeted by the maximum value of the RHS. Some manipulations yield the following sufficient condition for misclassification:

̸

Within the misclassification regimes in Conjecture 3.1, Proposition A.1 implies that the survivalto-contamination ratio SU / CN shrinks at a polynomial rate n -w for some w > 0 . By standard subgaussian maximal inequalities, the largest label-defining feature is O ( √ log( nk )) with high probability. Gaussian anticoncentration implies that for some β = α, β ∈ [ k ] , Z ( β ) outcompetes the LHS, which is bounded above by n -w , with probability at least 1 2 -o (1) . Hence, we conclude that the model will misclassify with rate at least 1 2 asymptotically.

Let us now describe how to boost the misclassification rate to 1 -o (1) . Notice that the above argument only considered the competition between the LHS of Eq. (48) and one of the Z ( β ) 's on the RHS instead of the maximum Z ( β ) . It's not hard to see from the definition of Z ( β ) in Eq. (31) that the Z ( β ) are jointly Gaussian. For intuition's sake, assuming the Z ( β ) were independent , then max β Z ( β ) would outcompete with probability ( 1 2 -o (1)) k -1 .

In reality, the Z ( β ) are correlated, but we are able to show that the maximum correlation between the Z ( β ) is 1 2 + o (1) with high probability. An application of Slepian's lemma (Slepian (1962)) and some explicit bounds on orthant probabilities (Pinasco et al. (2021)) implies that max β Z ( β ) > 0 with probability at least 1 -1 k 1+ o (1) . An application of anticoncentration for Gaussian maxima (Chernozhukov et al., 2015) implies that max β Z ( β ) > n -w with probability 1 -o (1) , which finishes off the proof.

To fill in the details of the above proof sketch, we will prove the following proposition in Appendix F.

Proposition A.5 (Correlation bound) . Assume we are in the bi-level ensemble model (Definition 1), the true data generating process is 1-sparse (Assumption 1), and the number of classes scales with n (i.e. t > 0 ). Then for every ϵ > 0 , we have

̸

for sufficiently large n and any u > 0 .

B Main tools

In this section we introduce our suite of technical tools that allow us to prove the desired rates of growth for survival, contamination, and correlation.

B.1 Hanson-Wright Inequality

As established in Section 4, we need to use the Hanson-Wright inequality to prove our tight characterization of generalization. For the sake of precision, we explicitly state our definitions of subgaussian and subexponential which we use throughout the rest of the paper.

The subgaussian norm ∥ ξ ∥ ψ 2 of a random variable ξ is defined as in Rudelson and Vershynin (2013),

The sub-exponential norm ∥ ξ ∥ ψ 1 is defined as in Vershynin (2018, Definition 2.7.5):

We will occassionally need to use the following variant of Hanson-Wright for nonsparse bilinear forms, first proved in Park et al. (2021).

Theorem B.1 (Hanson-Wright for bilinear forms without sparsity) . Let x = ( X 1 , . . . , X n ) ∈ R n and y ∈ ( Y 1 , . . . , Y n ) be random vectors such that the pairs ( X i , Y i ) are all independent of each other (however X i and Y i can be correlated). Assume also that E [ X i ] = E [ Y i ] = 0 and max { ∥ X i ∥ ψ 2 , ∥ Y i ∥ ψ 2 } ≤ K . Then there exists an absolute constant c > 0 such that for all M ∈ R n × n and ϵ ≥ 0 we have

Finally, we restate our new version of Hanson-Wright for bilinear forms with soft sparsity, which we prove in Appendix G.

Theorem 4.1 (Hanson-Wright for bilinear forms with soft sparsity) . Let x = ( X 1 , . . . , X n ) ∈ R n and y ∈ ( Y 1 , . . . , Y n ) ∈ R n be random vectors such that ( X i , Y i ) are independent pairs of (possibly correlated) centered random variables such that ∥ X i ∥ ψ 2 ≤ K and Y i has soft sparsity at level π , i.e. | Y i | ≤ 1 almost surely, and E [ Y 2 i ] ≤ π . Assume that conditioned on Y j , ∥ X j ∥ ψ 2 ≤ K . Then there exists an absolute constant c > 0 such that for all M ∈ R n × n and ϵ ≥ 0 we have

B.2 Gram matrices and the Woodbury formula

In order to apply Hanson-Wright to the bilinear form x ⊤ My , we need to have a deterministic matrix M such that the hypotheses are satisfied. However, in our setting we study bilinear forms such as z ⊤ j A -1 ∆ y . Here, the inverse Gram matrix A -1 is not independent of z j or ∆ y , so we cannot simply condition on A -1 . The way around this is to cleverly decompose A -1 using the so-called Woodbury inversion formula (stated formally below), which generalizes the leave-one-out trick and Sherman-Morrison used to study binary classification in Muthukumar et al. (2021). To that end, we will explicitly decompose the Gram matrix A ≜ ∑ j ∈ [ d ] λ j z j z ⊤ j based on whether the features z j are favored or not.

We now introduce some notation to keep track of which matrices contain or leave out which indices. In general, we use subscripts to denote which sets of features we preserve or leave out; we use a minus sign to signify leaving out. The k label-defining features are represented with a subscript k , whereas the s -k favored but not label defining features are represented with a subscript F . The rest of the d -s unfavored features are represented with a subscript U .

For notational convenience, we introduce some new notation for the weighted features, as the superscript w to denote weighted features is rather cumbersome. We denote the weighted labeldefining feature matrix by W k ≜ [ w 1 · · · w k ] ∈ R n × k , where the vectors w i ≜ √ λ i z i ∈ R n denote the weighted observations for feature i . Define the unweighted label-defining feature matrix Z k ≜ [ z 1 · · · z k ] ∈ R n × k . Similarly, define W F ≜ [ w k +1 · · · w s ] ∈ R n × ( s -k ) , which contains the rest of the weighted favored features and the corresponding unweighted version Z F .

Let A -k ≜ ∑ i ̸∈ [ k ] w i w ⊤ i denote the leavek -out Gram matrix which removes the k label-defining features. Similarly, let A -F ≜ ∑ i ̸∈ [ s ] \ [ k ] w i w ⊤ i ∈ R n × n to denote leave-( s -k ) -out Gram matrix which removes the favored but not label-defining features. Finally, let A U ≜ ∑ i ̸∈ [ s ] w i w ⊤ i ∈ R n × n denote the leaves -out matrix which only retains the unfavored features. We will also sometimes write A -s instead of A U to emphasize that the s favored features have all been removed.

Define the so-called hat matrices by

These hat matrices appear in the Woodbury inversion formula. For the sake of notational compactness, define

The Woodbury inversion formula yields

Left multiplying (58) by W ⊤ k yields

We can derive completely analogous identities using A -1 -F instead of A -1 -k . The above exposition is summarized by the following lemma.

Lemma B.2. We have

Lemma B.2 is quite powerful. Indeed, consider the action of the linear operator W ⊤ k A -1 : R n → R k on ∆ y . The action is identical to that of the linear operator W ⊤ k A -1 -k : R n → R k , up to some invertible transformation. This new linear operator is nice because A -1 -k is independent of W k and ∆ y , as it removes all of the label-defining features. Reclaiming independence sets the stage for using our variant of Hanson-Wright.

How does the invertible operator ( I k + H k ) -1 act? Our general strategy is to show that H k is itself close to a scaled identity matrix, i.e. H k ≈ ν I k for an appropriately defined ν . Then for any i ∈ [ k ] , we have that

Of course, there will be some error in this approximation, as H k is not exactly equal to ν I k . Nevertheless, we can bound away the error that arises from this approximation.

B.3 Concentration of spectrum

As foreshadowed in the previous section, we will leverage the fact that the hat matrices such as H k are close to a scaled identity. To formalize this, we appeal to random matrix theory and show that the spectra of various random matrices are very close to being flat (i.e. all eigenvalues are within 1 + o (1) of each other). To that end, we present the following standard characterization of the spectrum of a standard Wishart matrix, which is Equation 2.3 in Rudelson and Vershynin (2010).

Lemma B.3 (Concentration of spectrum for Wishart matrices) . Let M ∈ R M × m with M > m be a real matrix with iid N (0 , 1) entries. Then for any ϵ ≥ 0 , we have with probability at least 1 -2 e -ϵ 2 / 2 that √ √

In other words, the singular values of M satisfy subgaussian concentration.

Since µ m ( M ⊤ M ) = σ min ( M ) 2 and µ 1 ( M ⊤ M ) = σ max ( M ) 2 , we can conclude that if m = o ( M ) , then for any ϵ > 0 we have

with probability at least 1 -2 e -ϵ 2 / 2 .

On the other hand, consider MM ⊤ ∈ R M × M . Its spectrum is just that of M ⊤ M ∈ R m × m with an additional M -m zeros corresponding to the fact that m<M .

We can use Lemma B.3 to prove concentration of the spectrum of the various matrices introduced in Appendix B.2. Let us summarize some convenient forms of these results; their proofs are deferred to Appendix H.

Proposition B.4 (Gram matrices have a flat spectrum) . Recall that A U = A -s = ∑ j>s λ j z j z ⊤ j ∈ R n × n is the unfavored Gram matrix and A -k = ∑ j>k λ j z j z ⊤ j ∈ R n × n is the leavek -out Gram matrix.

Then the following hold with probability at least 1 -2 e -n -2 e - √ n ,

(a) For all i ∈ [ n ] , we have µ i ( A U ) = n p (1 ± O ( n -κ 7 )) .

(b) For all i ∈ [ s -k ] , we have

where κ 9 is a positive constant. Moreover, for all i ∈ [ n ] \ [ s -k ] , we have

where κ 7 is a positive constant.

As a simple corollary, we can obtain the following cruder bounds on the trace and spectral norm of A -1 -k and A -1 -s .

Corollary B.5 (Trace and spectral norm of A -1 -k ) . In the bi-level model, with probability at least 1 -2 e -n , we have

where c 2 , κ 7 , and κ 3 are all positive constants.

Proof. We prove the claim for A -1 -k ; the proof for A -1 U is similar or easier because A -1 U has a flat spectrum (Proposition B.4).

and

If q + r < 1 , the upper bound for the spectral norm similarly follows. For the trace bounds, we can apply Proposition B.4, we have

where and

as q +2 r < 2( q + r ) < 2 by assumption.

On the other hand, the claim is obviously true when q + r > 1 , as the entire spectrum of A -1 -k is (1 ± O ( n -κ 2 )) n -p with an appropriately defined positive constant κ 2 . The spectral norm bound follows by defining c 2 to be any positive constant greater than 1 which absorbs the o (1) deviation terms in the spectrum.

The proof concludes by setting κ 3 = min { κ 1 , κ 2 } .

Finally, we have the following proposition which controls the spectrum of hat matrices such as H k ≜ W ⊤ k A -1 -k W k ∈ R k × k . The intuition is that even though the spectrum of A -1 -k may be spiked, the spectrum of W ⊤ k A -1 -k W k is ultimately flat because we are taking an extremely low dimensional projection which is unlikely to see significant contribution from the spiked portion of A -1 -k .

̸

In fact, we can prove a more general statement, which will be useful for us in the proof. Let ∅ = T ⊆ S ⊆ [ s ] ; here T and S index nonempty subsets of the s favored features. Then we can define W T to be the matrix of weighted features in T and the leaveT -out Gram matrix

Now define the ( T, S ) hat matrix as H T,S ≜ W ⊤ T A -1 -S W T . Evidently we have H k = H [ k ] , [ k ] , so our notion is more general. The full proof is deferred to Appendix H.

Proposition B.6 (Generalized hat matrices are flat) . Assume we are in the bi-level ensemble Definition 1. For any nonempty T ⊆ S ⊆ [ s ] , with probability at least 1 -2 e - √ n -2 e -n , we have all the eigenvalues tightly controlled:

where c T,S and κ 11 are positive constants that depend on | T | and | S | .

C Utility bounds: applying the tools

Wishart concentration allows us to tightly bound the hat matrix and pass to studying bilinear forms of the form w ⊤ i A -1 -k ∆ y rather than w ⊤ i A -1 ∆ y . Since A -1 -k is independent of W k and ∆ y , we can condition on A -1 -k and then apply Hanson-Wright (Theorem 4.1) to these bilinear forms for every realization of A -1 -k . In this section, we will explicitly calculate the scaling of the typical value of these bilinear forms using the bi-level ensemble scaling; these will prove to be useful throughout the rest of the paper.

We first state the following proposition which bounds the correlation between the relevant labeldefining features and the label vectors; it is a combination of Propositions D.5 and D.6 in (Subramanian et al., 2022).

Proposition C.1. For any distinct α, β ∈ [ k ] , we have

With the above proposition in hand, we can prove the following lemma which gives concentration of the bilinear forms that we study.

̸

Lemma C.2. Let i ∈ [ d ] and ∆ y = y α -y β where α, β ∈ [ k ] and β = α . Let M ∈ R n × n be a (random) matrix which is independent of z i and ∆ y . Then conditioned on M , with probability at least 1 -1 /nk ,

and the same holds with ∆ y replaced with y α . Here, c 3 is an appropriately chosen universal positive constant.

Moreover, we have

For any distinct α, β ∈ [ k ] , we have

where c 7 and c 4 are positive constants.

For i ∈ [ d ] \ { α, β } , we have

For i ∈ [ d ] \ { α } , we have

where c 5 is a positive constant.

Proof. Let us check the conditions for our new variant of Hanson-Wright with soft sparsity (Theorem 4.1). We want to apply it to the random vectors ( z i , ∆ y ) = ( z i [ j ] , ∆ y [ j ]) n j =1 . Some of the hypotheses are immediate by definition. Evidently, ( z i [ j ] , ∆ y [ j ]) are independent across j , and are mean zero. Since z i [ j ] ∼ N (0 , 1) , it is subgaussian with parameter at most K = 2 . For the bounded and soft sparsity assumption, we clearly have | ∆ y [ j ] | ≤ 1 and y α [ j ] ≤ 1 almost surely. Also, since ∆ y [ j ] 2 ∼ Ber ( 2 k ) , we have E [ ∆ y [ j ] 2 ] = 2 k . Similarly, E [ y α [ j ] 2 ] = 1 k (1 -1 k ) 2 +(1 -1 k ) 1 k 2 ≤ 2 k .

The more complicated condition is the subgaussianity of z i [ j ] conditioned on the value of ∆ y [ j ] or y α [ j ] . Regardless of whether we're conditioning on ∆ y or y α , it suffices to instead prove that z i [ j ] is subgaussian conditioned on whether feature i won the competition for datapoint j . First, suppose i won, i.e. y oh i [ j ] = 1 . Then the Borell-TIS inequality (Adler et al., 2007, Theorem 2.1.1) implies that z i [ j ] satisfies a subgaussian tail inequality. By the equivalent conditions for subgaussianity Vershynin (2018, Proposition 2.5.2), it follows that z i [ j ] -E [ z i [ j ] | y oh i [ j ] = 1] conditionally has subgaussian norm bounded by some absolute constant K . If i doesn't win (or doesn't participate in the competition), then Proposition D.2 in Subramanian et al. (2022) implies that z i [ j ] -E [ z i [ j ] | y oh i [ j ] = 0] conditionally has subgaussian norm bounded by 6 .

Finally, since M is independent of z i and ∆ y , we can condition on M and apply Theorem 4.1 to the bilinear form for every realization of M .

Hence, we conclude that with probability at least 1 -1 /nk we have

where c 3 is an appropriately chosen absolute constant based on K and the constant c defined in Theorem 4.1.

Now we can compute E [ z ⊤ i M ∆ y | M ] to prove the rest of the theorem. If i = α , we have

Let us now compute E [ ∆ y z ⊤ α ] . From Eq. (76) in Proposition C.1, we have E [ y α z ⊤ α ] = c 4 √ log k k I n , where 1 √ π log 2 ≤ c 4 ≤ √ 2 . Similarly, we have E [ y β z ⊤ α ] = -c 5 √ log k k ( k -1) I n where 1 √ π log 2 ≤ c 5 ≤ √ 2 . It follows that E [ ∆ y z ⊤ α ] = Θ ( √ log k k ) I n .

For i ∈ [ d ] \ { α, β } , by symmetry we obtain E [ y α z ⊤ i ] = E [ y β z ⊤ i ] . This implies E [ ∆ y z ⊤ i ] = E [ y α z ⊤ i ] -E [ y β z ⊤ i ] = 0 , so we obtain

Plugging in the bi-level scaling, we obtain the following corollary.

Corollary C.3 (Asymptotic concentration of bilinear forms) . In the bi-level model, for any i ∈ [ k ] , we have with probability at least 1 -O (1 /nk ) that

Moreover, we have

For any distinct α, β ∈ [ k ]

The same statements hold (with different constants) if we replace A -1 -k with A -1 -s .

Proof. From Corollary B.5, we have ∥ ∥ A -1 -k ∥ ∥ 2 ≤ c 6 n -p , where c 6 is an appropriately chosen universal positive constant based on c 3 . Recall that A -k is obtained by removing the k label-defining features, so in particular A -1 -k is independent of ( z i , ∆ y ) for i ∈ [ k ] . Hence, the conditions for Lemma C.2 are satisfied. Then applying the union bound for the spectral norm bound on A -1 -k , we see that with probability at least 1 -O (1 /nk ) , the deviation term from Hanson-Wright is at most c 6 n 1 2 -p √ log( nk ) .

We now turn to calculating the asymptotic scalings for the expectations. From Lemma C.2, we know that E [ z ⊤ α A -1 -k ∆ y | A -1 -k ] = c 7 √ log k k tr ( A -1 -k ) . Applying the high probability bound on tr ( A -1 -k ) from Corollary B.5, we obtain that with probability at least 1 -O (1 /nk ) that

where in the second line we have applied Corollary B.5 and c 7 is an appropriately chosen positive constant. This proves (85).

With Corollary C.3 in hand, we are now in a position to do some straightforward calculations and bound some quantities which will pop up in the survival and contamination analysis.

Proposition C.4 (Worst-case bound based on Hanson-Wright) . Let T ⊆ [ s ] be a subset of favored features such that { α, β } ⊆ T . Assume that | T | = n τ for some τ ≤ r . Then with probability at least 1 -O (1 /nk ) , we have

Proof. WLOG, suppose α = 1 and β = 2 . By Corollary C.3 we have with probability at least 1 -1 /n that

Hence, the norm of this vector is at most

where c 8 is a positive constant.

D Bounding the survival

Recall that the relative survival was defined to be λ F ̂ h α,β [ α ] = λ F z ⊤ α A -1 ∆ y . The strategy is to apply our variant of Hanson-Wright to z ⊤ α A -1 ∆ y . Unfortunately, A -1 is not independent of z α or ∆ y , so we need to use Woodbury to extract out the independent portions and bound away the dependent portion. As we'll see shortly, the error from the dependent portions can also be controlled using Hanson-Wright. Let us now recall Proposition 4.2 for reference.

Proposition 4.2 (Bounds on relative survival) . Suppose we are in the bi-level model. With probability at least 1 -O (1 /n ) ,

Proof. Recall that ̂ h α,β [ α ] = z ⊤ α A -1 ∆ y . We first observe that for i ∈ [ k ] , the dependence between A -1 and z i as well as A -1 and ∆ y only comes through the k label defining features. Hence, we can use the Woodbury identity to extract out the independent portions of A -1 .

Indeed, our 'push through' lemma for Woodbury (Lemma B.2) and concentration of the hat matrix (Proposition B.6) implies that with extremely high probability

where ∥ E ∥ 2 = O ( n -κ 11 ) .

Let u α ∈ R k denote the α th row vector in E , and let u -α ∈ R k -1 denote the subvector of u α without index α . By reading off the α th row of Eq. (93), we see that

Since ∥ u α ∥ 2 ≤ ∥ E ∥ 2 = O ( n -κ 11 ) , it follows from Cauchy-Schwarz that

Let us pause for a moment and interpret Eq. (95). The term min { µ, 1 } is merely capturing the difference in behavior when regression works and fails; if regression works ( q + r < 1 ) then it becomes µ , and if regression fails ( q + r > 1 ), then it becomes 1 . This behavior should be expected: in the regression works case, we expect the effect of interpolation to be a regularizing one: the signals are attenuated by a factor of µ . The RHS of Eq. (95) is an error term, capturing how differently z ⊤ α A -1 ∆ y behaves from the expected behavior min { µ, 1 } z ⊤ α A -1 -k ∆ y .

Let us now bound the error term. From Proposition C.4 we have with probability at least 1 -O (1 /nk ) that

Let us do casework on t . For t < 1 2 , we have 1 2 -p < 1 -t -p , so we conclude that the error term is min { µ, 1 } O ( n 1 -t -p · n -κ 4 ) √ log( nk 2 ) , where κ 4 is a positive constant.

On the other hand, our Hanson-Wright calculations imply (Corollary C.3) that with probability at least 1 -O (1 /nk ) that

Again, since t < 1 2 , the deviation term is o ( n 1 -t -p ) √ log k .

Hence we conclude that with probability 1 -O (1 /nk ) we have

where κ 5 is a positive constant.

Completely analogous logic handles the bounds for z ⊤ β A -1 ∆ y . Let us now return back to the quantity of interest, λ F ̂ h α,β [ α ] . We can compute

On the other hand, if t ≥ 1 2 the error terms all dominate, and we replace n 1 -t -p with n 1 2 -p everywhere. We conclude that with probability at least 1 -O (1 /nk ) ,

where c 9 is a positive constant. Plugging in the scaling for λ F yields the desired result.

E Bounding the contamination

In this section we give a tight analysis of the contamination term. First, we rewrite the squared contamination term and separate it out into the contamination from the k -2 label-defining features which are not α or β , the rest of the s -k favored features, and the remaining d -s unfavored features. From Eq. (30), we have

Here, CN α,β,L corresponds to contamination from label defining features, CN α,β,F corresponds to contamination from favored features, and CN α,β,U corresponds to contamination from unfavored features. The reason for separating out the contamination into these three subterms is that we will need slightly different arguments to bound each of them, although Hanson-Wright and Woodbury are central to all of the arguments. In Appendix E.1 we prove the upper bound on CN α,β,L + CN α,β,F ; in Appendix E.2 we prove the lower bound. Finally, in Appendix E.3 we bound CN α,β,U . After putting these bounds together, we will obtain the main bounds on the contamination, which we restate here for reference.

Proposition 4.3 (Bounds on contamination) . Suppose we are in the bi-level model. Then with probability at least 1 -O (1 /n ) , the contamination satisfies

E.1 Upper bounding the contamination from label-defining+favored features

In this section, we upper bound the contamination coming from the s -2 favored features which are not α or β . This culminates in the following lemma.

Lemma E.1. In the same setting as Proposition 4.3, we have with probability 1 -O (1 /nk ) that

where c 12 is a positive constant.

Proof. Let W R ∈ R n × ( s -2) as the weighted feature matrix which includes all of the s -2 favored features aside from α, β . We can then define A -R ≜ A -W R W ⊤ R and H R = W ⊤ R A -1 -R W R . Using Woodbury, an analogous computation to Lemma B.2 implies that

The contamination from all of the s -2 favored features that are not α or β satisfies

Since Proposition B.6 implies that µ 1 (( I s -2 + H R ) -2 ) ≤ c 10 min { µ 2 , 1 } with extremely high probability where c 10 is a positive constant, we know the contamination is with extremely high probability upper bounded by the following quadratic form:

We still cannot apply Hanson-Wright, because A -1 -R is not independent of ∆ y . However, we can use Woodbury again to take out z α , z β from A -1 -R .

Define W α,β = [ w α w β ] and H ( s ) α,β = W ⊤ α,β A -1 -s W α,β . Then Woodbury implies that

Hence

We will use Hanson-Wright and Cauchy-Schwarz to argue that the second term in Eq. (106) above will be dominated by the first term. Indeed, Corollary C.3 implies that z ⊤ j A -1 -s ∆ y ≤ c 6 n 1 -t 2 -p √ log( nsk ) with probability at least 1 -O (1 /nsk ) , so it suffices to show that the other term is dominated by n 1 -t 2 -p . We will show that its contribution for each j is min { 1 , µ -1 } ˜ O ( n 1 2 -t -p ) . By Cauchy-Schwarz, the magnitude of the second term of Eq. (106) is at most

where c 11 is a positive constant. In the second line, we have used Proposition B.6 to upper bound ∥ ∥ ∥ ( I 2 + H ( s ) α,β ) -1 ∥ ∥ ∥ 2 ≤ O (min { µ, 1 } ) and we have used Theorem B.1 and Proposition B.4 to deduce that that ∣ ∣ z ⊤ j A -1 -s z a ∣ ∣ ≤ O ( n 1 2 -p ) √ log( nsk ) with probability at least 1 -O (1 /nsk ) . Similarly, we used the scaling from Corollary C.3 to deduce that ∣ ∣ z ⊤ α A -1 -s ∆ y ∣ ∣ ≤ O ( n 1 -t -p ) √ log k , and similarly for β .

Hence z ⊤ j A -1 -R ∆ y is O ( n 1 -t 2 -p ) log( nsk ) with probability 1 -O (1 /nsk ) . By union bounding over j and plugging our upper bound back into Eq. (104), we conclude that with probability at least 1 -O (1 /nk )

where c 12 is a positive constant, concluding the proof.

E.2 Lower bounding the contamination from label-defining+favored features

In this section, we upper bound the contamination coming from the s -2 favored features which are not α or β . This culminates in the following lemma.

Lemma E.2. In the same setting as Proposition 4.3, if t > 0 , with probability at least 1 -O (1 /nk ) , we have

where c 14 is a positive constant.

Proof. Following the beginning of the proof of Lemma E.1 and what we know about the flatness of the spectra of hat matrices from Proposition B.6, we can deduce that there is some positive constant c 13 such that with extremely high probability

We will further lower bound this by throwing out all label-defining j . In other words, the goal now is to lower bound

The main idea is to use Bernstein's inequality, but unfortunately A -1 -R is not independent of ∆ y , so we will again resort to Woodbury to take out z α and z β . As in the proof for upper bounding the favored contamination, we have W α,β = [ w α w β ] and H ( s ) α,β = W ⊤ α,β A -1 -s W α,β . Then we can deduce from another application of Woodbury that

Again, we can argue that with probability 1 -O (1 /nsk ) , the second term is upper bounded in magnitude by

where κ 6 is a positive constant because t > 0 . Since Hanson-Wright (Corollary C.3) implies that z ⊤ j A -1 -s ∆ y = ˜ O ( n 1 -t 2 -p ) , this implies that z ⊤ j A -1 -R ∆ y = ˜ O ( n 1 -t 2 -p ) , and similarly for β . Hence we have

We are now in a position to analyze the contribution from the first term of Eq. (119) to Eq. (114): its contribution is 〈 Z ⊤ F A -1 -s ∆ y, Z ⊤ F A -1 -s ∆ y 〉 . This does have all the independence required to apply Bernstein, because ( A -1 -s , ∆ y ) are independent of Z F . Hence conditioned on A -1 -s and ∆ y , 〈 Z ⊤ F A -1 -s ∆ y, Z ⊤ F A -1 -s ∆ y 〉 is a sum of s -k subexponential variables, and by Lemma 2.7.7 of Vershynin (2018) each of these random variables conditionally has subexponential norm at most ∥ ∥ A -1 -s ∆ y ∥ ∥ 2 2 and conditional mean 〈 A -1 -s ∆ y, A -1 -s ∆ y 〉 .

We can use Hanson-Wright (Theorem 4.1) to bound both of these quantities. Indeed, it implies that with probability at least 1 -O (1 /nk ) ,

Let us now compute the Hanson-Wright bound for 〈 A -1 -s ∆ y, A -1 -s ∆ y 〉 . Note that A -1 -s is independent of ∆ y , so we can condition on A -1 -s and conclude that with probability at least 1 -O (1 /nk )

where we have used the fact that ∆ y is mean zero and ∆ y [ i ] 2 ∼ Ber ( 2 k ) .

From Proposition B.4, we obtain the scaling for Tr ( A -2 -s ) and ∥ ∥ A -2 -s ∥ ∥ 2 . This implies that with probability at least 1 -O (1 /nk )

as t < 1 .

Bernstein and the union bound implies that with probability at least 1 -O (1 /nk ) ,

as r > 0 .

To wrap up, we will need to upper bound the contribution of the error term in Eq. (119). Its contribution from summing over j ∈ [ s ] \ [ k ] is min { 1 , µ -2 } o ( n r +1 -t -2 p ) , which is negligible compared to the Bernstein term, which as we just proved is Ω( n r +1 -t -2 p ) . Hence 〈 Z ⊤ F A -1 -R ∆ y, Z ⊤ F A -1 -R ∆ y 〉 ≥ Ω( n r +1 -t -2 p ) with high probability, and inserting this back into our lower bound Eq. (113), we see that

where c 14 is a positive constant.

E.3 Bounding the unfavored contamination

Finally, we wrap up the section by proving matching upper and lower bounds for the unfavored contamination CN α,β,U .

Lemma E.3 (Bounding unfavored contamination) . In the same setting as Proposition 4.3, if t > 0 , with probability 1 -O (1 /nk ) , the contamination from the unfavored features satisfies

where c 15 is a positive constant.

On the other hand, if t = 0 , then with probability 1 -O (1 /nk ) , the unfavored contamination satisfies

Proof. By Woodbury, we have where

and H s ≜ W ⊤ s A -1 -s W s .

Now we have

By Theorem 4.1, we have with probability at least 1 -O (1 /nk )

where c 15 is a positive constant.

On the other hand, we have that

Due to Proposition B.6, µ i (( I s + H s ) -1 ) = min { µ, 1 } (1 ± o (1)) for all i with very high probability. Hence to handle the error terms that are not ∆ y ⊤ A -1 U ∆ y , it suffices to asymptotically bound ∆ y ⊤ A -1 U M s A -1 U ∆ y . In turn, we can couple this to the quadratic form min { µ, 1 } (1 ± o (1)) ∆ y ⊤ A -1 U W s W s A -1 U ∆ y . By Proposition C.4, we have with probability at least 1 -O (1 /nk )

For t > 0 , we claim that the term in Eq. (139) is o ( n 1 -t -p ) , because 1 -2 t -p < 1 -t -p and r -t -p < 1 -t -p . Hence if t > 0 then by union bound we have with probability at least 1 -O (1 /nk ) that

as desired.

On the other hand, if t = 0 , we only have an issue if q + r < 1 , so that min { 1 , µ -1 } = 1 . In this case, the deviation term n 1 -2 t -p = n 1 -t -p . However, this won't affect the fact that the upper bound on contamination will still be ˜ O ( n 1 -t -p ) . More precisely, this bound concludes by arguing that

where c 16 is an appropriately defined positive constant.

It turns out we don't have to worry about this edge case at all for the lower bound on CN α,β,U , because the stated conditions for misclassification imply that t > 0 anyway. This completes the proof of the lemma.

F Obtaining tight misclassification rate

In this section, we will prove Proposition A.5. Let us restate the main proposition and sketch out its proof more formally.

Proposition A.5 (Correlation bound) . Assume we are in the bi-level ensemble model (Definition 1), the true data generating process is 1-sparse (Assumption 1), and the number of classes scales with n (i.e. t > 0 ). Then for every ϵ > 0 , we have

̸

for sufficiently large n and any u > 0 .

Proof sketch. Note that the Z ( β ) 's that must outcompete the decaying survival to contamination ratio are jointly Gaussian, as they are projections of a standard Gaussian vector x test ∈ R d . Hence if we want to study the probability that max β Z ( β ) outcompetes n -u , we have to understand the correlation structure of the Z ( β ) 's.

We will argue that for β, γ ∈ [ k ] with α, β, γ pairwise distinct, the correlation between Z ( β ) and Z ( γ ) is 1 2 ± o (1) with high probability. To that end, we want to look at the correlation (inner product) between the vectors { λ j ̂ h α,β [ j ] } for j ̸∈ { α, β } and { λ j ̂ h α,γ [ j ] } for j ̸∈ { α, γ } . However, note that by independence of the components of x test from every other random variable and the fact that they are mean zero, we have

Hence it suffices to look at the correlation for j ̸∈ { α, β, γ } .

We assume WLOG that α = 1 , β = 2 , γ = 3 . Let

represent the diagonal matrices containing the squared feature weights with indices α, β zeroed out. Next, let v α,β ∈ R d denote the vector with v α,β [ α ] = v α,β [ β ] = 0 and v α,β [ j ] = λ j ̂ h α,β [ j ] for j ∈ [ d ] , j ̸∈ { α, β } . Hence v α,β = Λ 1 / 2 α,β ( ̂ f α -̂ f β ) . Since Z ( β ) = ⟨ v α,β , x test ⟩ , in order to analyze the correlations between Z ( β ) and Z ( γ ) , it suffices to analyze v α,β . Indeed, we will show that the weighted halfspaces Λ 1 / 2 α,β ̂ f α ∈ R d and Λ 1 / 2 α,β ̂ f β ∈ R d are asymptotically orthogonal.

In other words, we need to show that

with probability at least 1 -O (1 /nk ) ; we can then union bound against all choices of β . This is the most technically involved part of the proof, and is the content of Proposition F.1.

̸

This in turn will imply (see Lemma F.2) that the maximum (and minimum) correlation between the v α,β for different β is 1 2 ± o (1) . Let ( Z β ) β ∈ [ k ] ,β = α be equicorrelated gaussians with correlation ρ = 1 2 + o (1) , and ( Z β ) β ∈ [ k ] ,β = α be equicorrelated gaussians with correlation ρ = 1 2 -o (1) . By Slepian's lemma, for any u > 0 , the probability of max β Z ( β ) losing to n -u is sandwiched as

̸

̸

where we have adopted the shorthand max β to denote max β ∈ [ k ] ,β = α .

Theorem 2.1 of Pinasco et al. (2021) shows that jointly gaussian vectors in R k with equicorrelation ρ lie in the positive orthant with probability Θ( k 1 -1 /ρ ) . In particular, applied to Z β , with correlation ρ = 1 2 + o (1) , we find that

and similarly for Z β . Anticoncentration for Gaussian maxima (Chernozhukov et al., 2015, Corollary 1) implies that we can transfer over the bound on Pr [ max β Z β ≤ 0 ] to a bound on Pr [ max β Z β ≤ n -u ] to show that for every ϵ > 0 , we have

for sufficiently large n . Taking the complement of the above event concludes the proof.

F.1 Main results for tight misclassification rates

The main result in this section is the following proposition, which states that the halfspace predictions are asymptotically orthogonal. Its proof is deferred to the subsequent sections.

Proposition F.1. Assume we are in the bi-level ensemble model (Definition 1), the true data generating process is 1-sparse (Assumption 1), and the number of classes scales with n (i.e. t > 0 ).

For any distinct α, β ∈ [ k ] , with probability at least 1 -O (1 /nk ) , we have

Given Proposition F.1, we can show that the Z ( β ) have correlations that approach 1 2 . The intuitive reason that this correleation approaches 1 2 is that the contribution from α is common. The following lemma formalizes this intuition.

Lemma F.2 (Correlation of relative differences of almost orthogonal vectors) . Suppose that we have n unit vectors x 1 , . . . , x n ∈ R d such that |⟨ x i , x j ⟩| ≤ γ for γ > 0 . Then for any distinct i, j, k ∈ [ n ] , we have

̸

Proof. For any i = j , we have ∥ x i -x j ∥ 2 = 2 -2 ⟨ x i , x j ⟩ . Hence we have

Also

Since ∥ x i -x j ∥ ≥ √ 2 -2 γ , we can rearrange and obtain that

Similarly we can reverse the inequalities and get

Combining Proposition F.1 with Lemma F.2 yields the following formal statement about the correlations between the Z ( β ) .

so

Lemma F.3 (Asymptotic correlation of relative survivals) . For any distinct α, β, β ′ ∈ [ k ] , under the same assumptions as Proposition F.1, as n →∞ , with probability at least 1 -O (1 /n ) , we have

As a consequence, the asymptotic correlation between the relative survivals approaches 1 2 at a polynomial rate.

Proof. Plugging in the result of Proposition F.1 into Lemma F.2, we obtain the stated result.

F.2 Lower bounding the denominator

Let us now begin to prove Proposition F.1. The first step is to bound the denominator of the normalized correlation. Writing out the definitions, we have

Note that these two terms are respectively analogous to CN 2 α,β,L + CN 2 α,β,F and CN 2 α,β,U . In fact, the proofs of the lower bounds for contamination essentially transfer over verbatim to the lower bounds on the denominator, because Hanson-Wright implies that we can show that ∥ ∥ A -1 -s y α ∥ ∥ 2 concentrates the same way that ∥ ∥ A -1 -s ∆ y ∥ ∥ 2 does. In essence, we are able to show the following proposition.

̸

Proposition F.4 (Lower bound on norm of scaled halfspaces) . Under the same assumptions as Proposition F.1, for any α, β ∈ [ k ] , with α = β , with probability at least 1 -O (1 /nk ) , we have

F.3 Upper bounding the numerator: the unnormalized correlation

We now turn to the more involved part of the bound: proving an upper bound on the numerator. As before, we can bound the split up the numerator into favored and unfavored terms. For each term, we will show that it is dominated by the denominator, in the precise sense that each term is

Now, let's look at the numerator, which is the bilinear form

We refer to the the first term as the label defining correlation cor α,β,L , the second term as the favored correlation cor α,β,F , and the last term as the unfavored correlation cor α,β,U . Here, we abuse terminology slightly and refer to these inner products as correlations , even though strictly speaking, they are unnormalized.

F.3.1 Bounding the favored correlation

We now bound the correlation coming from the favored features; we will ultimately show that its contribution is min { 1 , µ -2 } o ( n r -t -1 ) . Recall that W R ∈ R n × ( s -2) is the weighted feature matrix

for the s -2 favored features aside from α and β . Then the label-defining+favored correlation cor α,β,L + cor α,β,F can be written succinctly as

Why should we be able to bound this better than Cauchy-Schwarz? Intuitively, although there is a mild dependence between y α and y β , it is not strong enough to cause Z ⊤ R A -1 y α and Z ⊤ R A -1 y β to point in the same direction.

To formalize this argument, we will first follow the strategy to bound the favored contamination . In particular, using the push-through form of Woodbury (Lemma B.2) we see that

Now, we can apply Proposition B.6 to replace ( I s -2 + H R ) -2 with min { µ 2 , 1 } ( I s -2 + E ) , where ∥ E ∥ 2 = O ( n -κ 11 ) with extremely high probability. Cauchy-Schwarz yields that

The term in Eq. (149) can be bounded in the same way that we bounded the favored contamination. Indeed, since we can swap in y α and y β with ∆ y , the argument that proved the bounds on ∥ ∥ Z ⊤ R A -1 -R ∆ y ∥ ∥ 2 port over immediately. After using the scaling for λ F and the fact that ∥ E ∥ 2 = O ( n -κ 11 ) , we conclude that this Cauchy-Schwarz error term is at most min { 1 , µ -2 } o ( n r -t -1 ) with probability at least 1 -O (1 /nk ) .

Let us now turn to the term in Eq. (148). As in the proof for the lower bound for favored contamination Lemma E.2, to get better concentration than Cauchy-Schwarz, we want to use Bernstein. We can rewrite it suggestively as

We cannot immediately power through with the calculation, because A -1 -R is not independent of y α or y β . The main idea is to again use Woodbury and show that the dependent portions contribute negligibly to z ⊤ j A -1 -R y α . Therefore the dependent contributions get dominated by the lower bound on the correlation.

As in the proof for bounding the favored contamination, we can further define W α,β = [ w α w β ] and H ( s ) α,β = W ⊤ α,β A -1 -s W α,β . Then we can deduce from another application of Woodbury that

Again, we can argue that the second term is bounded in magnitude by

because t > 0 . Since Hanson-Wright (Corollary C.3) implies that z ⊤ j A -1 -s y α = ˜ O ( n 1 -t 2 -p ) , this implies that z ⊤ j A -1 -R y α = ˜ O ( n 1 -t 2 -p ) , and similarly for β . Hence we have

This implies that we can rewrite Eq. (150) as

Let us argue that the second term in Eq. (157) will be negligible compared to the denominator, which is min { 1 , µ -2 } Ω( n r -t -1 ) . Tracing back up the stack, we see that its contribution to the favored correlation will be at most

Turning back to the first term, we are now in a position to apply Bernstein. Note that ( A -1 -s , y α , y β ) are independent of Z R . Hence conditioned on A -1 -s , y α , and y β , 〈 Z ⊤ R A -1 -s y α , Z ⊤ R A -1 -s y β 〉 is a sum of s -2 subexponential variables, and by Lemma 2.7.7 of Vershynin (2018) each of these random variables conditionally has subexponential norm at most ∥ ∥ A -1 -s y α ∥ ∥ 2 ∥ ∥ A -1 -s y β ∥ ∥ 2 and conditional mean 〈 A -1 -s y α , A -1 -s y β 〉 .

We can use Hanson-Wright (Theorem 4.1) to bound both of these quantities. Indeed, it implies that with probability at least 1 -O (1 /nk ) ,

Let us now compute the Hanson-Wright bound for 〈 A -1 -s y α , A -1 -s y β 〉 . Note that A -1 -s is independent of y α and y β , so we can condition on A -1 -s and conclude that with probability at least 1 -O (1 /nk )

We can rewrite the expectation as

Clearly, E [ y β y ⊤ α ] is diagonal, and each diagonal entry is equal to . Let ρ = 1 k . Then since y α = 1 -ρ implies y β = -ρ and vice versa, we get

In other words, the expectation is negative , so we can neglect it in our upper bound.

On the other hand, the deviation term is with very high probability at most

Combining all of our bounds, Bernstein yields

Again, let's trace all the way back to Eq. (148) and then the favored correlation bound. We have shown that 〈 Z ⊤ R A -1 -s y α , Z ⊤ R A -1 -s y β 〉 's contribution to the favored correlation is at most

where the last line follows becuase 0 < t < r < 1 , and c 17 is a positive constant.

F.3.2 Bounding the unfavored correlation

Now, let us show that the unfavored correlation cor α,β,U is negligible; more precisely, we'll show that it's min { 1 , µ -2 } o ( n r -t -1 ) + o ( n 1 -t -p ) . We can rewrite cor α,β,U as

and play the same game with using Woodbury to replace A -1 with A -1 -A -1 U M s A -1 U , where we recall that

This yields

Let us first focus on the first term of Eq. (174). Hanson-Wright implies that with probability at least 1 -O (1 /n ) ,

Also, E [ y ⊤ α A -1 U y β | A -1 U ] = Tr ( A -1 U E [ y β y ⊤ α ] ) = Θ( n 1 -2 t -p ) with high probability, and ∥ ∥ A -1 U ∥ ∥ 2 ≤ n -p with extremely high probability. Hence we see that y ⊤ α A -1 U y β ≤ O ( n 1 -t 2 -p ) ≤ o ( n 1 -t -p ) as t > 0 .

Next, let's turn to the second and third terms of Eq. (174). We claim that only the second term will be relevant to bound asymptotically, and moreover that they are both min { 1 , µ -2 } o ( n r -t -1 ) . Since

the second and third term can be rewritten as

As we are going to use Hanson-Wright to bound the entries of Z ⊤ s A -1 U y α , it follows that only the second term of Eq. (174) is relevant asymptotically.

To bound the second term, we will use Cauchy-Schwarz. We see that

where in the second line we have used Proposition B.6.

Now, note that if regression works, this yields an upper bound of O ( n r -t -p ) log( ns ) . But since p > 1 , this is o ( n r -t -1 ) , which means this contribution is dominated by the denominator.

On the other hand, if regression fails, then the upper bound is now µ -1 O ( n r -t -p ) log( ns ) , which we claim is o ( µ -2 n r -t -1 ) . Indeed, from the definition of the bi-level ensemble Definition 1, we have p > q + r , so

as desired.

Let us now go back to Eq. (174) and combine our two bounds. Since λ U = O (1) , we have just shown that

as desired.

G A new variant of the Hanson-Wright inequality with soft sparsity

In this section, we prove Theorem 4.1. First, we outline a high level idea of the proof. The starting point of the proof is to explicitly decompose the quadratic form into diagonal and off-diagonal terms

̸

̸

where in the first line we have used the fact that for i = j , X i and Y j are independent and mean zero to conclude that E [ X i Y j ] = 0 .

We can start with the upper tail inequality P [ x ⊤ My -E [ x ⊤ My ] > t ] and conclude the lower tail inequality by replacing M with -M . To bound S diag and S offdiag , we will proceed by explicitly bounding the MGF and applying Chernoff's inequality.

G.1 Diagonal terms

For the diagonal terms, we want to bound the MGF of S diag = ∑ i m ii ( X i Y i -E [ X i Y i ]) . For λ 2 < 1 2 C 1 K 2 max i m 2 ii , we obtain

where we have applied Jensen's inequality in the second line and the subgaussian assumption on X i conditioned on Y i in the last line. Here, C 1 is a universal positive constant relating the equivalent formulations of subgaussianity Vershynin (2018). Continuing with our calculation, we have

where in the first line we have used the inequality exp( x ) ≤ 1 + 2 x valid for x ≤ 1 2 , in the second line we have used the soft sparsity assumption on Y i , and in the last line we have used the inequality 1 + x ≤ exp( x ) , valid for all x .

Now Markov's inequality yields for ϵ > 0 that

and optimizing λ in the region λ 2 ≤ 1 2 C 1 K 2 max i m 2 ii yields

Plugging in this value of λ into the Markov calculation yields the desired upper tail bound. We can repeat the argument with -M to get the lower tail bound. A union bound completes the proof.

G.2 Offdiagonal terms

Following Rudelson and Vershynin (2013), for the offdiagonal terms we can decouple the terms in the sum. More precisely, the terms in S offdiag involving indices i and j are precisely m ij X i Y j + m ji Y i X j . The issue is that Y i can be correlated with X i , which complicates the behavior of this random variable. Decoupling ensures that for any j ∈ [ n ] we will have exactly one term which involves either X j or Y j , so in particular we will regain independence of the terms, allowing us to bound the MGF more easily.

Let { δ i } i ∈ [ n ] denote iid Bernoulli's with parameter 1 / 2 , which are independent of all other random variables.

Let

Since E [ δ i (1 -δ j )] = 1 4 , we have

Hence, Jensen's inequality yields

where we have used the independence of δ and all other random variables. It follows that it suffices to upper bound the MGF of S δ .

Define the random set Λ δ = { i ∈ [ n ] : δ i = 1 } to denote the indices selected by δ . For a vector u ∈ R n we also introduce the shorthand u Λ δ to denote the subvector of u where δ i = 1 and u Λ c δ to denote the subvector of u where δ i = 0 .

Hence, we can rewrite S δ ≜ ∑ i ∈ Λ δ ,j ∈ Λ c δ m ij X i Y j . For | λ | ≤ 1 2 C 1 K ∥ M ∥ 2 , we have

Now we can use the fact that the X i and Y j are mean zero and independent because i ∈ Λ δ and j ∈ Λ c δ , to show that

In the first line, we have used the subgaussianity of X i ; in the second line, we have used the assumption on λ , in the third line, we have used the variance bound on Y j .

Again, we can apply Markov's inequality and to see that for ϵ > 0 ,

̸

Picking yields the desired result.

H Proofs of main lemmas for concentration of spectrum

̸

The goal of this section is ultimately to prove Proposition B.6, which asserts that for valid ( T, S ) , the hat matrix H T,S is a flat matrix whose spectrum is min { µ, 1 } (1 + o (1)) with extremely high probability. First, let us recall some notation. For any ∅ = T ⊆ S ⊆ [ s ] , we can define the ( T, S ) hat matrix as H T,S ≜ W ⊤ T A -1 -S W T . Here, W T is the n ×| T | matrix of weighted features in T , and A -T = A -W T W ⊤ T is the leaveT -out Gram matrix.

First, Wishart concentration applied to W ⊤ T W T yields the following result.

Lemma H.1. Recall that µ ≜ n q + r -1 and W ⊤ T W T ∈ R | T |×| T | . For any nonempty T ⊆ [ s ] , with probability at least 1 -2 e - √ n we have that for all i ∈ [ | T | ] ,

Proof. We can apply Lemma B.3 with M = Z T ∈ R n ×| T | , with M = n , m = | T | = o ( n ) , and ϵ = n 1 4 = o ( √ n | T | ) . Hence we have

Pluagging in the scaling λ F = n p -q -r and dividing through by n yields the desired result. Here, we define c T to be an appropriately defined positive constant which only depends on | T | (as the favored features are identically distributed).

Next, we can use Wishart concentration to bound the spectrum of A U .

Lemma H.2 (Concentration of spectrum for unfavored Gram matrix) . Throughout this theorem, assume we are in the bi-level model (Definition 1). Define µ ≜ n q + r -1 . Recall A U = λ U ∑ j>s z j z ⊤ j ∈ R n × n . With probability at least 1 -2 e -n , for i ∈ [ n ] we have

where c 18 and κ 7 are positive constants. In other words, the spectrum of the unfavored Gram matrix A U is flat.

Proof. Note that A U = λ U ∑ j>s z j z ⊤ j . Under the bi-level model, λ U = 1 + o (1) . Now we can apply Lemma B.3 with M = ∑ j>s z j z ⊤ j , M = d -s = n p -n r , m = n = o ( d ) , and ϵ = √ 2 n to conclude that with probability at least 1 -2 e -n , we have

We can obtain the spectrum of A U by multiplying through by

where in the last line we have used the power series expansion for 1 1 -x = 1 + x + o ( x ) . Preserving only first order terms for λ U and the spectrum of ∑ j>s z j z ⊤ j in Eq. (204) yields

In fact, we know 1 -p 2 > 1 -p > r -p , since r < 1 and p > 1 . This means we can neglect the r -p term in the max, define κ 7 = min { p -1 2 , -q } > 0 and c 18 to be an appropriately defined positive constant.

Since A -T = A U + W [ s ] \ T W ⊤ [ s ] \ T , we can apply Lemmas H.1 and H.2 to control the spectrum of A -T . We show that there is a (potentially) spiked portion of the spectrum corresponding to the s -| T | favored features which were not taken out, whereas the rest of the n -s + | T | eigenvalues are flat.

Lemma H.3. Recall that A -T ∈ R n × n . For any nonempty T ⊆ [ s ] , with probability at least 1 -2 e - √ n -2 e -n , we have that for all i ∈ [ s -| T | ] ,

For all i ∈ [ n ] \ [ s -| T | ] , we have

Weyl's inequality (Horn and Johnson, 2012, Corollary 4.3.15) implies that for any i ∈ [ n ] , we have

Then applying Lemmas H.1 and H.2, for i ∈ [ s -| T | ] we conclude that

which proves Eq. (208).

For i > s -| T | , applying Lemma H.2 and the fact that µ i ( W [ s ] \ T W ⊤ [ s ] \ T ) = 0 to Eq. (211) yields

Proof. We can write which proves Eq. (209).

By inverting the bounds proved above, we can also control the spectrum of A -1 -T .

Corollary H.4. Recall that A -T ∈ R n × n . For any nonempty T ⊆ [ s ] , with probability at least 1 -2 e - √ n -2 e -n , we have that for all i ∈ [ n -s + | T | ] ,

For all i ∈ [ n ] \ [ n -s + | T | ] , we have

where κ 8 is a positive constant depending on | T | .

Proof. By inverting the bounds in Lemma H.3, using the fact that µ i ( A -1 -T ) = 1 µ n -i +1 ( A -T ) we see that for i ∈ [ n -s + | T | ] ,

where we have used the power series expansion 1 1 -x = 1 + x + o ( x 2 ) and c 19 is a positive constant.

On the other hand, for i > n -s + | T | , we get

where c 20 and κ 8 are positive constants defined as follows. If q + r < 1 , i.e. regression works, then µ -1 = ω (1) , so the denominator becomes µ -1 ( 1 ± c T √ | T | n + µ (1 ± c 18 n -κ 7 ) ) . Then, since

| T | ≤ s = n r , we see that we can pick

On the other hand, if q + r > 1 , i.e. regression fails, then µ -1 = o (1) , and so we can define

Hence to cover both cases we can pick

The choice of c 20 is picked by again using the power series expansion for 1 1 -x .

Note that Corollary H.4 immediately implies Proposition B.4, with κ 9 defined based on picking T = [ k ] . We are now in a position to prove that the generalized hat matrices H T,S , and hence the Woodbury terms ( I | T | + H T,S ) -1 have a flat spectrum as well.

Proposition B.6 (Generalized hat matrices are flat) . Assume we are in the bi-level ensemble Definition 1. For any nonempty T ⊆ S ⊆ [ s ] , with probability at least 1 -2 e - √ n -2 e -n , we have all the eigenvalues tightly controlled:

where c T,S and κ 11 are positive constants that depend on | T | and | S | .

Proof. We seek to control the spectrum of the hat matrix H T,S = W ⊤ T A -1 -S W T . We cannot directly use naive eigenvalue bounds to bound the minimum and maximum eigenvalue, as this does not rule out the possibility that H T,S has a spike. Instead, we control the spectrum from first principles.

The spectrum of H T,S is flat: Since A -1 -S is symmetric, it has an eigendecomposition V DV ⊤ , where V is an orthogonal matrix. Because W T is a weighted subset of only the (equally) favored features, its law is rotationally invariant. Furthermore, since A -1 -S is independent of W T (as T ⊆ S ), we can absorb the rotation V into W T to reduce to the case where A -1 -S = D . Here, we have

where D flat ∈ R ( n -s + | T | ) × ( n -s + | T | ) and D spiked ∈ R ( s -| T | ) × ( s -| T | ) correspond to the flat and (downwards) spiked portions of the spectrum of A -1 -S . We can also correspondingly decompose

where B T ∈ R ( n -s + | T | ) ×| T | and C T ∈ R ( s -| T | ) ×| T | . Note that each entry of these matrices are i.i.d. N (0 , 1) variables.

By direct computation we have

We thus have by using standard eigenvalue inequalities that

where in the last line we have used µ n ( A -1 -S ) ≥ 0 .

Since n > s , we have n -s + | T | > | T | , so we can apply Wishart concentration (Lemma B.3) to B ⊤ T B T to obtain that with probability at least 1 -2 e - √ n we have

where c 21 is a positive constant.

On the other hand, we can deduce that

Lemma H.1 implies that with probability at least 1 -2 e - √ n

Similarly, Corollary H.4 implies that with probability at least 1 -2 e -n -2 e - √ n , µ 1 ( A -1 -S ) and µ n -s + | T | ( A -1 -S ) are both (1 ± c 20 n κ 8 ) n -p . Since | T | ≤ s = n r , Eqs. (227) and (229) together demonstrate that for all i ∈ [ | T | ] ,

Here, c 22 and κ 10 are positive constants defined as follows. Since | T | ≤ s = n r . Then κ 10 = min { 1 -r, 1 -r 2 , κ 8 } , and c 22 is a constant chosen appropriately based on c 20 and c 21 . Plugging in the scaling λ F = n p -q -r , we conclude that with extremely high probability, for all i ∈ [ | T | ] ,

From here, it is easy to compute the spectrum of ( I | T | + H T,S ) -1 . Indeed, reading off our result from Eq. (231) yields

Here, the positive constant c T,S is picked appropriately and κ 11 = min { κ 10 , | 1 -q -r |} > 0 . This completes the proof.

I Miscellaneous lemmas

Lemma I.1 (Coupling of quadratic forms) . Let B ∈ R n × m be an arbitrary real matrix and M ∈ R n × n be a PSD matrix. Then for any vector x ∈ R m , we have

Proof. For any PSD matrix C , the matrix B ⊤ CB is PSD. In particular, C has a unique square root C 1 / 2 ∈ R n × n with C 1 / 2 C 1 / 2 = ( C 1 / 2 ) ⊤ C 1 / 2 = C . We thus have

Hence

Since M ⪯ λ 1 ( M ) I n by definition, λ 1 ( M ) I n -M is a PSD matrix. Hence by applying Eq. (236), we conclude that

which gives the upper bound in Eq. (234).

Similarly, M ⪰ λ n ( M ) I n , so an analogous argument

which gives the lower bound in Eq. (234).

Next, we prove the elementary anti-concentration result that we will need.

Proposition I.2 (Gaussian anticoncentration) . Let x ∼ N (0 , I d ) be a standard Gaussian vector, and let v ∈ R d be arbitrary deterministic vector. Then

Proof. Note that ⟨ x , v ⟩ is a linear projection of a standard multivariate Gaussian, so it is itself a one-dimensional Gaussian. It is also clearly zero mean, and its variance is just give by the squared norm of v . So ⟨ x , v ⟩ ∼ N (0 , ∥ v ∥ 2 2 ) . Now we have

J Comparison to the straightforward non-interpolative scheme

In this section, we quickly give calculations for how well a straightforward non-interpolating scheme for learning classifiers can work asymptotically. However, a similar analysis using the tools developed to prove our main results should give a rigorous proof of the below derivation.

This scheme simply uses the sum/average of all positive training examples of a class as the vector we take an inner-product with to generate scores for classifying test points. For m ∈ [ k ] , define

To understand how well this will do asymptotically, it is easy to see that the for the true labeldefining direction, the positive exemplars in the bi-level model will be tightly concentrating around √ 2 log k √ λ F which, keeping only the polynomial-order scaling, will be like n p -q -r 2 . There will be roughly n k = n 1 -t positive examples for every class with high probability. For simplicity, let us just k t -1

look at m = 1 and consider n ̂ f 1 = n ̂ f 1 . We see

For the other directions that are not true-label defining, we will just have random Gaussians. The favored directions will be Gaussian with variance λ F = n p -q -r while the unfavored directions will essentially be Gaussian with unit variance. By averaging over n 1 -t examples, those variances will be reduced by that factor. This means that for the s = n r favored directions, the variance of the average will be n p -q -r -(1 -t ) each and for the essentially n p unfavored directions, the variance of the average will be n t -1 each.

On a test point, we are going to take the inner product of n t -1 ̂ f m with an independent random draw of x w test . For classification to succeed, we need this inner product to be dominated by the true

m -th feature-defining direction. When that happens, the correct label will win the comparison. One can easily see that the contribution from the true feature-defining direction will be a Gaussian with mean 0 and variance λ F · ( n p -q -r 2 ) 2 = λ 2 F = n 2 p -2 q -2 r . Meanwhile, the s favored features will have their scaled variances sum up in the score to give a total variance of n r · λ F · n p -q -r -(1 -t ) = n 2 p -2 q -r -(1 -t ) . And finally, the unfavored features will also have their variances sum up in the score to give a total variance of n p · 1 · n t -1 = n p + t -1 .

For the true-feature-defining direction to dominate the contamination from other favored directions, we need

which immediately gives the condition t < 1 -r .

For the true-feature-defining direction to dominate the contamination from other unfavored directions, we need

which gives the condition t < p +1 -2( q + r ) .

Here, there is no difference between regimes in which regression works or does not work. The condition for classification to asymptotically succeed is t < min(1 -r, p +1 -2( q + r )) .

Notice that when MNI regression does not work q + r > 1 , this is identical to the tight characterization given for MNI classification in (13). But in the regime where MNI regression does work q + r < 1 , this is different. For MNI classification, (13) tells us that we require t < min(1 -r, p -1) . Consider q = 0 . 1 , r = 0 . 5 and p = 1 . 1 . MNI classification can only allow t < 0 . 1 . Meanwhile, the noninterpolating average-of-positive-examples classifier will work as long as t < 0 . 5 . This demonstrates the potential for significant suboptimality (in terms of the number of distinct classes that can be learned) of MNI classifiers in this regime of benign overfitting for regression.

K Alternative framing of bi-level ensemble

In this section we provide an alternative exposition for the bi-level ensemble, which may be more intuitive for some readers.

The high level conceptual picture of the setup is as follows. The learner simply observes high dimensional jointly Gaussian zero-mean features with some unknown covariance Σ ∈ R d × d . It then performs min-norm interpolation of (essentially) one-hot-encoded labels to learn the score functions for the k different classes. These score functions are used at test-time to do multiclass classification.

Note that the learning algorithm has no knowledge of Σ , nor does it even know that the features are jointly Gaussian - all it has is the training data. For analysis purposes, we parameterize Σ . In the spirit of spiked covariance models, where a low-dimensional subspace has higher variance, we study the case that the eigenvalues of Σ follow the simplified bi-level model parameterized by ( p, q, r ) .

The bi-level model stipulates that (i) the number of features is d = n p and (ii) there are two discrete variance levels for the Gaussian features. The higher variance component resides in a low-dimensional subspace of dimension s = n r . For this bi-level model, we are able to prove sharp phase-transition style results telling us when successful generalization will happen. Here, the number of classes also matters, and that is where the parameter t enters.

To simplify notation for analysis in the paper, Subramanian et al. (2022) just assume that Σ is diagonal to begin with, instead of explicitly rotating coordinates to the eigenbasis of a general Σ . Because Gaussianity is preserved under rotations and min-norm interpolation only cares about norms, this transformation is without loss of generality. We reiterate that the learner is agnostic to all of these choices and does not know Σ at any point.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the asymptotic generalization error behavior of an overparameterized linear model and under the Gaussian covariates bi-level model. In this setup, the number of data points, features, and classes all grow together. The authors manage to fully characterize the regimes of the generalization error, which is surprisingly “polarized”. This solves the conjecture posed by Subramanian et al. (2022).\n\nSTRENGTHS:\nBoth the achieved theoretical result and also the used technical tools (like the newly established Hanson-Wright inequality) are highly novel and strong. Even though I am not familiar with the literature, I estimate that the work is quite valuable.\n\nWEAKNESSES:\n- The considered assumptions for the distribution of features are very simplistic. Both independence and having identical Gaussian distributions are very restrictive; which highly influences the practicality of the results.\n \n-  The paper studies the generalization error of linear models; which are far from the deep neural networks. In this sense, there is still a big gap in the practical aspects of the paper (and also the previous literature on this). However, it is totally understandable that these are the first steps toward that goal.\n\n- The paper lacks providing the needed intuitions about the obtained results.\n\nQUESTIONS:\n-  It would be useful to provide more intuition about the considered “bi-level ensemble” model. In this model, it seems that while we are considering $n^p$ dimensions, the ``effective dimensions’’ is the favored ones; meaning that the features in the rest of the dimensions (and their relative magnitude) somehow either add a “useful noise injection” or become dominant with respect to the “signal” which makes the prediction impossible. This might be inexact, but this is just to give an example of what kind of intuition I refer to.\n \n-  Similarly, it would be extremely useful to discuss the different regimes of the theorems. Why having $r$ close to 1 would fail the learner? Why having a small $p-(q+r)$ would do so?\n \n-   Finally, the scope of the paper remains a bit narrow (and purely technical) and the authors could probably use the intuitions derived from their results to discuss some potential understanding that these results could give about some more realistic setups. This is shortly done in the discussion section, but it would be appreciated to be extended.\n\nLIMITATIONS:\nAs discussed above, and I guess the authors would also agree, the considered model is quite restrictive.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper titled "Asymptotic Generalization of Overparameterized Linear Models for Multiclass Classification under Gaussian Covariates Bi-level Model" presents a study on the asymptotic generalization of overparameterized linear models for multiclass classification under the Gaussian covariates bi-level model. The authors provide an asymptotic characterization of the generalization of a linear model for multiclass classification in an idealized Gaussian setting where a) the number of data points, b) the dimension, and c) the number of classes diverge while their ratio remains fixed. An interesting result, in particular, is that the min-norm interpolating classifier can be suboptimal in this regime.

STRENGTHS:
The paper present many strong analytical results.

In particular, the authors have successfully resolved a conjecture posed in a previous works  (Subramanian et al. '22,) and established new lower bounds that demonstrate the misclassification rate either goes to 0 or 1 asymptotically.  The paper also introduces a new variant of the Hanson-Wright inequality, a tool used in high-dimensional probability, that is particularly useful for multiclass problems with sparse labels. Sparse labels refer to situations where only a small number of classes are represented in the dataset.

QUESTIONS:
How realistic is the  Bi-level feature weighting model ? I am thinking instead of the source/capacity setting where I would expect a more power-law-like behavior. How different would the conclusions be?

LIMITATIONS:
The paper is theoretical in nature and its limitations are stated in the theorems
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This is a theoretical paper that gives insight into how and when an overparametrized linear classification model, for multi-class classification, can be successfully generalized.    In particular, they look at multiclass classification under the Gaussian covariates bi-level model introduced by Subramamian et al. in 2022, and fully resolve a conjecture from that paper.  The key to their analysis is a new variant of the Hanson-Wright inequality.

STRENGTHS:
This paper builds on previous work to give tight bounds on the regions where generalization is possible, improving significantly on previous partial results.  It also makes rigorous previous analyses based on heuristic calculations.
This is a very well-written paper.
Based on the proof sketch given in the main paper, the technical level of the proofs seems high, involving both the use of existing techniques as well as the proof of a new version of the Hanson-Wright inequality,.

WEAKNESSES:
No significant weaknesses were noted.

QUESTIONS:
None.

LIMITATIONS:
It would be helpful to comment on the limitations of the bi-level model studied.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-09

comments: SUMMARY:
In their main result, Theorem 3.2, the authors establish Conjecture 3.1 which is a conjecture posed in 2022 describing the asymptotic misclassification probability of the bi-level ensemble model (Definition 1) under a sparsity assumption (Assumption 1).  They provide a rigorous and tight analysis, with very clear explanations both in-text and in the appendix.

STRENGTHS:
The paper is clearly written and explained wonderfully, the proofs are detailed, and the contributions are interesting.  The proofs are sound and clearly articulated.  


Minor point, the appendix is also very neatly organized which makes proofreading nice.

**Update:** My questions and concerns have been addressed.

WEAKNESSES:
Overall the paper is rigerous and clear, so I do not really have any significant weakness worth reporting; only a few minor comments.

* Only minor comments:*

 - In equations (36)-(40) could you explicitly write the polylogarithmic terms in the denominator, or at the very least, precisely defined them after the equation environments.

- Could you formally state that all r.v. are defined on the same probability space, at the beginning of the paper; for rigor.

- Very minor point, above equation (219), I guess $\boldsymbol{A}_{-\boldsymbol{S}}^{-1}$ is *block diagonal* not diagonal.

QUESTIONS:
There are two little details, which I didn't fully follow, so let me ask:

- Perhaps a silly question, but in equation (182), are you using a Brascamp–Lieb inequality (or something simpler which I'm possibly missing)?

- In equation (240) do you mean $n^{t-1}\hat{\boldsymbol{f}}_1[1]$ is $\Theta(n^{(p-q-2)/2})$?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors analyze the generalization of the linear multiclass classifiers in the overparametrized regime under the bi-level model with Gaussian covariates. In particular they prove a conjecture made in a previous paper about the region (characterized by the paramters of the bi-level model) under which the generalization error can go to zero. In fact, they establish a `0-1` law for generalization error: depending upon the parameter regime the probability of error either converges to zero or one. In the process of proving  this result the authors also establish a generalization of the usual Hanson Wight inequality for quadratic forms to exploit \"soft sparsity\". \n\nOverall, I feel that this paper contains a technically rigorous analysis of the overparametrized classification problem somewhat restrictive model assumptions.\n\nSTRENGTHS:\nThe paper is well written, and while the material is quite technical, the authors do make an effort to provide sufficient context and explanations to motivate them. In particular, the detailed overview of the argument used in proving the main result (Theorem 3.2) is quite helpful in getting the idea used in the proof.\n\nWEAKNESSES:\n1.  While this paper does not seem to have any obvious flaws, I feel that the model analyzed in this paper is a bit too restrictive. In a footnote on page 4, the authors mention that \"such models are widely used to study learning even beyond this particular thread of work\". It would be very helpful, if the authors include a detailed discussion of at least one such practical application, which is naturally modeled by the bi-level model studied in this paper.  \n2. Besides the model, even the construction of the classifier makes some strong assumptions. In particular, the classifier is constructed after reweighting the features with weights $(\\lambda_i)_{i \\geq 1}$ that  are tuned to the specific bi-level model.  Since in most practical problems, there is at least some level of misspecification, I feel that this reduces the practical utility of the results in this paper.\n\nQUESTIONS:\n1. Can you include a discussion of some practical tasks where the bi-level model arises naturally? \n\n2. Regarding the second point in the weaknesses section, can you discuss (at least informally) the effect of model misspecification on the generalization guarantees. More generally, do you think that the same performance guarantees as Theorem 3.2 (i.e., probability of error converging to zero for the same parameter ranges) can be achieved adaptively without using the knowledge of model parameters ($p, q, r, t$) to construct the classifier? Or there is a price to pay for achieving zero generalization adaptively?\n\nLIMITATIONS:\nI think that the strong model assumptions used in this paper might reduce the practical utility of the results of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}

comments: SUMMARY:
By resolving the conjecture posed by Subramanian et al. (2022), the authors address the asymptotic generalization for overparameterized minimum-norm interpolation (MNI) linear multi-class classifiers under two assumptions: (1) features are Gaussian vectors and labels are generated from $1$-sparse noiseless model; (2) the scaling follows bi-level ensemble. This paper can be seen as a completion to the previous result of Subramanian et al. (2022) since it captures the generalization for both the regime where the regressor fails and regime where it works. The technical contribution appears to be a new variant of the Hanson-Wright inequality.

STRENGTHS:
A sound completion to the previous theoretical result.

WEAKNESSES:
The setting of bi-level ensemble seems a bit weird to me, especially the eq (12). Maybe the authors can offer more explainations.

QUESTIONS:
Line 165: "it suffices for the maximum value of the LHS ......'' Is it the maximum instead of minimum here?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27