SlowGuess/NeurIPS_2023_OCR huggingface dataset:
David X. Wu
david_wu@berkeley.edu
Anant Sahai
sahai@eecs.berkeley.edu
We study the asymptotic generalization of an overparameterized linear model for multiclass classification under the Gaussian covariates bi-level model introduced in Subramanian et al. (2022), where the number of data points, features, and classes all grow together. We fully resolve the conjecture posed in Subramanian et al. (2022), matching the predicted regimes for generalization. Furthermore, our new lower bounds are akin to an information-theoretic strong converse: they establish that the misclassification rate goes to 0 or 1 asymptotically. One surprising consequence of our tight results is that the min-norm interpolating classifier can be asymptotically suboptimal relative to noninterpolating classifiers in the regime where the min-norm interpolating regressor is known to be optimal.
The key to our tight analysis is a new variant of the Hanson-Wright inequality which is broadly useful for multiclass problems with sparse labels. As an application, we show that the same type of analysis can be used to analyze the related multilabel classification problem under the same bi-level ensemble.
In this paper, we directly follow up on a specific line of work initiated by Subramanian et al. (2022); Wu and Sahai (2023). For the sake of self-containedness, we briefly reiterate the context, directing the reader to Subramanian et al. (2022) and the references cited therein for more. A broader story can be found in Bartlett et al. (2021); Belkin (2021); Dar et al. (2021); Oneto et al. (2023).
Classical statistical learning theory intuition predicts that highly expressive models, which can interpolate random labels (Zhang et al., 2016; 2021), ought not to generalize well. However, deep learning practice has seen such models performing well when trained with good labels. Resolving this apparent contradiction has recently been the focus of a multitude of works, and this paper builds on one particular thread of investigation that can be rooted in Bartlett et al. (2020); Muthukumar et al. (2020) where the concept of benign/harmless interpolation was crystallized in the context of overparameterized linear regression problems and conditions given for when this can happen. In Muthukumar et al. (2021), a specific toy "bi-level model" with Gaussian features was introduced to study overparameterized binary classification and show that successful generalization could happen even beyond the conditions for benign interpolation for regression. Following the introduction of the corresponding multi-class problem in Wang et al. (2021) with a constant number of classes, an asymptotic setting where the number of classes can grow with the number of training examples was introduced in Subramanian et al. (2022) where a conjecture was presented for when minimum-norm interpolating classifiers will generalize. We are now in a position to state our main contributions; afterwards, we expand on the related works.
Our contributions
Our main contribution is crisply identifying the asymptotic regimes where an overparameterized linear model which performs minimum-norm interpolation does and does not generalize for multiclass classification under a Gaussian features assumption, thus resolving the main conjecture posed by (Subramanian et al., 2022). We improve on the analysis of Subramanian et al. (2022); Wu and Sahai (2023), covering all regimes with the asymptotically optimal misclassification rate. When the model generalizes, it does so with a misclassification rate  $o(1)$ , and we show a matching "strong converse" establishing when it misclassifies, it does so with rate  $1 - o(1)$ , where the explicit rate is nearly identical to that of random guessing. The critical component of our analysis is a new variant of the Hanson-Wright inequality, which applies to bilinear forms between a vector with subgaussian entries and a vector that is bounded and has soft sparsity, a notion we will define in Section 4.2. We show how this tool can be used to analyze other multiclass problems, such as multilabel classification.
1.1 Brief treatment of related work
Our thread begins with a recent line of work that analyzes the generalization behavior of overparameterized linear models for regression (Hastie et al., 2022; Mei and Montanari, 2022; Bartlett et al., 2020; Belkin et al., 2020; Muthukumar et al., 2020). These simple models demonstrate how the capacity to interpolate noise can actually aid in generalization: training noise can be harmlessly absorbed by the overparameterized model without contaminating predictions on test points. In effect, extra features can be regularizing (in the context of descent algorithms' implicit regularization (Soudry et al., 2018; Ji and Telgarsky, 2019; Engl et al., 1996; Gunasekar et al., 2018)), but an excessive amount of such regularization causes regression to fail because even the true signal will not survive the training process. Although works in this thread focus on very shallow networks, Chatterji and Long (2023) established that deeper networks can behave similarly. Note that recently, Mallinar et al. (2022) called-out an alternative regime (behaving like 1-nearest-neighbor learning) called "tempered" overfitting in which training noise is not completely absorbed but the true signal does survive training.
The thread continues in a line of work that studies binary classification (Muthukumar et al., 2021; Chatterji and Long, 2021; Wang and Thrampoulidis, 2021) in similar overparameterized linear models. While confirming that the basic story is similar to regression, these works identify a further surprise: binary classification can work in some regimes where the corresponding regression problem would not work<sup>1</sup> due to the regularizing effect of overparameterization being too strong. Just as in the regression case, the results here are sharp in toy models: we can exactly characterize where binary classification using an interpolating classifier asymptotically generalizes.
With binary classification better understood, the thread continues to multiclass classification. After all, the current wave of deep learning enthusiasm originated in breakthrough performance in multiclass classification, and we have seen a decade of ever larger networks trained on ever larger datasets with ever more classes Kaplan et al. (2020). Using similar toy models (Muthukumar et al., 2020; 2021; Wang et al., 2022), the constant number of classes case was studied in Wang et al. (2021) to recover results similar to binary classification and subsequently generalized to general convex losses with regularization in Loureiro et al. (2021) and student-teacher networks in Cornacchia et al. (2023).
Subramanian et al. (2022) further introduced a model where the number of classes grows with the number of training points and proved an achievability result on how fast the number of classes can grow while still allowing the interpolating classifier to asymptotically generalize. While Subramanian et al. (2022) gave a conjecture for what the full region should be, there was no converse proof, and they could not show generalization in entire conjectured region. Wu and Sahai (2023) proved a partial weak converse; they showed that the misclassification rate is bounded away from 0 — rather than tending to 1 — in some of the predicted regimes.
The model, formally defined in the following section, is a stylized version of the well-known spiked covariance model (Johnstone, 2001; Donoho et al., 2018). On the theoretical front, it is related to several problems such as PCA variants (Montanari and Richard, 2015; Richard and Montanari, 2014)
<sup>1</sup>Regression failing in the overparameterized regime is linked to the empirical covariance of the limited data not revealing the spiked reality of the underlying covariance (Wang and Fan, 2017). See Appendix J of Subramanian et al. (2022). When regression doesn't generalize, we also get "support-vector proliferation" in classification problems (Muthukumar et al., 2021; Hsu et al., 2021) which is also intimately related to the phenomenon of "neural collapse" (Papyan et al., 2020) as discussed, for example, in Xu et al. (2023).
and community detection in the stochastic block model (Abbe, 2017). These models have also been applied in practice for climate studies and functional data analysis (Johnstone, 2001). At a high level, spiked covariance models can be interpreted as a linearized version of the manifold hypothesis.
The following exposition is lifted from Subramanian et al. (2022), which we include for the sake of staying consistent and self-contained. In Appendix K, we include an alternative high level framing of the problem which readers may find helpful for intuition.
We consider the multiclass classification problem with  $k$  classes. The training data consists of  $n$  pairs  $\{\pmb{x}_i,\ell_i\}_{i = 1}^n$  where  $\pmb {x}_i\in \mathbb{R}^d$  are i.i.d standard Gaussian vectors. We assume that the labels  $\ell_i\in [k]$  are generated as follows.
Assumption 1 (1-sparse noiseless model). The class labels  $\ell_{i}$  are generated based on which of the first  $k$  dimensions of a point  $\pmb{x}_i$  has the largest value,
$$
\ell_ {i} = \underset {m \in [ k ]} {\arg \max } \boldsymbol {x} _ {i} [ m ]. \tag {1}
$$
Let us emphasize at this point that the classifier that we analyze only observes the training data, and does not use any of the data-generating assumptions.
For a vector  $\pmb{x}$ , we index its  $j$ th entry with  $\pmb{x}[j]$ . Hence, under Assumption 1,  $\pmb{x}_i[m]$  can be interpreted as how representative of class  $m$  the  $i$ th training point is.
For clarity of exposition in the analysis, we make explicit a feature weighting that transforms the training points:
$$
\boldsymbol {x} _ {i} ^ {w} [ j ] = \sqrt {\lambda_ {j}} \boldsymbol {x} _ {i} [ j ] \quad \forall j \in [ d ]. \tag {2}
$$
Here  $\lambda \in \mathbb{R}^d$  contains the squared feature weights. The feature weighting serves the role of favoring the true pattern, something that is essential for good generalization. Again, we emphasize that the classifier does not do any reweighting of features; this explicit step is purely syntactic.
The weighted feature matrix  $X^w \in \mathbb{R}^{n \times d}$  is given by
$$
\boldsymbol {X} ^ {w} = \left[ \begin{array}{l l l} \boldsymbol {x} _ {1} ^ {w} & \dots & \boldsymbol {x} _ {n} ^ {w} \end{array} \right] ^ {\top} = \left[ \begin{array}{l l l} \sqrt {\lambda_ {1}} \boldsymbol {z} _ {1} & \dots & \sqrt {\lambda_ {d}} \boldsymbol {z} _ {d} \end{array} \right] \tag {3}
$$
where we introduce the notation  $\mathbf{z}_j \in \mathbb{R}^n$  to contain the  $j^{th}$  feature from the  $n$  training points. Note that  $\mathbf{z}_j \sim N(0, I_n)$  are i.i.d Gaussians. We use a one-hot encoding for representing the labels as the matrix  $Y^{\mathrm{oh}} \in \mathbb{R}^{n \times k}$
$$
\boldsymbol {Y} ^ {\mathrm {o h}} = \left[ \begin{array}{l l l} \boldsymbol {y} _ {1} ^ {\mathrm {o h}} & \dots & \boldsymbol {y} _ {k} ^ {\mathrm {o h}} \end{array} \right], \quad \text {w h e r e} \quad \boldsymbol {y} _ {m} ^ {\mathrm {o h}} [ i ] = \left\{ \begin{array}{l l} 1, & \text {i f} \ell_ {i} = m \\ 0, & \text {o t h e r w i s e} \end{array} . \right. \tag {4}
$$
Since we consider linear models, we center the one-hot encodings and define
$$
\boldsymbol {y} _ {m} \triangleq \boldsymbol {y} _ {m} ^ {\mathrm {o h}} - \frac {1}{k} \mathbf {1}. \tag {5}
$$
$^{2}$ Following previous work, we are staying within a Gaussian features framework. However, recent developments have confirmed that these models are actually predictive when the features arise from nonlinearities in a lifting, as long as there is enough randomness underneath (Hu and Lu, 2022; Lu and Yau, 2022; Goldt et al., 2022; Misiakiewicz, 2022; McRae et al., 2022; Pesce et al., 2023; Kaushik et al., 2023).
<sup>3</sup>Our weighted feature model is equivalent to other works (e.g. Muthukumar et al. (2021)) that assume that the covariates come from a  $d$ -dimensional anisotropic Gaussian with a covariance matrix  $\Sigma$  that favors the truly important directions (Wei et al., 2022). These directions do not have to be axis-aligned — we make that assumption only for notational convenience. In reality, the optimizer will never know these directions a priori.
Our classifier consists of  $k$  coefficient vectors  $\widehat{\pmb{f}}_m$  for  $m\in [k]$  that are learned by minimum-norm interpolation (MNI) of the zero-mean one-hot variants using the weighted features:4
$$
\widehat {\boldsymbol {f}} _ {m} = \arg \min  _ {\boldsymbol {f}} \| \boldsymbol {f} \| _ {2} \tag {6}
$$
$$
\text {s . t .} X ^ {w} f = y _ {m}. \tag {7}
$$
We can express these coefficients in closed form as
$$
\widehat {\boldsymbol {f}} _ {m} = \left(\boldsymbol {X} ^ {w}\right) ^ {\top} \left(\boldsymbol {X} ^ {w} \left(\boldsymbol {X} ^ {w}\right) ^ {\top}\right) ^ {- 1} \boldsymbol {y} _ {m}. \tag {8}
$$
On a test point  $\pmb{x}_{\mathrm{test}} \sim N(0, I_d)$  we predict a label as follows: First, we transform the test point into the weighted feature space to obtain  $\pmb{x}_{\mathrm{test}}^w$  where  $\pmb{x}_{\mathrm{test}}^w[j] = \sqrt{\lambda_j} \pmb{x}_{\mathrm{test}}[j]$  for  $j \in [d]$ . Then we compute  $k$  scalar "scores" and assign the class based on the largest score as follows:
$$
\hat {\ell} = \underset {1 \leq m \leq k} {\arg \max } \widehat {\boldsymbol {f}} _ {m} ^ {\top} \boldsymbol {x} _ {\text {t e s t}} ^ {w}. \tag {9}
$$
By assumption, a misclassification event  $\mathcal{E}_{\mathrm{err}}$  occurs whenever
$$
\underset {1 \leq m \leq k} {\arg \max } \boldsymbol {x} _ {\text {t e s t}} [ m ] \neq \underset {1 \leq m \leq k} {\arg \max } \widehat {\boldsymbol {f}} _ {m} ^ {\top} \boldsymbol {x} _ {\text {t e s t}} ^ {w}. \tag {10}
$$
We study where the MNI generalizes in an asymptotic regime where the number of training points, features, classes, and feature weights all scale according to the bi-level ensemble model:
Definition 1 (Bi-level ensemble). The bi-level ensemble is parameterized by  $p, q, r$  and  $t$  where  $p > 1, 0 \leq r < 1, 0 < q < (p - r)$  and  $0 \leq t < r$ . Here, parameter  $p$  controls the extent of overparameterization,  $r$  determines the number of favored features,  $q$  controls the weights on favored features and  $t$  controls the number of classes. The number of features  $(d)$ , number of favored features  $(s)$ , and number of classes  $(k)$  all scale with the number of training points  $(n)$  as follows:
$$
d = \lfloor n ^ {p} \rfloor , s = \lfloor n ^ {r} \rfloor , a = n ^ {- q}, k = c _ {k} \lfloor n ^ {t} \rfloor , \tag {11}
$$
where  $c_k$  is a positive integer. Define the feature weights by
$$
\sqrt {\lambda_ {j}} = \left\{ \begin{array}{l l} \sqrt {\frac {a d}{s}}, & 1 \leq j \leq s \\ \sqrt {\frac {(1 - a) d}{d - s}}, & \text {o t h e r w i s e} \end{array} . \right. \tag {12}
$$
We introduce the notation  $\lambda_F \triangleq \frac{ad}{s}$  and  $\lambda_U \triangleq \frac{(1 - a)d}{d - s}$  to distinguish between the (squared) favored and unfavored weights, respectively.
We visualize the bi-level model in Fig. 1, reproduced from Subramanian et al. (2022). Intuitively, the bi-level ensemble captures a simple family of overparameterized problems where learning can succeed. Although there are  $d = n^p$  features where  $d \gg n$ , there is a low dimensional subspace of favored, higher weight features of dimension  $s = n^r$ , and  $s \ll n$ . From this perspective, the bi-level model can be viewed as a parameterized version of an approximate linear manifold hypothesis. Depending on the signal strength, the noise added from the  $d - s$  unfavored features can either help generalization ("benign overfitting") or overwhelm the true signal and cause the classifier to fail.
In this section we state our main results and compare them to what was known and conjectured previously. Subramanian et al. (2022) use heuristic calculations to conjecture necessary and sufficient conditions for the bi-level model to generalize; we restate the conjecture here for reference.
4The classifier learned via this method is equivalent to those obtained by other natural training methods (SVMs or gradient-descent with exponential tailed losses like cross-entropy) under sufficient overparameterization (Wang et al., 2021; Kaushik et al., 2023). Recently, Lai and Muthukumar (2023) showed via an extension of Ji and Telgarsky (2021) that a much broader category of losses also asymptotically result in convergence to the same MNI solution for sufficiently overparameterized classification problems.
<sup>5</sup>Such models are widely used to study learning even beyond this particular thread of work. For example, Tan et al. (2023) uses this to understand the privacy/generalization tradeoff of overparameterized learning.
Figure 1: Bi-level feature weighting model. The first  $s$  features have a higher weight and are favored during minimum-norm interpolation. These can be thought of as the square-roots of the eigenvalues of the feature covariance matrix  $\Sigma$  in a Gaussian model for the covariates as in Bartlett et al. (2020).
Conjecture 3.1 (Conjectured bi-level regions). Under the bi-level ensemble model (Definition 1), when the true data generating process is 1-sparse (Assumption 1), as  $n \to \infty$ , the probability of misclassification  $\mathrm{Pr}[\mathcal{E}_{\mathrm{err}}]$  for MNI as described in Eq. (6) satisfies
$$
\Pr \left[ \mathcal {E} _ {\text {e r r}} \right]\rightarrow \left\{\begin{array}{l l}0,&\text {i f} t <   \min  \left\{1 - r, p + 1 - 2 \max  \left\{1, q + r \right\} \right\}\\1,&\text {i f} t > \min  \left\{1 - r, p + 1 - 2 \max  \left\{1, q + r \right\} \right\}\end{array}. \right. \tag {13}
$$
Our main theorem establishes that Conjecture 3.1 indeed captures the correct generalization behavior of the overparameterized linear model.
Theorem 3.2 (Generalization for bi-level-model). Under the bi-level ensemble model (Definition 1), when the true data generating process is 1-sparse (Assumption 1), Conjecture 3.1 holds.
For comparison, we quote the best known previous positive and negative results for the bi-level model, which only hold in the restricted regime where regression fails ( $q + r > 1$ ).
Theorem 3.3 (Generalization for bi-level model (Subramanian et al., 2022)). In the same setting as Conjecture 3.1, in the regime where regression fails ( $q + r > 1$ ), as  $n \to \infty$  we have  $\operatorname{Pr}[\mathcal{E}_{\mathrm{err}}] \to 0$  if
$$
t <   \min  \{1 - r, p + 1 - 2 (q + r), p - 2, 2 q + r - 2 \}. \tag {14}
$$
Theorem 3.4 (Misclassification in bi-level model (Wu and Sahai, 2023)). In the same setting as Conjecture 3.1, in the regime where regression fails ( $q + r > 1$ ), as  $n \to \infty$  we have  $\operatorname*{Pr}[\mathcal{E}_{\mathrm{err}}] \geq \frac{1}{2}$  if
$$
t > \min  \{1 - r, p + 1 - 2 (q + r) \}. \tag {15}
$$
Let us interpret the different conditions in Conjecture 3.1. To interpret the condition  $t < 1 - r$ , first rearrange it to  $t + r < 1$ . Recall that the parameter  $r$  controls the number of favored features, and hence is a proxy for the "effective dimension" of the problem. On the other hand, the parameter  $t$  controls the number of classes, so in a loose sense there are  $n^{t + r}$  parameters being learned. From this perspective, the condition  $t + r < 1$  says that the problem is "effectively underparameterized".
The other condition on  $p + 1 - 2\max \{1,q + r\}$  comes from looking at the noise from the unfavored features. To see why, recall that the squared favored feature weighting is  $\lambda_F = n^{p - q - r}$ . So for fixed  $p$ , the quantity  $q + r$  controls the level of favored feature weighting. When  $q + r > 1$ , the favored feature weighting is small enough that regression fails, and the empirical covariance becomes flat. In this case, the condition becomes  $t < p + 1 - 2(q + r) = (p - q - r) + 1 - (q + r)$ . As  $q + r$  increases, the amount of favoring decreases, making it harder to generalize.
For ease of comparison between our main result and Theorems 3.3 and 3.4, we visualize the regimes in Fig. 2, as in Subramanian et al. (2022); Wu and Sahai (2023). In particular, the blue starred and dashed regions in Fig. 2 indicate how Theorem 3.3 only applies where regression fails. In contrast, our new result holds regardless of whether regression fails or not, as in the the green diamond region and light blue triangle regions. The regions are also completely tight; the looseness between the prior Theorem 3.3 and our result can be seen in the light blue square region.
The weak converse in the prior Theorem 3.4 captures some of the correct conditions for misclassification, but again only when  $q + r > 1$ . As depicted in the maroon X region for  $r < 0.25$  in Fig. 2b, our main theorem gives a strong converse, whereas Theorem 3.4 has nothing to say because  $q + r < 1$ . Theorem 3.4 also only proves that the misclassification rate is asymptotically at least  $\frac{1}{2}$ . In the red circle and maroon X regions, we illustrate how our result pushes the misclassification rate to  $1 - o(1)$ , which requires a more refined analysis. We elaborate on this further in Section 4.
We remark that it is simpler to analyze the case where regression fails, as the random matrices that arise in the analysis are flat, i.e. approximately equal to a scaled identity matrix. However, in the regime where regression works, the same matrices have a spiked spectrum, which complicates the analysis. To smoothly handle both cases, we leverage a new variant of the Hanson-Wright inequality to show concentration of certain sparse bilinear forms; see Section 4.1 for more details.
Figure 2: Example of regimes for multiclass/binary classification and regression. The white regions correspond to invalid regimes under the bi-level model. The entirety of 2b and all the light blue regions are new to this paper, as is showing that the error tends to 1 in the maroon regions.
We now sketch out the proof for our main theorem. As in Subramanian et al. (2022); Wu and Sahai (2023), the starting point is writing out the necessary and sufficient conditions for misclassification.
Assume without loss of generality that the test point  $\boldsymbol{x}_{\mathrm{test}} \sim N(0, I_d)$  has true label  $\alpha$  for some  $\alpha \in [k]$ . Let  $\boldsymbol{x}_{\mathrm{test}}^w$  be the weighted version of this test point. From (10), an equivalent condition for misclassification is that for some  $\beta \neq \alpha, \beta \in [k]$ , we have  $\widehat{\boldsymbol{f}}_{\alpha}^{\top} \boldsymbol{x}_{\mathrm{test}}^w < \widehat{\boldsymbol{f}}_{\beta}^{\top} \boldsymbol{x}_{\mathrm{test}}^w$ , i.e. the score for  $\beta$  outcompetes the score for  $\alpha$ . Define the Gram matrix  $A \triangleq X^w(X^w)^{\top}$ , the relative label vector  $\Delta y \triangleq y_{\alpha} - y_{\beta} \in \{-1, 0, 1\}^n$ , and the relative survival vector  $\widehat{h}_{\alpha,\beta} \in \mathbb{R}^d$  which compares the signal from  $\alpha$  and  $\beta$ :
$$
\begin{array}{l} \widehat {\boldsymbol {h}} _ {\alpha , \beta} [ j ] \triangleq \lambda_ {j} ^ {- 1 / 2} \left(\widehat {\boldsymbol {f}} _ {\alpha} [ j ] - \widehat {\boldsymbol {f}} _ {\beta} [ j ]\right) (16) \\ = \boldsymbol {z} _ {j} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Delta} y, (17) \\ \end{array}
$$
where to obtain the last line we have used (8). By converting the misclassification condition into the unweighted feature space we see that we will have errors when
$$
\lambda_ {\alpha} \widehat {\boldsymbol {h}} _ {\alpha , \beta} [ \alpha ] \boldsymbol {x} _ {\text {t e s t}} [ \alpha ] - \lambda_ {\beta} \widehat {\boldsymbol {h}} _ {\beta , \alpha} [ \beta ] \boldsymbol {x} _ {\text {t e s t}} [ \beta ] <   \sum_ {j \notin \{\alpha , \beta \}} \lambda_ {j} \widehat {\boldsymbol {h}} _ {\beta , \alpha} [ j ] \boldsymbol {x} _ {\text {t e s t}} [ j ]. \tag {18}
$$
Define the contamination term  $\mathsf{CN}_{\alpha ,\beta}$
$$
\mathrm {C N} _ {\alpha , \beta} \triangleq \sqrt {\sum_ {j \notin \{\alpha , \beta \}} \lambda_ {j} ^ {2} (\hat {\boldsymbol {h}} _ {\beta , \alpha} [ j ]) ^ {2}}. \tag {19}
$$