SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Then applying Lemmas H.1 and H.2, for  $i \in [s - |T|]$  we conclude that
$$
\mu_ {i} (\boldsymbol {A} _ {- T}) = \left(1 \pm c _ {T} \sqrt {\frac {| T |}{n}}\right) \mu^ {- 1} n ^ {p} + \left(1 \pm c _ {1 8} n ^ {- \kappa_ {7}}\right) n ^ {p}. \tag {212}
$$
which proves Eq. (208).
For  $i > s - |T|$ , applying Lemma H.2 and the fact that  $\mu_i(\mathbf{W}_{[s]\backslash T}\mathbf{W}_{[s]\backslash T}^\top) = 0$  to Eq. (211) yields
$$
\mu_ {i} \left(\boldsymbol {A} _ {- T}\right) = \left(1 \pm c _ {1 8} n ^ {- \kappa_ {7}}\right) n ^ {p}. \tag {213}
$$
which proves Eq. (209).
By inverting the bounds proved above, we can also control the spectrum of  $A_{-T}^{-1}$ .
Corollary H.4. Recall that  $A_{-T} \in \mathbb{R}^{n \times n}$ . For any nonempty  $T \subseteq [s]$ , with probability at least  $1 - 2e^{-\sqrt{n}} - 2e^{-n}$ , we have that for all  $i \in [n - s + |T|]$
$$
\mu_ {i} \left(\boldsymbol {A} _ {- T} ^ {- 1}\right) = \left(1 \pm c _ {1 9} n ^ {- \kappa_ {\tau}}\right) n ^ {- p} \tag {214}
$$
For all  $i \in [n] \setminus [n - s + |T|]$ , we have
$$
\mu_ {i} \left(\boldsymbol {A} _ {- T} ^ {- 1}\right) = \min  \left\{\mu , 1 \right\} \left(1 \pm c _ {2 0} n ^ {- \kappa_ {8}}\right) n ^ {- p}. \tag {215}
$$
where  $\kappa_{8}$  is a positive constant depending on  $|T|$ .
Proof. By inverting the bounds in Lemma H.3, using the fact that  $\mu_i(\mathbf{A}_{-T}^{-1}) = \frac{1}{\mu_{n - i + 1}(\mathbf{A}_{-T})}$  we see that for  $i\in [n - s + |T|]$ ,
$$
\begin{array}{l} \mu_ {i} \left(\boldsymbol {A} _ {- T} ^ {- 1}\right) = \frac {1}{1 \pm c _ {1 8} n ^ {- \kappa_ {7}}} n ^ {- p} (216) \\ = \left(1 \pm c _ {1 9} n ^ {- \kappa_ {7}}\right) n ^ {- p}, (217) \\ \end{array}
$$
where we have used the power series expansion  $\frac{1}{1 - x} = 1 + x + o(x^2)$  and  $c_{19}$  is a positive constant.
On the other hand, for  $i > n - s + |T|$ , we get
$$
\begin{array}{l} \mu_ {i} \left(\boldsymbol {A} _ {- T} ^ {- 1}\right) = \frac {1}{\left(1 \pm c _ {T} \sqrt {\frac {| T |}{n}}\right) \mu^ {- 1} + \left(1 \pm c _ {1 8} n ^ {- \kappa_ {7}}\right)} n ^ {- p} (218) \\ = \min  \left\{\mu , 1 \right\} \left(1 \pm c _ {2 0} n ^ {- \kappa_ {8}}\right) n ^ {- p}, (219) \\ \end{array}
$$
where  $c_{20}$  and  $\kappa_{8}$  are positive constants defined as follows. If  $q + r < 1$ , i.e. regression works, then  $\mu^{-1} = \omega(1)$ , so the denominator becomes  $\mu^{-1}\left(1 \pm c_T \sqrt{\frac{|T|}{n}} + \mu(1 \pm c_{18} n^{-\kappa_7})\right)$ . Then, since  $|T| \leq s = n^r$ , we see that we can pick
$$
\kappa_ {8} = \min  \left\{\frac {1 - r}{2}, 1 - q - r \right\}.
$$
On the other hand, if  $q + r > 1$ , i.e. regression fails, then  $\mu^{-1} = o(1)$ , and so we can define
$$
\kappa_ {8} = \min  \left\{\kappa_ {7}, q + r - 1 \right\}.
$$
Hence to cover both cases we can pick
$$
\kappa_ {8} = \min  \left\{\frac {1 - r}{2}, \kappa_ {7}, | 1 - q - r | \right\}.
$$
The choice of  $c_{20}$  is picked by again using the power series expansion for  $\frac{1}{1 - x}$ .
Note that Corollary H.4 immediately implies Proposition B.4, with  $\kappa_{9}$  defined based on picking  $T = [k]$ . We are now in a position to prove that the generalized hat matrices  $\pmb{H}_{T,S}$ , and hence the Woodbury terms  $(\pmb{I}_{|T|} + \pmb{H}_{T,S})^{-1}$  have a flat spectrum as well.
$$
\mu_ {i} \left(\left(\boldsymbol {I} _ {| T |} + \boldsymbol {H} _ {T, S}\right) ^ {- 1}\right) = \min  \left\{\mu , 1 \right\} (1 \pm c _ {T, S} n ^ {- \kappa_ {1 1}}). \tag {75}
$$
Proof. We seek to control the spectrum of the hat matrix  $\pmb{H}_{T,S} = \pmb{W}_T^\top \pmb{A}_{-S}^{-1}\pmb{W}_T$ . We cannot directly use naive eigenvalue bounds to bound the minimum and maximum eigenvalue, as this does not rule out the possibility that  $\pmb{H}_{T,S}$  has a spike. Instead, we control the spectrum from first principles.
The spectrum of  $H_{T,S}$  is flat: Since  $A_{-S}^{-1}$  is symmetric, it has an eigendecomposition  $VDV^{\top}$ , where  $V$  is an orthogonal matrix. Because  $W_T$  is a weighted subset of only the (equally) favored features, its law is rotationally invariant. Furthermore, since  $A_{-S}^{-1}$  is independent of  $W_T$  (as  $T \subseteq S$ ), we can absorb the rotation  $V$  into  $W_T$  to reduce to the case where  $A_{-S}^{-1} = D$ . Here, we have
$$
\boldsymbol {D} \triangleq \left[ \begin{array}{l l} \boldsymbol {D} _ {\text {f l a t}} & \\ & \boldsymbol {D} _ {\text {s p i k e d}} \end{array} \right] \in \mathbb {R} ^ {n \times n} = \left[ \begin{array}{c c c} \mu_ {1} \left(\boldsymbol {A} _ {- S} ^ {- 1}\right) & & \\ & \ddots & \\ & & \mu_ {n} \left(\boldsymbol {A} _ {- S} ^ {- 1}\right) \end{array} \right], \tag {220}
$$
where  $D_{\mathrm{flat}} \in \mathbb{R}^{(n - s + |T|) \times (n - s + |T|)}$  and  $D_{\mathrm{spiked}} \in \mathbb{R}^{(s - |T|) \times (s - |T|)}$  correspond to the flat and (downwards) spiked portions of the spectrum of  $A_{-S}^{-1}$ . We can also correspondingly decompose
$$
\mathbf {Z} _ {T} = \left[ \begin{array}{l} \mathbf {B} _ {T} \\ \mathbf {C} _ {T} \end{array} \right], \tag {221}
$$
where  $\pmb{B}_{T} \in \mathbb{R}^{(n - s + |T|) \times |T|}$  and  $\pmb{C}_{T} \in \mathbb{R}^{(s - |T|) \times |T|}$ . Note that each entry of these matrices are i.i.d.  $N(0,1)$  variables.
By direct computation we have
$$
\boldsymbol {Z} _ {T} ^ {\top} \boldsymbol {D} \boldsymbol {Z} _ {T} = \boldsymbol {B} _ {T} ^ {\top} \boldsymbol {D} _ {\text {f l a t}} \boldsymbol {B} _ {T} + \boldsymbol {C} _ {T} ^ {\top} \boldsymbol {D} _ {\text {s p i k e d}} \boldsymbol {C} _ {T} \tag {222}
$$
We thus have by using standard eigenvalue inequalities that
$$
\mu_ {| T |} \left(\boldsymbol {Z} _ {T} ^ {\top} \boldsymbol {D} \boldsymbol {Z} _ {T}\right) \geq \mu_ {| T |} \left(\boldsymbol {B} _ {T} ^ {\top} \boldsymbol {D} _ {\text {f l a t}} \boldsymbol {B} _ {T}\right) + \mu_ {| T |} \left(\boldsymbol {C} _ {T} ^ {\top} \boldsymbol {D} _ {\text {s p i k e d}} \boldsymbol {C} _ {T}\right) \tag {223}
$$
$$
\geq \mu_ {| T |} \left(\boldsymbol {B} _ {T} ^ {\top} \boldsymbol {B} _ {T}\right) \mu_ {n - s + | T |} \left(\boldsymbol {A} _ {- S} ^ {- 1}\right) + \mu_ {| T |} \left(\boldsymbol {C} _ {T} ^ {\top} \boldsymbol {C} _ {T}\right) \mu_ {n} \left(\boldsymbol {A} _ {- S} ^ {- 1}\right) \tag {224}
$$
$$
\geq \mu_ {| T |} \left(\boldsymbol {B} _ {T} ^ {\top} \boldsymbol {B} _ {T}\right) \mu_ {n - s + | T |} \left(\boldsymbol {A} _ {- S} ^ {- 1}\right), \tag {225}
$$
where in the last line we have used  $\mu_n(A_{-S}^{-1})\geq 0$
Since  $n > s$ , we have  $n - s + |T| > |T|$ , so we can apply Wishart concentration (Lemma B.3) to  $B_T^\top B_T$  to obtain that with probability at least  $1 - 2e^{-\sqrt{n}}$  we have
$$
\mu_ {| T |} \left(\boldsymbol {B} _ {T} ^ {\top} \boldsymbol {B} _ {T}\right) \geq n - s + | T | - 2 \sqrt {(n - s + | T |) | T |} + o (\sqrt {(n - s + | T |) | T |}) \tag {226}
$$
$$
\geq n \left(1 - n ^ {r - 1} - c _ {2 1} \sqrt {\frac {| T |}{n}}\right), \tag {227}
$$
where  $c_{21}$  is a positive constant.
On the other hand, we can deduce that
$$
\mu_ {1} \left(\mathbf {Z} _ {T} ^ {\top} \mathbf {D} \mathbf {Z} _ {T}\right) \leq \mu_ {1} \left(\mathbf {Z} _ {T} ^ {\top} \mathbf {Z} _ {T}\right) \mu_ {1} \left(\mathbf {A} _ {- S} ^ {- 1}\right). \tag {228}
$$
Lemma H.1 implies that with probability at least  $1 - 2e^{-\sqrt{n}}$
$$
\mu_ {1} \left(\boldsymbol {Z} _ {T} ^ {\top} \boldsymbol {Z} _ {T}\right) \leq n \left(1 + c _ {T} \sqrt {\frac {| T |}{n}}\right) \tag {229}
$$
Similarly, Corollary H.4 implies that with probability at least  $1 - 2e^{-n} - 2e^{-\sqrt{n}}$ ,  $\mu_1(A_{-S}^{-1})$  and  $\mu_{n - s + |T|}(A_{-S}^{-1})$  are both  $(1\pm c_{20}n^{\kappa_8})n^{-p}$ . Since  $|T|\leq s = n^{r}$ , Eqs. (227) and (229) together demonstrate that for all  $i\in [|T|]$ ,
$$
\mu_ {i} \left(\boldsymbol {Z} _ {T} ^ {\top} \boldsymbol {D} \boldsymbol {Z} _ {T}\right) = n ^ {1 - p} \left(1 \pm c _ {2 2} n ^ {- \kappa_ {1 0}}\right). \tag {230}
$$
Here,  $c_{22}$  and  $\kappa_{10}$  are positive constants defined as follows. Since  $|T| \leq s = n^r$ . Then  $\kappa_{10} = \min \left\{1 - r, \frac{1 - r}{2}, \kappa_8\right\}$ , and  $c_{22}$  is a constant chosen appropriately based on  $c_{20}$  and  $c_{21}$ . Plugging in the scaling  $\lambda_F = n^{p - q - r}$ , we conclude that with extremely high probability, for all  $i \in [|T|]$ ,
$$
\mu_ {i} \left(\boldsymbol {H} _ {T, S}\right) = \mu^ {- 1} \left(1 \pm c _ {2 2} n ^ {- \kappa_ {1 0}}\right). \tag {231}
$$
From here, it is easy to compute the spectrum of  $(\pmb{I}_{|T|} + \pmb{H}_{T,S})^{-1}$ . Indeed, reading off our result from Eq. (231) yields
$$
\mu_ {i} \left(\left(\boldsymbol {I} _ {| T |} + \boldsymbol {H} _ {T, S}\right) ^ {- 1}\right) = \frac {1}{1 + \mu_ {n - i + 1} \left(\boldsymbol {H} _ {T , S}\right)} \tag {232}
$$
$$
= \min  \left\{\mu , 1 \right\} \left(1 \pm c _ {T, S} n ^ {- \kappa_ {1 1}}\right). \tag {233}
$$
Here, the positive constant  $c_{T,S}$  is picked appropriately and  $\kappa_{11} = \min \left\{\kappa_{10}, |1 - q - r| \right\} > 0$ . This completes the proof.
I Miscellaneous lemmas
Lemma I.1 (Coupling of quadratic forms). Let  $\pmb{B} \in \mathbb{R}^{n \times m}$  be an arbitrary real matrix and  $\pmb{M} \in \mathbb{R}^{n \times n}$  be a PSD matrix. Then for any vector  $\pmb{x} \in \mathbb{R}^m$ , we have
$$
\lambda_ {n} (\boldsymbol {M}) \boldsymbol {x} ^ {\top} \boldsymbol {B} ^ {\top} \boldsymbol {B} \boldsymbol {x} \leq \boldsymbol {x} ^ {\top} \boldsymbol {B} ^ {\top} M \boldsymbol {B} \boldsymbol {x} \leq \lambda_ {1} (\boldsymbol {M}) \boldsymbol {x} ^ {\top} \boldsymbol {B} ^ {\top} \boldsymbol {B} \boldsymbol {x}. \tag {234}
$$
Proof. For any PSD matrix  $C$ , the matrix  $B^\top CB$  is PSD. In particular,  $C$  has a unique square root  $C^{1/2} \in \mathbb{R}^{n \times n}$  with  $C^{1/2} C^{1/2} = (C^{1/2})^\top C^{1/2} = C$ . We thus have
$$
\begin{array}{l} \boldsymbol {x} ^ {\top} \boldsymbol {B} ^ {\top} \boldsymbol {C} \boldsymbol {B} \boldsymbol {x} = \boldsymbol {x} ^ {\top} \boldsymbol {B} ^ {\top} \boldsymbol {C} ^ {1 / 2} \boldsymbol {C} ^ {1 / 2} \boldsymbol {B} \boldsymbol {x} (235) \\ = \left\| \boldsymbol {C} ^ {1 / 2} \boldsymbol {B} \boldsymbol {x} \right\| _ {2} ^ {2} \geq 0. (236) \\ \end{array}
$$
$$
\lambda_ {1} (M) \boldsymbol {x} ^ {\top} B ^ {\top} B \boldsymbol {x} - \boldsymbol {x} ^ {\top} B ^ {\top} M B \boldsymbol {x} = \boldsymbol {x} ^ {\top} B ^ {\top} \left(\lambda_ {1} (M) I _ {n} - M\right) B \boldsymbol {x}. \tag {237}
$$
Since  $M \preceq \lambda_1(M)I_n$  by definition,  $\lambda_1(M)I_n - M$  is a PSD matrix. Hence by applying Eq. (236), we conclude that
$$
\lambda_ {1} (M) \boldsymbol {x} ^ {\top} B ^ {\top} B \boldsymbol {x} - \boldsymbol {x} ^ {\top} B ^ {\top} M B \boldsymbol {x} \geq 0, \tag {238}
$$
which gives the upper bound in Eq. (234).
Similarly,  $M\succeq \lambda_{n}(M)I_{n}$ , so an analogous argument
$$
\lambda_ {n} (\boldsymbol {M}) \boldsymbol {x} ^ {\top} \boldsymbol {B} ^ {\top} \boldsymbol {B} \boldsymbol {x} - \boldsymbol {x} ^ {\top} \boldsymbol {B} ^ {\top} \boldsymbol {M} \boldsymbol {B} \boldsymbol {x} \leq 0, \tag {239}
$$
which gives the lower bound in Eq. (234).
Next, we prove the elementary anti-concentration result that we will need.
Proposition 1.2 (Gaussian anticoncentration). Let  $\mathbf{x} \sim N(0, \mathbf{I}_d)$  be a standard Gaussian vector, and let  $\mathbf{v} \in \mathbb{R}^d$  be arbitrary deterministic vector. Then
$$
\Pr \left[ | \langle \boldsymbol {x}, \boldsymbol {v} \rangle | \leq \epsilon \right] \leq \frac {2 \epsilon}{\sqrt {2 \pi} \| v \| _ {2}}.
$$
Proof. Note that  $\langle \pmb{x},\pmb{v}\rangle$  is a linear projection of a standard multivariate Gaussian, so it is itself a one-dimensional Gaussian. It is also clearly zero mean, and its variance is just given by the squared norm of  $\pmb{v}$ . So  $\langle \pmb{x},\pmb{v}\rangle \sim N(0,\| \pmb{v}\|_2^2)$ . Now we have
$$
\Pr [ | \langle \boldsymbol {x}, \boldsymbol {v} \rangle | \leq \epsilon ] = \frac {1}{\sqrt {2 \pi} \| \boldsymbol {v} \| _ {2}} \int_ {- \epsilon} ^ {\epsilon} \exp \left(- \frac {x ^ {2}}{\| \boldsymbol {v} \| _ {2} ^ {2}}\right) d x \leq \frac {2 \epsilon}{\sqrt {2 \pi} \| \boldsymbol {v} \| _ {2}}.
$$
J Comparison to the straightforward non-interpolative scheme
In this section, we quickly give calculations for how well a straightforward non-interpolating scheme for learning classifiers can work asymptotically. However, a similar analysis using the tools developed to prove our main results should give a rigorous proof of the below derivation.
This scheme simply uses the sum/average of all positive training examples of a class as the vector we take an inner-product with to generate scores for classifying test points. For  $m \in [k]$ , define
$$
\widehat {\boldsymbol {f}} _ {m} = \sum_ {i: \ell_ {i} = m} \boldsymbol {x} _ {i} ^ {w}. \tag {240}
$$
To understand how well this will do asymptotically, it is easy to see that the true label-defining direction, the positive exemplars in the bi-level model will be tightly concentrating around  $\sqrt{2\log k}\sqrt{\lambda_F}$  which, keeping only the polynomial-order scaling, will be like  $n^{\frac{p - q - r}{2}}$ . There will be roughly  $\frac{n}{k} = n^{1 - t}$  positive examples for every class with high probability. For simplicity, let us just look at  $m = 1$  and consider  $\frac{k}{n}\widehat{f}_1 = n^{t - 1}\widehat{f}_1$ . We see
$$
n ^ {t - 1} \widehat {\boldsymbol {f}} _ {1} [ 1 ] = \Theta \left(n ^ {\frac {p - q - r}{2}}\right). \tag {241}
$$
For the other directions that are not true-label defining, we will just have random Gaussians. The favored directions will be Gaussian with variance  $\lambda_F = n^{p - q - r}$  while the unfavored directions will essentially be Gaussian with unit variance. By averaging over  $n^{1 - t}$  examples, those variances will be reduced by that factor. This means that for the  $s = n^r$  favored directions, the variance of the average will be  $n^{p - q - r - (1 - t)}$  each and for the essentially  $n^p$  unfavored directions, the variance of the average will be  $n^{t - 1}$  each.
On a test point, we are going to take the inner product of  $n^{t - 1}\widehat{f}_m$  with an independent random draw of  $x_{\mathrm{test}}^w$ . For classification to succeed, we need this inner product to be dominated by the true
$m$ -th feature-defining direction. When that happens, the correct label will win the comparison. One can easily see that the contribution from the true feature-defining direction will be a Gaussian with mean 0 and variance  $\lambda_F \cdot (n^{\frac{p - q - r}{2}})^2 = \lambda_F^2 = n^{2p - 2q - 2r}$ . Meanwhile, the  $s$  favored features will have their scaled variances sum up in the score to give a total variance of  $n^r \cdot \lambda_F \cdot n^{p - q - r - (1 - t)} = n^{2p - 2q - r - (1 - t)}$ . And finally, the unfavored features will also have their variances sum up in the score to give a total variance of  $n^p \cdot 1 \cdot n^{t - 1} = n^{p + t - 1}$ .
For the true-feature-defining direction to dominate the contamination from other favored directions, we need
$$
2 p - 2 q - 2 r > 2 p - 2 q - r - (1 - t) \tag {242}
$$
which immediately gives the condition  $t < 1 - r$ .
For the true-feature-defining direction to dominate the contamination from other unfavored directions, we need
$$
2 p - 2 q - 2 r > p + t - 1 \tag {243}
$$
which gives the condition  $t < p + 1 - 2(q + r)$ .
Here, there is no difference between regimes in which regression works or does not work. The condition for classification to asymptotically succeed is  $t < \min(1 - r, p + 1 - 2(q + r))$ .
Notice that when MNI regression does not work  $q + r > 1$ , this is identical to the tight characterization given for MNI classification in (13). But in the regime where MNI regression does work  $q + r < 1$ , this is different. For MNI classification, (13) tells us that we require  $t < \min(1 - r, p - 1)$ . Consider  $q = 0.1, r = 0.5$  and  $p = 1.1$ . MNI classification can only allow  $t < 0.1$ . Meanwhile, the non-interpolating average-of-positive-examples classifier will work as long as  $t < 0.5$ . This demonstrates the potential for significant suboptimality (in terms of the number of distinct classes that can be learned) of MNI classifiers in this regime of benign overfitting for regression.
K Alternative framing of bi-level ensemble
In this section we provide an alternative exposition for the bi-level ensemble, which may be more intuitive for some readers.
The high level conceptual picture of the setup is as follows. The learner simply observes high dimensional jointly Gaussian zero-mean features with some unknown covariance  $\Sigma \in \mathbb{R}^{d\times d}$ . It then performs min-norm interpolation of (essentially) one-hot-encoded labels to learn the score functions for the  $k$  different classes. These score functions are used at test-time to do multiclass classification.
Note that the learning algorithm has no knowledge of  $\Sigma$ , nor does it even know that the features are jointly Gaussian — all it has is the training data. For analysis purposes, we parameterize  $\Sigma$ . In the spirit of spiked covariance models, where a low-dimensional subspace has higher variance, we study the case that the eigenvalues of  $\Sigma$  follow the simplified bi-level model parameterized by  $(p,q,r)$ .
The bi-level model stipulates that (i) the number of features is  $d = n^p$  and (ii) there are two discrete variance levels for the Gaussian features. The higher variance component resides in a low-dimensional subspace of dimension  $s = n^r$ . For this bi-level model, we are able to prove sharp phase-transition style results telling us when successful generalization will happen. Here, the number of classes also matters, and that is where the parameter  $t$  enters.
To simplify notation for analysis in the paper, Subramanian et al. (2022) just assume that  $\Sigma$  is diagonal to begin with, instead of explicitly rotating coordinates to the eigenbasis of a general  $\Sigma$ . Because Gaussianity is preserved under rotations and min-norm interpolation only cares about norms, this transformation is without loss of generality. We reiterate that the learner is agnostic to all of these choices and does not know  $\Sigma$  at any point.