SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\text {w h e r e} \mathcal {X} (\alpha , \nu , W, \Lambda) \equiv \sqrt {\left(\alpha (W + 1) + \nu (\Lambda - W - 1)\right) ^ {2} + 4 \Lambda \nu^ {2} (W + 1)} \tag {198}
$$
making the same substitutions, we can then write
$$
\hat {q} = [ s (1 - c) ] ^ {- 1} \mathcal {Q} \tag {199}
$$
$$
\text {w h e r e} \quad \mathcal {Q} \equiv \frac {2 \alpha \nu}{(- \alpha (W + 1) + \nu (\Lambda + W + 1) + X)} \tag {200}
$$
Continuing to make substitutions in this manner, we arrive at:
$$
S = [ s (1 - c) ] ^ {- 1} \mathcal {S} \tag {201}
$$
$$
\text {w h e r e} \quad \mathcal {S} \equiv \frac {\mathcal {Q}}{\nu + \mathcal {Q} (1 + W)} \tag {202}
$$
Finally, we may express the generalization error in a reasonably compact form. In the special case at hand, the generalization error is written:
$$
E _ {k} = \frac {1}{k} E _ {r r} \left(\nu_ {r r} = \frac {1}{k}\right) + \frac {k - 1}{k} E _ {r r ^ {\prime}} \left(\nu_ {r r ^ {\prime}} = \frac {1}{k} \delta_ {r r ^ {\prime}}\right) \tag {203}
$$
Substituting 201 for S in eq. 16, and making further substitutions as necessary, we arrive at:
$$
E _ {r r} \left(\nu_ {r r} = \frac {1}{k}\right) = s (1 - c) \mathcal {E} _ {r r} (k, \alpha , \rho , \Lambda , H, W, Z) \tag {204}
$$
$$
\mathcal {E} _ {r r} = \frac {N _ {1} + N _ {2}}{D} \tag {205}
$$
$$
N _ {1} = \alpha (H + 1) k + \mathcal {S} (\mathcal {S} (W + 1) (\alpha + W Z + Z) - 2 \alpha) \tag {206}
$$
$$
N _ {2} = \alpha \rho^ {2} (k - S) (W (k + S) + k + S - 2) \tag {207}
$$
$$
D = \alpha k - \mathcal {S} ^ {2} (W + 1) ^ {2} \tag {208}
$$
We also obtain
$$
E _ {r r ^ {\prime}} \left(\nu_ {r r ^ {\prime}} = \frac {1}{k} \delta_ {r r ^ {\prime}}\right) = s (1 - c) \mathcal {E} _ {r r ^ {\prime}} (k, \alpha , \rho , \Lambda , H, W, Z) \tag {209}
$$
$$
\mathcal {E} _ {r r ^ {\prime}} = \frac {2 (\rho^ {2} - 1) \mathcal {S}}{k} - 2 \rho^ {2} + 1 \tag {210}
$$
Combining these, we obtain the error of the ensemble as:
$$
E _ {k} = s (1 - c) \mathcal {E} (k, \alpha , \rho , \Lambda , H, W, Z) \tag {211}
$$
$$
\mathcal {E} = \frac {1}{k} \mathcal {E} _ {r r} + \frac {k - 1}{k} \mathcal {E} _ {r r ^ {\prime}} \tag {212}
$$
These results are derived in the Mathematica notebook titled "EquiCorrParameterReduction.nb" included with the available code. It follows from eq. 211 that when  $\lambda = 0$  error can be written in the form of eq. 22. It follows from equations 211, 209 that at "locally optimal" regularization  $\lambda^{*}$ , error can be written in the form of eq. 23. To see this, note that the reduced regularization  $\Lambda^{*}$  which minimizes  $E_{rr}$  will only depend on the other arguments of  $\mathcal{E}_{rr}$ . Full expressions for the generalization error in the case  $\lambda \rightarrow 0$  and  $\lambda = \lambda^{*}$  can be found in the mathematica notebooks "EquiCorrPhaseDiagram_ZeroReg.nb" and "EquiCorrPhaseDiagram_ LocalReg.nb" included with the available code. These equations are long and difficult to interpret - nor are they directly used in our code - and so are omitted here.
H.4.1 The Intermediate to Noise-Dominated Transition
The transition between the intermediate regime where  $1 < k^{*} < \infty$  and the noise-dominated regime where  $k^{*} = \infty$  can be studied analytically relatively painlessly in the ridgeless limit  $\lambda \to 0$ . The strategy we employ to determine this phase boundary is to examine the large- $k$  asymptotic expansion of  $E_{k}$  to determine whether the error approaches its asymptotic value from below or above. If  $E_{k}$  approaches  $E_{\infty}$  from below, then  $k^{*}$  must be finite. If  $E_{k}$  approaches  $E_{\infty}$  from above, then  $k^{*}$  may be infinite - however, there is still the possibility of  $k^{*} < \infty$  if  $E_{k}$  is non-monotonic in  $k$ . In practice, we check the values of  $E_{k}$  for  $k = 1,2,\ldots,100$  and  $k = \infty$ .
Setting  $\Lambda = 0$  and expanding  $E_{k}$  around  $k = \infty$ , we find:
$$
\frac {E _ {k}}{s (1 - c)} = 1 - \rho^ {2} + \rho^ {2} W + \frac {\left((1 + W) ^ {2} \rho^ {2} + \alpha (- 2 + H + H W + 2 \rho^ {2})\right)}{(1 + W) \alpha k} + \mathcal {O} \left(\frac {1}{k ^ {2}}\right) \tag {213}
$$
Setting the coefficient of  $k^{-1}$  to zero, we find the phase boundary as:
$$
\alpha = \frac {(1 + W) ^ {2} \rho^ {2}}{2 (1 - \rho^ {2}) - H (1 + W)} \tag {214}
$$
This equation explains the shapes of the boundaries between the intermediate and noise-dominated regions in the phase diagrams of fig. 4 with  $\lambda = 0$  (see black dotted lines in panels b, c, d).
An analytical formula for the boundary between the intermediate and noise-dominated regimes at locally optimal regularization  $\lambda = \lambda^{*}$  cannot be easily obtained. To understand why, we can assess the large-  $k$  asymptotic expansion of the generalization error at locally optimal regularization:
$$
E _ {k} \left(\lambda = \lambda^ {*}\right) = s (1 - c) \left[ 1 - \rho^ {2} + \rho^ {2} W + \frac {H}{k} \right] + \mathcal {O} \left(\frac {1}{k ^ {2}}\right) \tag {215}
$$
This shows that at large  $k$ , when  $H > 0$ , error always approaches its asymptotic value from above (recall that when  $H = 0$  we always have  $k^{*} = 1$ , so that there is no phase boundary unless  $H > 0$ ). Thus, determining the phase boundary requires determining the value of  $k$  which minimizes  $E_{k}$  which is not analytically tractable.
H.5 Infinite Data Limit
In this section we consider the behavior of generalization error in the equicorrelated data model as  $\alpha \to \infty$  while keeping the  $\lambda \sim \mathcal{O}(1)$ . For simplicity, we assume  $\nu_{rr'} = 0$  for  $r \neq r'$ , isotropic features ( $c = 0$ ), no feature noise ( $\omega = 0$ ) and uniform readout noise  $\eta_r = \eta$  as in main text Fig. 3. This limit corresponds to data-rich learning, where the number of training examples is large relative to the number of model parameters. In this case, the saddle point equations reduce to:
$$
\hat {q} _ {r} \rightarrow \frac {\alpha}{\lambda} \tag {216}
$$
$$
q _ {r} \rightarrow \frac {\nu_ {r r} \lambda}{\alpha} \tag {217}
$$
In this limit, we find that  $\gamma_{rr^{\prime}}\rightarrow 0$  . Using this, we can simplify the generalization error as follows:
$$
E _ {g} = \frac {1}{k ^ {2}} \sum_ {r r ^ {\prime} = 1} ^ {k} E _ {r r ^ {\prime}} = s \left[ 1 - \left(2 - \frac {1}{k}\right) \left(\frac {1}{k} \sum_ {r = 1} ^ {k} \nu_ {r r}\right) \right] + \frac {\eta^ {2}}{k} \tag {218}
$$
Interestingly, we find that the readout error in this case depends on the subsampling fractions  $\nu_{rr}$  only through their mean. Therefore, with infinite data, there will be no distinction between homogeneous and heterogeneous subsampling.
I Theoretical Learning Curves and Optimal Subsampling Phase Diagrams
Here, we provide additional learning curves and phase diagrams of  $k^*$  such as those in Fig. 4a,b,c,d, exploring more parameter values for the task-model alignment  $\rho$  and the Reduced noises  $H$ ,  $W$ , and  $Z$ . Generalization errors are calculated for a homogeneous ensemble of  $k$  linear regressors, as described in sections 5, H.4. We also show diagrams of generalization error  $E_{k^*}$ .
Figure S6: (a) Optimal ensemble size  $k^*$  (eqs. 25, 26) in the parameter space of sample size  $\alpha$  and reduced readout noise scale  $H$  setting  $W = Z = 0$ . Grey indicates  $k^* = 1$  and white indicates  $k^* \to \infty$ , with intermediate values given by the colorbar. Appended vertical bars show  $\alpha \to \infty$ .  $\rho$  and  $\lambda$  indicated in panel titles.  $\lambda = \lambda^*$  denotes the 'locally optimal' regularization (see section H.3) (b) Optimal generalization error  $E_{k^*}$  for the same parameter values in (a).
Figure S7: (a) Optimal ensemble size  $k^*$  (eqs. 25, 26) in the parameter space of sample size  $\alpha$  and reduced feature noise scale  $W$  setting  $H = Z = 0$ . Grey indicates  $k^* = 1$  and white indicates  $k^* \to \infty$ , with intermediate values given by the colorbar. Appended vertical bars show  $\alpha \to \infty$ .  $\rho$  and  $\lambda$  indicated in panel titles.  $\lambda = \lambda^*$  denotes the 'locally optimal' regularization (see section H.3) (b) Optimal generalization error  $E_{k^*}$  for the same parameter values in (a).
Figure S8: (a) Optimal ensemble size  $k^{*}$  (eqs. 25, 26) in the parameter space of sample size  $\alpha$  and reduced label noise scale  $Z$  setting  $H = W = 0$ . Grey indicates  $k^{*} = 1$  and white indicates  $k^{*} \to \infty$ , with intermediate values given by the colorbar. Appended vertical bars show  $\alpha \to \infty$ .  $\rho$  and  $\lambda$  indicated in panel titles.  $\lambda = \lambda^{*}$  denotes the 'locally optimal' regularization (see section H.3) (b) Optimal generalization error  $E_{k^{*}}$  for the same parameter values in (a).
Figure S9: Reduced generalization errors  $E_{g} / s(1 - c)$  with  $\lambda = 0$  and  $W = Z = 0$  (given by eq. 22) for linear ridge ensembles of varying size  $k$ .  $\rho$  and  $H$  values indicated above plots. Grey lines indicate  $k = 1$ , dashed black lines  $k \to \infty$ , and intermediate  $k$  values by the colorbar.
Figure S10: Reduced generalization errors  $E_{g} / s(1 - c)$  with  $\lambda = \lambda^{*}$  and  $W = Z = 0$  (given by eq. 23) for linear ridge ensembles of varying size  $k$ .  $\rho$  and  $H$  values indicated above plots. Grey lines indicate  $k = 1$ , dashed black lines  $k \to \infty$ , and intermediate  $k$  values by the colorbar.
Figure S11: Reduced generalization errors  $E_{g} / s(1 - c)$  with  $\lambda = 0$  and  $H = Z = 0$  (given by eq. 22) for linear ridge ensembles of varying size  $k$ .  $\rho$  and  $W$  values indicated above plots. Grey lines indicate  $k = 1$ , dashed black lines  $k \to \infty$ , and intermediate  $k$  values by the colorbar.
Figure S12: Reduced generalization errors  $E_{g} / s(1 - c)$  with  $\lambda = \lambda^{*}$  and  $H = Z = 0$  (given by eq. 23) for linear ridge ensembles of varying size  $k$ .  $\rho$  and  $W$  values indicated above plots. Grey lines indicate  $k = 1$ , dashed black lines  $k \to \infty$ , and intermediate  $k$  values by the colorbar.
Figure S13: Reduced generalization errors  $E_{g} / s(1 - c)$  with  $\lambda = 0$  and  $H = W = 0$  (given by eq. 22) for linear ridge ensembles of varying size  $k$ .  $\rho$  and  $Z$  values indicated above plots. Grey lines indicate  $k = 1$ , dashed black lines  $k \to \infty$ , and intermediate  $k$  values by the colorbar
Figure S14: Reduced generalization errors  $E_{g} / s(1 - c)$  with  $\lambda = \lambda^{*}$  and  $H = W = 0$  (given by eq. 23) for linear ridge ensembles of varying size  $k$ .  $\rho$  and  $Z$  values indicated above plots. Grey lines indicate  $k = 1$ , dashed black lines  $k \to \infty$ , and intermediate  $k$  values by the colorbar.