SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\Rightarrow \hat {Q} _ {k k ^ {\prime}} = \frac {1}{\alpha} \hat {V} _ {k} \hat {V} _ {k ^ {\prime}} \left[ \zeta^ {2} + \rho - m _ {k} - m _ {k ^ {\prime}} + Q _ {k k ^ {\prime}} \right] \tag {152}
$$
$$
\hat {\boldsymbol {m}} = \alpha (\boldsymbol {I} + \boldsymbol {V}) ^ {- 1} \mathbf {1} \quad \Rightarrow \quad \hat {m} _ {k} = \frac {\alpha}{\left(1 + V _ {k}\right)} = \hat {V} _ {k} \tag {153}
$$
Simplifying the prior equations we obtain (through some tedious but straightforward algebra):
$$
Q _ {k k ^ {\prime}} = \hat {Q} _ {k k ^ {\prime}} J _ {k k ^ {\prime}} + \hat {V} _ {k} \hat {V} _ {k ^ {\prime}} \Lambda_ {k k ^ {\prime}} \tag {154}
$$
$$
m _ {k} = \hat {V} _ {k} R _ {k} \tag {155}
$$
Where we have defined:
$$
\boldsymbol {G} _ {k} \equiv \nu_ {k k} \lambda_ {k} \boldsymbol {I} _ {p} + \hat {V} _ {k} \bar {\Sigma} _ {k k} \tag {156}
$$
$$
J _ {k k ^ {\prime}} \equiv \frac {1}{p} \operatorname {t r} \left[ \bar {\boldsymbol {\Sigma}} _ {k k ^ {\prime}} \boldsymbol {G} _ {k ^ {\prime}} ^ {- 1} \bar {\boldsymbol {\Sigma}} _ {k ^ {\prime} k} \boldsymbol {G} _ {k} ^ {- 1} \right] \tag {157}
$$
$$
\Lambda_ {k k ^ {\prime}} \equiv \frac {1}{p} \boldsymbol {\theta} ^ {\top} \boldsymbol {\Sigma} _ {s} \bar {\boldsymbol {A}} _ {k} ^ {\top} \boldsymbol {G} _ {k} ^ {- 1} \bar {\boldsymbol {\Sigma}} _ {k k ^ {\prime}} \boldsymbol {G} _ {k ^ {\prime}} ^ {- 1} \bar {\boldsymbol {A}} _ {k ^ {\prime}} \boldsymbol {\Sigma} _ {s} \boldsymbol {\theta} \tag {158}
$$
$$
R _ {k} \equiv \boldsymbol {\theta} ^ {\top} \Sigma_ {s} \bar {A} _ {k} ^ {\top} G _ {k} ^ {- 1} \bar {A} _ {k} \Sigma_ {s} \boldsymbol {\theta} \tag {159}
$$
Solving eq's 154,152 for  $Q$ , we obtain:
$$
Q _ {k k ^ {\prime}} = \frac {\gamma_ {k k ^ {\prime}}}{1 - \gamma_ {k k ^ {\prime}}} \left(\zeta^ {2} + \rho - m _ {k} - m _ {k ^ {\prime}}\right) + \frac {\hat {V} _ {k} \hat {V} _ {k ^ {\prime}}}{1 - \gamma_ {k k ^ {\prime}}} \Lambda_ {k k ^ {\prime}} \quad (k \neq k ^ {\prime}) \tag {160}
$$
Combining these results, we arrive at a formula for  $E_{kk'}$ :
$$
E _ {k k ^ {\prime}} = \frac {1}{1 - \gamma_ {k k ^ {\prime}}} \left(\gamma_ {k k ^ {\prime}} \zeta^ {2} + \rho - \hat {V} _ {k} R _ {k} - \hat {V} _ {k ^ {\prime}} R _ {k ^ {\prime}} + \hat {V} _ {k} \hat {V} _ {k ^ {\prime}} \Lambda_ {k k ^ {\prime}}\right) + \frac {1}{K ^ {2}} \sum_ {k} \eta_ {k} ^ {2} \quad (k \neq k ^ {\prime}) \tag {161}
$$
$$
\text {w h e r e} \gamma_ {k k ^ {\prime}} \equiv \frac {1}{\alpha} \hat {V} _ {k} \hat {V} _ {k ^ {\prime}} J _ {k k ^ {\prime}} \tag {162}
$$
Where the order parameters  $\hat{V}_k$ ,  $k = 1, \dots, K$  satisfy the fixed-point equations given by eq's 149, 150.
G.2.2 Diagonal Terms
To calculate  $E_{rr}$ , we cannot ignore the presence of readout noise in the loss function. However, as the loss function is separable over the readouts  $k$ , we may calculate  $E_{rr}$  using the results of [31] in
the special case where  $K = 1$ , incorporating the readout noise into the label noise. Concretely, we may calculate  $E_{rr}$  as the error of a single linear predictor under the same setup as the off-diagonal terms except that in the training set  $y^{\mu} \sim P_{y}^{0}(y|\frac{1}{\sqrt{p}}\pmb{\theta}^{\top}\pmb{x}^{\mu})$  where  $P_{y}^{0}(y|x) = \mathcal{N}(x,\zeta^{2} + \eta_{k}^{2})$ . This may be recovered from the calculation for the off-diagonal terms by setting  $k = k'$  and re-scaling  $\zeta^2 \to \zeta^2 + \eta_k^2$ , giving:
$$
E _ {k k} = \frac {1}{1 - \gamma_ {k k}} \left(\gamma_ {k k} \left(\zeta^ {2} + \eta_ {r} ^ {2}\right) + \rho - 2 \hat {V} _ {k} R _ {k} + \hat {V} _ {k} ^ {2} \Lambda_ {k k}\right) + \frac {1}{K ^ {2}} \sum_ {k} \eta_ {k} ^ {2} \quad (k \neq k ^ {\prime}) \tag {163}
$$
where the definitions of  $\gamma_{kk}$ ,  $\rho$ ,  $R_k$ , and  $\Lambda_{kk}$  can be inherited from the off-diagonal case, as well as the saddle-point equations 149, 150.
G.2.3 Full Error
The results obtained here are equivalent to the results of our main theorem, up to a reshuffling of additive constants  $\eta_k^2$  among the error terms, and a trivial re-scaling of the order parameters as follows:  $V_{k}\to \frac{1}{\lambda_{k}} V_{k},\hat{V}_{k}\to \lambda_{k}\hat{V}_{k}$
H Equicorrelated Data Model
To gain an intuition for the joint effects of correlated data, subsampling, ensembling, feature noise, and readout noise, we simplify the formulas for the generalization error in the following special case:
$$
\boldsymbol {\Sigma} _ {s} = s \left[ (1 - c) \boldsymbol {I} _ {M} + c \mathbf {1} _ {M} \mathbf {1} _ {M} ^ {\top} \right] \tag {164}
$$
$$
\boldsymbol {\Sigma} _ {0} = \omega^ {2} \boldsymbol {I} _ {M} \tag {165}
$$
Here  $s$  is a parameter which sets the overall scale of the data and  $c \in [0,1]$  tunes the correlation structure in the data and  $\omega^2$  sets the scale of an isotropic feature noise. We consider an ensemble of  $k$  readouts, each of which sees a subset of the features. Due to the isotropic nature of the equicorrelated data model and the pairwise decomposition of the generalization error, we expect that the generalization error will depend on the partition of features among the readout neurons through only:
- The number of features sampled by each readout:  $N_{r} \equiv \nu_{rr}M$ , for  $r = 1, \dots, k$
- The number of features jointly sampled by each pair of readouts  $n_{rr'} \equiv \nu_{rr'} M$  for  $r, r' \in \{1, \ldots, k\}$
Here, we have introduced the subsampling fractions  $\nu_{rr'} = \frac{N_r}{M}$  and the overlap fractions  $\nu_{rr'}' = \frac{n_{rr'}}{M}$ .
We will average the generalization error over readout weights drawn randomly from the space perpendicular to  $\mathbf{1}_M$ , with an added spike along the direction of  $\mathbf{1}_M$ :
$$
\boldsymbol {w} ^ {*} = \sqrt {1 - \rho^ {2}} \mathbb {P} _ {\perp} \boldsymbol {w} _ {0} ^ {*} + \rho \mathbf {1} _ {M} \tag {166}
$$
$$
\boldsymbol {w} _ {0} ^ {*} \sim \mathcal {N} (0, \boldsymbol {I} _ {M}) \tag {167}
$$
The projection matrix may be written  $\mathbb{P}_{\perp} = I_M - \frac{1}{N}\mathbf{1}_M\mathbf{1}_M^\top$ . The two components of the ground truth weights will yield independent contributions to the generalization error in the sense that
$$
\left\langle E _ {r r ^ {\prime}} \right\rangle = \left(1 - \rho^ {2}\right) E _ {r r ^ {\prime}} (\rho = 0) + \rho^ {2} E _ {r r ^ {\prime}} (\rho = 1) \tag {168}
$$
Calculating  $E_{rr}$  and  $E_{rr'}$  is an exercise in linear algebra which is straightforward but tedious. To assist with the tedious algebra, we wrote a Mathematica package which can handle multiplication, addition, and inversion of matrices of symbolic dimension of the specific form encountered in this problem. This form consists of block matrices, where the blocks may be written as  $a\delta_{MN}\mathbf{I}_M + b\mathbf{1}_M\mathbf{1}_N^\top$ , where  $a, b$  are scalars and  $\delta_{MN}$  ensures that there is only a diagonal component for square blocks (when  $M = N$ ). This package is included as supplemental material to this publication.
H.1 Diagonal Terms and Saddle-Point Equations
Here, we solve for the dominant values of  $q_{r}$  and  $\hat{q}_r$  and simplify the expressions for  $E_{rr}$  in the case of equicorrelated features described above. In this isotropic setting,  $E_{rr}, q_r, \hat{q}_r$  will depend on the subsampling only through  $N_{r} = \nu_{rr}M$ . We may then write, without loss of generality  $A_{r} = (I_{N_{r}}\quad 0)\in \mathbb{R}^{N_{r}\times M}$  where  $\mathbf{0}$  denotes a matrix of all zeros, of the appropriate dimensionality.
We start by simplifying the saddle-point equations 53,54. Expanding  $\frac{1}{M}\mathrm{tr}\left(\pmb{G}_r^{-1}\tilde{\pmb{\Sigma}}_{rr}\right)$  and keeping only leading order terms, the saddle-point equations for  $q_{r}$  and  $\hat{q}_r$  reduce to:
$$
q _ {r} = \frac {\nu_ {r r} (s (1 - c) + \omega^ {2})}{\hat {q} _ {r} (s (1 - c) + \omega^ {2}) + \nu_ {r}} \tag {169}
$$
$$
\hat {q} _ {r} = \frac {\alpha}{\lambda + q _ {r}} \tag {170}
$$
Defining  $a \equiv s(1 - c) + \omega^2$  and solving this system of equations, we find:
$$
q _ {r} = \frac {\sqrt {a ^ {2} \alpha^ {2} + 2 a \alpha (\lambda - a) \nu_ {r} + (a + \lambda) ^ {2} \nu_ {r} ^ {2}} - a \alpha + (a - \lambda) \nu_ {r}}{2 \nu_ {r}} \tag {171}
$$
$$
\hat {q} _ {r} = \frac {\sqrt {a ^ {2} \alpha^ {2} + 2 a \alpha (\lambda - a) \nu_ {r} + (a + \lambda) ^ {2} \nu_ {r} ^ {2}} + a \alpha - (a + \lambda) \nu_ {r}}{2 a \lambda} \tag {172}
$$
We have selected the solution with  $q_r > 0$  because self-overlaps must be at least as large as overlaps between different replicas. This solution to the saddle-point equations can be applied to each of the  $k$  readouts.
Next, we calculate  $E_{rr}$ . Expanding  $\gamma_{rr} \equiv \frac{\alpha}{M\kappa^2}\mathrm{tr}\left[(G^{-1}\tilde{\Sigma})^2\right]$  to leading order in  $M$ , we find:
$$
\gamma_ {r r} = \frac {a ^ {2} \alpha \nu_ {r}}{\left(\lambda + q _ {r}\right) ^ {2} \left(a \hat {q} _ {r} + \nu_ {r}\right) ^ {2}} \tag {173}
$$
$$
\begin{array}{l} \langle E _ {r r} \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} (\rho = 0) = \frac {1}{1 - \gamma_ {r r}} \frac {1}{M} \operatorname {t r} \left[ \mathbb {P} _ {\perp} \left(\boldsymbol {\Sigma} _ {s} - \frac {2}{\nu_ {r r}} \hat {q} _ {r} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r} ^ {- 1} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} + \frac {1}{\nu_ {r r}} \hat {q} _ {r} ^ {2} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} \boldsymbol {G} _ {r} ^ {- 1} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s}\right) \mathbb {P} _ {\perp} \right] \\ + \frac {\gamma_ {r r}}{1 - \gamma_ {r r}} \zeta^ {2} + \eta_ {r} ^ {2}, \tag {174} \\ \end{array}
$$
$$
\begin{array}{l} \langle E _ {r r} \rangle_ {\mathcal {D}, \pmb {w} ^ {*}} (\rho = 1) = \frac {1}{1 - \gamma_ {r r}} \frac {1}{M} \mathbf {1} _ {M} ^ {\top} \left[ \pmb {\Sigma} _ {s} - \frac {2}{\nu_ {r r}} \hat {q} _ {r} \pmb {\Sigma} _ {s} \pmb {A} _ {r} ^ {\top} \pmb {G} _ {r} ^ {- 1} \pmb {A} _ {r} \pmb {\Sigma} _ {s} + \frac {1}{\nu_ {r r}} \hat {q} _ {r} ^ {2} \pmb {\Sigma} _ {s} \pmb {A} _ {r} ^ {\top} \pmb {G} _ {r} ^ {- 1} \tilde {\pmb {\Sigma}} \pmb {G} _ {r} ^ {- 1} \pmb {A} _ {r} \pmb {\Sigma} _ {s} \right] \mathbf {1} _ {m} \\ + \frac {\gamma_ {r r}}{1 - \gamma_ {r r}} \zeta^ {2} + \eta_ {r} ^ {2}, \tag {175} \\ \end{array}
$$
With the aid of our custom Mathematica package, we calculate the traces and contractions in these expressions and expand them to leading order in  $M$ , finding:
$$
\langle E _ {r r} \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} (\rho = 0) = \frac {1}{1 - \gamma_ {r r}} \left(s (1 - c) \left(1 - \frac {(1 - c) s \hat {q} _ {r} \nu_ {r} \left(\hat {q} _ {r} (s (1 - c) + \omega^ {2}) + 2 \nu_ {r}\right)}{\left(\hat {q} _ {r} (s (1 - c) + \omega^ {2}) + \nu_ {r}\right) ^ {2}}\right)\right) + \frac {\gamma_ {r r} \zeta^ {2} + \eta_ {r} ^ {2}}{1 - \gamma_ {r r}} \tag {176}
$$
$$
\langle E _ {r r} \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} (\rho = 1) = \frac {1}{1 - \gamma_ {r r}} \left(\frac {s (1 - c) (1 - \nu_ {r r}) + \omega^ {2}}{\nu_ {r r}}\right) + \frac {\gamma_ {r r} \zeta^ {2} + \eta_ {r} ^ {2}}{1 - \gamma_ {r r}} \tag {177}
$$
In the "ridgeless" limit where  $\lambda \to 0$ , we obtain:
$$
\gamma_ {r r} = \frac {4 \alpha \nu_ {r r}}{(\alpha + \nu_ {r r} + | \alpha - \nu_ {r r} |) ^ {2}} \tag {178}
$$
$$
\langle E _ {r r} (\rho = 0) \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} = \left\{\begin{array}{l l}\frac {s (1 - c) \nu_ {r r}}{\nu_ {r r} - \alpha} \left(1 + \frac {s \alpha (1 - c) (\alpha - 2 \nu_ {r r})}{\nu_ {r r} [ s (1 - c) + \omega^ {2} ]}\right) + \frac {\alpha \zeta^ {2} + \nu_ {r r} \eta_ {r} ^ {2}}{\nu_ {r r} - \alpha},&\text {i f} \alpha <   \nu_ {r r}\\\frac {s (1 - c) \alpha}{\alpha - \nu_ {r r}} \left(1 - \frac {s (1 - c) \nu_ {r r}}{s (1 - c) + \omega^ {2}}\right) + \frac {\nu_ {r r} \zeta^ {2} + \alpha \eta_ {r} ^ {2}}{\alpha - \nu_ {r r}},&\text {i f} \alpha > \nu_ {r r}\end{array}\right\} \quad (\lambda \rightarrow 0) \tag {179}
$$
$$
\langle E _ {r r} (\rho = 1) \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} = \left\{\begin{array}{l l}\frac {\nu_ {r r}}{\nu_ {r r} - \alpha} \left(\frac {s (1 - c) (1 - \nu_ {r r}) + \omega^ {2}}{\nu_ {r r}}\right) + \frac {\alpha \zeta^ {2} + \nu_ {r r} \eta_ {r} ^ {2}}{\nu_ {r r} - \alpha},&\text {i f} \alpha <   \nu_ {r r}\\\frac {\alpha}{\alpha - \nu_ {r r}} \left(\frac {s (1 - c) (1 - \nu_ {r r}) + \omega^ {2}}{\nu_ {r r}}\right) + \frac {\nu_ {r r} \zeta^ {2} + \alpha \eta_ {r} ^ {2}}{\alpha - \nu_ {r r}},&\text {i f} \alpha > \nu_ {r r}\end{array}\right\} \quad (\lambda \rightarrow 0) \tag {180}
$$
H.2 Off-Diagonal Terms
In this section, we calculate the off-diagonal error terms  $E_{rr'}$  for  $r \neq r'$ , again making use of our custom Mathematica package to simplify contractions of block matrices of the prescribed form. By the isotropic nature of the equicorrelated data model,  $E_{rr'}$  can only depend on the subsampling scheme through  $\nu_{rr'}$ ,  $\nu_{r'r'}$ , and  $\nu_{rr'}$ . We can thus, without loss of generality, write:
$$
\boldsymbol {A} _ {r} = \left( \begin{array}{l l l l} \boldsymbol {I} _ {n _ {r} \times n _ {r}} & \boldsymbol {0} _ {n _ {r} \times n _ {r ^ {\prime}}} & \boldsymbol {0} _ {n _ {r} \times n _ {s}} & \boldsymbol {0} _ {n _ {r} \times l} \\ \boldsymbol {0} _ {n _ {s} \times n _ {r}} & \boldsymbol {0} _ {n _ {s} \times n _ {r ^ {\prime}}} & \boldsymbol {I} _ {n _ {s} \times n _ {s}} & \boldsymbol {0} _ {n _ {s} \times l} \end{array} \right) \in \mathbb {R} ^ {N _ {r} \times M} \tag {181}
$$
$$
\boldsymbol {A} _ {r ^ {\prime}} = \left( \begin{array}{c c c c} \mathbf {0} _ {n _ {r ^ {\prime}} \times n _ {r}} & \boldsymbol {I} _ {n _ {r ^ {\prime}} \times n _ {r ^ {\prime}}} & \mathbf {0} _ {n _ {r ^ {\prime}} \times n _ {s}} & \mathbf {0} _ {n _ {r ^ {\prime}} \times l} \\ \mathbf {0} _ {n _ {s} \times n _ {r}} & \mathbf {0} _ {n _ {s} \times n _ {r ^ {\prime}}} & \boldsymbol {I} _ {n _ {s} \times n _ {s}} & \mathbf {0} _ {n _ {s} \times l} \end{array} \right) \in \mathbb {R} ^ {N _ {r ^ {\prime}} \times M} \tag {182}
$$
where we have defined  $n_s$  to be the number of features shared between the readouts,  $n_r = N_r - n_s$  and  $n_{r'} = N_{r'} - n_s$  and the count of remaining features  $l = M - n_r - n_{r'} - n_s$ .
Then, to leading order in  $M$ , we find:
$$
\gamma_ {r r ^ {\prime}} = \frac {\alpha \nu_ {r r ^ {\prime}} (s (1 - c) + \omega^ {2}) ^ {2}}{(\lambda + q _ {r}) (\lambda + q _ {r ^ {\prime}}) (\nu_ {r r} + (s (1 - c) + \omega^ {2}) \hat {q} _ {r}) (\nu_ {r ^ {\prime} r ^ {\prime}} + (s (1 - c) + \omega^ {2}) \hat {q} _ {r ^ {\prime}})} \tag {183}
$$
Averaging  $E_{rr'}$  over  $\pmb{w}_0^* \sim \mathcal{N}(0, \pmb{I}_M)$ , we get:
$$
\begin{array}{l} \langle E _ {r r ^ {\prime}} (\mathcal {D}) \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} (\rho = 0) = \frac {\gamma_ {r r ^ {\prime}}}{1 - \gamma_ {r r ^ {\prime}}} \zeta^ {2} + \frac {1}{1 - \gamma_ {r r ^ {\prime}}} \left(\frac {1}{M} \operatorname {t r} \left[ \mathbb {P} _ {\perp} \boldsymbol {\Sigma} _ {s} \mathbb {P} _ {\perp} \right]\right) \\ - \frac {1}{M (1 - \gamma_ {r r ^ {\prime}})} \operatorname {t r} \left[ \mathbb {P} _ {\perp} \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\nu_ {r r}} \hat {q} _ {r} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r} ^ {- 1} \boldsymbol {A} _ {r} + \frac {1}{\nu_ {r ^ {\prime} r ^ {\prime}}} \hat {q} _ {r ^ {\prime}} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {G} _ {r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}}\right) \boldsymbol {\Sigma} _ {s} \mathbb {P} _ {\perp} \right] \\ + \frac {\hat {q} _ {r} \hat {q} _ {r ^ {\prime}}}{M \left(1 - \gamma_ {r r ^ {\prime}}\right)} \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \operatorname {t r} \left[ \mathbb {P} _ {\perp} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \boldsymbol {G} _ {r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \mathbb {P} _ {\perp} \right], \tag {184} \\ \end{array}
$$
$$
\begin{array}{l} \langle E _ {r r ^ {\prime}} (\mathcal {D}) \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} (\rho = 1) = \frac {\gamma_ {r r ^ {\prime}}}{1 - \gamma_ {r r ^ {\prime}}} \zeta^ {2} + \frac {1}{M (1 - \gamma_ {r r ^ {\prime}})} \left(\mathbf {1} _ {M} ^ {\top} \boldsymbol {\Sigma} _ {s} \mathbf {1} _ {M}\right) \\ - \frac {1}{M (1 - \gamma_ {r r ^ {\prime}})} \mathbf {1} _ {M} ^ {\top} \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\nu_ {r r}} \hat {q} _ {r} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r} ^ {- 1} \boldsymbol {A} _ {r} + \frac {1}{\nu_ {r ^ {\prime} r ^ {\prime}}} \hat {q} _ {r ^ {\prime}} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {G} _ {r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}}\right) \boldsymbol {\Sigma} _ {s} \mathbf {1} _ {M} ^ {\top} \\ + \frac {\hat {q} _ {r} \hat {q} _ {r ^ {\prime}}}{M \left(1 - \gamma_ {r r ^ {\prime}}\right)} \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \mathbf {1} _ {M} ^ {\top} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \boldsymbol {G} _ {r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \mathbf {1} _ {M} \tag {185} \\ \end{array}
$$
Calculating these contractions and traces and expanding to leading order in  $M$ , we get:
$$
\begin{array}{l} \langle E _ {r r ^ {\prime}} (\mathcal {D}) \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} (\rho = 0) = \frac {s (1 - c)}{1 - \gamma_ {r r ^ {\prime}}} \left(1 - \frac {s (1 - c) \nu_ {r r} \hat {q} _ {r}}{\nu_ {r r} + (s (1 - c) + \omega^ {2}) \hat {q} _ {r}} - \frac {s (1 - c) \nu_ {r ^ {\prime} r ^ {\prime}} \hat {q} _ {r ^ {\prime}}}{\nu_ {r ^ {\prime} r ^ {\prime}} + (s (1 - c) + \omega^ {2}) \hat {q} _ {r ^ {\prime}}}\right). \\ \left. + \frac {s (1 - c) \left(s (1 - c) + \omega^ {2}\right) \nu_ {r r ^ {\prime}} \hat {q} _ {r} \hat {q} _ {r ^ {\prime}}}{\left(\nu_ {r r} + \left(s (1 - c) + \omega^ {2}\right) \hat {q} _ {r}\right) \left(\nu_ {r ^ {\prime} r ^ {\prime}} + \left(s (1 - c) + \omega^ {2}\right) \hat {q} _ {r ^ {\prime}}\right)}\right) \\ + \frac {\gamma_ {r r ^ {\prime}}}{1 - \gamma_ {r r ^ {\prime}}} \zeta^ {2} \tag {186} \\ \end{array}
$$
$$
\langle E _ {r r ^ {\prime}} (\mathcal {D}) \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} (\rho = 1) = \frac {1}{1 - \gamma_ {r r ^ {\prime}}} \left(\frac {s (1 - c) \left(\nu_ {r r ^ {\prime}} - \nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}\right) + \omega^ {2} \nu_ {r r ^ {\prime}}}{\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}\right) + \frac {\gamma_ {r r ^ {\prime}}}{1 - \gamma_ {r r ^ {\prime}}} \zeta^ {2} \tag {187}
$$
Taking  $\lambda \to 0$  we get the ridgeless limit:
$$
\gamma_ {r r ^ {\prime}} \rightarrow \frac {4 \alpha \nu_ {r r ^ {\prime}}}{(\alpha + \nu_ {r r} + | \alpha - \nu_ {r r} |) (\alpha + \nu_ {r ^ {\prime} r ^ {\prime}} + | \alpha - \nu_ {r ^ {\prime} r ^ {\prime}} |)} \quad (\lambda \rightarrow 0) \tag {188}
$$
$$
\langle E _ {r r ^ {\prime}} (\mathcal {D}) \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} (\rho = 0) = \frac {1}{1 - \gamma_ {r r ^ {\prime}}} F _ {0} (\alpha) + \frac {\gamma_ {r r ^ {\prime}}}{1 - \gamma_ {r r ^ {\prime}}} \zeta^ {2} \quad (r \neq r ^ {\prime}) \tag {189}
$$
$$
F _ {0} (\alpha) \equiv \left\{ \begin{array}{l l} \frac {(c - 1) s \left(\nu_ {r} \nu_ {r ^ {\prime}} ((2 \alpha - 1) (c - 1) s + \omega^ {2}) - \alpha^ {2} (c - 1) s \nu_ {r r ^ {\prime}}\right)}{\nu_ {r} ((c - 1) s - \omega^ {2}) \nu_ {r ^ {\prime}}}, & \text {i f} \alpha \leq \nu_ {r r} \leq \nu_ {r ^ {\prime} r ^ {\prime}} \\ \frac {(c - 1) s \left(\nu_ {r ^ {\prime}} ((c - 1) s \nu_ {r} + (\alpha - 1) (c - 1) s + \omega^ {2}) - \alpha (c - 1) s \nu_ {r r ^ {\prime}}\right)}{((c - 1) s - \omega^ {2}) \nu_ {r ^ {\prime}}}, & \text {i f} \nu_ {r r} \leq \alpha \leq \nu_ {r ^ {\prime} r ^ {\prime}} \\ \frac {(c - 1) s \left((c - 1) s \nu_ {r ^ {\prime}} - c s \nu_ {r r ^ {\prime}} + (c - 1) s \nu_ {r} - c s + s \nu_ {r r ^ {\prime}} + s + \omega^ {2}\right)}{(c - 1) s - \omega^ {2}}, & \text {i f} \nu_ {r r} \leq \nu_ {r ^ {\prime} r ^ {\prime}} \leq \alpha \end{array} \right\} \tag {190}
$$
$$
\langle E _ {r r ^ {\prime}} (\mathcal {D}) \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} (\rho = 1) = \frac {1}{1 - \gamma_ {r r ^ {\prime}}} \left(\frac {s (1 - c) \left(\nu_ {r r ^ {\prime}} - \nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}\right) + \omega^ {2} \nu_ {r r ^ {\prime}}}{\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}\right) + \frac {\gamma_ {r r ^ {\prime}}}{1 - \gamma_ {r r ^ {\prime}}} \zeta^ {2} \quad (\lambda \rightarrow 0) \tag {191}
$$
H.3 Optimal Regularization
Here, we derive the "locally" optimal regularization which minimizes the prediction error of the ensemble members independently. Equivalently we derive the optimal regularization for a single readout ( $k = 1$ ) or the regularization which minimizes the diagonal terms  $E_{rr}$  of the generalization error. By differentiating  $E_{rr}$  with respect to the regularization  $\lambda$ , one can show that for the optimal regularization, we will have
$$
\frac {1}{(1 - \gamma_ {r r})} \frac {d S _ {r}}{d \lambda} \left[ \frac {a ^ {2} \nu_ {r r}}{\alpha (1 - \gamma_ {r r})} S _ {r} \left((1 - \rho^ {2}) I _ {r r} ^ {0} + \rho^ {2} I _ {r r} ^ {1} + \zeta^ {2} + \eta_ {r} ^ {2}\right) + (1 - \rho^ {2}) s ^ {2} (1 - c ^ {2}) \nu_ {r r} (a S _ {r} - 1) \right] = 0 \tag {192}
$$
It is easy to show that  $\frac{dS_r}{d\lambda}$  cannot be equal to 0. Setting the term in brackets to zero and solving for  $\lambda$ , we find with the aid of the computer algebra system Mathematica [54]:
$$
\lambda^ {\star} = a \left(\frac {a \left(\nu_ {r r} \left(\zeta^ {2} + \eta^ {2}\right) + a \rho^ {2} + s (1 - c) \nu_ {r r} \left(1 - 2 \rho^ {2}\right)\right)}{(1 - c) ^ {2} \nu_ {r r} ^ {2} \left(1 - \rho^ {2}\right) s ^ {2}} - 1\right), \quad (0 <   c \leq 1) \tag {193}
$$
In the limiting case of isotropic data  $(c = 0)$ , the optimal regularization reduces to:
$$
\lambda^ {\star} = \frac {1}{\nu_ {r r}} \left(\zeta^ {2} + \eta^ {2}\right) + s \left(\frac {1}{\nu_ {r r}} - 1\right), \quad (c = 0) \tag {194}
$$
We note that while generalization error is discontinuous at  $c = 0$ , this result can be quickly obtained from eq. 193 by noticing that the formula for the generalization error with  $0 < c \leq 1$  reduces to the formula for generalization error with  $c = 0$  when we set  $\rho = c = 0$  (when  $c = 0$ , the generalization error does not depend on  $\rho$  because there is no special direction in the data).
H.4 Homogeneous Ensembling with Resource Constraints
We make further simplifications in the special case where  $\lambda_r = \lambda$ ,  $\eta_r = \eta$ ,  $\nu_{rr} = \frac{1}{k}$  for all  $r = 1, \ldots, k$ . For simplicity, we also set  $\nu_{rr'} = 0$  for all  $r \neq r'$  so that ensemble members sample mutually exclusive subsets of the features. We will consider both the ridgeless limit  $\lambda \to 0$  and the case of "locally optimal" regularization  $\lambda = \lambda^*$  (see eq. 193). Concretely, we show here that under these special cases, generalization error has the forms given in eqs. 25, 26.
We start by analyzing the saddle-point equations 111, 112. Note that in this special case the saddle point equations will be identical for all  $r = 1, \ldots, k$ . We will therefore suppress the  $r$  index. Solving this quadratic system of equations explicitly, we encounter the following radical, which we assign variable name  $x$ :
$$
x \equiv \sqrt {\left(a \alpha - a \nu + \lambda \nu\right) ^ {2} + 4 a \lambda \nu^ {2}} \tag {195}
$$
In order to simplify this radical to begin extracting the factor of  $s(1 - c)$  that appears in eqs. 25, 26, we define a reduced regularization:
$$
\Lambda \equiv \frac {\lambda}{s (1 - c)} \tag {196}
$$
And substitute  $\Lambda, H, W$ , and  $Z$  (recall eq. 24) for  $\lambda, \eta, \omega$ , and  $\zeta$ , giving
$$
x = s (1 - c) \mathcal {X} (\alpha , \nu , W, \Lambda) \tag {197}
$$