SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To construct the dataset, we start with a set of  $n$  images  $\{\pmb{x}^{\mu}\}_{\mu = 1}^{n}$  from a subset of  $C = 10$  classes of the imagenet dataset. For each image  $\pmb{x}^{\mu}$ , we obtain a corresponding feature vector  $\psi^{\mu} \in \mathbb{R}^{M}$  as the last-hidden-layer activation of the ResNext [46], which has been pre-trained on the imagenet classification task [44]. The architecture we use produces  $M = 2048$  features per image. These features will serve as the data input to the downstream linear classifier. The corresponding labels  $\pmb{y}^{\mu} \in \mathbb{R}^{C}$  are one-hot vectors.
We construct two datasets using this method, using images from the "Imagenette" and "Imageloof" datasets [45]. For the "Imagenette" task, we a construct a training set of size  $n_{tr} = 9469$  and a test set of size  $n_{test} = 3925$  containing features corresponding to images from 10 unrelated classes (tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute). For the "Imageloof" task, we a construct a training set of size  $n_{tr} = 9025$  and a test set of size  $n_{test} = 3929$  containing features corresponding to images of 10 different dog breeds (Australian terrier, Border terrier, Samoyed, Beagle, Shih-Tzu, English foxhound, Rhodesian ridgeback, Dingo, Golden retriever, Old English sheepdog). The imageloof classification task is naturally more difficult. The statistics of the resulting datasets are described in Fig. S1, where we plot the data-data covariance matrix, feature-feature covariance matrix, and the eigenvalue spectrum for both the "Imagenette" and "Imageloof" tasks.
E.2 Model Training
At training time a dataset of  $P$  examples is constructed:  $\mathcal{D} = \{\pmb{\psi}^{\mu},\pmb{y}^{\mu}\}_{\mu = 1}^{P}$ . We represent the training set with a "design matrix"  $\Phi \in \mathbb{R}^{P\times M}$  and a label matrix  $\pmb{Y}\in \{0,1\}^{P\times C}$ . The loss function for each ensemble member is generalized to multi-class regression. With ridge regularization, the objective for each ensemble member  $r$  becomes:
$$
\hat {\boldsymbol {W}} _ {r} = \underset {\boldsymbol {W} _ {r} \in \mathbb {R} ^ {N _ {r} \times C}} {\arg \min} \left[ \sum_ {\mu = 1} ^ {P} \| \frac {1}{\sqrt {N _ {r}}} \boldsymbol {A} _ {r} \bar {\psi} ^ {\mu} \boldsymbol {W} _ {r} + \boldsymbol {\xi} _ {r} ^ {\mu} - \boldsymbol {y} ^ {\mu} \| _ {2} ^ {2} + \lambda_ {r} \| \boldsymbol {W} _ {r} \| _ {F} ^ {2} \right],
$$
Imagewoof
Data Covariance Matrix
b
C
Eigenvalue Spectrum
Figure S1: In numerical experiments, we train linear classifiers to predict labels of imagenet images based on their last-hidden-layer representations in a pre-trained ResNext deep learning architecture [46]. Here, we show the structure of the datasets constructed using the ResNext feature map for the Imagenette task (left), which consists of categorizing images from 10 unrelated categories, and the Imagewoof task (right), which consists of categorizing images from 10 different dog breeds. (a) Gram matrix of the centered ResNext features defined as  $\frac{1}{P}\left(\Phi -\bar{\Phi}\right)^{\top}\left(\Phi -\bar{\Phi}\right)$  for data matrix  $\Phi \in \mathbb{R}^{P\times M}$  where  $P$  is the total size of the dataset. Dataset is sorted by label and tick marks show the boundaries between classes. (b) The covariance eigenspectrum of the ResNext features is well described by a power law decay. (c) Generalization error of Linear classification with a single linear predictor with access to a fraction  $\nu = N / M$  of the ResNext features averaged over 100 trials (see discussion in section E.4.1)
where  $\| \cdot \| _F$  denotes the Frobenius norm and where the readout noise vector  $\pmb{\xi}_r^\mu \sim \mathcal{N}(0,\eta^2\pmb{I}_C)$  is a  $C$  - dimensional readout noise which corrupts the model's prediction. The weights  $\hat{W}_r$  can be determined in closed form as follows:
$$
\hat {\boldsymbol {W}} _ {r} = \frac {1}{\sqrt {N _ {r}}} \left(\frac {1}{N _ {r}} \boldsymbol {A} _ {r} \boldsymbol {\Phi} ^ {\top} \boldsymbol {\Phi} \boldsymbol {A} _ {r} ^ {\top} + \lambda_ {r} \boldsymbol {I}\right) ^ {- 1} \left(\boldsymbol {A} _ {r} \boldsymbol {\Phi} ^ {\top} (\boldsymbol {Y} - \boldsymbol {\Xi} _ {r})\right).
$$
where we have defined  $[\Xi_r]_{\mu c} = [\xi_r^\mu ]_c$ . When  $\lambda = 0$  we instead use a pseudoinverse rule. In all experiments presented here, we use measurement matrices  $A_{r}$  which implement an ordinary subsampling of the features, so that the rows and columns of  $A_{r}$  consist of one-hot vectors.
E.3 Model Prediction
Once trained, the learned weights may be used to predict the label of a new example  $\psi$  as follows. For  $r = 1,\dots ,k$  we calculate the prediction of each ensemble member by first assigning each class a "score". The scores of predictor  $r$  are stored in a vector  $\pmb {f}_r\in \mathbb{R}^C$  ..
$$
\boldsymbol {f} _ {r} (\psi) = \frac {1}{\sqrt {N _ {r}}} \boldsymbol {A} _ {r} \psi \hat {\boldsymbol {W}} _ {r} + \boldsymbol {\xi} _ {r}
$$
Where  $\xi_r \sim \mathcal{N}(0, \eta^2 I_C)$  is drawn randomly at model evaluation. Each ensemble member's "vote" corresponds to the class with the largest score. The prediction of the ensemble is then calculated as a majority vote of the ensemble members. Generalization error is then calculated as the probability of misclassifying an example from the test set.
E.4 Linear Classification Experiments
We apply the described majority-vote linear classifier ensembles to the ResNext-Imagenette and ResNext-Imagewoof tasks in three different experiments.
E.4.1 Reduncancy of ResNext Features
In the first experiment, we investigate the performance of a single linear classifier  $(k = 1)$  as the fraction of features  $\nu$  which it has access to varies. We set  $P = 1000$  and vary  $\nu$  over 50 values on a logarithmic scale from  $10^{-3}$  to 1. We also vary the regularization strength over 0 and a logarithmic scale from  $10^{-3}$  to  $10^{4}$ . We average over 100 trials. At each trial the particular subset of  $P = 1000$  training examples and the particular subset of  $\nu * M$  features is randomly re-sampled. We find that ResNext features are highly redundant - classification accuracy is very robust to subsampling of the features. For example, in the Imagewoof classification task with the best regularization tested  $(\lambda = 0.1)$ , test error increases from about  $1\%$  to about  $2\%$  as the subsampling fraction decreases from  $\nu = 1$  to  $\nu = 0.1$  (meaning  $90\%$  of the features are ignored) (see Fig. 3d). Similarly, for the imagenette task, test error increases from about  $0.1\%$  to about  $0.2\%$  as the subsampling fraction decreases from  $\nu = 1$  to  $\nu = 0.1$  (see Fig. S1c). Code used to run these experiments may be found in the folder "DeepNet_Subsamp" in the GitHub repository.
E.4.2 Heterogeneous Ensembling Mitigates Double-Descent
In the second experiment, we compare learning curves for homogeneous ensembling and heterogeneous ensembling applied to the ResNext-Imagewoof classification task. In each trial, we train ensembles of  $k = \{1, 4, 8, 16, 32\}$  linear predictors whose subsampling fractions  $\{\nu_{rr}, \dots, \nu_{kk}\}$  are assigned either with Homogeneous ensembling or with Heterogeneous Ensembling with  $\sigma = 0.75 / k$ . C. After subsampling fraction are assigned, the training set is randomly shuffled. We iterate over 50 sample sizes  $P$  logarithmically distributed from 400 to 4000, and then add the values  $M / k$  for each  $k$  to the list of  $P$  values. We repeat for 100 trials for both  $\lambda_r = 0$  (pseudoinverse rule) and  $\lambda = 0.1$ ,
Figure S2: Results of the experiment described in section E.4.2. Learning curves for ResNext-Imagenewoof classification task in linear classifier ensembles. In Homogeneous ensembling (left) all  $\nu_{rr} = 1 / k$  with  $k$  indicated by the legend. In heterogeneous ensembling  $\nu_{rr}$  are drawn from a Dirichlet distribution as described in section C, with  $\sigma = 3 / (4k)$ . We use regularization  $\lambda = 0$  (top) and  $\lambda = 0.1$  (bottom).  $\lambda = 0, 0.1$ . Lines represent an average over 100 trials, shaded regions show standard deviation. We set  $\nu_{rr'} = 0$  for  $r \neq r'$ .
which was found to mitigate double-descent by the parameter sweep in Fig. S1c. In main text Fig. 3e, we plot the mean learning curves over 100 trials. In Fig. S2 we show standard deviation over the 100 trials as shaded error bars. When  $\lambda = 0$ , heterogeneous ensembling mitigates double-descent, leading to a monotonically decreasing learning curve for sufficiently high  $k$ . When  $\lambda = 0.1$ , homogeneous and heterogeneous ensembles of size  $k$  perform similarly. Code used to run these experiments may be found in the folder "DeepNet_HomVHet" in the GitHub repository.
E.4.3 Readout Noise Encourages Ensembling
In the third experiment, we test the effect of a readout noise which is independent across the members of the ensemble on generalization error. We do this by sweeping over the readout noise scale  $\eta$  as defined in sections E.2, E.3. For  $\lambda = 0, 0.1$  and  $\nu = 0, .1, \dots, 1$  we compute the learning curves of linear predictors with  $k = 1, 2, 3, 4, 5, 10, 15, 20, 25, 30, 100, 200$  and all  $\nu_{rr} = 1 / k$ , averaged over 50 trials. These learning curves are shown in Fig. S3 for both the ResNext-Imagenette and ResNext-Imagewoof task. In Figs. S4a and S5a, we plot the value  $k^*$  which minimizes error as pase diagrams in the parameter space of  $\alpha$  and  $\eta$ , analogous to the phase diagrams in Fig. 4b. We see that the qualitative shape of these phase diagrams is similar to the equicorrelated model. The differences may be attributed to the nonlinear nature of the classification task. Furthermore, we find
that, in general the optimal  $k^*$  tends to be higher with the Imagenette dataset than with the Imageloof dataset, in agreement with our finding in the equicorrelated regression model that as  $\rho$  increases (making the classification task easier), optimal ensemble size tends to increase (Fig. 4b and S6, S7, S8). In Fig S3 we see that there are often a number of  $k$  values for which test error is at or near to its lowest. To quantify this, we also plot diagrams of the minimum and maximum values of  $k$  that are within an small margin  $\epsilon$  of the minimum measured error. For the ResNext-Imagenette task, we use  $\epsilon = 0.001$  and for ResNext-Imagewoof  $\epsilon = 0.01$ . We see that there is a wide array of  $k$  values which bring error near-to-minimum in practice (Figs S4b,c, S5b,c). We also plot the minimum achieved error, and the difference between minimum errors at  $\lambda = 0.1, 0$  (Figs. S4d, S5d). Code used to run these experiments may be found in the folder "DeepNet_PD" in the GitHub repository.
Figure S3: Learning curves for ensembles of linear classifiers with homogeneous subsampling for  $\lambda = 0, 0.1$  and readout noise  $\eta$  values indicated in the figure. Results are averaged over 50 trials. Experiments are described in section E.4.3
$K^{\star}\lambda = 0.1$  imagenette2
$K_{min}\lambda = 0.1$  imagenette2
$K_{max}\lambda = 0.1$  imagenette2
$\alpha \equiv P / M$
d
$\lambda = 0$  imagenette2 1.77 1.14 1.5  $\alpha \equiv P / M$
Optimal E  $\lambda = 0.1$  imagenette2
e
$\alpha \equiv P / M$
Figure S4: Diagrams described in section E.4.3 for the ResNext-Imagenette experiment. Using learning curves in Fig. S3 (and for additional values of  $\eta$  not shown there), we plot (a) Optimal  $k^{*}$  in the parameter space of  $\alpha$  and  $\eta$ , (b) Minimum value of  $k$  for which error is within a tolerance  $\epsilon = .001$  of its value for  $k^{*}$ , (b) Maximum value of  $k$  for which error is within a tolerance  $\epsilon = .001$  of its value for  $k^{*}$ , (d) the value of the minimum error  $E(k^{*})$ , and (e) the difference between this optimal error for  $\lambda = 0.1$  and  $\lambda = 0$ .
Figure S5: Diagrams described in section E.4.3 for the ResNext-Imagewoof experiment. Using learning curves in Fig. S3 (and for additional values of  $\eta$  not shown there), we plot (a) Optimal  $k^{*}$  in the parameter space of  $\alpha$  and  $\eta$ , (b) Minimum value of  $k$  for which error is within a tolerance  $\epsilon = .001$  of its value for  $k^{*}$ , (b) Maximum value of  $k$  for which error is within a tolerance  $\epsilon = .001$  of its value for  $k^{*}$ , (d) the value of the minimum error  $E(k^{*})$ , and (e) the difference between this optimal error for  $\lambda = 0.1$  and  $\lambda = 0$ .
F Generalization error of ensembled linear regression from the replica trick
Here we use the Replica Trick from statistical physics to derive analytical expressions for  $E_{r r^{\prime}}$ . We treat the cases where  $r = r^{\prime}$  and  $r \neq r^{\prime}$  separately. Following a statistical mechanics approach, we calculate the average generalization error over a Gibbs measure with inverse temperature  $\beta$ ;
$$
Z = \int \prod_ {r} d \boldsymbol {w} \exp \left(- \frac {\beta}{2} \sum_ {r} E _ {t} ^ {r} - \frac {M \beta}{2} \sum_ {r, r ^ {\prime}} J _ {r r ^ {\prime}} E _ {r r ^ {\prime}} \left(\boldsymbol {w} _ {r}, \boldsymbol {w} _ {r ^ {\prime}}\right)\right) \tag {30}
$$
$$
E _ {t} ^ {r} = \sum_ {\mu = 1} ^ {P} \left(\frac {1}{\sqrt {N _ {r}}} \boldsymbol {w} _ {r} ^ {\top} \boldsymbol {A} _ {r} \bar {\psi} _ {\mu} + \xi_ {r} - y _ {\mu}\right) ^ {2} + \lambda | \boldsymbol {w} _ {r} ^ {2} | \tag {31}
$$
In the limit where  $\beta \rightarrow \infty$  the gibbs measure will concentrate around the weight vector  $\hat{\pmb{w}}_r$  which minimizes the regularized loss function. The replica trick relies on the following identity:
$$
\left\langle \log (Z [ \mathcal {D} ]) \right\rangle_ {\mathcal {D}} = \lim  _ {n \rightarrow 0} \frac {1}{n} \log \left(\left\langle Z ^ {n} \right\rangle_ {\mathcal {D}}\right) \tag {32}
$$
where  $\langle \cdot \rangle_{\mathcal{D}}$  represents an average over all quenched disorder in the system. In this case, quenched disorder - the disorder which is fixed prior to and throughout training of the weights - consists of the selected training examples along with their feature noise and label noise:  $\mathcal{D} = \{\psi_{\mu},\sigma^{\mu},\epsilon^{\mu}\}_{\mu = 1}^{P}$ . The calculation proceeds by first computing the average of the replicated partition function assuming  $n$  is a positive integer. Then, in a non-rigorous but standard step, we analytically extend the result to  $n\to 0$ .
F.1 Diagonal Terms
We start by calculating  $E_{rr}$  for some fixed choice of  $r$ . This derivation partially follows section D.3 from [36], with the addition of readout noise and label noise. Noting that the diagonal terms of the generalization error  $E_{rr}$  only depend on the learned weights  $w_r$ , and the loss function separates over the readouts, we may consider the Gibbs measure over only these weights:
$$
Z = \int d \boldsymbol {w} _ {r} \exp \left(- \frac {\beta}{2 \lambda} E _ {r} ^ {t} - \frac {J M \beta}{2} E _ {r r} (\boldsymbol {w} _ {r})\right) \tag {33}
$$
$$
\begin{array}{l} \langle Z ^ {n} \rangle_ {\mathcal {D}} = \int \prod_ {a} d \boldsymbol {w} _ {r} ^ {a} \mathbb {E} _ {\{\psi_ {\mu}, \sigma^ {\mu}, \epsilon^ {\mu} \}} \\ \exp \left(- \frac {\beta M}{2 \lambda} \sum_ {\mu , a} \frac {1}{M} \left[ \frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {w} _ {r} ^ {a ^ {\top}} \boldsymbol {A} _ {r} \left(\psi_ {\mu} + \sigma^ {\mu}\right) - \boldsymbol {w} ^ {* ^ {\top}} \psi_ {\mu} - \sqrt {M} \left(\epsilon^ {\mu} - \xi_ {r} ^ {\mu}\right) \right] ^ {2} \right. \tag {34} \\ \left. - \frac {\beta}{2} \sum_ {a} | \boldsymbol {w} _ {r} ^ {a} | ^ {2} - \frac {J M \beta}{2} \sum_ {a} E _ {r r} (\boldsymbol {w} ^ {a})\right) \\ \end{array}
$$
Next we must perform the averages over quenched disorder. We first integrate over  $\{\psi_{\mu},\sigma^{\mu},\xi_r^\mu ,\epsilon^\mu \}_{\mu = 1}^P$  . Noting that the scalars
$$
h _ {\mu} ^ {r a} \equiv \frac {1}{\sqrt {M}} \left[ \frac {1}{\sqrt {\nu_ {r r}}} \pmb {w} _ {r} ^ {a \top} \pmb {A} _ {r} (\pmb {\psi} _ {\mu} + \pmb {\sigma} ^ {\mu}) - \pmb {w} ^ {* \top} \pmb {\psi} _ {\mu} - \sqrt {M} (\epsilon^ {\mu} - \xi_ {r} ^ {\mu}) \right]
$$
are Gaussian random variables (when conditioned on  $A_r$ ) with mean zero and covariance:
$$
\begin{array}{l} \left\langle h _ {\mu} ^ {r a} h _ {\nu} ^ {r b} \right\rangle = \delta_ {\mu \nu} Q _ {a b} ^ {r r} (35) \\ Q _ {a b} ^ {r r} = \frac {1}{M} \left[ \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} - \boldsymbol {w} ^ {* \top}\right) \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {w} _ {r} ^ {b} - \boldsymbol {w} ^ {*}\right) \right. (36) \\ \left. \left. + \frac {1}{\nu_ {r r}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} \Sigma_ {0} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {w} _ {r} ^ {b} + M \left(\zeta^ {2} + \eta_ {r} ^ {2}\right) \right] \right. \\ \end{array}
$$
To perform this integral we re-write in terms of  $\{\pmb{H}_{\mu}^{r}\}_{\mu = 1}^{P}$ , where
$$
\boldsymbol {H} _ {\mu} ^ {r} = \left[ \begin{array}{c} h _ {\mu} ^ {r 1} \\ h _ {\mu} ^ {r 2} \\ \vdots \\ h _ {\mu} ^ {r n} \end{array} \right] \in \mathbb {R} ^ {n} \tag {37}
$$
$$
\langle Z ^ {n} \rangle_ {\mathcal {D}} = \int \prod_ {a} d \boldsymbol {w} _ {r} ^ {a} \mathbb {E} _ {\left\{\psi_ {\mu}, \sigma^ {\mu}, \epsilon^ {\mu} \right\}} \exp \left(- \frac {\beta}{2 \lambda} \sum_ {\mu} \boldsymbol {H} _ {\mu} ^ {r \top} \boldsymbol {H} _ {\mu} ^ {r} - \frac {\beta}{2} \sum_ {a} | \boldsymbol {w} _ {r} ^ {a} | ^ {2} - \frac {J M \beta}{2} \sum_ {a} E _ {r r} (\boldsymbol {w} ^ {a})\right) \tag {38}
$$
Integrating over the  $H_{\mu}^{r}$  we get:
$$
\langle Z ^ {n} \rangle_ {\mathcal {D}} = \int \prod_ {a} d \boldsymbol {w} _ {r} ^ {a} \exp \left(- \frac {P}{2} \log \det  \left(\boldsymbol {I} _ {n} + \frac {\beta}{\lambda} \boldsymbol {Q} ^ {r r}\right) - \frac {\beta}{2} \sum_ {a} | \boldsymbol {w} _ {r} ^ {a} | ^ {2} - \frac {J M \beta}{2} \sum_ {a} E _ {r r} (\boldsymbol {w} _ {r})\right) \tag {39}
$$
Next we integrate over  $Q^r$  and add constraints. We use the following identity:
$$
\begin{array}{l} 1 = \prod_ {a b ^ {\prime}} \int d Q _ {a b} ^ {r r} \delta \left(Q _ {a b} ^ {r r} - \frac {1}{M} \left[ \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} - \boldsymbol {w} ^ {* \top}\right) \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {w} _ {r} ^ {b} - \boldsymbol {w} ^ {*}\right) \right. \right. \tag {40} \\ \left. \left. + \frac {1}{\nu_ {r r}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {0} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {w} _ {r} ^ {b} + M \left(\zeta^ {2} + \eta_ {r} ^ {2}\right) \right]\right) \\ \end{array}
$$
Using the Fourier representation of the delta function, we get:
$$
\begin{array}{l} 1 = \prod_ {a b} \int \frac {1}{4 \pi i / M} d Q _ {a b} ^ {r r} d \hat {Q} _ {a b} ^ {r r} \exp \left(\frac {M}{2} \hat {Q} _ {a b} ^ {r r} \left(Q _ {a b} ^ {r r} - \frac {1}{M} \left[ \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} - \boldsymbol {w} ^ {* \top}\right) \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {w} _ {r} ^ {b} - \boldsymbol {w} ^ {*}\right) \right. \right. \right. \\ \left.\left. + \frac {1}{\nu_ {r r}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {0} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {w} _ {r} ^ {b} + M \left(\zeta^ {2} + \eta_ {r} ^ {2}\right)\right]\right)\left. \right) \tag {41} \\ \end{array}
$$
Inserting this identity into the replicated partition function and substituting  $E_{rr}(\pmb{w}_r^a) = Q_{aa}^{rr} - \zeta^2$  we find:
$$
\begin{array}{l} \langle Z ^ {n} \rangle_ {\mathcal {D}} \propto \\ \int \prod_ {a b} d Q _ {a b} ^ {r r} d \hat {Q} _ {a b} ^ {r r} \exp \left(- \frac {P}{2} \log \det  \left(\boldsymbol {I} _ {n} + \frac {\beta}{\lambda} \boldsymbol {Q} ^ {r r}\right) + \frac {1}{2} \sum_ {a b} M \hat {Q} _ {a b} ^ {r r} Q _ {a b} ^ {r r} - \frac {J M \beta}{2} \sum_ {a} \left(Q _ {a a} ^ {r r} - \zeta^ {2}\right)\right) \\ \int \prod_ {a} d \boldsymbol {w} _ {r} ^ {a} \exp \left(- \frac {\beta}{2} \sum_ {a} | \boldsymbol {w} _ {r} ^ {a} | ^ {2} - \frac {1}{2} \sum_ {a b} \hat {Q} _ {a b} ^ {r r} \left[ \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} - \boldsymbol {w} ^ {* \top}\right) \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {w} _ {r} ^ {b} - \boldsymbol {w} ^ {*}\right) \right. \right. \\ \left. \left. + \frac {1}{\nu_ {r r}} \boldsymbol {w} _ {r} ^ {a ^ {\top}} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {0} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {w} _ {r} ^ {b} + M \left(\zeta^ {2} + \eta_ {r} ^ {2}\right) \right]\right) \tag {42} \\ \end{array}
$$
In order to perform the Gaussian integral over the  $\{\pmb{w}_r^a\}$ , we unfold over the replica index  $a$ . We first define the following:
$$
\boldsymbol {w} _ {r} ^ {\cdot} \equiv \left[ \begin{array}{c} \boldsymbol {w} _ {r} ^ {1} \\ \vdots \\ \boldsymbol {w} _ {r} ^ {n} \end{array} \right] \tag {43}
$$
$$
T ^ {r} \equiv \beta \boldsymbol {I} _ {n} \otimes \boldsymbol {I} _ {N _ {r}} + \hat {\boldsymbol {Q}} ^ {r r} \otimes \left(\frac {1}{\nu_ {r r}} \boldsymbol {A} _ {r} \left(\boldsymbol {\Sigma} _ {s} + \boldsymbol {\Sigma} _ {0}\right) \boldsymbol {A} _ {r} ^ {\top}\right) \tag {44}
$$
$$
V ^ {r} \equiv \left(\hat {\boldsymbol {Q}} ^ {r r} \otimes \boldsymbol {I} _ {N _ {r}}\right) \left(\mathbf {1} _ {n} \otimes \frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*}\right) \tag {45}
$$
We then have for the integral over  $w$
$$
\int d \boldsymbol {w} _ {r} \exp \left(- \frac {1}{2} \boldsymbol {w} _ {r} ^ {\cdot \top} T ^ {r} \boldsymbol {w} _ {r ^ {\prime}} + V ^ {r \top} \boldsymbol {w} _ {r}\right) \tag {46}
$$
$$
= \exp \left(\frac {1}{2} V ^ {r \top} \left(T ^ {r}\right) ^ {- 1} V ^ {r} - \frac {1}{2} \log \det \left(T ^ {r}\right)\right) \tag {47}
$$
We can finally write the replicated partition function as:
$$
\langle Z ^ {n} \rangle_ {\mathcal {D}} \propto
$$
$$
\int \prod_ {a b} d Q _ {a b} ^ {r r} d \hat {Q} _ {a b} ^ {r} \exp \left(- \frac {P}{2} \log \det  \left(\boldsymbol {I} _ {n} + \frac {\beta}{\lambda} \boldsymbol {Q} ^ {r r}\right) + \frac {1}{2} \sum_ {a b} M \hat {Q} _ {a b} ^ {r r} Q _ {a b} ^ {r r} - \frac {J M \beta}{2} \sum_ {a} \left(Q _ {a a} ^ {r r} - \zeta^ {2}\right)\right)
$$
$$
\exp \left(\frac {1}{2} V ^ {r \top} \left(T ^ {r}\right) ^ {- 1} V ^ {r} - \frac {1}{2} \log \det  \left(T ^ {r}\right) - \frac {1}{2} \sum_ {a b} \hat {Q} _ {a b} ^ {r r} \left(M \left(\zeta^ {2} + \eta_ {r} ^ {2}\right) + \boldsymbol {w} ^ {* \top} \Sigma_ {s} \boldsymbol {w} ^ {*}\right)\right) \tag {48}
$$
We now make the following replica-symmetric ansatz:
$$
Q _ {a b} ^ {r r} = \beta^ {- 1} q \delta_ {a b} + q _ {0} \tag {49}
$$
$$
\hat {Q} _ {a b} ^ {r r} = \beta \hat {q} \delta_ {a b} + \beta^ {2} \hat {q} _ {0} \tag {50}
$$
which is well-motivated because the loss function is convex. We will verify that the chosen scalings are self-consistent in the zero-temperature limit where  $\beta \rightarrow \infty$ . We may then rewrite the partition function as follows:
$$
\langle Z ^ {n} \rangle_ {\mathcal {D}} = \exp \left(- \frac {n M}{2} \mathfrak {g} [ q, q _ {0}, \hat {q}, \hat {q} _ {0} ]\right) \tag {51}
$$
Where the effective action is written: