SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\partial \mathfrak {g}}{\partial R} = 0 = \beta \frac {\alpha (\lambda + q)}{(\lambda + q) (\lambda + r) - c ^ {2}} - \beta \hat {r} \tag {107}
$$
$$
\frac {\partial \mathfrak {g}}{\partial \hat {R}} = 0 = - \beta r + \beta \frac {1}{M} \operatorname {t r} \left(\boldsymbol {G} ^ {- 1} \left[ \begin{array}{c c} \boldsymbol {0} & \boldsymbol {0} \\ \boldsymbol {0} & \tilde {\boldsymbol {\Sigma}} _ {r ^ {\prime} r ^ {\prime}} \end{array} \right]\right) \tag {108}
$$
$$
\frac {\partial \mathfrak {g}}{\partial C} = 0 = - \beta \frac {2 \alpha c}{(\lambda + q) (\lambda + r) - c ^ {2}} - 2 \beta \hat {c} + \beta J \tag {109}
$$
$$
\frac {\partial \mathfrak {g}}{\partial \hat {C}} = 0 = - 2 \beta c + \beta \frac {1}{M} \operatorname {t r} \left(\boldsymbol {G} ^ {- 1} \left[ \begin{array}{c c} \boldsymbol {0} & \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \\ \tilde {\boldsymbol {\Sigma}} _ {r ^ {\prime} r} & \boldsymbol {0} \end{array} \right]\right) \tag {110}
$$
Note that when  $J = 0$ , the saddle point equations 109, 110 are solved by setting  $c = \hat{c} = 0$ , and in this case the remaining saddle-point equations decouple over the readouts (as expected for independently trained ensemble members) giving: For readout  $r$ :
$$
0 = \frac {\alpha}{(\lambda + q)} - \hat {q} \tag {111}
$$
$$
0 = - q + \frac {1}{M} \operatorname {t r} \left(\boldsymbol {G} _ {r r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r}\right) \tag {112}
$$
and for readout  $r'$ :
$$
0 = \frac {\alpha}{(\lambda + r)} - \hat {r} \tag {113}
$$
$$
0 = - r + \frac {1}{M} \operatorname {t r} \left(\boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r ^ {\prime} r ^ {\prime}}\right) \tag {114}
$$
These are equivalent to the saddle-point equations for a single readout given in equation 53, 54 as expected for independently trained readouts. It is physically sensible that  $c = 0$  when  $J = 0$ , because at zero source, there is no term in the replicated system energy function which would distinguish the overlap between two readouts from the same replica from the overlap between two readouts in separate replicas (we expect that the total overlap between readouts is non-zero, as we may still have  $C > 0$ ).
The saddle-point equations obtained by setting the derivatives  $\frac{\partial\mathfrak{g}}{\partial q} = \frac{\partial\mathfrak{g}}{\partial\dot{q}} = \frac{\partial\mathfrak{g}}{\partial r} = \frac{\partial\mathfrak{g}}{\partial\dot{r}} = 0$  will similarly decouple to recover two copies of the diagonal case 57 55. We will not re-write the expressions here as they are not necessary to determine the off-diagonal error term  $E_{rr^{\prime}}$ .
The remaining saddle-point equations are obtained by setting  $\frac{\partial\mathfrak{g}}{\partial c} = \frac{\partial\mathfrak{g}}{\partial\hat{c}} = 0$
$$
\left. \frac {\partial \mathfrak {g}}{\partial c} \right| _ {c = \hat {c} = J = 0} = - \frac {2 \alpha \beta C}{(\lambda + q) (\lambda + r)} - 2 \beta \hat {C} \quad \Rightarrow \quad \hat {C} = - \frac {\alpha C}{(\lambda + q) (\lambda + r)} \tag {115}
$$
$$
\begin{array}{l} \left. \frac {\partial \mathfrak {g}}{\partial \hat {c}} \right| _ {c = \hat {c} = J = 0} = 0 = - 2 \beta C + 2 \beta \left(\frac {1}{M} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} + \zeta^ {2}\right) \\ - \frac {2 \beta}{M} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \left[ \frac {1}{\nu_ {r r}} \hat {q} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r r} ^ {- 1} \boldsymbol {A} _ {r} + \frac {1}{\nu_ {r ^ {\prime} r ^ {\prime}}} \hat {r} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}} \right] \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} \tag {116} \\ + \frac {2 \beta \hat {q} \hat {r}}{M} \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} \\ - \frac {2 \hat {C} \beta}{M} \operatorname {t r} \left[ \boldsymbol {G} _ {r r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r ^ {\prime} r} \right] \\ \end{array}
$$
Solving equations 115 and 116 for  $C$ , we obtain:
$$
\begin{array}{l} C = \frac {1}{1 - \gamma} \zeta^ {2} + \frac {1}{1 - \gamma} \left(\frac {1}{M} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*}\right) \\ - \frac {1}{M (1 - \gamma)} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \left[ \frac {1}{\nu_ {r r}} \hat {q} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r r} ^ {- 1} \boldsymbol {A} _ {r} + \frac {1}{\nu_ {r ^ {\prime} r ^ {\prime}}} \hat {r} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}} \right] \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} (117) \\ + \frac {1}{M (1 - \gamma)} \hat {q} \hat {r} \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} \\ \text {w h e r e} \quad \gamma \equiv \frac {\alpha}{(\lambda + q) (\lambda + r)} \operatorname {t r} \left[ \boldsymbol {G} _ {r r ^ {\prime}} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r ^ {\prime} r} \right] (118) \\ \end{array}
$$
We can obtain the generalization error as:
$$
E _ {r r ^ {\prime}} = \lim  _ {\beta \rightarrow \infty} \frac {1}{\beta} \frac {\partial}{\partial J} \mathfrak {g} [ q, Q, r, R, c, C, \hat {q}, \hat {Q}, \hat {r}, \hat {R}, \hat {c}, \hat {C} ] = \lim  _ {\beta \rightarrow \infty} \frac {1}{\beta} (c + \beta C - \beta \zeta^ {2}) = C - \zeta^ {2} \tag {119}
$$
We may then simplify the expression for the  $E_{rr'}$  error as follows:
$$
\begin{array}{l} E _ {r r ^ {\prime}} = \frac {\gamma}{1 - \gamma} \zeta^ {2} + \frac {1}{1 - \gamma} \left(\frac {1}{M} \pmb {w} ^ {* \top} \pmb {\Sigma} _ {s} \pmb {w} ^ {*}\right) \\ - \frac {1}{M (1 - \gamma)} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \left[ \frac {1}{\nu_ {r r}} \hat {q} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r r} ^ {- 1} \boldsymbol {A} _ {r} + \frac {1}{\nu_ {r ^ {\prime} r ^ {\prime}}} \hat {r} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}} \right] \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} \tag {120} \\ + \frac {1}{M (1 - \gamma)} \hat {q} \hat {r} \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} \\ \end{array}
$$
Re-labeling the order parameters:  $\hat{q} \rightarrow \hat{q}_r$ ,  $\hat{r} \rightarrow \hat{q}_{r'}$ ,  $\gamma \rightarrow \gamma_{rr'}$  and  $G_{rr} \rightarrow G_r$ , we obtain the result given in the main text.
G Derivation of Proposition 1 from [31]
In the case where the data and noise covariance matrices  $\pmb{\Sigma}_s$  and  $\pmb{\Sigma}_0$  have bounded spectra, our main result may be derived using Theorem 4.1 from Loureiro et. al. [31], with a few additional arguments to incorporate a readout noise which varies across ensemble members, and to allow for the presence of label noise in the training set but not at test time. Rather than reproducing their very lengthy statements here, we direct the reader to theorem 4.1 and corollary 4.2 in [31].
G.1 Altered Expectations at Evaluation
In [31], labels are generated at both training and test time through the same statistical process  $y \sim P_y^0(y|\nu)$  where  $\nu = \frac{\langle x|\theta\rangle}{\sqrt{d}}$ . However, their results may be easily extended to the case where data are generated through a different statistical process for the training set and at evaluation. This will allow the application of their results to the case where label noise is present during training but not at evaluation as studied in this work. We therefore introduce separate distributions of the labels  $y$  at training and evaluation. During training, we still have  $y \sim P_y^0(y|\nu)$ . At evaluation, we put  $y \sim P_y^g(y|\nu)$ . This leads to the updated formula:
$$
\mathbb {E} _ {(y, \boldsymbol {x})} \left[ \varphi \left(y, \frac {\langle \langle \hat {\boldsymbol {W}} \mid \boldsymbol {U} \rangle \rangle}{\sqrt {p}}\right) \right] \xrightarrow {\mathrm {P}} \mathbb {E} _ {(\nu , \boldsymbol {\mu})} \left[ \int d y P _ {y} ^ {g} (y | \nu) \varphi (y, \boldsymbol {\mu}) \right] \tag {121}
$$
when the expectation is over data-label pairs  $(y, x)$  at evaluation.
G.2 Rigorous Proof of Proposition 1
We now restate the problem setup of our main theorem using notation consistent with [31]. We study generalization error in an ensemble of estimators  $\{\hat{\pmb{w}}_k\}$ ,  $k = 1, \dots, K$ . We say  $\hat{\pmb{w}}_k \in \mathbb{R}^p$  for all  $k = 1, \dots, K$ . The weights are trained independently such that each minimizes a ridge loss function:
$$
\begin{array}{l} \hat {\boldsymbol {w}} _ {k} = \arg \min  _ {\boldsymbol {w}} \sum_ {\mu = 1} ^ {n} \left(y ^ {\mu} - \frac {1}{\sqrt {N _ {k}}} \boldsymbol {u} _ {k} ^ {\top} \boldsymbol {w} - \xi_ {k} ^ {\mu}\right) ^ {2} + \lambda_ {k} | \boldsymbol {w} | ^ {2} \quad k = 1, \dots , K \\ = \arg \min  _ {\boldsymbol {w}} \sum_ {\mu = 1} ^ {n} \left(y ^ {\mu} - \frac {1}{\sqrt {p}} \boldsymbol {u} _ {k} ^ {\top} \left(\frac {1}{\sqrt {\nu_ {k k}}} \boldsymbol {w}\right) - \xi_ {k} ^ {\mu}\right) ^ {2} + \nu_ {k k} \lambda_ {k} | \frac {1}{\sqrt {\nu_ {k k}}} \boldsymbol {w} | ^ {2} \tag {122} \\ \end{array}
$$
So that our results will correspond to the results of [31] after re-scaling the regularizations  $\lambda_k \to \nu_{kk} \lambda_k$ . The training labels are drawn as  $y^\mu \sim P_y^0(y|\frac{1}{\sqrt{p}} \boldsymbol{\theta}^\top \boldsymbol{x}^\mu)$  where  $P_y^0(y|x) = \mathcal{N}(x, \zeta^2)$ ,  $\xi_k^\mu \sim \mathcal{N}(0, \eta_r^2)$  and  $\theta$  represent the ground truth weights. For  $k = 1, \dots, k$  we have a "measurement matrix"  $\mathbf{A}_k \in \mathbb{R}^{N_k \times d}$ , and we may set  $d = p \geq \max_k N_k$  (so that  $\gamma = \frac{d}{p} = 1$ ). The feature vectors  $\mathbf{u}_k(x)$  are then drawn as
$$
\boldsymbol {u} _ {k} (\boldsymbol {x}) = \left[ \begin{array}{c} \boldsymbol {A} _ {k} (\boldsymbol {x} + \boldsymbol {\sigma}) \\ \boldsymbol {0} _ {(p - N _ {k})} \end{array} \right] = \bar {\boldsymbol {A}} _ {k} (\boldsymbol {x} + \boldsymbol {\sigma})
$$
where  $\pmb{x} \sim \mathcal{N}(0, \pmb{\Sigma}_s)$  and  $\pmb{\sigma} \sim \mathcal{N}(0, \pmb{\Sigma}_0)$ . For convenience, we have defined the auxiliary matrices
$$
\bar {\boldsymbol {A}} _ {k} \equiv \left[ \begin{array}{c} \boldsymbol {A} _ {k} \\ \mathbf {0} _ {(p - N _ {k}) \times p} \end{array} \right] \in \mathbb {R} ^ {p \times p}
$$
By constructing the feature vectors as p-dimensional vectors with only  $N_{k}$  non-zero components, we may apply the results of [31] while preserving structural heterogeneity (we may have  $N_{k} \neq N_{k'}$  for  $k \neq k'$ ) as is present in our main result. Because  $[\pmb{u}_k]_i = 0$  for all  $i > N_k$ , these auxiliary dimensions will not affect model predictions or generalization error. We then have  $\mu_{k} = \frac{1}{\sqrt{p}}\pmb{u}_{k}^{\top}\hat{\pmb{w}}_{k}$ , and labels are generated at evaluation according to:
$$
y \sim P _ {y} ^ {g} (y | \frac {1}{K} \sum_ {k} \mu_ {k}) \text {w h e r e} P _ {y} ^ {g} (y | x) = \mathcal {N} \left(x, \frac {1}{K ^ {2}} \sum_ {k} \eta_ {k} ^ {2}\right) \tag {123}
$$
The generalization error may then be decomposed as  $E_{g} = \frac{1}{K^{2}}\sum_{k,k^{\prime}}E_{kk^{\prime}}$  where  $E_{kk^{\prime}} = \mathbb{E}_{(y,\pmb {x})}\left[(y - \mu_k)(y - \mu_{k'})\right]$ . We will apply eq. 121 separately to calculate  $E_{rr'}$  in the cases where  $r\neq r'$  and  $r = r'$ .
G.2.1 Off-Diagonal Terms
Eq. 121 cannot be used to calculate  $E_{rr'}$  directly in the case where  $r \neq r'$  due to the presence of noises  $\xi_k^\mu$  which vary over elements of the ensemble (indexed by  $k$ ) in the loss function (eq. 122). We may argue, however, that the presence of readout noise noise has no effect on the expected value of  $E_{rr'}$  when  $r \neq r'$ , then apply eq. 121 with  $\eta_k = \eta_{k'} = 0$ . To see this, we examine the analytical form of the minimizer of the loss function (eq. 122):
$$
\hat {\boldsymbol {w}} _ {k} = \boldsymbol {U} _ {k} ^ {\top} \left(\boldsymbol {U} _ {k} \boldsymbol {U} _ {k} ^ {\top} + \lambda_ {k} \boldsymbol {I}\right) ^ {- 1} (\boldsymbol {y} - \boldsymbol {\xi} _ {k}) \tag {124}
$$
where we have defined the design matrices  $[U_k]_{i\mu} = \frac{1}{\sqrt{p}} [u_k(x^\mu)]_i$  and the vectors  $[\pmb {y}]_{\mu} = y^{\mu}$  and  $[\pmb {\xi}_k]_\mu = \xi_k^\mu$ . We then have
$$
\begin{array}{l} E _ {k k ^ {\prime}} = \left(y - \frac {1}{\sqrt {p}} \boldsymbol {y} ^ {\top} \left(\boldsymbol {U} _ {k} \boldsymbol {U} _ {k} ^ {\top} + \lambda_ {k} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {U} _ {k}\right) \left(y - \frac {1}{\sqrt {p}} \boldsymbol {y} ^ {\top} \left(\boldsymbol {U} _ {k ^ {\prime}} \boldsymbol {U} _ {k ^ {\prime}} ^ {\top} + \lambda_ {k ^ {\prime}} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {U} _ {k ^ {\prime}}\right) \\ + \left(y - \frac {1}{\sqrt {p}} \boldsymbol {y} ^ {\top} \left(\boldsymbol {U} _ {k} \boldsymbol {U} _ {k} ^ {\top} + \lambda_ {k} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {U} _ {k}\right) \left(\frac {1}{\sqrt {p}} \boldsymbol {\xi} _ {k ^ {\prime}} ^ {\top} \left(\boldsymbol {U} _ {k ^ {\prime}} \boldsymbol {U} _ {k ^ {\prime}} ^ {\top} + \lambda_ {k ^ {\prime}} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {U} _ {k ^ {\prime}}\right) \\ + \left(y - \frac {1}{\sqrt {p}} \boldsymbol {y} ^ {\top} \left(\boldsymbol {U} _ {k ^ {\prime}} \boldsymbol {U} _ {k ^ {\prime}} ^ {\top} + \lambda_ {k ^ {\prime}} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {U} _ {k ^ {\prime}}\right) \left(\frac {1}{\sqrt {p}} \boldsymbol {\xi} _ {k} ^ {\top} \left(\boldsymbol {U} _ {k} \boldsymbol {U} _ {k} ^ {\top} + \lambda_ {k} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {U} _ {k}\right) \\ + \left(\frac {1}{\sqrt {p}} \boldsymbol {\xi} _ {k} ^ {\top} \left(\boldsymbol {U} _ {k} \boldsymbol {U} _ {k} ^ {\top} + \lambda_ {k} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {U} _ {k}\right) \left(\frac {1}{\sqrt {p}} \boldsymbol {\xi} _ {k ^ {\prime}} ^ {\top} \left(\boldsymbol {U} _ {k ^ {\prime}} \boldsymbol {U} _ {k ^ {\prime}} ^ {\top} + \lambda_ {k ^ {\prime}} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {U} _ {k ^ {\prime}}\right) \tag {125} \\ \end{array}
$$
Taking the expectation value over the readout noise in the training set we get:
$$
\mathbb {E} _ {\left\{\boldsymbol {\xi} _ {1}, \dots , \boldsymbol {\xi} _ {K} \right\}} E _ {k k ^ {\prime}} = \left(y - \frac {1}{\sqrt {p}} \boldsymbol {y} ^ {\top} \left(\boldsymbol {U} _ {k} \boldsymbol {U} _ {k} ^ {\top} + \lambda_ {k} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {U} _ {k}\right) \left(y - \frac {1}{\sqrt {p}} \boldsymbol {y} ^ {\top} \left(\boldsymbol {U} _ {k ^ {\prime}} \boldsymbol {U} _ {k ^ {\prime}} ^ {\top} + \lambda_ {k ^ {\prime}} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {U} _ {k ^ {\prime}}\right) \quad (k \neq k ^ {\prime}) \tag {126}
$$
This is identical to  $E_{kk'}$  when  $\xi_k = 0$  for all  $k$ . We may therefore calculate the off-diagonal error terms by setting  $\xi_k = 0$  in eq. 122, which gives a problem compatible with the theorem 4 of [31]. To calculate  $E_{rr'}$ , we appeal to theorem 4.1 and corollary 4.2 of [31]. The following objects defined in 121 are given the following definitions:
$$
r \left(\left\{\hat {\boldsymbol {w}} _ {1}, \dots , \hat {\boldsymbol {w}} _ {K} \right\}\right) = \frac {1}{2} \sum_ {k = 1} ^ {K} \nu_ {k k} \lambda_ {k} \left| \hat {\boldsymbol {w}} _ {k} \right| ^ {2} \tag {127}
$$
$$
\Delta (y, \hat {y} (\boldsymbol {x})) = (y - \hat {y} (\boldsymbol {x})) ^ {2} \tag {128}
$$
$$
\hat {\ell} (y, \boldsymbol {\mu}) = \frac {1}{2} | \boldsymbol {\mu} - y \mathbf {1} | ^ {2} \tag {129}
$$
$$
\mathcal {Z} ^ {0} (y, \mu , \sigma) := \int \frac {P _ {y} ^ {0} (y \mid x) \mathrm {d} x}{\sqrt {2 \pi \sigma}} \mathrm {e} ^ {- \frac {(x - \mu) ^ {2}}{2 \sigma}} = \frac {1}{\sqrt {2 \pi (\zeta^ {2} + \sigma)}} \exp \left(- \frac {(y - \mu) ^ {2}}{2 (\zeta^ {2} + \sigma)}\right) \tag {130}
$$
$$
\begin{array}{l} E _ {k k ^ {\prime}} = \mathbb {E} _ {(y, \boldsymbol {x})} [ (y - \mu_ {k}) (y - \mu_ {k ^ {\prime}}) ] \rightarrow \mathbb {E} _ {(\nu , \boldsymbol {x})} \left[ \int d y P _ {y} ^ {g} (y | \nu) (y - \mu_ {k}) (y - \mu_ {k ^ {\prime}}) \right] (131) \\ = \mathbb {E} _ {(\nu , \boldsymbol {x})} \left[ (\nu - \mu_ {k}) \left(\nu - \mu_ {k ^ {\prime}}\right) \right] + \frac {1}{K ^ {2}} \sum_ {k = 1} ^ {K} \eta_ {k} ^ {2} (132) \\ = \rho - [ \boldsymbol {m} ] _ {k} - [ \boldsymbol {m} ] _ {k ^ {\prime}} + [ \boldsymbol {Q} ] _ {k k ^ {\prime}} + \frac {1}{K ^ {2}} \sum_ {k = 1} ^ {K} \eta_ {k} ^ {2} (133) \\ \end{array}
$$
Where  $\rho, m$  and  $Q$  are defined as in [31]:
$$
(\nu , \boldsymbol {\mu}) \sim \mathcal {N} \left(\mathbf {0} _ {1 + K}, \left( \begin{array}{c c} \rho & \boldsymbol {m} ^ {\top} \\ \boldsymbol {m} & \boldsymbol {Q} \end{array} \right)\right) \tag {134}
$$
What remains is to determine the values of the order parameters  $\rho$ ,  $m$  and  $Q$ . We next define the covariance matrices which characterize the feature maps. The feature-feature covariance is:
$$
\begin{array}{l} \boldsymbol {\Omega} _ {k k ^ {\prime}} ^ {i j} = \mathbb {E} _ {\boldsymbol {x}} \left[ \left(\boldsymbol {u} _ {k} (\boldsymbol {x})\right) _ {i} \left(\boldsymbol {u} _ {k ^ {\prime}} (\boldsymbol {x})\right) _ {j} \right] = \mathbb {1} _ {\{1 \leq i \leq N _ {k}, 1 \leq j \leq N _ {k ^ {\prime}} \}} \left[ \boldsymbol {A} _ {k} \left(\boldsymbol {\Sigma} _ {s} + \boldsymbol {\Sigma} _ {0}\right) \boldsymbol {A} _ {k ^ {\prime}} ^ {\top} \right] _ {i j} (135) \\ = \sqrt {\nu_ {k k} \nu_ {k ^ {\prime}} k ^ {\prime}} \mathbb {1} _ {\{1 \leq i \leq N _ {k}, 1 \leq j \leq N _ {k ^ {\prime}} \}} \left[ \tilde {\Sigma} _ {k k ^ {\prime}} \right] _ {i j} = \left[ \bar {\Sigma} _ {k k ^ {\prime}} \right] _ {i j} (136) \\ \end{array}
$$
where we have defined  $\nu_{kk} = \frac{N_k}{p}$ ,  $\tilde{\Sigma}_{kk'} = \frac{1}{\sqrt{\nu_{kk'}\nu_{k'}k'}} A_k(\Sigma_s + \Sigma_0)A_{k'}^\top$ , and  $\bar{\Sigma}_{kk'} = \sqrt{\nu_{kk'}\nu_{k'}k'}\left[ \begin{array}{cc}\tilde{\Sigma}_{kk'} & 0\\ 0 & 0 \end{array} \right] \in \mathbb{R}^{p\times p}$ . Note that while in [31], all  $\Omega_{kk}$  must have strictly positive eigenvalues, their result can be easily extended to cover the case where some eigenvalues are zero by a continuity argument.
The feature-label covariance is given by:
$$
\left[ \hat {\boldsymbol {\Phi}} \right] _ {k} ^ {i} = \mathbb {E} _ {\boldsymbol {x}} \left[ \boldsymbol {u} _ {k} (\boldsymbol {x}) \boldsymbol {x} ^ {\top} \boldsymbol {\theta} \right] _ {i} = \mathbb {1} _ {\{1 \leq i \leq N _ {k} \}} \left[ \boldsymbol {A} _ {k} \boldsymbol {\Sigma} _ {s} \boldsymbol {\theta} \right] _ {i} = \left[ \bar {\boldsymbol {A}} _ {k} \boldsymbol {\Sigma} _ {s} \boldsymbol {\theta} \right] _ {i} \tag {137}
$$
$$
\left[ \boldsymbol {\Theta} \right] _ {k k ^ {\prime}} ^ {i j} = \left[ \hat {\boldsymbol {\Phi}} \right] _ {k} ^ {i} \left[ \hat {\boldsymbol {\Phi}} \right] _ {k ^ {\prime}} ^ {j} = \left[ \bar {\boldsymbol {A}} _ {k} \boldsymbol {\Sigma} _ {s} \boldsymbol {\theta} \right] _ {i} \left[ \bar {\boldsymbol {A}} _ {k ^ {\prime}} \boldsymbol {\Sigma} _ {s} \boldsymbol {\theta} \right] _ {j} \tag {138}
$$
Recalling  $\omega \coloneqq Q^{1 / 2}\xi$ , we can now obtain an explicit form for the proximal  $h$ :
$$
\boldsymbol {h} := \underset {\boldsymbol {u}} {\operatorname {a r g m i n}} \left[ \frac {(\boldsymbol {u} - \boldsymbol {\omega}) \boldsymbol {V} ^ {- 1} (\boldsymbol {u} - \boldsymbol {\omega})}{2} + \hat {\ell} (\boldsymbol {y}, \boldsymbol {u}) \right] = \boldsymbol {V} (\boldsymbol {I} + \boldsymbol {V}) ^ {- 1} \left(\boldsymbol {V} ^ {- 1} \boldsymbol {Q} ^ {1 / 2} \boldsymbol {\xi} + \boldsymbol {y} \mathbf {1}\right) \tag {139}
$$
The proximal  $G$  should not arise in this special case.
Next, we will simplify the saddle-point equations. Simplifying where possible, we may write:
$$
\boldsymbol {f} = \boldsymbol {V} ^ {- 1} (\boldsymbol {h} - \boldsymbol {w}) = (\boldsymbol {I} + \boldsymbol {V}) ^ {- 1} (y \mathbf {1} - \boldsymbol {\omega}) \tag {140}
$$
$$
\partial_ {\boldsymbol {\omega}} \boldsymbol {f} = - (\boldsymbol {I} + \boldsymbol {V}) ^ {- 1} \tag {141}
$$
$$
\rho = \mathbb {E} _ {\boldsymbol {x}} \left[ \left(\frac {1}{\sqrt {d}} \boldsymbol {\theta} ^ {\top} \boldsymbol {x}\right) ^ {2} \right] = \frac {1}{d} \boldsymbol {\theta} ^ {\top} \boldsymbol {\Sigma} _ {s} \boldsymbol {\theta} \tag {142}
$$
$$
\omega_ {0} \equiv \boldsymbol {m} ^ {\top} \boldsymbol {Q} ^ {- 1 / 2} \boldsymbol {\xi} \tag {143}
$$
$$
\sigma_ {0} = \rho - \boldsymbol {m} ^ {\top} \boldsymbol {Q} ^ {- 1} \boldsymbol {m} = \frac {1}{d} \boldsymbol {\theta} ^ {\top} \boldsymbol {\Sigma} _ {s} \boldsymbol {\theta} - \boldsymbol {m} ^ {\top} \boldsymbol {Q} ^ {- 1} \boldsymbol {m} \tag {144}
$$
$$
\begin{array}{l} \hat {\boldsymbol {V}} = - \alpha \int d y \mathbb {E} _ {\boldsymbol {\xi}} \left[ Z ^ {0} \left(y, \omega_ {0}, \sigma_ {0}\right) \left(- \left(\boldsymbol {I} + \boldsymbol {V}\right) ^ {- 1}\right) \right] = \alpha (\boldsymbol {I} + \boldsymbol {V}) ^ {- 1} \mathbb {E} _ {\boldsymbol {\xi}} \left[ \underbrace {\int d y Z ^ {0} \left(y , \omega_ {0} , \sigma_ {0}\right)} _ {1} \right] (145) \\ \Rightarrow \hat {\boldsymbol {V}} = \alpha (\boldsymbol {I} + \boldsymbol {V}) ^ {- 1} (146) \\ \end{array}
$$
We can simplify the prior equation for  $\hat{V}$  to the following set of equations:
$$
\boldsymbol {V} _ {k k} = \frac {1}{p} \operatorname {t r} \left[ \bar {\boldsymbol {\Sigma}} _ {k k} \left[ \nu_ {k k} \lambda_ {k} \boldsymbol {I} _ {p} + \hat {\boldsymbol {V}} _ {k k} \bar {\boldsymbol {\Sigma}} _ {k k} \right] ^ {- 1} \right] \quad k = 1, \dots , K \tag {147}
$$
$$
\boldsymbol {V} _ {k k ^ {\prime}} = \frac {1}{p} \operatorname {t r} \left[ \bar {\boldsymbol {\Sigma}} _ {k k ^ {\prime}} \left[ \hat {\boldsymbol {V}} _ {k ^ {\prime} k} \bar {\boldsymbol {\Sigma}} _ {k ^ {\prime} k} \right] ^ {- 1} \right] \quad k ^ {\prime} \neq k \tag {148}
$$
Equations 146 and 148 are solved by setting  $\hat{\mathbf{V}}_{kk'} = \mathbf{V}_{kk'} = 0$  for all  $k \neq k'$  so that  $\hat{\mathbf{V}}_{kk'} = \hat{V}_k\delta_{kk'}$  and  $\mathbf{V}_{kk'} = V_k\delta_{kk'}$ . The diagonal components then satisfy
$$
\hat {V} _ {k} = \frac {\alpha}{1 + V _ {k}} \tag {149}
$$
$$
V _ {k} = \frac {1}{p} \operatorname {t r} \left[ \bar {\boldsymbol {\Sigma}} _ {k k} \left[ \nu_ {k k} \lambda_ {k} \boldsymbol {I} _ {p} + \hat {V} _ {k} \bar {\boldsymbol {\Sigma}} _ {k k} \right] ^ {- 1} \right] \tag {150}
$$
Which may be solved separately for each  $k = 1, \dots, K$ . Simplifying the remaining channel equations we have:
$$
\hat {\boldsymbol {Q}} = \alpha (\boldsymbol {I} + \boldsymbol {V}) ^ {- 1} \left[ \left(\zeta^ {2} + \rho\right) \mathbf {1 1} ^ {\top} - \mathbf {1 m} ^ {\top} - \mathbf {m 1} ^ {\top} + \boldsymbol {Q} \right] (\boldsymbol {I} + \boldsymbol {V}) ^ {- 1} \tag {151}
$$