SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathfrak {g} [ q, q _ {0}, \hat {q}, \hat {q} _ {0} ] = \alpha \left[ \log (1 + \frac {q}{\lambda}) + \frac {\beta q _ {0}}{\lambda + q} \right] - (q \hat {q} + \beta q \hat {q} _ {0} + \beta q _ {0} \hat {q}) + \beta J [ (\beta^ {- 1} q + q _ {0}) - \zeta^ {2} ] \\ - \frac {\beta}{\nu_ {r r} M} \hat {q} ^ {2} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} ^ {- 1} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} + \frac {1}{M} \left[ \log \det  (\boldsymbol {G}) + \beta \hat {q} _ {0} \operatorname {t r} [ \boldsymbol {G} ^ {- 1} \tilde {\boldsymbol {\Sigma}} ] \right] + \beta \hat {q} \left(\zeta^ {2} + \eta_ {r} ^ {2} + \frac {1}{M} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*}\right) \tag {52} \\ \end{array}
$$
Where  $\pmb {G}\equiv \pmb{I}_{N_r} + \hat{q}\tilde{\pmb{\Sigma}}$  and  $\tilde{\Sigma}\equiv \frac{1}{\nu_{rr}} A_r(\Sigma_s + \Sigma_0)\bar{A}_r^\top$
To determine the values of the order parameters we set the derivatives of  $\mathfrak{g}[q, q_0, \hat{q}, \hat{q}_0]$ , evaluated at zero source ( $J = 0$ ), to zero:
$$
\frac {\partial \mathfrak {g}}{\partial q _ {0}} = 0 = \frac {\alpha \beta}{\lambda + q} - \beta \hat {q} \quad \Rightarrow \hat {q} = \frac {\alpha}{\lambda + q} \tag {53}
$$
$$
\frac {\partial \mathfrak {g}}{\partial \hat {q} _ {0}} = 0 = - \beta q + \frac {\beta}{M} \operatorname {t r} \left[ \boldsymbol {G} ^ {- 1} \tilde {\boldsymbol {\Sigma}} \right] \quad \Rightarrow q = \frac {1}{M} \operatorname {t r} \left[ \boldsymbol {G} ^ {- 1} \tilde {\boldsymbol {\Sigma}} \right] \tag {54}
$$
$$
\frac {\partial \mathfrak {g}}{\partial q} = 0 = \frac {\alpha}{\lambda + q} - \frac {\alpha q _ {0} \beta}{(\lambda + \beta q) ^ {2}} - \hat {q} - \beta \hat {q} _ {0} \quad \Rightarrow \hat {q} _ {0} = - \frac {\alpha q _ {0}}{(\lambda + \beta q) ^ {2}} \tag {55}
$$
$$
\begin{array}{l} \frac {\partial \mathfrak {g}}{\partial \hat {q}} = 0 = - q - \beta q _ {0} + \beta \zeta^ {2} + \beta \eta_ {r} ^ {2} + \frac {1}{M} \operatorname {t r} \left[ \boldsymbol {G} ^ {- 1} \tilde {\boldsymbol {\Sigma}} \right] - \frac {\beta \hat {q} _ {0}}{M} \operatorname {t r} \left[ \left(\boldsymbol {G} ^ {- 1} \tilde {\boldsymbol {\Sigma}}\right) ^ {2} \right] (56) \\ + \frac {\beta}{M} \boldsymbol {w} ^ {* \top} \left[ \boldsymbol {\Sigma} _ {s} - \frac {2}{\nu_ {r r}} \hat {q} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} ^ {- 1} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} + \frac {1}{\nu_ {r r}} \hat {q} ^ {2} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} ^ {- 1} \tilde {\boldsymbol {\Sigma}} \boldsymbol {G} ^ {- 1} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \right] \boldsymbol {w} ^ {*} \\ \Rightarrow q _ {0} = \frac {1}{1 - \gamma} \left(\zeta^ {2} + \eta_ {r} ^ {2} + \frac {1}{M} \boldsymbol {w} ^ {* \top} \left[ \boldsymbol {\Sigma} _ {s} - \frac {2}{\nu_ {r r}} \hat {q} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} ^ {- 1} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} + \frac {1}{\nu_ {r r}} \hat {q} ^ {2} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} ^ {- 1} \tilde {\boldsymbol {\Sigma}} \boldsymbol {G} ^ {- 1} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \right] \boldsymbol {w} ^ {*}\right) (57) \\ \end{array}
$$
where  $\gamma \equiv \frac{\alpha}{M\kappa^2}\mathrm{tr}\left[(G^{-1}\tilde{\Sigma})^2\right]$  and  $\kappa \equiv \lambda +q$  . We retroactively confirm that the chosen  $\beta$  -scalings are correct by noticing that the saddle-point equations permit a solution where all order parameters remain  $\mathcal{O}(1)$  as  $\beta \rightarrow \infty$  . The error is given as:
$$
E _ {r r} = \lim  _ {\beta \rightarrow \infty} \frac {1}{\beta} \frac {\partial}{\partial J} \mathfrak {g} [ q, q _ {0}, \hat {q}, \hat {q} _ {0} ] = \lim  _ {\beta \rightarrow \infty} \beta^ {- 1} q + q _ {0} - \zeta^ {2} = q _ {0} - \zeta^ {2} \tag {58}
$$
Where  $q, q_0, \hat{q}, \hat{q}_0$  are the solutions to the saddle-point equations 53, 54, 55, 57. Substituting eq. 57 for  $q_0$ , we obtain:
$$
\begin{array}{l} E _ {r r} = \frac {1}{1 - \gamma} \frac {1}{M} \boldsymbol {w} ^ {\star \top} \left[ \boldsymbol {\Sigma} _ {s} - \frac {2}{\nu_ {r r}} \hat {q} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} ^ {- 1} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} + \frac {1}{\nu_ {r r}} \hat {q} ^ {2} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} ^ {- 1} \tilde {\boldsymbol {\Sigma}} \boldsymbol {G} ^ {- 1} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \right] \boldsymbol {w} ^ {*} + \frac {\gamma \zeta^ {2} + \eta_ {r} ^ {2}}{1 - \gamma} (59) \\ = \frac {1}{1 - \gamma} \frac {1}{M} \boldsymbol {w} ^ {\star \top} \left[ \boldsymbol {\Sigma} _ {s} - \frac {1}{\nu_ {r r}} \hat {q} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} ^ {- 1} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} - \frac {1}{\nu_ {r r}} \hat {q} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} ^ {- 2} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \right] \boldsymbol {w} ^ {*} + \frac {\gamma \zeta^ {2} + \eta_ {r} ^ {2}}{1 - \gamma} (60) \\ \end{array}
$$
F.2 Off-Diagonal Terms
We now calculate  $E_{rr'}$  for  $r \neq r'$ . We now must consider the joint Gibbs Measure over  $\boldsymbol{w}_r$  and  $\boldsymbol{w}_{r'}$ :
$$
Z = \int d \boldsymbol {w} _ {r} d \boldsymbol {w} _ {r ^ {\prime}} \exp \left(- \frac {\beta}{2 \lambda} \left(E _ {r} ^ {t} + E _ {r ^ {\prime}} ^ {t}\right) - \frac {J M \beta}{2} E _ {r r ^ {\prime}} \left(\boldsymbol {w} _ {r}, \boldsymbol {w} _ {r ^ {\prime}}\right)\right) \tag {61}
$$
$$
\begin{array}{l} \langle Z ^ {n} \rangle_ {\mathcal {D}} = \int \prod_ {a} d \boldsymbol {w} _ {r} ^ {a} d \boldsymbol {w} _ {r ^ {\prime}} ^ {a} \mathbb {E} _ {\{\psi_ {\mu}, \boldsymbol {\sigma} ^ {\mu}, \epsilon^ {\mu} \}} \\ \exp \left(- \frac {\beta M}{2 \lambda} \sum_ {\mu , a} \frac {1}{M} \left[ \left(h _ {\mu} ^ {r a}\right) ^ {2} + \left(h _ {\mu} ^ {r ^ {\prime} a}\right) ^ {2} \right] - \frac {\beta}{2} \sum_ {a} \left[ | \boldsymbol {w} _ {r} ^ {a} | ^ {2} + | \boldsymbol {w} _ {r ^ {\prime}} ^ {a} | ^ {2} \right] - \frac {J M \beta}{2} \sum_ {a} E _ {r r ^ {\prime}} \left(\boldsymbol {w} _ {r} ^ {a}, \boldsymbol {w} _ {r ^ {\prime}} ^ {a}\right)\right) \tag {63} \\ \end{array}
$$
Where the  $h_\mu^{ra}$  are defined as before. Next we must perform the averages over quenched disorder. To do so, we note that the  $h_\mu^{ra}$  are Gaussian random variables with covariance structure:
$$
\begin{array}{l} \left\langle h _ {\mu} ^ {r a} h _ {\nu} ^ {r ^ {\prime} b} \right\rangle = \delta_ {\mu \nu} Q _ {a b} ^ {r r ^ {\prime}} (64) \\ Q _ {a b} ^ {r r ^ {\prime}} = \frac {1}{M} \left[ \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} - \boldsymbol {w} ^ {* \top}\right) \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\sqrt {\nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {w} _ {r ^ {\prime}} ^ {b} - \boldsymbol {w} ^ {*}\right) \right. (65) \\ \left. + \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} \Sigma_ {0} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {w} _ {r ^ {\prime}} ^ {b} + M \zeta^ {2} \right] \\ \end{array}
$$
To perform this integral we re-write in terms of  $\{H_{\mu}\}_{\mu = 1}^{P}$ , where
$$
\boldsymbol {H} _ {\mu} = \left[ \begin{array}{l} \boldsymbol {H} _ {\mu} ^ {r} \\ \boldsymbol {H} _ {\mu} ^ {r ^ {\prime}} \end{array} \right] \in \mathbb {R} ^ {2 n} \tag {66}
$$
$$
\begin{array}{l} \langle Z ^ {n} \rangle_ {\mathcal {D}} = \int \prod_ {a} d \boldsymbol {w} _ {r} ^ {a} d \boldsymbol {w} _ {r ^ {\prime}} ^ {a} \mathbb {E} _ {\{\psi_ {\mu}, \sigma^ {\mu}, \epsilon^ {\mu} \}} \\ \exp \left(- \frac {\beta}{2 \lambda} \sum_ {\mu} \boldsymbol {H} _ {\mu} ^ {\top} \boldsymbol {H} _ {\mu} - \frac {\beta}{2} \sum_ {a} \left[ | \boldsymbol {w} _ {r} ^ {a} | ^ {2} + | \boldsymbol {w} _ {r ^ {\prime}} ^ {a} | ^ {2} \right] - \frac {J M \beta}{2} \sum_ {a} E _ {r r ^ {\prime}} \left(\boldsymbol {w} _ {r} ^ {a}, \boldsymbol {w} _ {r ^ {\prime}} ^ {a}\right)\right) \tag {67} \\ \end{array}
$$
Integrating over  $H_{\mu}$  we get:
$$
\begin{array}{l} \langle Z ^ {n} \rangle_ {\mathcal {D}} = \int \prod_ {a} d \boldsymbol {w} _ {r} ^ {a} d \boldsymbol {w} _ {r ^ {\prime}} ^ {a} \\ \exp \left(- \frac {P}{2} \log \det  \left(\boldsymbol {I} _ {2 n} + \frac {\beta}{\lambda} \boldsymbol {Q}\right) - \frac {\beta}{2} \sum_ {a} \left[ | \boldsymbol {w} _ {r} ^ {a} | ^ {2} + | \boldsymbol {w} _ {r ^ {\prime}} ^ {a} | ^ {2} \right] - \frac {J M \beta}{2} \sum_ {a} E _ {r r} \left(\boldsymbol {w} _ {r} ^ {a}, \boldsymbol {w} _ {r ^ {\prime}} ^ {a}\right)\right) \tag {68} \\ \end{array}
$$
Where we have defined the matrix  $Q$  so that:
$$
\boldsymbol {Q} = \left[ \begin{array}{l l} \boldsymbol {Q} ^ {r r} & \boldsymbol {Q} ^ {r r ^ {\prime}} \\ \boldsymbol {Q} ^ {r r ^ {\prime}} & \boldsymbol {Q} ^ {r ^ {\prime} r ^ {\prime}} \end{array} \right] \tag {69}
$$
Next we integrate over  $\mathbf{Q}$  and add constraints. We use the following identity:
$$
\begin{array}{l} 1 = \prod_ {a b} \int d Q _ {a b} ^ {r r ^ {\prime}} \delta \left(Q _ {a b} ^ {r r ^ {\prime}} - \frac {1}{M} \left[ \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} - \boldsymbol {w} ^ {* \top}\right) \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\sqrt {\nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {w} _ {r ^ {\prime}} ^ {b} - \boldsymbol {w} ^ {*}\right) \right. \right. \tag {70} \\ \left. \left. + \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {0} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {w} _ {r ^ {\prime}} ^ {b} + M \zeta^ {2} \right] \right) \\ \end{array}
$$
$$
\begin{array}{l} 1 = \prod_ {a b} \int \frac {1}{4 \pi i / M} d Q _ {a b} ^ {r r ^ {\prime}} d \hat {Q} _ {a b} ^ {r r ^ {\prime}} \exp \left(\frac {M}{2} \hat {Q} _ {a b} ^ {r r ^ {\prime}}\right) \left(Q _ {a b} ^ {r r ^ {\prime}} - \frac {1}{M} \left[ \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} - \boldsymbol {w} ^ {* \top}\right) \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\sqrt {\nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {w} _ {r ^ {\prime}} ^ {b} - \boldsymbol {w} ^ {*}\right) \right. \right. \\ \left.\left. + \frac {1}{\nu_ {r r}} \boldsymbol {w} _ {r} ^ {a \top} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {0} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {w} _ {r ^ {\prime}} ^ {b} + M \zeta^ {2} \right]\right)\left. \right) \tag {71} \\ \end{array}
$$
Inserting this identity and the corresponding statements for  $Q_{ab}^{rr'}$  and  $Q_{ab}^{r'r'}$  into the replicated partition function and substituting  $E_{r'r'}(\boldsymbol{w}^a) = Q_{aa}^{rr'} - \zeta^2$  we find:
$$
\begin{array}{l} \langle Z ^ {n} \rangle_ {\mathcal {D}} \propto \int \prod_ {a b r _ {1} r _ {2}} d Q _ {a b} ^ {r _ {1} r _ {2}} d \hat {Q} _ {a b} ^ {r _ {1} r _ {2}} \\ \exp \left(- \frac {P}{2} \log \det  \left(\boldsymbol {I} _ {2 n} + \frac {\beta}{\lambda} \boldsymbol {Q}\right) + \frac {1}{2} \sum_ {a b r _ {1} r _ {2}} M \hat {Q} _ {a b} ^ {r _ {1} r _ {2}} Q _ {a b} ^ {r _ {1} r _ {2}} - \frac {J M \beta}{2} \sum_ {a} \left(Q _ {a a} ^ {r r ^ {\prime}} - \zeta^ {2}\right)\right) \\ \int \prod_ {a} d \boldsymbol {w} _ {r} ^ {a} d \boldsymbol {w} _ {r ^ {\prime}} ^ {a} \exp \left( \right.- \frac {\beta}{2} \sum_ {a} \left[ | \boldsymbol {w} _ {r} ^ {a} | ^ {2} + | \boldsymbol {w} _ {r ^ {\prime}} ^ {a} | ^ {2} \right] - \frac {1}{2} \sum_ {a b r _ {1} r _ {2}} \hat {Q} _ {a b} ^ {r _ {1} r _ {2}} \left[\left(\frac {1}{\sqrt {\nu_ {r _ {1}}}} \boldsymbol {w} _ {r _ {1}} ^ {a \top} \boldsymbol {A} _ {r _ {1}} - \boldsymbol {w} ^ {* \top}\right) \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\sqrt {\nu_ {r _ {2}}}} \boldsymbol {A} _ {r _ {2}} ^ {\top} \boldsymbol {w} _ {r _ {2}} ^ {b} - \boldsymbol {w} ^ {*}\right)\right. \\ \left. \left. + \frac {1}{\sqrt {\nu_ {r _ {1}} \nu_ {r _ {2}}}} \boldsymbol {w} _ {r _ {1}} ^ {a \top} \boldsymbol {A} _ {r _ {1}} \boldsymbol {\Sigma} _ {0} \boldsymbol {A} _ {r _ {2}} ^ {\top} \boldsymbol {w} _ {r _ {2}} ^ {b} + M \zeta^ {2} \right]\right) \tag {72} \\ \end{array}
$$
Where sums over  $r_1$  and  $r_2$  run over  $\{r,r^{\prime}\}$
In order to perform the Gaussian integral over the  $\{\pmb{w}_r^a\}$ , we unfold in two steps. We first define the following:
$$
\boldsymbol {w} _ {r} ^ {\cdot} \equiv \left[ \begin{array}{c} \boldsymbol {w} _ {r} ^ {1} \\ \vdots \\ \boldsymbol {w} _ {r} ^ {n} \end{array} \right] \tag {73}
$$
$$
\left[ \hat {\boldsymbol {Q}} ^ {r r ^ {\prime}} \right] _ {a b} \equiv \hat {Q} _ {a b} ^ {r r ^ {\prime}} \tag {74}
$$
$$
\tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \equiv \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {A} _ {r} \left[ \boldsymbol {\Sigma} _ {s} + \boldsymbol {\Sigma} _ {0} \right] \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \tag {75}
$$
$$
T ^ {r r ^ {\prime}} \equiv \beta \delta_ {r r ^ {\prime}} \boldsymbol {I} _ {n} \otimes \boldsymbol {I} _ {N _ {r}} + \hat {\boldsymbol {Q}} ^ {r r ^ {\prime}} \otimes \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \tag {76}
$$
Unfolding over the replica indices, we then get:
$$
\begin{array}{l} \langle Z ^ {n} \rangle_ {\mathcal {D}} \propto \int \prod_ {a b r _ {1} r _ {2}} d Q _ {a b} ^ {r _ {1} r _ {2}} d \hat {Q} _ {a b} ^ {r _ {1} r _ {2}} \\ \exp \left(- \frac {P}{2} \log \det  \left(\boldsymbol {I} _ {2 n} + \frac {\beta}{\lambda} \boldsymbol {Q}\right) + \frac {1}{2} \sum_ {a b r _ {1} r _ {2}} M \hat {Q} _ {a b} ^ {r _ {1} r _ {2}} Q _ {a b} ^ {r _ {1} r _ {2}} - \frac {J M \beta}{2} \sum_ {a} \left(Q _ {a a} ^ {r r ^ {\prime}} - \zeta^ {2}\right)\right) \\ \exp \left(- \frac {1}{2} \sum_ {a b r _ {1} r _ {2}} \hat {Q} _ {a b} ^ {r _ {1} r _ {2}} \left(\boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} + M \zeta^ {2}\right)\right) \\ \int d \boldsymbol {w} _ {r} ^ {\prime} d \boldsymbol {w} _ {r ^ {\prime}} \exp \left(- \frac {1}{2} \sum_ {r _ {1} r _ {2}} \boldsymbol {w} _ {r _ {1}} ^ {\top} T ^ {r _ {1} r _ {2}} \boldsymbol {w} _ {r _ {2}} + \sum_ {r _ {1} r _ {2}} \left[ \left(\hat {\boldsymbol {Q}} ^ {r _ {1} r _ {2}} \otimes \boldsymbol {I} _ {N _ {r _ {1}}}\right) \left(\boldsymbol {1} _ {n} \otimes \frac {1}{\sqrt {\nu_ {r _ {1}}}} \boldsymbol {A} _ {r _ {1}} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*}\right) \right] ^ {\top} \boldsymbol {w} _ {r _ {1}}\right) \tag {77} \\ \end{array}
$$
Note that the dimensionality of  $T^{r_1r_2}$  varies for different choices of  $r_1$  and  $r_2$ . Next, we unfold over the two readouts:
$$
\boldsymbol {w} \equiv \left[ \begin{array}{l} \boldsymbol {w} _ {r ^ {\prime}} \\ \boldsymbol {w} _ {r ^ {\prime}} ^ {\prime} \end{array} \right] \tag {78}
$$
$$
T \equiv \left[ \begin{array}{l l} T ^ {r r} & T ^ {r r ^ {\prime}} \\ T ^ {r ^ {\prime} r} & T ^ {r ^ {\prime} r ^ {\prime}} \end{array} \right] \tag {79}
$$
$$
V \equiv \left[ \begin{array}{c} \left(\left(\hat {\boldsymbol {Q}} ^ {r r} + \hat {\boldsymbol {Q}} ^ {r r ^ {\prime}}\right) \otimes \boldsymbol {I} _ {N _ {r}}\right) \left(\mathbf {1} _ {n} \otimes \frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*}\right) \\ \left(\left(\hat {\boldsymbol {Q}} ^ {r ^ {\prime} r ^ {\prime}} + \hat {\boldsymbol {Q}} ^ {r ^ {\prime} r}\right) \otimes \boldsymbol {I} _ {N _ {r ^ {\prime}}}\right) \left(\mathbf {1} _ {n} \otimes \frac {1}{\sqrt {\nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {A} _ {r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*}\right) \end{array} \right] \tag {80}
$$
The integral over  $w$  then becomes:
$$
\int d \boldsymbol {w} \exp \left(- \frac {1}{2} \boldsymbol {w} ^ {\top} T \boldsymbol {w} + V ^ {\top} \boldsymbol {w}\right) \propto \exp \left(\frac {1}{2} V ^ {\top} T ^ {- 1} V - \frac {1}{2} \log \det  T\right) \tag {81}
$$
We are now ready to make a replica-symmetric ansatz. The order parameter that we wish to constrain is  $Q_{ab}^{rr'}$ . Overlaps go between the weights from different replicas of the system as well as different readouts. The scale of the overlap between two measurements depends on their overlap with each other and with the principal components of the data distribution. An ansatz which is replica-symmetric but makes no assumptions about the overlaps between different measurements is as follows:
$$
Q _ {a b} ^ {r _ {1} r _ {2}} = \beta^ {- 1} q ^ {r _ {1} r _ {2}} \delta_ {a b} + Q ^ {r _ {1} r _ {2}} \tag {82}
$$
$$
\hat {Q} _ {a b} ^ {r _ {1} r _ {2}} = \beta \hat {q} ^ {r _ {1} r _ {2}} \delta_ {a b} + \beta^ {2} \hat {Q} ^ {r _ {1} r _ {2}} \tag {83}
$$
Next step is to plug the RS ansatz into the free energy and simplify. To make calculations more transparent, we re-label the paramters in the RS ansatz as follows:
$$
\boldsymbol {Q} ^ {r r} = \beta^ {- 1} q \boldsymbol {I} + Q \mathbf {1 1} ^ {\top} \tag {84}
$$
$$
\boldsymbol {Q} ^ {r ^ {\prime} r ^ {\prime}} = \beta^ {- 1} r \boldsymbol {I} + R \mathbf {1 1} ^ {\top} \tag {85}
$$
$$
\boldsymbol {Q} ^ {r r ^ {\prime}} = \beta^ {- 1} c \boldsymbol {I} + C \mathbf {1 1} ^ {\top} \tag {86}
$$
$$
\hat {\boldsymbol {Q}} ^ {r r} = \beta \hat {q} \boldsymbol {I} + \beta^ {2} \hat {\boldsymbol {Q}} \mathbf {1 1} ^ {\top} \tag {87}
$$
$$
\hat {\boldsymbol {Q}} ^ {r ^ {\prime} r ^ {\prime}} = \beta \hat {r} \boldsymbol {I} + \beta^ {2} \hat {R} \mathbf {1 1} ^ {\top} \tag {88}
$$
$$
\hat {\boldsymbol {Q}} ^ {r r ^ {\prime}} = \beta \hat {c} \boldsymbol {I} + \beta^ {2} \hat {C} \mathbf {1 1} ^ {\top} \tag {89}
$$
In order to simplify  $\log \det (\lambda I_{2n} + \beta Q)$ , we note that this is a symmetric 2-by-2-block matrix, where each block commutes with all other blocks. We may then use [53]'s result to simplify.
$$
\log \det  (\lambda I _ {2 n} + \beta Q) = n \left[ \log \left((\lambda + q) (\lambda + r) - c ^ {2}\right) + \beta \frac {(\lambda + q) R + (\lambda + r) Q - 2 c C}{(\lambda + q) (\lambda + r) - c ^ {2}} \right] + \mathcal {O} (n ^ {2}) \tag {90}
$$
$$
\sum_ {a b r _ {1} r _ {2}} \hat {\boldsymbol {Q}} _ {a b} ^ {r _ {1} r _ {2}} \boldsymbol {Q} _ {a b} ^ {r _ {1} r _ {2}} = n \left[ \left(q \hat {q} + \beta \hat {q} Q + \beta q \hat {Q}\right) + \left(r \hat {r} + \beta \hat {r} R + \beta r \hat {R}\right) + 2 \left(c \hat {c} + \beta \hat {c} C + \beta c \hat {C}\right) \right] + \mathcal {O} (n ^ {2}) \tag {91}
$$
$$
\sum_ {a} \left(\boldsymbol {Q} _ {a a} ^ {r r ^ {\prime}} - \zeta^ {2}\right) = n \left[ \frac {1}{\beta} c + C - \zeta^ {2} \right] + \mathcal {O} \left(n ^ {2}\right) \tag {92}
$$
$$
\sum_ {a b r _ {1} r _ {2}} \hat {Q} _ {a b} ^ {r _ {1} r _ {2}} = \beta n [ \hat {q} + \hat {r} + 2 \hat {c} ] + \mathcal {O} \left(n ^ {2}\right) \tag {93}
$$
$$
\log \det  (T) = n \left[ \log (\beta) + \log \det  \left[ \begin{array}{l l} \boldsymbol {G} _ {r r} & \boldsymbol {G} _ {r r ^ {\prime}} \\ \boldsymbol {G} _ {r ^ {\prime} r} & \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} \end{array} \right] + \beta \operatorname {t r} \left(\left[ \begin{array}{l l} \boldsymbol {G} _ {r r} & \boldsymbol {G} _ {r r ^ {\prime}} \\ \boldsymbol {G} _ {r ^ {\prime} r} & \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} \end{array} \right] ^ {- 1} \left[ \begin{array}{l l} \hat {Q} \tilde {\Sigma} _ {r r} & \hat {C} \tilde {\Sigma} _ {r r ^ {\prime}} \\ \hat {C} \tilde {\Sigma} _ {r ^ {\prime} r} & \hat {R} \tilde {\Sigma} _ {r ^ {\prime} r ^ {\prime}} \end{array} \right]\right) \right] + \mathcal {O} (n ^ {2}) \tag {94}
$$
$$
\text {w h e r e} \boldsymbol {G} _ {r r} = \boldsymbol {I} _ {N _ {r}} + \hat {q} \tilde {\boldsymbol {\Sigma}} _ {r r} \quad \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} = \boldsymbol {I} _ {N _ {r ^ {\prime}}} + \hat {r} \tilde {\boldsymbol {\Sigma}} _ {r ^ {\prime} r ^ {\prime}} \quad \boldsymbol {G} _ {r r ^ {\prime}} = \hat {c} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \quad \boldsymbol {G} _ {r ^ {\prime} r} = \hat {c} \tilde {\boldsymbol {\Sigma}} _ {r ^ {\prime} r} \tag {95}
$$
$$
V ^ {\top} T ^ {- 1} V = n \beta \boldsymbol {w} ^ {* \top} \left[ \begin{array}{l} \frac {1}{\sqrt {\nu_ {r r}}} (\hat {q} + \hat {c}) \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \\ \frac {1}{\sqrt {\nu_ {r ^ {\prime} r ^ {\prime}}}} (\hat {r} + \hat {c}) \boldsymbol {A} _ {r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \end{array} \right] ^ {\top} \left[ \begin{array}{l l} \boldsymbol {G} _ {r r} & \boldsymbol {G} _ {r r ^ {\prime}} \\ \boldsymbol {G} _ {r ^ {\prime} r} & \boldsymbol {G} _ {r ^ {\prime} r ^ {\prime}} \end{array} \right] ^ {- 1} \left[ \begin{array}{l} \frac {1}{\sqrt {\nu_ {r r}}} (\hat {q} + \hat {c}) \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \\ \frac {1}{\sqrt {\nu_ {r ^ {\prime} r ^ {\prime}}}} (\hat {r} + \hat {c}) \boldsymbol {A} _ {r r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \end{array} \right] \boldsymbol {w} ^ {*} + \mathcal {O} (n ^ {2}) \tag {96}
$$
Collecting these terms, we may write the replicated partition function as follows:
$$
\langle Z ^ {n} \rangle_ {\mathcal {D}} = \exp \left(- \frac {n M}{2} \mathfrak {g} [ q, Q, r, R, c, C, \hat {q}, \hat {Q}, \hat {r}, \hat {R}, \hat {c}, \hat {C} ]\right) \tag {97}
$$
Where the free energy is written:
$$
\begin{array}{l} \mathfrak {g} \left[ q, Q, r, R, c, C, \hat {q}, \hat {Q}, \hat {r}, \hat {R}, \hat {c}, \hat {C} \right] = (98) \\ \alpha \left[ \log \left((\lambda + q) (\lambda + r) - c ^ {2}\right) + \beta \frac {(\lambda + q) R + (\lambda + r) Q - 2 c C}{(\lambda + q) (\lambda + r) - c ^ {2}} \right] (99) \\ - \left[ \left(q \hat {q} + \beta \hat {q} Q + \beta q \hat {Q}\right) + \left(r \hat {r} + \beta \hat {r} R + \beta r \hat {R}\right) + 2 \left(c \hat {c} + \beta \hat {c} C + \beta c \hat {C}\right) \right] (100) \\ + J (c + \beta C - \beta \zeta^ {2}) (101) \\ + \beta [ \hat {q} + \hat {r} + 2 \hat {c} ] \left(\frac {1}{M} \boldsymbol {w} ^ {* \top} \Sigma \boldsymbol {w} ^ {*} + \zeta^ {2}\right) (102) \\ - \frac {1}{M} \boldsymbol {\beta} \boldsymbol {w} ^ {* \top} \left[ \begin{array}{l} \frac {1}{\sqrt {\nu_ {r r}}} (\hat {q} + \hat {c}) \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \\ \frac {1}{\sqrt {\nu_ {r ^ {\prime} r ^ {\prime}}}} (\hat {r} + \hat {c}) \boldsymbol {A} _ {r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \end{array} \right] ^ {\top} \boldsymbol {G} ^ {- 1} \left[ \begin{array}{l} \frac {1}{\sqrt {\nu_ {r r}}} (\hat {q} + \hat {c}) \boldsymbol {A} _ {r} \boldsymbol {\Sigma} _ {s} \\ \frac {1}{\sqrt {\nu_ {r ^ {\prime} r ^ {\prime}}}} (\hat {r} + \hat {c}) \boldsymbol {A} _ {r r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \end{array} \right] \boldsymbol {w} ^ {*} (103) \\ + \frac {1}{M} \left[ \right. \log (\beta) + \log \det  \boldsymbol {G} + \beta \operatorname {t r} \left(\boldsymbol {G} ^ {- 1} \left[\begin{array}{l l}\hat {Q} \tilde {\boldsymbol {\Sigma}} _ {r r}&\hat {C} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}}\\\hat {C} \tilde {\boldsymbol {\Sigma}} _ {r ^ {\prime} r}&\hat {R} \tilde {\boldsymbol {\Sigma}} _ {r ^ {\prime} r ^ {\prime}}\end{array}\right]\right)\left. \right)\left. \right] (104) \\ \end{array}
$$
where we have defined  $G \equiv \left[ \begin{array}{cc} G_{rr} & G_{rr'} \\ G_{r'r} & G_{r'r'} \end{array} \right]$ .
The saddle-point equations for the replica-diagonal order parameters are:
$$
\frac {\partial \mathfrak {g}}{\partial Q} = 0 = \beta \frac {\alpha (\lambda + r)}{(\lambda + q) (\lambda + r) - c ^ {2}} - \beta \hat {q} \tag {105}
$$
$$
\frac {\partial \mathfrak {g}}{\partial \hat {Q}} = 0 = - \beta q + \beta \frac {1}{M} \operatorname {t r} \left(\boldsymbol {G} ^ {- 1} \left[ \begin{array}{l l} \tilde {\boldsymbol {\Sigma}} _ {r r} & \boldsymbol {0} \\ \boldsymbol {0} & \boldsymbol {0} \end{array} \right]\right) \tag {106}
$$