SlowGuess/NeurIPS_2023_OCR huggingface dataset:
if  $\pmb{x}^{\mu}$  belongs to the  $k$ -th cluster. By definition of the Gaussian mixture model, this is just a Gaussian variable:  $\pmb{\eta}^{\mu} \sim \mathcal{N}(0, \pmb{\Sigma}_k)$ , and the average over the data set  $\{x^{\mu}\}$  reduces to averaging over the cluster index  $k$  and  $\{\pmb{\eta}^{\mu}\}$ . For notational brevity, we shall name in this appendix  $\xi^{\mu}$  what is called  $\sqrt{\Delta}\xi^{\mu}$  in the main text (i.e., absorbing the variance in the definition of  $\xi^{\mu}$ ). The replicated partition function then reads
$$
\begin{array}{l} \mathbb {E} _ {\mathcal {D}} Z ^ {s} = \int \prod_ {a = 1} ^ {s} d \boldsymbol {w} _ {a} d b _ {a} e ^ {- \beta \sum_ {a = 1} ^ {s} g (\boldsymbol {w} ^ {a})} \\ \times \prod_ {\mu = 1} ^ {n} \sum_ {k = 1} ^ {K} \rho_ {k} \mathbb {E} _ {\boldsymbol {\eta}, \boldsymbol {\xi}} e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left\| \boldsymbol {\mu} _ {k} + \boldsymbol {\eta} - \left[ b _ {a} \times \left(\sqrt {1 - \Delta} \left(\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}\right) + \boldsymbol {\xi}\right) + \frac {\boldsymbol {w} _ {a} ^ {\top}}{\sqrt {d}} \sigma \left(\frac {\boldsymbol {w} _ {a} \left(\sqrt {1 - \Delta} \left(\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}\right) + \boldsymbol {\xi}\right)}{\sqrt {d}}\right) \right] \right\| ^ {2}}. \tag {21} \\ \end{array}
$$
One can expand the exponent  $(\star)$  as
$$
\begin{array}{l} e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left[ \mathrm {T r} \left[ \frac {\pmb {w} _ {a} \pmb {w} _ {a} ^ {\top}}{d} \sigma \left(\frac {\pmb {w} _ {a} (\sqrt {1 - \Delta} (\pmb {\mu} _ {k} + \pmb {\eta}) + \pmb {\xi})}{\sqrt {d}}\right) ^ {\otimes 2} \right] - 2 \sigma \left(\frac {\pmb {w} _ {a} (\sqrt {1 - \Delta} (\pmb {\mu} _ {k} + \pmb {\eta}) + \pmb {\xi})}{\sqrt {d}}\right) ^ {\top} \frac {\pmb {w} _ {a} ((1 - b _ {a} \sqrt {1 - \Delta}) (\pmb {\mu} _ {k} + \pmb {\eta}) - b _ {a} \pmb {\xi})}{\sqrt {d}} \right]} \\ \times e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left[ (1 - \sqrt {1 - \Delta b _ {a}}) ^ {2} \left(\| \boldsymbol {\mu} _ {k} \| ^ {2} + \| \boldsymbol {\eta} \| ^ {2} + 2 \boldsymbol {\mu} _ {k} ^ {\top} \boldsymbol {\eta}\right) + b _ {a} ^ {2} \| \boldsymbol {\xi} \| ^ {2} + 2 b _ {a} (\sqrt {1 - \Delta b _ {a}} - 1) \left(\boldsymbol {\mu} _ {k} ^ {\top} \boldsymbol {\xi} + \boldsymbol {\eta} ^ {\top} \boldsymbol {\xi}\right) \right]}. \tag {22} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {s, \boldsymbol {\eta}, \boldsymbol {\xi}} (\star) = \sum_ {k = 1} ^ {K} \rho_ {k} e ^ {- \frac {\beta}{2} \| \boldsymbol {\mu} _ {k} \| ^ {2}} \sum_ {a = 1} ^ {s} (\sqrt {1 - \Delta} b _ {a} - 1) ^ {2} \int \frac {d \boldsymbol {\eta} d \boldsymbol {\xi}}{(2 \pi \sqrt {\Delta}) ^ {d} \sqrt {\det  \boldsymbol {\Sigma} _ {k}}} \\ \underbrace {e ^ {- \frac {1}{2} \boldsymbol {\eta} ^ {\top} \left(\boldsymbol {\Sigma} _ {k} ^ {- 1} + \beta \sum_ {a = 1} ^ {s} (\sqrt {1 - \Delta b _ {a}} - 1) ^ {2} \mathbb {I} _ {d}\right) \boldsymbol {\eta} - \frac {1}{2} \| \boldsymbol {\xi} \| ^ {2} \left(\frac {1}{\Delta} + \beta \sum_ {a = 1} ^ {s} b _ {a} ^ {2}\right) - \beta \sum_ {a = 1} ^ {s} \left((1 - \sqrt {1 - \Delta b _ {a}}) ^ {2} \boldsymbol {\mu} _ {k} ^ {\top} \boldsymbol {\eta} + b _ {a} (\sqrt {1 - \Delta b _ {a}} - 1) \boldsymbol {\mu} _ {k} ^ {\top} \boldsymbol {\xi} + b _ {a} (\sqrt {1 - \Delta b _ {a}} - 1) \boldsymbol {\eta} ^ {\top} \boldsymbol {\xi}\right)} _ {\text {P e f f .}} \\ \times e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left[ \operatorname {T r} \left[ \frac {w _ {a} w _ {a} ^ {\top}}{d} \sigma \left(\frac {w _ {a} (\sqrt {1 - \Delta} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}) + \boldsymbol {\xi})}{\sqrt {d}}\right) ^ {\otimes 2} \right] - 2 \sigma \left(\frac {w _ {a} (\sqrt {1 - \Delta} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}) + \boldsymbol {\xi})}{\sqrt {d}}\right) ^ {\top} \frac {w _ {a} ((1 - b _ {a} \sqrt {1 - \Delta}) (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}) - b _ {a} \boldsymbol {\xi})}{\sqrt {d}} \right]}. \tag {23} \\ \end{array}
$$
The effective prior over the noises  $\eta, \xi$  is Gaussian with means  $\mu_{\xi}, \mu_{\eta}$  and covariance  $C$ :
$$
P _ {\text {e f f .}} (\boldsymbol {\eta}, \boldsymbol {\xi}) = \mathcal {N} \left(\boldsymbol {\eta}, \boldsymbol {\xi}; \boldsymbol {\mu} _ {\eta}, \boldsymbol {\mu} _ {\xi}, \left[ \begin{array}{l l} C _ {\eta \eta} ^ {- 1} & C _ {\xi \eta} ^ {- 1} \mathbb {I} _ {d} \\ C _ {\xi \eta} ^ {- 1} \mathbb {I} _ {d} & C _ {\xi \xi} ^ {- 1} \mathbb {I} _ {d} \end{array} \right] ^ {- 1}\right). \tag {24}
$$
Identifying the covariance components leads to
$$
C _ {\xi \xi} ^ {- 1} = \frac {1}{\Delta} + \beta \sum_ {a = 1} ^ {s} b _ {a} ^ {2}, \quad C _ {\eta \eta} ^ {- 1} = \Sigma_ {k} ^ {- 1} + \beta \sum_ {a = 1} ^ {s} (\sqrt {1 - \Delta} b _ {a} - 1) ^ {2} \mathbb {I} _ {d}, \quad C _ {\xi \eta} ^ {- 1} = \beta \sum_ {a = 1} ^ {s} b _ {a} (\sqrt {1 - \Delta} b _ {a} - 1). \tag {25}
$$
Note that all the sums indexed by  $a$  introduce terms of  $\mathcal{O}(s)$ , which can safely be neglected. Identifying the means leads to the following system of equations:
$$
\left\{ \begin{array}{l} C _ {\xi \xi} ^ {- 1} \boldsymbol {\mu} _ {\xi} + C _ {\xi \eta} ^ {- 1} \boldsymbol {\mu} _ {\eta} = \beta \sum_ {a = 1} ^ {s} b _ {a} (1 - \sqrt {1 - \Delta} b _ {a}) \boldsymbol {\mu} _ {k} \\ C _ {\eta \eta} ^ {- 1} \boldsymbol {\mu} _ {\eta} + C _ {\xi \eta} ^ {- 1} \boldsymbol {\mu} _ {\xi} = - \beta \sum_ {a = 1} ^ {s} (1 - \sqrt {1 - \Delta} b _ {a}) ^ {2} \boldsymbol {\mu} _ {k} \end{array} , \right. \tag {26}
$$
which is solved as
$$
\begin{array}{l} \boldsymbol {\mu} _ {\xi} = \underbrace {\beta \left(\boldsymbol {C} _ {\eta \eta} ^ {- 1} C _ {\xi \xi} ^ {- 1} - (C _ {\xi \eta} ^ {- 1}) ^ {2} \mathbb {I} _ {d}\right) \left(\sum_ {a = 1} ^ {s} \left(\boldsymbol {C} _ {\eta \eta} ^ {- 1} b _ {a} (1 - \sqrt {1 - \Delta} b _ {a}) + C _ {\xi \eta} ^ {- 1} (1 - \sqrt {1 - \Delta} b _ {a}) ^ {2} \mathbb {I} _ {d}\right)\right)} _ {\equiv g _ {\xi} ^ {\mu}} \boldsymbol {\mu} _ {k}, \\ \boldsymbol {\mu} _ {\eta} = \underbrace {- \beta \left(\boldsymbol {C} _ {\eta \eta} ^ {- 1} C _ {\xi \xi} ^ {- 1} - \left(C _ {\xi \eta} ^ {- 1}\right) ^ {2} \mathbb {I} _ {d}\right) \left(\sum_ {a = 1} ^ {s} \left(C _ {\xi \xi} ^ {- 1} \left(1 - \sqrt {1 - \Delta} b _ {a}\right) ^ {2} + C _ {\xi \eta} ^ {- 1} b _ {a} \left(1 - \sqrt {1 - \Delta} b _ {a}\right) \mathbb {I} _ {d}\right)\right)} _ {\equiv g _ {\eta} ^ {\mu}} \boldsymbol {\mu} _ {k}. \tag {27} \\ \end{array}
$$
Again, observe that  $g_{\eta}^{\mu}, g_{\xi}^{\mu} = \mathcal{O}(s)$  and will be safely neglected later in the derivation. Therefore
$$
\begin{array}{l} \mathbb {E} _ {s, \boldsymbol {\eta}, \xi} (\star) = \sum_ {k = 1} ^ {K} \rho_ {k} \frac {e ^ {- \frac {\beta}{2} \| \boldsymbol {\mu} _ {k} \| ^ {2}} \sum_ {a = 1} ^ {s} (\sqrt {1 - \Delta b _ {a} - 1}) ^ {2} + \frac {1}{2} C _ {\xi \xi} ^ {- 1} \| \boldsymbol {\mu} _ {\xi} \| ^ {2} + \frac {1}{2} \boldsymbol {\mu} _ {\eta} ^ {\top} C _ {\eta \eta} ^ {- 1} \boldsymbol {\mu} _ {\eta} + C _ {\xi \eta} ^ {- 1} \boldsymbol {\mu} _ {\xi} ^ {\top} \boldsymbol {\mu} _ {\eta}}{\sqrt {\det \boldsymbol {\Sigma} _ {k}} \Delta^ {\frac {d}{2}} \det \big (C _ {\eta \eta} ^ {- 1} C _ {\xi \xi} ^ {- 1} - (C _ {\xi \eta} ^ {- 1}) ^ {2} \mathbb {I} _ {d} \big) ^ {\frac {1}{2}}} \\ \times \underbrace {\left\langle e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left[ \operatorname {T r} \left[ \frac {w _ {a} w _ {a} ^ {\top}}{d} \sigma \left(\frac {w _ {a} (\sqrt {1 - \Delta} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}) + \boldsymbol {\xi})}{\sqrt {d}}\right) ^ {\otimes 2} \right] - 2 \sigma \left(\frac {w _ {a} (\sqrt {1 - \Delta} (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}) + \boldsymbol {\xi})}{\sqrt {d}}\right) ^ {\top} \frac {w _ {a} ((1 - b _ {a} \sqrt {1 - \Delta}) (\boldsymbol {\mu} _ {k} + \boldsymbol {\eta}) - b _ {a} \boldsymbol {\xi})}{\sqrt {d}} \right]} _ {(a)} \right\rangle_ {P _ {\text {e f f .}} (\boldsymbol {\eta}, \boldsymbol {\xi})}. \tag {28} \\ \end{array}
$$
We are now in a position to introduce the local fields:
$$
\lambda_ {a} ^ {\xi} = \frac {\boldsymbol {w} _ {a} (\boldsymbol {\xi} - \boldsymbol {\mu} _ {\xi})}{\sqrt {d}}, \quad \lambda_ {a} ^ {\eta} = \frac {\boldsymbol {w} _ {a} (\boldsymbol {\eta} - \boldsymbol {\mu} _ {\eta})}{\sqrt {d}}, \quad h _ {a} ^ {k} = \frac {\boldsymbol {w} _ {a} \boldsymbol {\mu} _ {k}}{\sqrt {d}}, \tag {29}
$$
$$
\left\langle \lambda_ {a} ^ {\eta} \lambda_ {b} ^ {\eta} \right\rangle \approx \frac {\boldsymbol {w} _ {a} \boldsymbol {\Sigma} _ {k} \boldsymbol {w} _ {b} ^ {\top}}{d}, \quad \left\langle \lambda_ {a} ^ {\xi} \lambda_ {b} ^ {\eta} \right\rangle = C _ {\xi \eta} \frac {\boldsymbol {w} _ {a} \boldsymbol {w} _ {b} ^ {\top}}{d} \approx 0, \quad \left\langle \lambda_ {a} ^ {\xi} \lambda_ {b} ^ {\xi} \right\rangle \approx \Delta \frac {\boldsymbol {w} _ {a} \boldsymbol {w} _ {b} ^ {\top}}{d}. \tag {30}
$$
We used the leading order of the covariances  $C_{\xi \xi, \eta \eta, \eta \xi}$ . One therefore has to introduce the summary statistics:
$$
Q _ {a b} = \frac {\boldsymbol {w} _ {a} \boldsymbol {w} _ {b} ^ {\top}}{d} \in \mathbb {R} ^ {p \times p}, \quad S _ {a b} ^ {k} = \frac {\boldsymbol {w} _ {a} \boldsymbol {\Sigma} _ {k} \boldsymbol {w} _ {b} ^ {\top}}{d} \in \mathbb {R} ^ {p \times p}, \quad m _ {a} ^ {k} = \frac {\boldsymbol {w} _ {a} \boldsymbol {\mu} _ {k}}{\sqrt {d}} \in \mathbb {R} ^ {p}. \tag {31}
$$
Note that the local fields  $\lambda_{a}^{\eta ,\xi}$  thus follow a Gaussian distribution:
$$
\left(\lambda_ {a} ^ {\xi}, \lambda_ {a} ^ {\eta}\right) _ {a = 1} ^ {s} \sim \mathcal {N} \left(0, \underbrace {\left[ \begin{array}{c c} S ^ {k} & 0 \\ 0 & \Delta Q \end{array} \right]} _ {\equiv \Omega_ {k}}\right). \tag {32}
$$
Going back to the computation,  $(a)$  can be rewritten as
$$
\begin{array}{l} \left\langle e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \mathrm {T r} \left[ Q _ {a a} \sigma \left(m _ {a} ^ {k} (\sqrt {1 - \Delta} (1 + g _ {\eta} ^ {\mu}) + g _ {\xi} ^ {\mu}) + \sqrt {1 - \Delta} \lambda_ {a} ^ {\eta} + \lambda_ {a} ^ {\xi}\right) ^ {\otimes 2} \right]} \right. \\ \times e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} \left[ - 2 \sigma \left(m _ {a} ^ {k} (\sqrt {1 - \Delta} (1 + g _ {\eta} ^ {\mu}) + g _ {\xi} ^ {\mu}) + \sqrt {1 - \Delta} \lambda_ {a} ^ {\eta} + \lambda_ {a} ^ {\xi}\right) ^ {\top} \left(m _ {a} ^ {k} ((1 - \sqrt {1 - \Delta} b _ {a}) (1 + g _ {\eta} ^ {\mu}) - b _ {a} g _ {\xi} ^ {\mu}) + (1 - \sqrt {1 - \Delta} b _ {a}) \lambda_ {a} ^ {\eta} - b _ {a} \lambda_ {a} ^ {\xi}\right) \right]} \rangle_ {\{\lambda_ {a} ^ {\xi}, \lambda_ {a} ^ {\eta} \} _ {a = 1} ^ {s}}. \tag {33} \\ \end{array}
$$
A.3 Reformulating as a saddle-point problem
Introducing Dirac functions enforcing the definitions of  $Q_{ab}, m_a$  brings the replicated function in the following form:
$$
\begin{array}{l} \mathbb {E} Z ^ {s} = \int \prod_ {a = 1} ^ {s} d b _ {a} \prod_ {a, b} d Q _ {a b} d \hat {Q} _ {a b} \prod_ {k = 1} ^ {K} \prod_ {a = 1} ^ {s} d m _ {a} d \hat {m} _ {a} \prod_ {k = 1} ^ {K} \prod_ {a, b} d S _ {a b} ^ {k} d \hat {S} _ {a b} ^ {k} \\ \underbrace {e ^ {- d \sum_ {a \leq b} Q _ {a b} \hat {Q} _ {a b} - d \sum_ {k = 1} ^ {K} \sum_ {a \leq b} S _ {a b} ^ {k} \hat {S} _ {a b} ^ {k} - \sum_ {k = 1} ^ {K} \sum_ {a} d m _ {a} ^ {k} \hat {m} _ {a} ^ {k}}} _ {e ^ {\beta s d} \Psi_ {t}} \\ \underbrace {\int \prod_ {a = 1} ^ {s} d w _ {a} e ^ {- \beta \sum_ {a} g \left(\boldsymbol {w} ^ {a}\right) + \sum_ {a \leq b} \hat {Q} _ {a b} \boldsymbol {w} _ {a} \boldsymbol {w} _ {b} ^ {\top} + \sum_ {k = 1} ^ {K} \sum_ {a \leq b} \hat {S} _ {a b} ^ {k} \boldsymbol {w} _ {a} \boldsymbol {\Sigma} _ {k} \boldsymbol {w} _ {b} ^ {\top} + \sum_ {k = 1} ^ {K} \sum_ {a} \hat {m} _ {a} ^ {k} \sqrt {d} \boldsymbol {w} _ {a} \boldsymbol {\mu} _ {k}} _ {e ^ {\beta s d} \Psi_ {w}} \\ \underbrace {\left[ \sum_ {k = 1} ^ {K} \rho_ {k} \frac {e ^ {\mathcal {O} (s ^ {2}) e ^ {- \frac {\beta}{2} \| \mu_ {k} \| ^ {2}} \sum_ {a = 1} ^ {s} (\sqrt {1 - \Delta b _ {a}} - 1) ^ {2}}}{\sqrt {\det \boldsymbol {\Sigma} _ {k}} \Delta^ {\frac {d}{2}} \det \left(\boldsymbol {C} _ {\eta \eta} ^ {- 1} C _ {\xi \xi} ^ {- 1} - \mathcal {O} (s ^ {2})\right) ^ {\frac {1}{2}}} (a) \right] ^ {\alpha d}} _ {e ^ {\alpha s d ^ {2} \Psi_ {\mathrm {q u a d .}} + \beta s d \Psi_ {y}}}. \tag {34} \\ \end{array}
$$
We introduced the trace, entropic and energetic potentials  $\Psi_t, \Psi_w, \Psi_y$ . Since all the integrands scale exponentially (or faster) with  $d$ , this integral can be computed using a saddle-point method. To proceed further, note that the energetic term encompasses two types of terms, scaling like  $sd^2$  and  $sd$ . More precisely,
$$
\begin{array}{l} \left[ \sum_ {k = 1} ^ {K} \rho_ {k} \frac {e ^ {- \frac {\beta}{2} \| \boldsymbol {\mu} _ {k} \| ^ {2} \sum_ {a = 1} ^ {s} (\sqrt {1 - \Delta} b _ {a} - 1) ^ {2}}}{\sqrt {\det \boldsymbol {\Sigma} _ {k}} \Delta^ {\frac {d}{2}} \det \left(\boldsymbol {C} _ {\eta \eta} ^ {- 1} \boldsymbol {C} _ {\xi \xi} ^ {- 1}\right) ^ {\frac {1}{2}}} (a) \right] ^ {\alpha d} \\ = \left[ 1 - s d \sum_ {k = 1} ^ {K} \rho_ {k} \frac {1}{s d} \ln \left(\sqrt {\det  \boldsymbol {\Sigma} _ {k}} \Delta^ {\frac {d}{2}} \det  \left(\boldsymbol {C} _ {\eta \eta} ^ {- 1} C _ {\xi \xi} ^ {- 1}\right) ^ {\frac {1}{2}}\right) + s \sum_ {k = 1} ^ {K} \rho_ {k} \frac {1}{s} \ln (a) - \frac {\beta}{2} \sum_ {k = 1} ^ {K} \rho_ {k} \| \boldsymbol {\mu} _ {k} \| ^ {2} \sum_ {a = 1} ^ {s} (\sqrt {1 - \Delta} b _ {a} - 1) ^ {2} \right] ^ {\alpha d} \\ \approx e ^ {- s \alpha d ^ {2} \sum_ {k = 1} ^ {K} \rho_ {k} \frac {1}{s d} \ln \left(\sqrt {\det \boldsymbol {\Sigma} _ {k}} \Delta^ {\frac {d}{2}} \det \left(\boldsymbol {C} _ {\eta \eta} ^ {- 1} C _ {\xi \xi} ^ {- 1}\right) ^ {\frac {1}{2}}\right) + s \alpha d \sum_ {k = 1} ^ {K} \rho_ {k} \frac {1}{s} \ln (\alpha)}. \tag {35} \\ \end{array}
$$
Note that the saddle-point method involves an intricate extremization over  $s \times s$  matrices  $Q, S_{k}$ . To make further progress, we assume the extremum is realized at the Replica Symmetric (RS) fixed point
$$
\forall a, \qquad b _ {a} = b
$$
$$
\forall a, \qquad m _ {a} ^ {k} = m _ {k}, \qquad \hat {m} _ {a} ^ {k} = \hat {m} _ {k}
$$
$$
\forall a, b, \qquad Q _ {a b} = (r - q) \delta_ {a b} + q, \qquad \hat {Q} _ {a b} = - \left(\frac {\hat {r}}{2} + \hat {q}\right) \delta_ {a b} + \hat {q}
$$
$$
\forall a, b, \quad S _ {a b} ^ {k} = \left(r _ {k} - q _ {k}\right) \delta_ {a b} + q _ {k}, \quad \hat {S} _ {a b} ^ {k} = - \left(\frac {\hat {r} _ {k}}{2} + \hat {q} _ {k}\right) \delta_ {a b} + \hat {q} _ {k} \tag {36}
$$
This is a standard assumption known as the RS ansatz [21, 22, 23, 24].
Quadratic potential The quadratic potential  $\Psi_{\mathrm{quad}}$ , which correspond to the leading order term in  $d$  in the exponent, needs to be extremized first in the framework of our saddle-point analysis. Its expression can be simplified as
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \rho_ {k} \ln \left(\sqrt {\det  \boldsymbol {\Sigma} _ {k}} \Delta^ {\frac {d}{2}} \det  \left(\boldsymbol {C} _ {\eta \eta} ^ {- 1} C _ {\xi \xi} ^ {- 1}\right) ^ {\frac {1}{2}}\right) = \frac {1}{2} \sum_ {k = 1} ^ {K} \rho_ {k} \ln \det  \left[ (1 + \Delta \beta s b ^ {2}) \left(\mathbb {I} _ {d} + \boldsymbol {\Sigma} _ {k} \beta s (\sqrt {1 - \Delta} b - 1) ^ {2}\right) \right] \\ = \frac {\beta s}{2} \sum_ {k = 1} ^ {K} \rho_ {k} \operatorname {T r} \left[ \Delta b ^ {2} \mathbb {I} _ {d} + \Sigma_ {k} (\sqrt {1 - \Delta} b - 1) ^ {2} \right] \\ = \frac {\beta d s}{2} \left[ \Delta b ^ {2} + (\sqrt {1 - \Delta} b - 1) ^ {2} \frac {1}{d} \sum_ {k = 1} ^ {K} \rho_ {k} \operatorname {T r} \boldsymbol {\Sigma} _ {k} \right], \tag {37} \\ \end{array}
$$
which is extremized for
$$
b = \frac {\frac {1}{d} \left(\sum_ {k = 1} ^ {K} \rho_ {k} \operatorname {T r} \boldsymbol {\Sigma} _ {k}\right) \sqrt {1 - \Delta}}{\frac {1}{d} \left(\sum_ {k = 1} ^ {K} \rho_ {k} \operatorname {T r} \boldsymbol {\Sigma} _ {k}\right) (1 - \Delta) + \Delta}. \tag {38}
$$
This fixes the skip connection strength  $b$ .
Entropic potential We now turn to the entropic potential  $\Psi_w$ . It is convenient to introduce the variance order parameters
$$
\hat {V} \equiv \hat {r} + \hat {q}, \quad \hat {V} _ {k} \equiv \hat {r} _ {k} + \hat {q} _ {k}. \tag {39}
$$
The entropic potential can then be expressed as
$$
e ^ {\beta s d \Psi_ {w}}
$$
$$
\begin{array}{l} = \int \prod_ {a = 1} ^ {s} d w _ {a} e ^ {- \beta \sum_ {a} g (\boldsymbol {w} ^ {a}) - \frac {1}{2} \sum_ {a = 1} ^ {s} \operatorname {T r} \left[ \hat {V} \boldsymbol {w} _ {a} \boldsymbol {w} _ {a} ^ {\top} \right] + \frac {1}{2} \sum_ {a, b} \operatorname {T r} \left[ \hat {q} \boldsymbol {w} _ {a} \boldsymbol {w} _ {b} ^ {\top} \right] + \hat {m} \sum_ {k = 1} ^ {K} \sum_ {a = 1} ^ {s} \sqrt {d} \hat {m} _ {k} ^ {\top} \boldsymbol {w} _ {a} ^ {\top} \boldsymbol {\mu} _ {k}} \\ \times e ^ {- \frac {1}{2} \sum_ {k = 1} ^ {K} \sum_ {a = 1} ^ {s} \operatorname {T r} \left[ \hat {V} _ {k} \boldsymbol {w} _ {a} \boldsymbol {\Sigma} _ {k} \boldsymbol {w} _ {a} ^ {\top} \right] + \frac {1}{2} \sum_ {k = 1} ^ {K} \sum_ {a, b} \operatorname {T r} \left[ \hat {q} _ {k} \boldsymbol {w} _ {a} \boldsymbol {\Sigma} _ {k} \boldsymbol {w} _ {b} ^ {\top} \right]} \\ \end{array}
$$
$$
\begin{array}{l} = \int D \Xi_ {0} \prod_ {k = 1} ^ {K} D \Xi_ {k} \\ \left[ \int d w e ^ {- \beta g (\boldsymbol {w}) - \frac {1}{2} \operatorname {T r} \left[ \hat {V} \boldsymbol {w} \boldsymbol {w} ^ {\top} + \sum_ {k = 1} ^ {K} \hat {V} _ {k} \boldsymbol {w} \boldsymbol {\Sigma} _ {k} \boldsymbol {w} ^ {\top} \right] + \left(\sum_ {k = 1} ^ {K} \sqrt {d} \hat {m} _ {k} \boldsymbol {\mu} ^ {\top} + \sum_ {k = 1} ^ {K} \Xi_ {k} \odot (\hat {q} _ {k} \otimes \boldsymbol {\Sigma} _ {k}) ^ {\frac {1}{2}} + \Xi_ {0} \odot (\hat {q} \otimes \mathbb {I} _ {d}) ^ {\frac {1}{2}}\right) \odot \boldsymbol {w}} \right] ^ {s} \\ \end{array}
$$
$$
\begin{array}{l} = \int \underbrace{D\Xi_{0}\prod_{k = 1}^{K}D\Xi_{k}}_{\equiv D\Xi} \\ \left[ \int d w e ^ {- \beta g (\boldsymbol {w}) - \frac {1}{2} \boldsymbol {w} \odot \left[ \hat {V} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {V} _ {k} \otimes \boldsymbol {\Sigma} _ {k} \right] \odot \boldsymbol {w} + \left(\sum_ {k = 1} ^ {K} \sqrt {d} \hat {m} _ {k} \boldsymbol {\mu} ^ {\top} + \sum_ {k = 1} ^ {K} \boldsymbol {\Xi} _ {k} \odot (\hat {q} _ {k} \otimes \boldsymbol {\Sigma} _ {k}) ^ {\frac {1}{2}} + \boldsymbol {\Xi} _ {0} \odot (\hat {q} \otimes \mathbb {I} _ {d}) ^ {\frac {1}{2}}\right) \odot \boldsymbol {w}} \right] ^ {s}. \tag {40} \\ \end{array}
$$
$$
\beta \Psi_ {w} = \frac {1}{d} \int D \boldsymbol {\Xi} \ln \left[ \int d w e ^ {- \beta g (\boldsymbol {w}) - \frac {1}{2} \boldsymbol {w} \odot \left[ \hat {V} \otimes \mathbb {I} _ {d} + \sum_ {k = 1} ^ {K} \hat {V} _ {k} \otimes \boldsymbol {\Sigma} _ {k} \right] \odot \boldsymbol {w} + \left(\sum_ {k = 1} ^ {K} \sqrt {d} \hat {m} _ {k} \boldsymbol {\mu} ^ {\top} + \sum_ {k = 1} ^ {K} \boldsymbol {\Xi} _ {k} \odot (\hat {q} _ {k} \otimes \boldsymbol {\Sigma} _ {k}) ^ {\frac {1}{2}} + \boldsymbol {\Xi} _ {0} \odot (\hat {q} \otimes \mathbb {I} _ {d}) ^ {\frac {1}{2}}\right) \odot \boldsymbol {w}} \right]. \tag {41}
$$
For a matrix  $\Xi \in \mathbb{R}^{p\times d}$  and tensors  $A,B\in \mathbb{R}^{p\times d}\otimes \mathbb{R}^{p\times d}$ , we denoted  $(\Xi \odot A)_{kl} = \sum_{ij}\Xi^{ij}A_{ij,kl}$  and  $(A\odot B)_{ij,kl} = \sum_{rs}A_{ij,rs}B_{rs,kl}$ .
Energetic potential In order to compute the energetic potential  $\Psi_y$ , one must first compute the inverse of the covariance  $\Omega_k$ , given by
$$
\Omega_ {k} ^ {- 1} = \left[ \begin{array}{c c} S _ {k} ^ {- 1} & 0 \\ 0 & \frac {1}{\Delta} Q ^ {- 1} \end{array} \right]. \tag {42}
$$
It is straightforward to see that  $S_k^{-1}, Q^{-1}$  share the same block structure as  $S_k, Q$ , and we shall name for clarity
$$
\left(S _ {k} ^ {- 1}\right) _ {a b} = \left(\tilde {r} _ {k} - \tilde {q} _ {k}\right) \delta_ {a b} + \tilde {q} _ {k},
$$
$$
\left(Q ^ {- 1}\right) _ {a b} = (\tilde {r} - \tilde {q}) \delta_ {a b} + \tilde {q}. \tag {43}
$$
From the Sherman-Morisson lemma, and noting
$$
V _ {k} \equiv r _ {k} - q _ {k} \in \mathbb {R} ^ {p \times p}, \quad V \equiv r - q \in \mathbb {R} ^ {p \times p}, \tag {44}
$$
one reaches
$$
\left\{ \begin{array}{l} \tilde {r} = V ^ {- 1} - (V + s q) ^ {- 1} q V ^ {- 1} \\ \tilde {q} = - (V + s q) ^ {- 1} q V ^ {- 1} \end{array} , \right. \quad \left\{ \begin{array}{l} \tilde {r} _ {k} = V _ {k} ^ {- 1} - \left(V _ {k} + s q _ {k}\right) ^ {- 1} q _ {k} V _ {k} ^ {- 1} \\ \tilde {q} _ {k} = - \left(V _ {k} + s q _ {k}\right) ^ {- 1} q _ {k} V _ {k} ^ {- 1} \end{array} . \right. \tag {45}
$$
We will also need the expression of the determinants
$$
\ln \det  S _ {k} = s \det  V _ {k} + s \operatorname {T r} \left[ V _ {k} ^ {- 1} q _ {k} \right], \quad Q = s \det  V + s \operatorname {T r} \left[ V ^ {- 1} q \right]. \tag {46}
$$
One can finally proceed in analyzing the term  $(a)$ :
$$
\begin{array}{l} (a) = \int_ {\mathbb {R} ^ {p}} \frac {\prod_ {a = 1} ^ {s} d \lambda_ {a} ^ {\eta} d \lambda_ {a} ^ {\xi}}{(2 \pi) ^ {p s} \sqrt {\det Q} \det S _ {k} \sqrt {\Delta} ^ {p s}} \\ \times e ^ {- \frac {1}{2} \sum_ {a = 1} ^ {s} (\lambda_ {a} ^ {\eta}) ^ {\top} (\tilde {r} _ {k} - \tilde {q} _ {k}) \lambda_ {a} ^ {\eta} - \frac {1}{2} \sum_ {a, b} (\lambda_ {a} ^ {\eta}) ^ {\top} \tilde {q} _ {k} \lambda_ {b} - \frac {1}{2 \Delta} \sum_ {a = 1} ^ {s} (\lambda_ {a} ^ {\xi}) ^ {\top} (\tilde {r} - \tilde {q}) \lambda_ {a} ^ {\xi} - \frac {1}{2 \Delta} \sum_ {a, b} (\lambda_ {a} ^ {\xi}) ^ {\top} \tilde {q} \lambda_ {b}} e ^ {- \frac {\beta}{2} \sum_ {a = 1} ^ {s} (*)} \\ = \int_ {\mathbb {R} ^ {p}} \frac {D \xi D \eta}{(2 \pi) ^ {p s} \sqrt {\det Q \det S _ {k}} \sqrt {\Delta} ^ {p s}} \\ \times \left[ \int_ {\mathbb {R} ^ {p}} d \lambda_ {\eta} d \lambda_ {\xi} e ^ {- \frac {1}{2} \lambda_ {\eta} ^ {\top} V _ {k} ^ {- 1} \lambda_ {\eta} + \lambda_ {\eta} ^ {\top} V _ {k} ^ {- 1} q _ {k} ^ {\frac {1}{2}} \eta - \frac {1}{2 \Delta} \lambda_ {\xi} ^ {\top} V ^ {- 1} \lambda_ {\xi} + \frac {1}{\sqrt {\Delta}} \lambda_ {\xi} ^ {\top} V ^ {- 1} q ^ {\frac {1}{2}} \xi - \frac {\beta}{2} (*) \right] ^ {s} \\ = \int_ {\mathbb {R} ^ {p}} D \xi D \eta \left[ \int \mathcal {N} \left(\lambda_ {\xi}; \sqrt {\Delta} q ^ {\frac {1}{2}} \xi , \Delta V\right) \mathcal {N} \left(\lambda_ {\eta}; q _ {k} ^ {\frac {1}{2}} \xi , V _ {k}\right) e ^ {- \frac {\beta}{2} (*)} \right] ^ {s} \\ = 1 + s \int_ {\mathbb {R} ^ {p}} D \xi D \eta \ln \left[ \int \mathcal {N} \left(\lambda_ {\xi}; \sqrt {\Delta} q ^ {\frac {1}{2}} \xi , \Delta V\right) \mathcal {N} \left(\lambda_ {\eta}; q _ {k} ^ {\frac {1}{2}} \xi , V _ {k}\right) e ^ {- \frac {\beta}{2} (*)} \right] \tag {47} \\ \end{array}
$$
where we noted with capital  $D$  an integral over a  $p-$  dimensional Gaussian distribution  $\mathcal{N}(0,\mathbb{I}_p)$ . Therefore
$$
\beta \Psi_ {y} = \sum_ {k = 1} ^ {K} \rho_ {k} \int_ {\mathbb {R} ^ {p}} D \xi D \eta \ln \left[ \int \mathcal {N} \left(\lambda_ {\xi}; \sqrt {\Delta} q ^ {\frac {1}{2}} \xi , \Delta V\right) \mathcal {N} \left(\lambda_ {\eta}; q _ {k} ^ {\frac {1}{2}} \xi , V _ {k}\right) e ^ {- \frac {\beta}{2} (*)} \right]. \tag {48}
$$
A.4 zero-temperature limit
In this subsection, we take the zero temperature  $\beta \rightarrow \infty$  limit. Rescaling
$$
\frac {1}{\beta} V \leftarrow V, \quad \beta \hat {V} \leftarrow \hat {V}, \quad \beta^ {2} \hat {q} \leftarrow \hat {q},
$$
$$
\frac {1}{\beta} V _ {k} \leftarrow V _ {k}, \quad \beta \hat {V} _ {k} \leftarrow \hat {V} _ {k}, \quad \beta^ {2} \hat {q} _ {k} \leftarrow \hat {q} _ {k}, \quad \beta \hat {m} _ {k} \leftarrow \hat {m} _ {k}, \tag {49}
$$
one has that