SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \frac {1}{| C _ {a} ^ {N} |} \sum_ {i \in C _ {a} ^ {N}} \left(\psi \left(s _ {i} ^ {t} + \mu_ {a} ^ {t} x _ {i} ^ {\star}\right) - \psi \left(x _ {i} ^ {t}\right)\right) \right| \\ \leq \frac {2 L}{C _ {a} ^ {N}} (\sqrt {C _ {a} ^ {N}} \| \boldsymbol {s} _ {a} ^ {t} + \mu_ {a} ^ {t} \boldsymbol {x} _ {a} ^ {\star} - \boldsymbol {x} _ {a} ^ {t} \| _ {2} + \| \boldsymbol {s} _ {a} ^ {t} + \mu_ {a} ^ {t} \boldsymbol {x} _ {a} ^ {\star} \| _ {2} \| \boldsymbol {s} _ {a} ^ {t} + \mu_ {a} ^ {t} \boldsymbol {x} _ {a} ^ {\star} - \boldsymbol {x} _ {a} ^ {t} \| _ {2} + \| \boldsymbol {s} _ {a} ^ {t} + \mu_ {a} ^ {t} \boldsymbol {x} _ {a} ^ {\ast} - \boldsymbol {x} _ {a} ^ {t} \| _ {2} ^ {2}) \\ \end{array}
$$
where  $\pmb{s}_a^t = (s_i^t)_{i\in C_a^N}\in \mathbb{R}^{|C_a^N|}$ ,  $\pmb{x}_a^t = (x_i^t)_{i\in C_a^N}\in \mathbb{R}^{|C_a^N|}$ . Therefore, to prove (47) it suffices to prove that for all  $t\geq 0$
$$
\lim  _ {N \rightarrow \infty} \frac {1}{\left| C _ {a} ^ {N} \right|} \| \boldsymbol {s} _ {a} ^ {t} + \mu_ {a} ^ {t} \boldsymbol {x} _ {a} ^ {\star} - \boldsymbol {x} _ {a} ^ {t} \| _ {2} ^ {2} \rightarrow 0 \tag {48}
$$
$$
\lim  _ {N \rightarrow \infty} \frac {1}{\left| C _ {a} ^ {N} \right|} \| \boldsymbol {s} _ {a} ^ {t} + \mu_ {a} ^ {t} \boldsymbol {x} _ {a} ^ {\star} \| _ {2} ^ {2} \rightarrow 0 \tag {49}
$$
Clearly, if we initialize  $\boldsymbol{x}^0$ ,  $s^0$  at zero then (48) and (49) are satisfied by our state evolution equations (5). Notice that (49) follows directly from (45) applied to the square function. We use here that we assumed that the second moment of  $x^{\star}$  is finite.
We now focus on proving (48) through strong induction. By definition of the iterates (43) and (46),
$$
\begin{array}{l} \left(\boldsymbol {s} _ {a} ^ {t} + \mu_ {a} ^ {t} \boldsymbol {x} _ {a} ^ {\star} - \boldsymbol {x} _ {a} ^ {t}\right) \\ = \left[ \left(\frac {1}{\sqrt {N} \sqrt {\boldsymbol {\Delta}}} \odot \boldsymbol {A}\right) f _ {t - 1} \left(\boldsymbol {s} ^ {t - 1} + \boldsymbol {\mu} ^ {t - 1} \odot \boldsymbol {x} ^ {\star}\right) - \left(\frac {1}{\sqrt {N} \sqrt {\boldsymbol {\Delta}}} \odot \boldsymbol {A}\right) f _ {t - 1} (\boldsymbol {x} ^ {t - 1}) \right. \\ + \mu^ {t} \odot \boldsymbol {x} ^ {\star} - \left(\frac {1}{N \boldsymbol {\Delta}} \odot \boldsymbol {x} ^ {\star} (\boldsymbol {x} ^ {\star}) ^ {\boldsymbol {T}}\right) f _ {t - 1} (\boldsymbol {x} ^ {t - 1}) \\ \left. + \mathbf {b} _ {t - 1} ^ {x} \odot f _ {t - 2} \left(\boldsymbol {x} ^ {t - 2}\right) - \mathbf {b} _ {t - 1} ^ {s} \odot f _ {t - 2} \left(\boldsymbol {s} ^ {t - 2} + \boldsymbol {\mu} ^ {\boldsymbol {t} - 2} \odot \boldsymbol {x} ^ {\star}\right) \right] _ {i \in C _ {a} ^ {N}} \\ \end{array}
$$
where  $\left[\cdot\right]_i$  corresponds to the  $i$ th row of a vector and  $\mathbf{b}_{t-1}^x$  and  $\mathbf{b}_t^s$  are the Onsager terms defined in (14) with respect to  $\boldsymbol{x}^{t-1}$  and  $\boldsymbol{s}^{t-1}$  respectively. The Cauchy-Schwarz inequality and Jensen's inequality imply that there exists some universal constant  $C$  such that
$$
\begin{array}{l} \frac {1}{| C _ {a} ^ {N} |} \| \boldsymbol {s} _ {a} ^ {t} + \mu_ {a} ^ {t} \boldsymbol {x} _ {a} ^ {\star} - \boldsymbol {x} _ {a} ^ {t} \| _ {2} ^ {2} \\ \leq \frac {C}{\left| C _ {a} ^ {N} \right|} \sum_ {i \in C _ {a} ^ {N}} \frac {1}{N} \left\| \left[ \frac {1}{\sqrt {\boldsymbol {\Delta}}} \odot \boldsymbol {A} \right] _ {i} \right\| _ {2} ^ {2} \| [ f _ {t - 1} (\boldsymbol {s} ^ {t - 1} + \boldsymbol {\mu} ^ {t - 1} \odot \boldsymbol {x} ^ {\star}) - f _ {t - 1} (\boldsymbol {x} ^ {t - 1}) ] _ {i} \| _ {2} ^ {2} \\ + \frac {C}{\left| C _ {a} ^ {N} \right|} \sum_ {i \in C _ {a} ^ {N}} \left(\mu_ {a} ^ {t} - \left[ \frac {1}{N \boldsymbol {\Delta}} \left(f _ {t - 1} (\boldsymbol {x} ^ {t - 1}) \odot \boldsymbol {x} ^ {\star}\right) \right] _ {i}\right) ^ {2} \left(x _ {i} ^ {\star}\right) ^ {2} \\ + \frac {C}{\left| C _ {a} ^ {N} \right|} \sum_ {i \in C _ {a} ^ {N}} \left(\left[ b _ {t - 1} ^ {x} \right] _ {i} - \left[ b _ {t - 1} ^ {s} \right] _ {i}\right) ^ {2} \left[ f _ {t - 2} \left(\boldsymbol {s} ^ {t - 2} + \boldsymbol {\mu} ^ {\boldsymbol {t} - \boldsymbol {2}} \odot \boldsymbol {x} ^ {\star}\right) \right] _ {i} ^ {2} \\ + \frac {C}{\left| C _ {a} ^ {N} \right|} \sum_ {i \in C _ {a} ^ {N}} \left[ \mathbf {b} _ {t - 1} ^ {x} \right] _ {i} ^ {2} \left(\left[ f _ {t - 2} (\boldsymbol {x} ^ {t - 2}) \right] _ {i} - \left[ f _ {t - 2} (\boldsymbol {s} ^ {t - 2} + \boldsymbol {\mu} ^ {t - 2} \odot \boldsymbol {x} ^ {\star}) \right] _ {i}\right) ^ {2} \\ \end{array}
$$
We now control each term separately.
1. To control the first term, notice that the matrix  $\frac{1}{N}\left[\frac{1}{\sqrt{\Delta}}\odot A\right]$  has iid entries within blocks and the sizes of the blocks diverge with the dimension, so we can control the sums of the squares of within each block using standard operator norm bounds Anderson et al. [2010]. The first term vanishes in the limit because  $f$  is pseudo-Lipschitz so we can apply the induction hypothesis bound which controls (48) at time  $t - 1$ .
2. To control the second term, notice that for  $i \in C_a^N$  by Lemma 2.4 applied to the pseudo-Lipschitz function  $yf_{t - 1}(x)$  that
$$
\left[ \frac {1}{N \boldsymbol {\Delta}} \left(f _ {t - 1} \left(\boldsymbol {x} ^ {t - 1}\right) \odot \boldsymbol {x} ^ {\star}\right)\right] _ {i} \rightarrow \mu_ {a}
$$
almost surely. This implies that the average of such terms vanishes since we assumed that the second moment  $\mathbb{E}[x_0^\star ]^2$  is finite.
3. To control the third and fourth terms, we can expand the definition of the Onsager terms and use the assumption that  $f'$  is pseudo-Lipschitz and almost surely bounded. Both terms vanish because our strong induction hypothesis gives us control of (48) at time  $t - 2$ .
Since all terms vanish in the limit, we have proven (48) for all  $a \in [q]$ , which finishes the proof of statement (47) and the proof of Theorem 1.2.
B Comparison with a naive PCA spectral method
In this appendix, we wish to show how the spectral method we propose differs, in practice, from a naive PCA. We provide an example of the spectrums of  $\mathbf{Y}$  and  $\tilde{\mathbf{Y}}$  before and after the transition at
Figure 3: Illustration of the spectrum of  $Y \in \mathbb{R}^{2500 \times 2500}$  evaluated at noise profiles with snr  $\lambda(\Delta) = 0.7$  (left, before the transition) and on the left and 1.8 on the right (after the transition). There is no outlying eigenvalue in contrast to the transformed matrix: the transition for a naive spectral method is sub-optimal.
Figure 4: Illustration of the spectrum of  $\tilde{Y} \in \mathbb{R}^{2500 \times 2500}$  evaluated at noise profiles with snr  $\lambda(\Delta) = 0.7$  (left, before the transition) and on the left and 1.8 on the right (after the transition), with the outlying eigenvector correlated with the spike arises at eigenvalue one. This is at variance with the results of the naive method in Fig.3
$\mathrm{SNR}(\Delta) = 1$ . In Figure 3 there is no clear separation of the extremal eigenvalue of  $\mathbf{Y}$  from the bulk around this transition. This is in contrast to Figure 4 where there is an extremal eigenvalue of  $\tilde{\mathbf{Y}}$  appearing at value one.