SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Spherical Harmonic Representation of the Population Loss. We express the data distribution  $\nu$  as a mixture of uniform measures  $\nu = \int_0^\infty \tau_{r,d}\rho (dr)$ , where  $\tau_{r,d} = \mathrm{Unif}(r\mathcal{S}_{d - 1})$ . Let  $\tau_{d} = \tau_{1,d}$  and  $u_{d}\in \mathcal{P}([-1,1])$  be the projection of  $\tau_{d}$  onto one direction, with density given in close form by  $u_{d}(dt) = Z^{-1}(1 - t^{2})^{(d - 3) / 2}\mathbf{1}(|t|\leq 1)dt$ , where  $Z$  is a normalizing factor. Let  $\{P_{j,d}\}_{j\in \mathbb{N}}$  be the orthogonal basis of Gegenbauer polynomials of  $L_{u_d}^2 ([-1,1])$ , normalized such that  $P_{j,d}(1) = 1$  for all  $j,d$ . For each  $r > 0$ , consider
$$
l _ {r} (\theta) := \left\langle \phi_ {\theta}, \phi_ {\theta^ {*}} \right\rangle_ {\tau_ {r, d}} = \left\langle \phi_ {\theta} ^ {(r)}, \phi_ {\theta^ {*}} ^ {(r)} \right\rangle_ {\tau_ {d}},
$$
where we define  $\phi^{(r)}: [-1,1] \to \mathbb{R}$  such that  $\phi^{(r)}(t) := \phi(rt)$ . We write its decomposition in  $L_{u_d}^2([-1,1])$  as  $\phi^{(r)} = \sum_j \alpha_{j,r,d} P_{j,d}$ , with  $\alpha_{j,r,d} = \frac{\langle \phi^{(r)}, P_{j,d} \rangle}{\|P_{j,d}\|^2}$ . Let  $\Omega_d$  be the Lebesgue measure of  $S_{d-1}$ , and  $N(j,d) = \frac{2d+j-2}{d} \binom{d+j-3}{d-1}$  the so-called dimension on the spherical harmonics of degree  $j$  in dimension  $d$ . From the Hecke-Funk representation formula [Frye and Efthimiou, 2012, Lemma 4.23] and the chosen normalization  $P_{j,d}(1) = 1$ , we have  $\|P_{j,d}\|_2 = \left(\frac{\Omega_{d-1}}{\Omega_{d-2}N(j,d)}\right)^{1/2}$  [Frye and Efthimiou, 2012, Proposition 4.15] and obtain finally
$$
l _ {r} (\theta) = \sum_ {j} \bar {\alpha} _ {j, r, d} ^ {2} P _ {j, d} \left(\theta \cdot \theta^ {*}\right), \tag {13}
$$
where we defined for convenience  $\bar{\alpha}_{j,r,d} = \alpha_{j,r,d} / \sqrt{N(j,d)}$ . As a result, it follows that the overall loss writes as solely the correlaton  $m_{\theta} = \theta \cdot \dot{\theta}^{*}$  as
$$
L (\theta) = 2 \| \phi \| _ {L _ {\nu} ^ {2}} ^ {2} - 2 \sum_ {j} \beta_ {j, d} P _ {j, d} (m _ {\theta}) := \ell (m), \tag {14}
$$
where  $\beta_{j,d} = \int_0^\infty \bar{\alpha}_{j,r,d}^2\rho (dr)\geq 0$ . Unsurprisingly, we observe that, thanks to the spherical symmetry and analogous to the Gaussian case, the loss still admits  $m = \theta \cdot \theta^{*}$  as a summary statistics. Yet, it is represented in terms of Gegenbauer polynomials, rather than monomials as in the Gaussian case. The monomial representation is a consequence of the stability of the Gaussian measure, as seen by the fact that  $\langle \phi_{\theta},\phi_{\tilde{\theta}}\rangle_{\gamma_d} = \langle \phi ,\mathsf{A}_{\theta .\tilde{\theta}}\phi \rangle_{\gamma}$ , where  $(\mathsf{A}_mf)(t) = \mathbb{E}_{z\sim \gamma}[f(mt + \sqrt{1 - m^2} z)]$  has a semi-group structure (it is even known in fact as the Ornstein-Ulthenbeck semi-group).
Let  $\eta \in \mathcal{P}(\mathbb{R})$  be the marginal of  $\nu$  along any direction. The following proposition characterizes the coefficients  $\beta_{j,d}$  as integrals over the radial distribution  $\rho$  and projections of the link function  $\phi$ :
Proposition 4.1 (Loss representation). The  $\beta_{j,d}$  defined in (14) have the integral representation
$$
\beta_ {j, d} = \left\langle \phi , \mathcal {K} _ {j} \phi \right\rangle_ {L _ {\eta} ^ {2}}, \tag {15}
$$
where  $\mathcal{K}_j$  is a positive semi-definite integral operator of  $L_{\eta}^{2}$  that depends solely on  $\rho$  and  $\phi$ .
Note that a closed form expression of  $\kappa_{j}$  can be found in Appendix D.2. The above proposition is in fact the stepping stone to calculate properly the information exponent that plays a crucial role in the property LPG. This is given through the link between the spectrum  $\beta_{j,d}$  and the decomposition of  $\phi$  in the  $L_{\eta}^{2}$  orthogonal basis of polynomials, that we denote by  $\{q_j\}_{j}$ .
Proposition 4.2. Let  $s = \inf \{j; \beta_{j,d} > 0\}$  and  $\tilde{s} = \inf \{j; \langle \phi, q_j \rangle_\eta \neq 0\}$ . Then  $s \leq \tilde{s}$ .
Thus, the number of vanishing moments of  $\phi$  with respect to the data marginal  $\eta$  provides an upper bound on the 'effective' information exponent of the problem  $s$ , as we will see next.
Local Polynomial Growth. From (14), and as  $\nabla_{\theta}^{S}L(\theta) = \ell^{\prime}(m)(\theta^{*} - m\theta)$ , we directly obtain
$$
- \nabla_ {\theta} ^ {S} L (\theta) \cdot \theta^ {*} = - (1 - m ^ {2}) \ell^ {\prime} (m) = 2 (1 - m ^ {2}) \sum_ {j} \beta_ {j, d} P _ {j, d} ^ {\prime} (m), \tag {16}
$$
which is the quantity we want to understand to exhibit the property LPG in this case. Hence, we now turn into the question of obtaining sufficient guarantees on the coefficients  $(\beta_{j,d})_j$  that ensure
local polynomial growth. Since the typical scale of initialization for  $m$  is  $\Theta(1/\sqrt{d})$ , our goal is to characterize sufficient conditions of local polynomial growth with  $b = O(1/\sqrt{d})$ .
For that purpose, let us define two key quantities of Gegenbauer polynomials:
$$
v _ {j, d} := - \min  _ {t \in (0, 1)} P _ {j, d} (t), \quad \text {(s m a l l e s t v a l u e)} \tag {17}
$$
$$
z _ {j, d} := \arg \max  \left\{t \in (0, 1); P _ {j, d} (t) = 0 \right\}. \quad (\text {l a r g e s t}
$$
We have the following sufficient condition based on the spectrum  $(\beta_{j,d})_j$
Proposition 4.3 (Spectral characterization of LPG). Suppose there exist constants  $K, C > 0$  and  $s \in \mathbb{N}$  such that we both have  $\beta_{s,d} \geq C$  and  $\sum_{j > s} \beta_{j,d} j(j + d - 2)v_{j - 1,d + 2} \leq Kd^{(3 - s) / 2}$ . Then, taking  $s^*$  as the infimum of such  $s$ ,  $L$  has the property LPG  $(s^* - 1, z_{s^*,d})$ . In particular, whenever  $s^* \ll d$ , we have  $z_{s^*,d} \leq 2\sqrt{s^* / d}$ .
This proposition thus establishes that, modulo a mild regularity assumption expressed though the decay of the coefficients  $\beta_{j,d}$ , the spherically symmetric non-Gaussian setting has the same geometry as the Gaussian setting, for correlations slightly above the equator. Crucially, the required amount of correlation to 'feel' the local polynomial growth is a  $O(\sqrt{s})$  factor from the typical initialization, and can be thus obtained with probability  $\simeq e^{-s}$  over the initialization, according to Lemma 2.1, a lower bound which is independent of  $d$ .
The sufficient condition for  $\beta_{j,d}$  appearing in Proposition 4.3 involves the minimum values  $v_{j,d}$  of Gegenbauer polynomials  $P_{j,d}$ , as well as sums of the form  $\sum_{j} j^{2} \beta_{j,d}$ . In order to obtain a more user-friendly condition, we now provide an explicit control of  $v_{j,d}$ , and leverage mild regularity of  $\phi$  to control  $\sum_{j} j^{2} \beta_{j,d}$ . This motivates the following assumption on  $\phi$  and  $\nu$ :
Assumption 4.4. The link function  $\phi$  satisfies  $\phi \in L_{\eta}^{2}$  and  $\phi' \in L_{\eta'}^{4}$ , and the radial distribution  $\rho$  has finite fourth moment  $\mathbb{E}_{\rho}[r^4] < \infty$ .
Theorem 4.5 (LPG for symmetric distributions). Assume that  $\phi$  and  $\nu$  satisfy Assumption 4.4, and let  $s^* = \inf \{j; \langle \phi, q_j \rangle_\eta \neq 0\}$ . Then  $L$  has the property LPG( $s^* - 1, 2\sqrt{s^* / d}$ ).
The proof is provided in Appendix D.5. At a technical level, the main challenge in proving Theorem 4.5 is to achieve a uniform control of  $v_{j,d}$  in  $j$ , a result which may be of independent interest. We address it by combining state-of-the-art bounds on the roots of the Gegenbauer polynomials, allowing us to cover the regime where  $j$  is small or comparable to  $d$ , together with integral representations via the Cauchy integral formula, providing control in the regime of large  $j$ . On the other hand, we relate the sum  $\sum_{j} j^{2}\beta_{j,d}$  to a norm of  $\phi'$  using a Cauchy-Schwartz argument, where we leverage the fourth moments from Assumption 4.4.
Remark 4.6. Since we are in a setting where  $\phi$  is known, an alternative to the original recovery problem from Eq (2) is to consider a pure Gegenbauer 'student' link function of the form  $\tilde{\phi} = P_{s,d}$ , where  $s$  is the information exponent from Proposition 4.2. Indeed, the resulting population loss  $\tilde{L}(\theta) = \mathbb{E}[(\tilde{\phi}(x \cdot \theta) - \tilde{\phi}(x \cdot \theta^{*}))^{2}]$  satisfies the LPG property, as easily shown in Fact D.5.
For the sake of completeness, we describe more precisely two concrete case studies below.
Example 4.7 (Uniform Measure on the Sphere). When  $\nu = \text{Unif}(\sqrt{d}\mathcal{S}_{d-1})$ , we have  $\rho = \delta_{\sqrt{d}}$ , and therefore  $\beta_{j,d} = \bar{\alpha}_{j,\sqrt{d},d}^2$ . In that case, the orthogonal polynomial basis  $\{q_j(t)\}_j$  of  $L_{\eta}^{2}$  coincides with the rescaled Gegenbauer polynomials,  $q_j(t) = P_{j,d}(t / \sqrt{d})$ . Consider now a link function  $\phi$  with  $s - 1$  vanishing moments with respect to  $L_{\eta}^{2}$ , i.e., such that  $\bar{\alpha}_{j,d} = \langle \phi, q_j \rangle_{\eta} = 0$  for  $j < s$  and  $\bar{\alpha}_{s,d} = \langle \phi, q_s \rangle_{\eta} \neq 0$ ; and with sufficient decay in the higher harmonics as to satisfy the bound on the sum presented in Proposition 4.3 (for example,  $\phi(t) = q_s(t)$  trivially satisfies this condition). Then Proposition 4.3 applies and we conclude that the resulting population landscape satisfies LPG  $(s - 1, O(\sqrt{s / d}))$ .
In Yehudai and Shamir [2020], Wu [2022] it is shown that monotonically increasing link functions lead to a benign population landscape, provided the data distribution  $\nu$  satisfies mild anti-concentration properties. We verify that in our framework.
1 or link functions where their monotonic behavior dominates; see Wu [2022].
Example 4.8 (Non-decreasing  $\phi$ ). Indeed, Proposition 4.3 is verified with  $s = 1$ , provided  $\phi' \in L_{\eta}^{2}$ . Indeed, if  $\phi \neq 0$  is monotonic, then we have  $\beta_{1} = \langle \phi, \mathcal{K}_{1}\phi \rangle_{\eta} = C_{d}(\mathbb{E}_{\eta}[t\phi(t)])^{2} \neq 0$ , since we can assume without loss of generality that  $\phi(t) \geq 0$  for  $t \geq 0$  and  $\phi(t) \leq 0$  for  $t \leq 0$ .
We emphasize that the results of Yehudai and Shamir [2020], Wu [2022] extend beyond the spherically symmetric setting, which is precisely the focus of next section.
4.2 Non-Spherically Symmetric Case
We now turn to the setting where  $\nu$  is no longer assumed to have spherical symmetry. By making further regularity assumptions on  $\phi$ , our main insight is that distributions that are approximately symmetric (defined in an appropriate sense) still benefit from a well-behaved optimization landscape.
Two-dimensional Wasserstein Distance. When  $\nu$  is not spherically symmetric, the machinery of spherical harmonics does not apply, and we thus need to rely on another structural property. Consider a centered and isometric data distribution  $\nu \in \mathcal{P}_2(\mathbb{R}^d)$ , i.e. such that  $\mathbb{E}_{\nu}x = 0$  and  $\Sigma_{\nu} = \mathbb{E}_{\nu}[xx^{\top}] = I_d$ . We consider the two-dimensional 1-Wassertein distance [Niles-Weed and Rigollet, 2022, Definition 1] -see also Paty and Cuturi [2019]- between a pair of distributions  $\nu_{a},\nu_{b}\in \mathbb{P}(\mathbb{R}^{d})$ , defined as
$$
\widetilde {W} _ {1, 2} \left(\nu_ {a}, \nu_ {b}\right) := \sup  _ {P \in \operatorname {G r} (2, d)} W _ {1} \left(P _ {\#} \nu_ {a}, P _ {\#} \nu_ {b}\right), \tag {19}
$$
where the supremum runs for any two-dimensional subspace  $P \in \mathrm{Gr}(2, d)$ , and  $P_{\#} \nu \in \mathcal{P}(\mathbb{R}^2)$  is the projection (or marginal) of  $\nu$  onto the span of  $P$ .  $\widetilde{W}_{1,2}$  is a distance ([Paty and Cuturi, 2019, Proposition 1]) and measures the largest 1-Wasserstein distance over all two-dimensional marginals.
We are in particular interested in the setting where  $\nu_{a} = \nu$  is our data distribution, and  $\nu_{b}$  is a reference symmetric measure - for instance the standard Gaussian measure  $\gamma_{d}$ . Consider the fluctuations
$$
\Delta_ {L} (\theta) := | L (\theta) - \bar {\ell} (m _ {\theta}) | \text {a n d}, \tag {20}
$$
$$
\Delta_ {\nabla L} (\theta) := \left| \nabla_ {\theta} ^ {S} L (\theta) \cdot \theta^ {*} - \bar {\ell} ^ {\prime} \left(m _ {\theta}\right) \left(1 - m _ {\theta} ^ {2}\right) \right|, \tag {21}
$$
where  $\bar{\ell}(m)$  is the Gaussian loss defined in Section 2 and  $L(\theta) = \mathbb{E}_{\nu} [|\phi(x \cdot \theta) - \phi(x \cdot \theta^*)|^2]$ .  $\Delta_L$  and  $\Delta_{\nabla L}$  thus measure respectively the fluctuations of the population loss and the relevant (spherical) gradient direction. By making additional mild regularity assumptions on the link function  $\phi$ , we can obtain a uniform control of the population loss geometry using the dual representation of the 1-Wasserstein distance.
Assumption 4.9 (Regularity of link function). We assume that  $\phi, \phi'$  are both  $B$ -Lipschitz, and that  $\phi''(t) = O(1/t)$ .
Assumption 4.10 (Subgaussianity). The data distribution  $\nu$  is  $M$ -subgaussian: for any  $v \in S_{d-1}$ , we have  $\|x \cdot v\|_{\psi_2} \leq M$ , where  $\|z\|_{\psi_2} \coloneqq \inf \{t > 0; \mathbb{E}[\exp(z^2 / t^2) \leq 2]\}$  is the Orlitz-2 norm.
Proposition 4.11 (Uniform gradient approximation). Under Assumptions 4.9 and 4.10, for all  $\theta \in S_{d-1}$ ,
$$
\Delta_ {\nabla L} (\theta) = (1 - m ^ {2}) O \left(\widetilde {W} _ {1, 2} (\nu , \gamma) \log \left(\widetilde {W} _ {1, 2} (\nu , \gamma) ^ {- 1}\right)\right) \tag {22}
$$
where the  $O(\cdot)$  notation only hides constants appearing in Assumptions 4.9 and 4.10.
In words, the population gradient under  $\nu$  is viewed as a perturbation of the population gradient under  $\gamma$ , which has the well-behaved geometry already described in Section 2. These perturbations can be uniformly controlled by the projected 1-Wasserstein distance, thanks to the subgaussian tails of  $\nu$ .
Our focus will be in situations where  $\widetilde{W}_{1,2}(\nu, \gamma) = O(1 / \sqrt{d})$ . This happens to be the 'natural' optimistic scale for this metric in the class of isotropic distributions  $\Sigma_{\nu} = I_d$ , as can be seen for instance when  $\nu = \mathrm{Unif}(\sqrt{d}\mathcal{S}_{d-1})$ . Under such conditions, it turns out that link functions with information exponent  $s \leq 2$  can be recovered with simple gradient-based methods, by paying an additional polynomial (in  $d$ ) cost in time complexity.
Assumption 4.12. The Gaussian information exponent of  $\phi$ ,  $s := \arg \min \{j; \langle \phi, H_j \rangle \neq 0\}$  satisfies  $s \leq 2$ .
Assumption 4.13. The projected Wasserstein distance satisfies  $\widetilde{W}_{1,2}(\nu, \gamma) \leq M' / \sqrt{d}$ .
Proposition 4.14 (LPG, non-symmetric setting). Under Assumptions 4.9, 4.10, 4.12 and 4.13,  $L$  verifies LPG  $\left(1,O\left(\sqrt{\frac{(\log d^{\kappa})}{d}}\right)\right)$ , where  $\kappa$  depends only on  $B,M,M^{\prime}$ .
This proposition illustrates the cost of breaking spherical symmetry in two aspects: (i) it requires additional regularity on  $\phi$ , and notably restricts its (Gaussian) information exponent to  $s = 2$ , and (ii) the scale to reach LPG is now no longer dimension-free, but has a polynomial dependency on dimension, since from Lemma 2.1, picking  $\delta$  any positive constant we have
$$
\mathbb {P} _ {\theta_ {0}} \left(m _ {\theta_ {0}} \geq \sqrt {\log d ^ {\kappa}} / \sqrt {d}\right) \geq \frac {\delta}{4} e ^ {- (\sqrt {\log d ^ {\kappa}} + \delta) ^ {2}} \geq \Omega \left(d ^ {- (1 + o (1)) \kappa}\right).
$$
We are not able to rule this out as a limitation of our proof; whether this polynomial dependency on dimension is inherent to the symmetry breaking is an interesting question for future work.
While assumptions 4.9, 4.10 and 4.12 are transparent and impose only mild conditions on the link function and tails of  $\nu$ , the 'real' assumption of Proposition 4.14 is the concentration of  $\widetilde{W}_{1,2}(\nu, \gamma)$  (Assumption 4.13). The ball  $\{\nu; \widetilde{W}_{1,2}(\nu, \gamma) = O(1 / \sqrt{d})\}$  contains many non-symmetric measures, for instance empirical measures sampled from  $\gamma$  with  $n = \omega(d^2)$  [Niles-Weed and Rigollet, 2022, Proposition 8], and we suspect it contains many other examples, such as convolutions of the form  $\nu * \gamma_\sigma$  arising for instance in diffusion models. That said, one should not expect the distance  $\widetilde{W}_{1,2}(\nu, \gamma)$  to be of order  $1 / \sqrt{d}$  for generic 'nice' distributions  $\nu$ ; for instance, log-concave distributions are expected to satisfy  $W_1(P_\# \nu, \gamma) \simeq 1 / \sqrt{d}$  for most subspaces  $P$ , as captured in Klartag's CLT for convex bodies Klartag [2007].
Localized Growth via Stein's method. To illustrate the mileage of the previous techniques beyond this 'quasi-symmetric' case, we consider now an idealised setting where the data is drawn from a product measure  $\nu = \eta^{\otimes d}$ , with  $\eta \in \mathcal{P}(\mathbb{R})$  and  $W_{1}(\eta, \gamma_{1}) = \Theta(1)$ . In other words,  $x = (x_{1}, \ldots, x_{d}) \sim \nu$  if  $x_{i} \sim \eta$  are i.i.d. In this setting, the distances  $W_{1}(P_{\#} \nu, \gamma)$  reflect a CLT phenomena, which requires the subspace  $P$  to 'mix' across independent variables. Consequently, one may expect the expression of the hidden direction  $\theta^{*}$  in the canonical basis to play a certain role so we make the following additional regularity assumption on the tails of  $\phi$ :
Assumption 4.15 (Additional Regularity in third derivatives).  $\phi$  admits four derivatives bounded by  $L$ , with  $|\phi^{(3)}(t)| = O(1 / t)$  and  $|\phi^{(4)}(t)| = O(1 / t^2)$ . Moreover, the third moment of the data distribution is finite:  $\tau_{3} = \mathbb{E}_{t\sim \eta}[t^{3}] < \infty$ .
Stein's method provides a powerful control on  $\Delta_L(\theta)$  and  $\Delta_{\nabla L}(\theta)$ , as shown by the following result:
Proposition 4.16 (Stein's method for product measure). Let  $\chi(\theta, \theta^{*}) \coloneqq \| \theta \|_{4}^{2} + \| \theta^{*} \|_{4}^{2}$ . Under Assumptions 4.9, 4.10 and 4.15, there exists a universal constant  $C = C(M, B, \tau_{3})$  such that
$$
\Delta_ {L} (\theta) \leq C \chi \left(\theta , \theta^ {*}\right), a n d \Delta_ {\nabla L} (\theta) \leq C \sqrt {1 - m ^ {2}} \chi \left(\theta , \theta^ {*}\right). \tag {23}
$$
The proof is based on the Stein's method for multivariate variables [Röllin, 2013, Theorem 3.1] with independent entries, which provides a quantitative CLT bound. Contrary to the quasi-symmetric case, here the concentration is not uniform over the sphere, but crucially depends on the sparsity of both  $\theta$  and  $\theta^{*}$ , measured via the  $\ell_4$  norms  $\| \theta \| _4$ ,  $\| \theta^{*}\|_{4}$ : for incoherent, non-sparse directions, we have  $\| \theta \| _4^2\simeq 1 / \sqrt{d}$ , recovering the concentration rate that led to Proposition 4.14, while for sparse directions we have  $\| \theta \| _4^2 = \Theta (1)$ , indicating an absence of concentration to the Gaussian landscape.
Therefore, the natural conclusion is to assume a planted model where  $\theta^{*}$  is incoherent with the data distribution, i.e.  $\| \theta^{*}\|_{4} = O(d^{1 / 4})$ . While the LPG property does not directly apply in this setting, we outline an argument that suggests that the single-index model can still be efficiently solved using gradient-based methods. For that purpose, we assume that  $\theta^{*}$  is drawn uniformly in  $S_{d - 1}$ , which implies that its squared- $L^4$  norm  $\| \theta^{*}\|_{4}^{2}$  is of order  $d^{-1 / 2}$  with high probability:
Fact 4.17. Assume  $\theta^{*}\sim \mathrm{Unif}(\mathcal{S}_{d - 1})$ . Then  $\mathbb{P}(\| \theta^{*}\| _4^2\leq C / \sqrt{d})\geq 1 - C'\exp (-C)$
Because  $\theta_0$  is also drawn uniformly on the sphere, the typical value of  $\chi (\theta ,\theta^{*})$  is of order  $d^{-1 / 2}$ . For Gaussian information exponent  $s = 2$ , the population gradient under Gaussian data satisfies
$-\nabla_{\theta}^{S_{d-1}} L_{\gamma}(\theta) \cdot \theta^* \geq C m_{\theta}$ . As a consequence, by Proposition 4.16, whenever  $|\theta_0 \cdot \theta^*| > c \sqrt{d}$  (which happens with constant probability lower bounded by  $e^{-c}$ ), we enter a 'local' LPG region where  $-\nabla_{\theta}^{S_{d-1}} L(\theta) \cdot \theta^* \geq C (m - c / \sqrt{d}) > 0$ . While this condition is sufficient in the quasi-symmetric setting to start accumulating correlation (Theorem 3.2), now this event is conditional on  $\theta$  being dense, ie so that  $\chi(\theta, \theta^*) = O(1 / \sqrt{d})$ .
Since the typical value of  $\chi (\theta ,\theta^{*})$  is of scale  $1 / \sqrt{d}$ , one would expect that SGD will rarely visit sparse points where  $\chi (\theta ,\theta^{*})\gg O(1 / \sqrt{d})$ , and thus that the local LPG property will be valid for most times during the entropic phase of weak recovery, summarized by the following conjecture:
Conjecture 4.18 (SGD avoids sparse points). Assume  $\theta^{*},\theta_{0}$  are drawn from the uniform measure, and let  $\theta_t$  be the  $t$ -th iterate of SGD with  $\delta \simeq 1 / (d\log d)$ . There exists a universal constant  $C$  such that for any  $\xi >0$ , we have
$$
\mathbb {P} \left(\sup  _ {t \leq T} \| \theta_ {t} \| _ {4} ^ {2} \geq \sqrt {\frac {\xi \log T}{d}}\right) \leq C \exp \left(- \xi^ {2} d\right). \tag {24}
$$
Since the time to escape mediocrity in the case  $s = 2$  is  $T \simeq d\log (d)^{2}$ , this conjecture would imply that SGD does not effectively 'see' any sparse points, and thus escapes mediocrity. If one assumed that in this phase the dynamics is purely noisy, now pretending that  $\theta_{i}$  were drawn independently from the uniform measure, and that  $\| \theta \|_4^4$  is approximately Gaussian with mean  $d^{-1}$  and variance  $d^{-3}$ , the result follows by simple concentration. The challenging aspect of Conjecture 4.18 is precisely to handle the dependencies across iterates, as well as the spherical projection steps.
In order to validate our theory, and inspect the degree to which our bounds may be pessimistic, we consider empirical evaluation of the training process in our two primary settings. Specifically, we consider random initialization on the half-sphere (with the sign chosen to induce positive correlation as in Arous et al. [2021]), and investigate how often strong recovery occurs relative to the information exponent of the link function. Ultimately the empirical evidence corroborates the theory, showing the symmetric case succeeds when the initial correlation is above a polynomial threshold and the nonsymmetric case succeeds as long as  $s = 2$ . Due to space constraints, the details of the experimental setup and further discussion appear in Section A.
6 Conclusion and Perspectives
In this work, we have asked whether the remarkable properties of high-dimensional Gaussian SGD regression of single-index models are preserved as one loses some key aspects that make Gaussian distributions so special (and so appealing for theorists). Our results are mostly positive, indicating a robustness of the Gaussian theory, especially within the class of spherically symmetric distributions, where a rich spherical harmonic structure is still available. As one loses spherical symmetry, the situation becomes more dire, motivating a perturbative analysis that we have shown is effective via projected Wasserstein and Stein couplings.