Title: From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGD
Venue: NeurIPS 2025
Accepted: yes
Keywords: Feature learning

stochastic gradient descent

single-index model
Abstract: To understand feature learning dynamics in neural networks, recent theoretical works have focused on gradient-based learning of Gaussian single-index models, where the label is a nonlinear function of a latent one-dimensional projection of the input. While the sample complexity of online SGD is determined by the *information exponent* of the link function, recent works improved this by performing multiple gradient steps on the same sample with different learning rates — yielding a *non-correlational* update rule — and instead are limited by the (potentially much smaller) *generative exponent*. However, this picture is only valid when these learning rates are sufficiently large. In this paper, we characterize the relationship between learning rate(s) and sample complexity for a broad class of gradient-based algorithms that encapsulates both correlational and non-correlational updates. We demonstrate that, in certain cases, there is a phase transition from an "information exponent regime" with small learning rate to a "generative exponent regime" with large learning rate. Our framework covers prior analyses of one-pass SGD and SGD with batch reuse, while also introducing a new layer-wise training algorithm that leverages a two-timescales approach (via different learning rates for each layer) to go beyond correlational queries without reusing samples or modifying the loss from squared error. Our theoretical study demonstrates that the choice of learning rate is as important as the design of the algorithm in achieving statistical and computational efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe study builds upon a series of works concerning sample complexity estimates for the high-dimensional limit of single index models. The authors consider the case of a two-layer neural network trained by algorithms belonging to a class of optimization methods based on generalized versions of the gradient of the MSE loss $\\psi_\\eta$, where $\\eta$ is an additional learning rate parameter. The main theorem establishes a formula for the sample complexity, which depends on the learning rates $\\gamma$ and $\\eta$, as well as on the Hermite coefficients of the student's and teacher's link functions.\n\nThe authors argue that the novelty lies in the generality of the algorithmic class to which the theorem applies. They then show that both _vanilla SGD_  and  _batch reuse SGD_ belong to this class, and they establish the relationship between the new estimates and the information and generative exponents. They demonstrate how, by tuning $\\eta$, it is possible to transition from a regime where learning is governed by the information exponent to one where learning is faster and governed by the generative exponent.\n\nIn addition, they introduce a new variant of SGD, termed _Alternating SGD_, and show how it can improve standard SGD when $\\eta$ is sufficiently large. This variant can be further enhanced when applied to deeper networks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strenghts:\n- well written and clear.\n- theorem 3.3 is a general results provides a unified framework in which generative and information exponents appear as limit cases.\n- the class of algorithm to which theorem 3.3 applies seems to be very broad, and Alternating SGD seems to be a nice example that could be of interest for practitioners\n- the sketch of the proof insection 5 manages to convey clearly the key idea of the argument, showing how all the terms in the Hermite expansion of the Loss can become relevant depending on the scaling of $\\eta$. \n- the technical appendices contain thorough proofs of the main results of the paper, and even rigorous analyses that go beyond the main focus of the paper, such as appendix C.3 that examines the case of a deeper network. Or appendix B.6 and B.7 that provide already known, but necessary context on how to rigorously bridge the gap between weak and strong recovery.\n\n### Weaknesses:\n- cumbersome notation for hermite coefficients, especially formula 3.2, which is one of the central definitions in the paper. Writing the definition in terms of the integral over 2 independent gaussian variables, instead of this complicated notation with two nested $\\to$, would have been shorter. I understand that in the paper you need to Hermite expand different functions, and need a compact notation for that. Nevertheless I suggest to provide the explicit intregrals at least once for the important formulas.\n- since the area of single index model has now many research works that investigate similar questions of sample complexity, it is hard to assess which contributions are original, and which overlap with previous work. I do not see much novelty on the side of the analysis of _batch reuse SGD_, however the overall analysis and _alternating SGD_ are completely new, to the best of my knowledge (even though possible overlaps with the very recent [CLW+25] could be investigated more).\n- just polynomial link functions is a rather restrictive assumption, I believe simulations that test the phenomenology on non-polynomial link functions should be performed.\n- there are no comments and simulations that test the proposed _alternating SGD_ in more realistic settings.\n\nQUESTIONS:\n- About figure 1, I would strongly suggest to plot results for different values of $d$, to show how the phenomenology portrayed passes to the high-dimensional limit. Should we see a sharp vertical transition between blue and white region for $d\\to \\infty$? It may also help to continue the plot for even larger values of $\\eta$.\n- does your analysis offer practical insigths for tuning learning rate in networks, comments on this and simulations on more realistic and complex task may be interesting to practitioners.\n\n### Questions and comments:\n- In Theorem 3.3 you need to require an upper bound on $\\gamma$. What happens for larger values? From the proof sketch, it seems that it is not possible to ignore the normalization term of the iterate expansion. Is that the tight value for such constant, beyond which the dynamics become too noisy? \n- About Figure 1, I would strongly suggest plotting results for different values of $d$, to show how the portrayed phenomenology extends to the high-dimensional limit. Should we see a sharp vertical transition between the blue and white regions for $d \\to \\infty$? It may also help to extend the plot to even larger values of $\\eta$.\n- Does your analysis offer practical insights for tuning learning rates in networks? Comments on this aspect, together with simulations on more realistic and complex tasks, may be useful to practitioners.\n- Can you clarify exactly what are your contributions in the batch reuse SGD setting? \n- Can you comment on whether a non-polynomial $\\sigma$ introduces just technical difficulties, or could give rise to different phenomenologies? \n- The deep network case discussed in Appendix C.3 seems to be one of the strongest contributions of the paper, showing how depth can enhance learning. In particular, the final discussion in Appendix C.3 seems relevant, concerning which properties of the Hermite coefficients of $\\sigma$ and $\\sigma_*$ would make the deep framework better or worse than multi-pass SGD. You make comments only about the three-layer case; can they be generalized to larger $D$?\n- Can your analysis be coupled with the smoothing techniques from [DNGL23] to reach the optimal sample complexity thresholds?\n\n### Suggestion\nFrom the communication point of view, in my opinion there is a bit too much emphasis on the fact that this work establishes the importance of the learning rate. This point is stressed both in the abstract, introduction, and conclusions. For instance, the conclusions section starts with:\n\n> This work demonstrates that the learning rate is a fundamental factor in determining the sample complexity of gradient-based algorithms for learning single-index models with neural networks.\n\nHowever, I think most readers already know that the learning rate is an impactful factor in determining the sample complexity in any learning problem. It is a core aspect of gradient methods, and most of the referenced works on single index models have detailed discussions about learning rates. I believe that the excessive emphasis on this aspect could risk drawing attention away from other important contributions of the paper. Moreover, it is even technically a bit misleading, since the usual SGD learning rate $\\gamma$ is treated similarly to previous work, and it is the additional parameter $\\eta$ that is the true focus of this work.\n\nIn my opinion, the merits of this work lie in the definition of a general framework that allows studying and finding optimal learning rates for a broad class of gradient-based optimization methods, providing a unified framework in which information and generative exponents appear as limit cases. I would suggest emphasizing more the generality and versatility of these results, and how the additional learning rate $\\eta$ introduced in this novel class of learning algorithms can impact the sample complexity.\n\nLIMITATIONS:\nThe authors address the main limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papers investigate the sample complexity of training two-layer neural networks to weakly learn intrinsic features with data generated from Gaussian single-index models. \n\nSpecifically, the authors focus on a set of online gradient-based algorithms that can be expressed with a general gradient oracle $\\psi_\\eta(y,\\langle x, w\\rangle)$ , where $\\eta$ is a hyperparameter that determines the non-linearity of the gradient query. \n\nThen, they stated the tight sample complexity for this general algorithm with a measure of non-correlational in $\\eta$. Using this main result, they provided the tight sample complexity for vanilla online SGD, alternating SGD and batch-reused SGD. For alternating SGD and batch-reused SGD, they revealed a novel phase-transition in the intrinsic learning rate $\\eta.$\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n1. The writing is generally clear and easy to follow. Proof sketch in the maintext is essential to capture the key components. \n2. The framework of the general gradient oracle is theoretically convenient and the characterization of the non-correlational property with $\\mu(\\eta)$ is quite critical in capturing the hardness of the problem (in terms of $\\eta$).\n3. The sample complexity phase transition in learning rate is a novel perspective for the batch-reused SGD in single index models. Previous work all focus on the cases where the psudo-step is taken with sufficiently large but reasonable step size, but the sample complexity for the batch-reused SGD with smaller learning rate remains unclear. \n\nCons\n\n1. The definition of $\\mu(\\eta)$ is a bit obsecure at first glance. It would be better if the author can preview some examples of $\\psi_\\eta$  in Section 3 and see how $\\eta$  appears. Also it would be better if authors can briefly explain how this quantity reigns the sample complexity in the main text & discuss if similar notion appears in other literatures. \n2. The result is novel by completing the whole picture of batch-reusing, but quite incremental in terms of matching the lower bound, as the prior works have fully settled the optimal algorithms and sample complexity of various online SGD in single-index models.\n\nQUESTIONS:\n1. In the perspective of exploration, the pseudo-step is used to find a better gradient near the current point and smoothing can be viewed as a similar technique (however smoothing is used in some other scenarios with higher generative exponent). In that case, the polarization parameter $\\lambda$ (see e.g. [1,2]) may plays a role. Would it be possible to analyze the analogous phenomenon for $\\lambda$? \n2. Would it be possible to extend this analysis to the general link functions, instead of the polynomial links?  \n3. The main result essentially depends on $\\gamma$ (the learning rate for the spherical gradient) and there is certain notion of the optimal choice of $\\gamma.$ But in the result of [2], they only require the learning rate to be sufficiently large. Is the choice of $\\gamma$ for the technical reason or it matters in whether the sample complexity is optimal? \n4. This type of model typically relates to the SQ-problem, where I tend to think that closing the computational statistical gap is more important (see the second con). Does this learning-rate phase transition relate to some more practical phenomenon or has practical implications? (I think it’s very different to EoS which is more realistic). Or potentially your characteriaztion can provides some other insights (see the first question).\n\nLIMITATIONS:\nSee cons.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper characterizes the relationship between the learning rate and sample complexity for a class of gradient-based algorithms including both correlational and non-correlational updates. A phase transition is observed when the learning rate increases from small to large. Examples include online SGD, alternating SGD, batch reuse SGD are discussed. Some very simple numerical experiments are included.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n(1) The main results are easy to follow, and the meaning of phase transition is clear in this context. \n\n(2) Various examples are included in the class of gradient-based algorithms, and they are properly discussed. \n\n(3) Figure 1 seems to be in line with what the theory suggests.\n\nWeaknesses:\n\n(1) The class of models considered in this paper still seem to be a bit special. It would be interesting to see if phase transitions can occur in more general settings.\n\n(2) Figure 1 is based on a toy example. More numerical experiments will be helpful, especially in terms of whether the phase transitions claimed in the paper can be observed in various numerical settings in practice, which will also make the theory more convincing. For example Figure 1 illustrates the toy example for Corollary 4.2., but there is no numerical illustration for Corollary 4.3. for batch reuse SGD, which according to Figure 3, should lead to multiple phase transitions. It would be great to see numerically that it is indeed the case. For Corollary 4.1. for online SGD, the theory does not suggest there will be phase transitions, and it will also be useful to numerically check that indeed no phase transitions will be observed numerically.\n\nQUESTIONS:\n(1) It would be great if you can discuss when Theorem 3.3. will lead to phase transitions. I understand that you have discussed several examples in Section 4, but it would also be interesting to discuss for the general setup as in Theorem 3.3, when you will have phase transitions, when you will not, depending on the properties of $\\mu_{i}(\\eta)$.\n\n(2) Some references need to be cleaned up. For example, in [AVPVF23], sgd should be SGD, in [Cla87], gronwall should be Gronwall, in [CCM11], gaussian should be Gaussian.\n\nLIMITATIONS:\nIt seems the author(s) did not discuss the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes how the learning rate affects the sample complexity for neural networks learning gaussian single-index models. It establishes a general framework to show that for many gradient-based algorithms, the learning rate induces a phase transition in performance. Specifically, a small learning rate confines the algorithm to a complexity determined by the \"information exponent,\" while a sufficiently large learning rate allows it to achieve a better complexity determined by the \"generative exponent\" for polynomial link function. \n\nThe authors further apply the framework to online SGD, SGD with square label transformation, and batch-reusing, and demonstrate how the phase transition occurs in each case.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis paper presents a well-structured and technically rigorous analysis that builds upon the extensive recent literature on learning single-index models. Its key contribution is investigating how the learning rate dictates the sample complexity, a crucial aspect that previous works often simplified by assuming either a sufficiently large or small learning rate.\n\n1. The key contribution of this work is its investigation into the influence of the learning rate on sample complexity. The authors demonstrate how adjusting the learning rate, including inner learning rates for batch-reusing steps, allows the sample complexity to interpolate between two critical bounds:\n\n     - The information exponent-related bound of $d^{p-1}$\n \n     - The SQ bound of $\\tilde O(d)$ for polynomial functions. \n\n2. If my understanding is correct, this research can be viewed as an extension of the work by Lee et al. (2024), providing a more nuanced understanding of how learning rate selection directly impacts the sample efficiency of learning in single-index models.\n\n**Weaknesses**:\n\nA potential limitation of the proposed framework is the cumbersome process required to determine the optimal learning rate. This selection is not only sensitive, with Remark 3.4 indicating that changes in $\\mu_i(\\eta)$ can alter the optimal rate by a polynomial factor in the dimension, but it also presupposes detailed knowledge of the interplay between the link and activation functions. This dependence on specific model information may impede the practical implementation of these algorithms.\n\nAnother possible issue is that the proposed algorithms in Section 4.2 are still a little bit artificial, but the batch-reusing part seems already more natural then the version in Lee et al. (2024). Meanwhile, I'm also aware this is a common issue in the existing literature. \n\n[1] Jason D. Lee, Kazusato Oko, Taiji Suzuki, and Denny Wu. Neural network learns low-dimensional 442 polynomials with SGD near the information-theoretic limit. In The Thirty-eighth Annual Confer443 ence on Neural Information Processing Systems, 2024.\n\nQUESTIONS:\n1. On Determining the Learning Rate: Beyond the specific analysis in this paper, is there a more general principle or rule of thumb for setting the optimal learning rate, or is it fundamentally tied to the specific properties of the model?\n\n2. On Extending to Broader Function Classes: The framework in (3.1) appears general enough to accommodate non-polynomial link functions. Could the current results be extended to such cases? What are the primary technical barriers to this extension? Would criteria similar to $\\mu_i(\\eta)$ still be applicable for analyzing the sample complexity in a non-polynomial setting?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}