Title: Revisiting Glorot Initialization for Long-Range Linear Recurrences
Venue: NeurIPS 2025
Accepted: yes
Keywords: Recurrent Networks

Initialization

Signal Propagation
Abstract: Proper initialization is critical for Recurrent Neural Networks (RNNs), particularly in long-range reasoning tasks, where repeated application of the same weight matrix can cause vanishing or exploding signals.
A common baseline for linear recurrences is Glorot initialization, designed to ensure stable signal propagation---but derived under the infinite-width, fixed-length regime—an unrealistic setting for RNNs processing long sequences. In this work, we show that Glorot initialization is in fact unstable: small positive deviations in the spectral radius are amplified through time and cause the hidden state to explode. Our theoretical analysis demonstrates that sequences of length $t = O(\sqrt{n})$, where $n$ is the hidden width, are sufficient to induce instability. To address this, we propose a simple, dimension-aware rescaling of Glorot that shifts the spectral radius slightly below one, preventing rapid signal explosion or decay. These results suggest that standard initialization schemes may break down in the long-sequence regime, motivating a separate line of theory for stable recurrent initialization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLinear RNNs can only be trained if the recurrence matrix eigenvalues are in the unit disk.  Glorot initialization guarantees that eigenvalues are in the unit disk if the number of hidden nodes is infinite, but experimental results often show that this result fails to hold for finite numbers of hidden nodes.  This paper proves that the maximum eigenvalue converges in distribution to a Gumbel variable that is quite likely to be greater than one, and that therefore, the probability of such a divergent eigenvalue can be precisely controlled by rescaling the Glorot initialization by an amount that depends on the critical values of the Gumbel distribution.  Numerical simulations show that standard Glorot often leads to divergent state variables, and this divergence is eliminated by the proposed scaling. Tests using simple linear RNNs for toy tasks with very long sequences (1k to 3k time steps) demonstrate that standard Glorot often diverges, but that the proposed method often outperforms recent empirically-motivated solutions (half-Glorot and diagonal-uniform).\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nFrom a practical point of view, the literature has been in desperate need of better study of long-sequence behaviors.  The entire deep learning literature has been limited to short sequence lengths for the past forty years, and this is a big problem in practice: Real-world sequences are not short, so the limitation to short sequences means that essentially we are modeling short extracts of the real world instead of modeling reality.\n\nThis study proposes studying long-sequence problems by studying the distribution of the maximum eigenvalue, rather than studying the distribution of all eigenvalues as Glorot did.  This step is very well motivated by 21st-century results in linear algebra, and in my opinion, the work in this paper is long overdue in the machine learning literature.\n\nThe empirical demonstrations in Fig. 1 are convincing.  Experiments include only numerical simulations and toy problems, which would be a weakness if the theoretical results were less interesting, but is not a weakness in the context of this paper: I treat these as a visualization of the key theoretical results.\n\nWeaknesses:\n\nExplanation of Eq. (18) in Eq. (19) is very useful, but I would have appreciated a similar explanation of the RHS of Eq. (20).  The approximation (1/t)exp(t^2/2n) is obviously true, but the factor of n suggests a more careful degree of precision that is not obvious.\n\nQUESTIONS:\nCould you add a sentence giving a brief explanation for the constants in the RHS of Eq. (20)?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits Glorot initialization used for initialization parameters of a Linear Recurrent Unit (LRU) in the context of long range dependency tasks. It first analyzes the instability caused by Glorot initialization in linear RNNs. It makes an observation that spectral radius of matrices initialized using Glorot initialization, typically converges to one from above, resulting in exponential growth in hidden states over long range sequences. To mitigate this issue, this work proposes a rescaled version of the Glorot initialization using the dimension dependent constant, reducing the probability by one standard deviation below expectation for the fact that spectral radius exceeds one.  For the complex Gaussian matrices, it provides some theoretical justification for the fact that with rescaled Glorot initialization, the hidden states explode with less likely probability compared to the standard Glorot initialization. Numerical experiments have been conducted to validate the efficacy of the proposed Rescaled Glorot Initialization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed Rescaled Glorot Initialization is easy to implement and decreases the probability of having eigenvalues above 1 during initialization compared to Glorot initialization.\n- Theoretical analysis shows that linear RNN initialization with complex Glorot matrix results in hidden state norm explosion when the sequence length is at least of the order of sqrt(n) where n is the hidden state dimension. \n\nWeaknesses:\n- Table 1 shows the performance of linear recurrent models for Long Range Arena benchmark. It’s unclear why the Glorot initialization yields exploding baseline when the original linear RNNs do not have such an issue in these benchmarks?\n- Contribution seems to be a bit lacking since the paper only proposes a simple rescaling of the standard Glorot initialization and shows stability of the initial hidden states in two setups (a) infinite width and finite length, and (b) infinite width and length. The experiments do not reveal a lot of insights into the proposed scaling vs standard Glorot initialization.\n\nQUESTIONS:\n- In Tab.1, in dense matrix case, for the longest sequence task (IMDB with 8K tokens), Rescale variants of the initialization do not always yield better performance compared to Glorot / 2 or uniform variants. How do you explain this behavior? Since the hidden state explosion is already handled at the initialization by all these schemes. \n- In Fig1, how does the density and hidden states evolve for Glorot/2 and Diagonal Uniform scaling?  \n- In Tab.1, the error bars correspond to how many repetitions for the experiments?\n- Although this work is concerned on long-sequences, does the proposed Glorot Rescaling hurt the performance of linear RNNs on short sequences since eigenvalues are less likely to be beyond 1 ?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper revisits the Glorot initialization scheme for Recurrent Neural Networks (RNNs), specifically in the context of long-sequence processing with linear recurrences. The authors make the key theoretical point that for finite-width matrices, the spectral radius of a Glorot-initialized matrix does not converge to one from below, but rather from above. This means the largest eigenvalue modulus is typically slightly greater than one. While this deviation is small, it gets amplified exponentially over long sequences, leading to exploding hidden states. To address this instability, the authors propose a simple, dimension-aware rescaling of the Glorot variance.  The empirical results on a few tasks from the Long Range Arena (LRA) benchmark, showing that their rescaled initialization is stable and performs better than standard Glorot (which fails to train) and a naive Glorot/2 baseline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe paper's main strength is its clear and well-articulated theoretical motivation. It correctly identifies a subtle but critical flaw in the common wisdom surrounding Glorot initialization for recurrent models: the convergence of the spectral radius to 1 from above. This is an important detail from random matrix theory that is often overlooked, and the paper does a good job of explaining its severe consequences for long-sequence stability.\n\nWeakness\n\n1. The central contribution is an incremental adjustment to the 15-year-old Glorot/Xavier initialization. While the theoretical motivation is sound, its practical impact in 2025 is questionable. The community has developed more powerful and general techniques to ensure training stability in deep models, most notably normalization layers (e.g., Layer Normalization), which are standard practice in nearly all modern recurrent models and Transformers. Furthermore, the work focuses on vanilla linear RNNs, a class of models that is not state-of-the-art. The paper itself acknowledges that recent advances center on State-Space Models (SSMs) and that its models do not compete with them. Improving an old initialization for a non-competitive model architecture feels like a solution to an already-solved problem, which severely limits the paper's significance.\n2. The empirical validation for the method's downstream performance is extremely thin, consisting of a single small table (Table 1). The lack of results on more complex, gated architectures (like LSTMs/GRUs) or modern architectures (like LRUs or SSMs) where initialization could interact with other components is a major omission. Also, a much more relevant and challenging baseline would be a standard Glorot-initialized model stabilized with a standard technique like Layer Normalization. Without this comparison, it's impossible to know if the proposed rescaling offers any benefit over what is already common practice.\n3. The theoretical analysis, while clean, rests on the simplified setting of linear RNNs with i.i.d. Gaussian inputs. This is a significant departure from real-world sequential data, which exhibits complex temporal dependencies.\n\nQUESTIONS:\nThe authors may need to justify the significance of their work with more empirical results to make sure this paper fits the NeurIPS community.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits the widely used Glorot initialization scheme in the context of Recurrent Neural Networks (RNNs), focusing on long-range linear recurrence architectures. The authors conduct a theoretical and empirical investigation into the stability of Glorot initialization, especially in deep or long-sequence settings.\n\nThe key insight is that Glorot initialization becomes unstable for sequences longer than √d, where d is the hidden state dimension. While Glorot aims to ensure variance stability and spectral norm convergence to 1, the authors highlight a transient phase during training where the spectral norm exceeds 1, causing unstable dynamics, particularly in deep RNNs.\n\nTo address this, the paper proposes a spectral radius-aware modification to Glorot initialization, explicitly bounding the spectral norm of the weight matrix below 1 to guarantee long-range stability and mitigate hidden state explosion.\n\nThe theoretical claims are validated through empirical results on Long Range Arena (LRA) benchmarks, including sequence classification tasks such as ListOps, seq-CIFAR, and IMDB. These experiments demonstrate that the proposed initialization leads to improved stability and convergence in long-sequence settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* Clear theoretical contribution:\n The paper provides a well-motivated and mathematically grounded critique of Glorot initialization when applied to long-range recurrent models. The authors identify a specific limitation - transient spectral norm inflation - and rigorously link it to practical instability in deep or long-sequence architectures.\n\n\n* Novel solution grounded in spectral theory:\n The spectral radius-aware variant of Glorot initialization is simple, well-motivated, and practically useful. It’s presented in a way that makes it readily implementable in standard training pipelines.\n\n\n* Insightful hidden state norm analysis:\n The contrast between stability in the infinite-width, shallow limit and practical deep architectures is clearly articulated, helping to bridge theory and practice.\n\n* Comprehensive empirical validation:\n The paper evaluates the proposed method across a variety of standard long-range sequence tasks in the LRA benchmark. The results consistently show improved stability and, in many cases, better task performance or faster convergence.\n\nWeaknesses and Areas for Improvement:\n\n* Evaluation limited to sequence classification tasks:\nThe experiments focus on classification tasks. It would be beneficial to consider sequence generation or forecasting settings as well, where recurrent stability over long horizons is even more critical.\n\nQUESTIONS:\nnone\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-12"}