Revisiting Glorot Initialization for Long-Range Linear Recurrences
Authors: Noga Bar
Mariia Seleznova
Yotam Alexander
Gitta Kutyniok
Raja Giryes
Venue: NeurIPS 2025 poster ()
Keywords: Recurrent Networks
Initialization
Signal Propagation
Primary area: theory
TL;DR: Standard Glorot initialization becomes unstable when used in RNNs with long sequences, leading to exploding hidden states. To address this, we propose a simple rescaling that effectively mitigates the instability.

Abstract: Proper initialization is critical for Recurrent Neural Networks (RNNs), particularly in long-range reasoning tasks, where repeated application of the same weight matrix can cause vanishing or exploding signals.
A common baseline for linear recurrences is Glorot initialization, designed to ensure stable signal propagation---but derived under the infinite-width, fixed-length regime—an unrealistic setting for RNNs processing long sequences. In this work, we show that Glorot initialization is in fact unstable: small positive deviations in the spectral radius are amplified through time and cause the hidden state to explode. Our theoretical analysis demonstrates that sequences of length $t = O(\sqrt{n})$, where $n$ is the hidden width, are sufficient to induce instability. To address this, we propose a simple, dimension-aware rescaling of Glorot that shifts the spectral radius slightly below one, preventing rapid signal explosion or decay. These results suggest that standard initialization schemes may break down in the long-sequence regime, motivating a separate line of theory for stable recurrent initialization.

Revisiting Glorot Initialization for Long-Range Linear Recurrences
Noga Bar"
Mariia Seleznova*
Yotam Alexander
Gitta Kutyniok
Raja Giryes
Equal contribution
Tel Aviv University
Maaiiig-axmmmliann---o Universität München
University of Tromso
DLR-German Aerospace Center
Muuncc Center for Machine Learning
Abstract
Proper initialization is critical for Recurrent Neural Networks (RNNs), particularly in long-range reasoning tasks, where repeated application of the same weight matrix can cause vanishing or exploding signals. A common baseline for linear recurrences is Glorot initialization, designed to ensure stable signal propagation but derived under the infinite-winn width, Exed-length regime- processing long sequences. In this work, we show that Glorot initialization is fact unstable: small positive deviations in the spectral radius amplified through time and cause the hidden state to explode. Our theoretical analysis demonstrates that sequences of length é O(vn), where is the hidden width, are sufficient induce instability. To address this, we propose a simple, dimension-awar rescaling of Glorot that shifts the spectral radius slightly below one, preventing rapid signal explosion or decay. These results suggest that standard initialization schemes may break down in the long-sequence regime, motivating separate line of theory for stable recurrent initialization.
unrealistic setting for RNNs
1 Introduction
Recurrent Neural Networks (RNNs) have long been foundational models for sequential data, powering applications that range from time-series forecasting [Salinas al., 2020, Rangapuram et al., 2018] to natural-language processing [Mikolov et al., 2010, Sundermeyer et al., 2015, 2012, Gers and Schmidhuber, 2001]. Recent advances in sequence modeling have centered on State-Space Models (SSMs) [Gu et al., 2021, Gu and Dao, 2023, De et al., 2024], and prominent concurrent work has demonstrated that a class of RNNs known as Linear Recurrent Units (LRUs) [Orvieto al., 2023] can match the performance of SSMs with simpler architectures. In both LRUs and SSMs, the recurrent blocks are linear, enabling efficient parallel-scan computations and thereby fast training and inference. The success of these models on long-range reasoning tasks reaffirms the relevance of RNNs modern deep learning.
Despite this potential, RNNs remain difficult to train on long sequences due to the well-known problem of vanishing and exploding gradients [Bengio al., 1994, Pascanu et al., 2013]. The core issue lies in the repeated application of the same weight matrix at every time step: even slight spectral imbalance is exponentially amplified, causing hidden states-and and hence gradients- to either explode or vanish. While deep feedforward networks suffer from similar pathologies, each layer in such architectures uses an independent weight matrix, making the dynamics of signal propagation easier to analyze and control.
In deep feedforward networks, instability typically mitigated by principled initialization schemes such as Glorot [Glorot and Bengio, 2010] or He [He et al., 2015] initialization. These methods are derived under the infiitt-widtth limit, where the width tends infinity while the depth is kept fixed. In this regime, one can compute the exact signal propagation statistics and tune the weights
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

variance so that activations and gradients remain of the same magnitude [Glorot and Bengio, 2010]. Although these initialization schemes have been immensely successful practice, recent theoretical work reveals that their underlying infinite- wiith approximation becomes increasingly inaccurate as depth grows [Hanin and Nica, 2019, 2020, et al., 2021, Seleznova and Kutyniok, 2022]. This raises questions about the applicabiliit of such schemes to RNNs, which can be viewed as extremely deep networks in time-typically of modest width but unbounded depth. As in the feedforward case, analyzing RNNs under the joint limit of infinite width and infinite input length poses significant theoretical challenges: standard random matrix theory techniques used the infinite-width setting no longer apply when both dimensions grow [Tao, 2012]. In RNNs, this limitation is further exacerbated by the repeated application of the same weight matrix at each time step.
Yet, recent approaches to long-range RNNs continue to be guided by infinite-widie wiith heuristics. Notably, the LRU method [Orvieto et al., 2023], whose success stems from its initialization and parametrization, is inspired by the Glorot scheme. The goal of LRU initialization to construct diagonal matrix whose eigenvalue spectrum matches that of iiottot-nitialized weight matrix. This implicitly assumes that replicating Glorot's ' spectral behavior is sufficient to ensure stable recurrent dynamics. In contrast, we show that standard Glorot initialization becomes unstable when iterated over long sequences in linear recurrent networks, thereby challenging this assumption.
Our main contributions are as follows:
Prior work has correctly noted that, in the infinite-width limit, the spectrum of large non-Hermitian Gaussian matrices (real or complex) used in Glorot initialization lies within the unit circle, in accordance with the circular law [Bai, 1997, Girko, 1985]. However, highlight a subtle but critical point often overlooked in prior works: the spectral radius of such matrices converges to one from above (see Section 4). In other words, the largest eigenvalue typically has modulus slightly greater than one, While this deviation is asymptotically small, it is sufficient to cause exponential growth in the hidden state over long sequences. This makes the standard Glorot initialization inherently unstable in long- sequence recurrent settings.
Instability of Glorot:
Rescaled Glorot initialization: To address this instability, we propose a simple rescaling of the weight matrix by dimension-dependeer constant. This adjustment lowers the probability that the radius exceeds one by shifting it approximately one standard deviation below its original expectation (see Section 5). As a result, the rescaled initialization prevents exponential growth in hidden state norms, even over long sequences. Consistent with prior work, our initialization favors eigenvalues close one from below, as slow signal decay is generally preferable amplification for memory retention in RNNs [Orvieto et al., 2023, White et al., 2004, Ganguli et al., 2008]. We therefore argue that this rescaling provides more principled and robust baseline for long-sequence modeling than the standard Glorot scheme.
Signal propagation over long sequences: We analyze the hidden state norm in linear RNNs with complex Glorot initialization, beginning with lower bound in the finite-width, finite-sequence setting under i.i.d. Gaussian inputs (Section 1)... In the infinitewwitth-ne widthh-and-length regime, our analysis guarantees exponential growth of the hidden state with time (see Section 6.1.).. Notably, we show that a sequence length of (VV)) (where n is the width) suffices to induce explosion. In contrast, in the infinite-width but fixed-length regime, the norm of the hidden states grows only slowly with time (see Section 6.1.1), failing to reflect senu-ne behavior.
Numerical experiments: We validate our theoretical findings empirically, evaluating both the statistical behavior of hidden states at initialization and the downstream performance of linear RNNs on real-world sequential data (see Section 7). As expected, standard Glorot initialization leads to signal explosion long-range reasoning tasks, while our rescaled initialization remains stable and trainable across multiple tasks.
Overall, our work introduces a simple yet theoretically grounded modification to Glorot initialization, establishing it as a principled baseline for long-range reasoning tasks. In doing so, we underscore the ofven-overlooked importance of the theoretical sarting-paaticularti assumptions about width, depth, and scaling- that underpin initialization schemes in recurrent architectures.

---

2 Related Work
Existing initialization theory primarily addresses feedforward networks with fixed depth. Extending these insights to recurrent architectures, especially in the long-sequence regime, remains an open problem. In this section, we survey works signal propagation, stability in RNNs, and recent efforts to understand the double-scaling limit of neural networks.
Initialization and signal propagation in feedforward networks. Glorot and He initialization schemes [Glorot and Bengio, 2010, He et al., 2015] were originally developed to prevent vanishing and exploding gradients in deep feedforward networks. Their theoretical justification typically relies on the analysis of signal propagation in the infinite-width, fixed-depth regime. This framework was formalized for fully connected networks by Schoenholz et al. [2016] and Poole et al. [2016], and later extended to other architectures [Xiao et al., 2018, Tarnowski et al., 2019, Gilboa et al., 2019]. Overall, signal propagation theory has had significant impact on network design and initialization, contributing both theoretical insights and strong empirical performance.
Initialization and stability in RNNs. Extending initialization theory to recurrent nets is challenging due to weight sharing across time and the unbounded effective depth introduced by repetitively applying the recurrent matrix. While signal propagation techniques have been adapted to vanilla RNNs and simple gated units in the infinite-width, fixed-length setting [Chen et al., 2018, Gilboa et al., 2019, Alemohammad et al., 2021], they do not fully capture long-sequence processing properties.
Recent work has begun to address lnfitite-length regimes in simplified settings, such as single-neuron or diagonal recurrent systems [Zucchet and Orvieto, 2024], revealing new instability mechanisms not accounted for by standard initialization theory. However, in the absence of a general frame- work, stability in earlier RNNs often enforced through architectural interventions -scch as LSTM [Hochreiter and Schmidhuber, 1997] and GRU [Cho et al., 2014] gating-alongsin alongside nor- malization layers [Ba et al., 2016, Gu et al., 2021], gradient clipping [Pascanu et al., 2013, Zhang et al., 2019), or spectral regularization during training [Jose et al., 2018, Zhang et al., 2018, Kanai al., 2017]. While these methods rely on nonlinear operations, the linearity of SSMs and LRUs is crucial for computational efficiency because it enables parallel scan operations. Consequently, modern recurrent models typically avoid complex gating (as in LSTMs and GRUs) and normalization within the recurrent block. This linearity, which prevents the use of earlier stabilization techniques, can reintroduce the problem of exploding signals [Orvieto et al., 2023, Gu et al., 2021]. To mitigate this issue, modern SSMs typically adopt structured initializations with predefined spectra to preserve stability of the recurrent unit [Gu et al., 2020, Fu al., 2023, Gu and Dao, 2023, Gu al., 2021]. Some works also proposed orthogonal or unitary initializations [Biegun et al., 2024, Henaff et al., 2016, Mikolov et al., 2014, Le et al., 2015] to preserve the norm over time and enable stable propaga- tion, but these can be difficult to maintain throughout training and may be suboptimal for memory retention [White et al., 2004, Ganguli et al., 2008].
Recent interest in linear RNNs (in particular, LRUs [Orvieto et al., 2023]) has renewed attention on initialization strategies for long-horizon regimes. Despite their effectiveness, these models are sensitive to initialization, and often adopt Glorot-style variance scaling as a baseline, inherited from feedforward networks. As we discussed, this practice lacks theoretical justification in recurrent contexts, especially under long sequences. A systematic understanding of initialization in linear recurrent models under such conditions remains an open problem.
Double scaling limit and random matrix theory. The theoretical challenge at the heart of signal propagation in long-range RNNs is the double scaling limit, where both the network's 's width n and the sequence length (or depth) t become large. This regime is natural for RNNs and other deep architectures, but it lies outside the reach of standard random matrix theory tools. Classical methods- Sachhods-uuhhaae as free probability, spectral theory, or genus expansions assume independent matrices or fixed depth, and thus fail when the same matrix is applied repeatedly [Tao,, 2012]. Although recent work has addressed the double-scaling limit in specific architectures [Hanin and Nica, 2020, 2019, Li et al., 2021, Roberts et al., 2022, Razin et al., 2024, Seleznova and Kutyniok, 2022], to the best of our knowledge, the recurrent setting has not been explored. Our work addresses this gap by characterizing the failure of Glorot-style initialization in linear RNNs under double scaling, and proposing a theoretically grounded correction.
3

---

3 Problem Setup
Before presenting our contributions, we review the theoretical background relevant to this work: the linear recurrent layer, Glorot initialization, and its spectral behavior in the infinite-width regime.
3.1 Linear Recurrent Layer
While classical RNNs rely on non-linear activations, recent work on Linear Recurrent Units (LRUs) has shown that linear recurrences can effectively model long-range dependencies in sequential data [Orvicto et al., 2023]. In addition to their empirical performance, linear RNNs allow for more tractable theoretical analyses, particularly with respect the spectral properties of the weight matrix.
In this work, we focus on this class of models.
Given an input sequence (Xi...., xx) with x; E R", the corresponding hidden states (h)..... h,) are computed via a time-invariant weight matrix W E Buxn as:
h = Wh
W x
where we set the initial state ho 0 for simplicity. Here, E N denotes the sequence length, which may be large in long-context applications. This formulation corresponds to a vanilla linear RNN without gating.
As evident from the power series expansion, each input X₃- k is amplified (or attenuated) by W meaning the overall dynamics are highly sensitive to the spectral radius of W. This highlights the susceptibility of linear RNNs to the exploding or vanishing gradients problem [Bengio et al., 1994, Pascanu et al., 2013], especially the sequence length increases. The initialization of W thus plays a critical role determining the stability of the model over long horizons.
Remark 3.1. A more general formulation of the recurrent layer includes an input projection term: h. = Wh--1 + Bx,, where B Rax* is a non-recurrent input matrix. In this work, we omit B without loss of generality, as it does not affect our main results. For analytical purposes, we treat x, as the effective input, implicitly assuming the identity or absorbed into the distribution of x,-
3.2 Glorot Initialization
Glorot initialization was originally proposed for feedforward networks to preserve the variance of activations and gradients during forward and backward propagation [Glorot and Bengio, 2010]. prescribes initializing dense weight matrices with i.i.d. entries drawn from normal distribution (O )²), where a = 22 + nout), and Rin, Pout denote the input and output widths of the layer. When applied to a square recurrent matrix W E R2x" this yields:
Welloro ((0, i.i.d.
In classical RNNs, the recurrent matrix W real and -on-Hermitian. However, some recent works including the LRU architecture [Orvieto et al., 2023]- employed complex-valued initialization, primarily because of its analytically convenient spectral properties. Motivated by this, we also consider the complex Glorot initialization:
Welcert
71 2
V2" Zz,
where Zı, ~ ~(0, i.d.d
This complex formulation enables using tools from oon-Hemmttian random matrix theory and yields cleaner spectral statistics than its real-valued counterpart. Below, we review existing random matrix theory results on the eigenvalue distribution of W under both real and complex Glorot initializations.
3.3 Spectral Distribution of Large Gaussian Matrices
There substantial body of mathematical literature analyzing the eigenvalue distribution of Gaussian random matrices in the large-n limit [Ginibre, 1965, Bai, 1997, Tao and Vu, 2008, Rider, 2003, Rider and Sinclair, 2014]. One of the most well-known results in this area the circular law, first conjectured by Girko [1985] and later made rigorous by Bai [1997] and others:

---

Theorem 3.2 (Circular law). Assume matrix W E Fux" with F E (R. is sampled from complex or real Glorot initialization. Then, the empirical distribution of the eigenvalues of W, denoted Pm (X), converges almost surely to a uniform distribution on unit disk in the complex plane:
An (A)
E C: {()
This result implies that, in the infinite-width limit, the spectrum of W is contained within the unit disk, and the spectral radius converges to 1. Consequently, each term in the linear recurrence formula (Eq. (1)) maintains bounded magnitude in expectation for any fixed k, suggesting that Glorot initialization yields stable signal propagation in the infinite-widhh setting.
Although prior work reported the empirical need to rescale Gaussia--nnitialized matrices to prevent hidden state explosion [Gu et al., 2021], the LRU architecture [Orvieto et al., 2023] motivates its initialization scheme based on Glorot and the circular law. particular, they aim to imitate the spectral behavior of dense Gaussian matrices via complex diagonalized initialization. This reflects a common assumption in the RNN literature that using Glorot is enough to ensure stability.
However, this assumption overlooks the fact that recurrent networks apply the same weight matrix repeatedly, and small deviations in spectral radius radius-though negligible each time step- cep-can can quickly compound over long sequences. In this work, we show that Glorot initialization is generally unstable when applied linear recurrent layers, especially under the double-scaling limit of large width and sequence length.
4 Spectral Radius Deviations in Glorot-Initialized Matrices
While the spectral radius of Glorot initialization converges one in the infinite-width limit, recent results from random matrix theory provide a more refined characterization of the spectral edge. In particular, Rider and Sinclair [2014] and Rider [2003] derive the asymptotic statistics for the largest eigenvalue of Gaussian matrices in the real and complex cases, respectively.
Theorem 4.1 (Spectral radius, Theorem 1.1 of Rider and Sinclair [2014] and Theorem I of Rider [2003]). Assume that W Fax" n is sampled from real or complex Glorot (Eqs. (2) and (3)), and denote the spectral radius by
==
max([N) : X is an eigenvalue of W).
(5)
Then, we have the following convergence in distribution:
44pm
[Amax - 1
Pn
G,
(6)
4n
n
where Pn :=
G a Gumbel variable with CDF Fa(x) = e -11--b/2))e and dg {0, 1} is one for real W and zero otherwise.
This result shows that the spectral radius converges to I from above, and the expected size of the largest eigenvalue of Glorot initialization for sufficiently large n is given by:
Pn 7 - ÖR In2
Ammax
W)] 11+
4n
V4pm.
where y is the Masch-Mascheroni constant. In words, the expectation of the largest eigenvalue exceeds one by ((1/vT), Additionally, Theorem 4.1 implies the following about the likelihood of stability: Corollary 4.2. For real and complex lies inside the unit disk satisfies:
W, the probability that the largest eigenvalue
PP P(PSmmx (i) (1)<1) < <P(D (P() (W) < 1
Pn 4n
V
2 e
cl-ba/2"
Thus, even though the bulk spectrum lies within the unit disk, eigenvalues outside the unit circle are likely for large n, particularly in the complex case.
5

---

Eigenvalues Radius
Matrix Powers
Hidden States
140
Glorot
435
10"
sfageensknn
Rescaled
330.
146
5.3
431
12.
135
18
175
380
Dadia Imagallide clugearabend
Figure 1: Top vs bottom: Glorot and rescaled initialization, respectively. Left: Empirical density of spectral radius for 100 independent samples of W E 5500x600 with entries drawn i.dd. from NO. 1/n). Glorot often yields eigenvalues with magnitudes exceeding one. Middle: Norms of |[W*x||2 function of k with each curve corresponding are sampled i.i.d. from (0,, Tsoo). We report the mean and variance over 50 random input samples. Glorot leads to exploding norms; the rescaled variant produces slowly decaying norms. Right: Norms of hidden states ||h: l22 in a recurrent layer with i.i.d. Gaussian inputs. The mean and variance are computed over 50 input realizations. Glorot initialization results in unstable (exploding) hidden states, while the rescaled initialization maintains stability over time.
different realization of W. Inputs
Connection to the double-scaling limit of neural networks. The correction to the largest eigen- value of order ((L/vn) implies that sequence lengths on the order of t O(vn" l+( are sufficient to induce exploding hidden states in linear LRUs. We make this statement precise in Section 6. This stands in sharp contrast the circular law intuition underlying Glorot initialization, which assumes stability based on the bulk spectrum being contained in the unit disk. However, our observation fits naturally into the emerging body of theoretical work on the double-scaling limit, where both network width and depth grow large [Hanin and Nica, 2020, Li et al., 2021, Roberts et al., 2022]. In this regime, has been shown that depth scaling of L = (n)) suffices to cause the explosion of fourth moments in MLPs [Seleznova and Kutyniok, 2022]. Thus, our observation with respect to the spectral radius does not only reveal flaw in the stability assumption behind Glorot initialization, but also quantifies the faster onset of instability in RNNs-arising from the repeated application of shared weight matrix compared to MLPs with independently sampled weights.
Rescaled Glorot Initialization for Long-Range Stability
We now introduce corrected initialization scheme with Gaussian matrices, aimed at stabilizing signal propagation in long-sequence settings. As shown in the previous section, the spectral radius of Glorot- initialized matrices often exceeds one by small but non-negligible amount of order O(1/ (()/vn). When the same weight matrix is applied repeatedly linear recurrent layers, this small bias is sufficient to induce exponential growth of hidden states. Our goal is to modify the initialization that the spectral radius remains slightly below one, thus suppressing explosion while simultaneously avoiding premature vanishing.
To achieve this, we propose a variance rescaling that shifts the spectral radius to lie below one with a specified confidence level. This shift is guided by the quantiles of the Gumbel distribution, which govern the asymptotic behavior of the spectral radius under Glorot initialization (see Theorem 4.1). Let ap denote the p-quantile of the Gumbel distribution, that is, P(G < a,) p. Then, for sufficiently

---

large n, the spectral radius of Glorot-initillied matrix satisfies:
G < Op
Amax (Wellorot) 2 1
1+
Pr 4n
+
Op
(9)
V4
Since scaling the entries of a Gaussian matrix rescales all its eigenvalues proportionally, we define our corrected initialization by reducing the variance accordingly:
(
Wrescaled ~N 00,
1
Pn
Gp
i..d.
(10)
4n
v
44pm
By design, this initialization has the following property for large enough
P (maxx (Wreseale)) )
<
= p.
(11)
and the following holds for the expectations:
E max (Wrescaled)]
[] = [ (Weloro))
||
1
ap
I
1.
(12)
4n
V4pmm
Choice of ap parameter. The value of ap should be chosen so that the spectral radius is likely to remain below one, but close enough to one to avoid rapid signal decay- vanishing dynamics limit the memory capacity. Importantly, the consequences of positive and negative deviations from the unit spectral radius are asymmetric: while slight contraction reduces memory gradually, a slight expansion leads to rapid exponential growth due the repeated accumulation of unstable terms in the hidden state. This asymmetry makes explosion far more damaging than mild vanishing.
In our work, we set a, = 8g In2 - π G/V corresponding to one standard deviation above the Gumbel mean, and yielding p 0.86. This choice significantly increases the likelihood that the spectral radius lies within the unit disk: from roughly p E 1/2 x 0.61 in the real Glorot case and p = e x 0.37 in the complex case, to p R 0.86 after rescaling. Notably, at the infinite- width limit, the largest eigenvalue modulus still converges one, preserving compatibility with the circular law.
Empirical validation. Numerical simulations in Fig. 1 (Left) clearly show that real Glorot- initialized matrices often exhibit a spectral radius exceeding one, whereas our corrected initialization makes this much less likely. As shown in Fig. 1 (Middle), the proposed scaling also prevents the norm ||W*x| from exploding for sequence lengths of order k = ((T)), in contrast to the standard Glorot initialization, where ||W*'] grows rapidly. Furthermore, we compare the effect of both initializations on the evolution the hidden states, demonstrating that our scaling mitigates instability in both **x x and the hidden states ha- A rigorous analysis covering both finite and infinite input sequences provided in Section 6, theoretically supporting the observed numerical behavior.
Explosion of Hidden States with Long Sequences
We analyze forward signal propagation in linear recurrent layers initialized with complex Glorot. Our key result is a lower bound on the hidden state norm in the setting of finite width and finite input length. By examining how this bound behaves two regimes-one one where width tends to infinity with fixed sequence length, and another where both grow simultaneously-wve we show that hidden states remain stable the former, but grow exponentially in the double-scaling regime.
6.1 Lower Bound on Hidden State Growth
Since the distribution of cigenvalues of a complex Glorot matrix is known exactly from random matrix theory and has relatively tractable form [Tao, 2012], we can use lower bound the growth of the hidden state summands for finite n and k in the following theorem (see proof in Appendix B).
Theorem 6.1 (Variance lower bound). Suppose W E C is sampled from a complex Glorot initialization (Eq. (3)), and x ~ NO, I,) is independent of W. Then the following holds for any k n. E N:
1
(n + R:).
E
(13)
n 1+ ml

---

Suppose the inputs sequence (xi....,X) is such that each input is sampled i..d. from IO) Then the above result is directly related the hidden state as follows:
E b
Σ E[x] K=
W X
Σ E BBllWixxxi x Am=0
(14)
Since the result is derived for finite width and sequence length, both the infinite-wide width and double- scaling regimes arise as special cases of this general setting. Finite-width results remain relatively rare in deep learning theory, as they often require tedious combinatorial arguments- path counting techniques used for ReLU networks [Hanin and Nica, 2019]. In contrast, our proof avoids path counting entirely and relies instead on the distributional density of eigenvalues, enabling concise and tractable analysis.
as the
In the following, we examine the implications of this result for the behavior of hidden states both the infinite-width and double-scaling regimes.
6.1.1 Stability with Infinite Width and Finite Length
We first consider the fixed-length and infinite-width regime, i.e. ((1) and oc, which is the prevalent setting in the literature. In this regime, we get the following for a single summand of the hidden state variance, for each ls
lim n
(15)
k: ++
From this, follows that the hidden state variance lower bound is described by the harmonic series:
lim 1 BlIlh, 12
1
log(t 1) + 2t
O
(16)
F2
The hidden states exhibit logarithmic growth with respect to t. Although this implies that the variance of the hidden states is theoretically unbounded, in practice the growth remains mild and is unlikely to significantly disrupt stability when sequences are short. This finding aligns with existing literature on signal propagation in recurrent networks. However, benign behavior does not extend to the more realistic scenario of long sequence lengths.
demonstrate in the next section, this
6.1.2 Instability with Infinite -eet--Withhhh-eeee
We analyze the behavior in the infinite-length regime, which is the central focus of our work. To obtain a meaningful bound in this setting, we consider the limit as t, n + oo with t = θ(vn) Under this scaling, use the following asymptotic approximation:
(k n)!
II
n +
II d=1
1
d
60 2n D 1//2m
(17)
d=1
where we observed that the terms d satisfy 0 < 1 for all d d k < t and vanish in the limit. This approximation can be seen as the following first- expansion of the logarithm, and we make more precise Appendix B.3:
a
n
d
k(k
log
1
log
1+
(18)
π
n
n2
2n
Therefore, under standard Glorot initialization, the expected norm || [ grows with the power k < t when the sequence length is sufficiently large unlike in the infinite- width regime (Eq. (15)):
lim
A:
W /2n
(19)
V/V
This asymptotic growth matches empirical observations both the real and complex settings (see Fig. 1 and Appendix A). Consequently, we show in Appendix B.3, the hidden state norm for large t and n can be approximated follows:
exp(k2/2n)
22 2n
lim 1,0000
E
exp
(20)
n
t/
k=0
8

---

Hence, when the sequence length t is at least of order (Vv)), the hidden state norm begins to diverge and this behavior becomes more pronounced under more aggressive scaling. For instance, when t = avn ++ for any c > O, the lower bound diverges rapidly.
Remark 6.2 (Real case). While the analysis in this section focuses on the analytically tractable complex Gaussian setting, we note that real Gaussian matrices exhibit qualitatively similar behavior. This is due to the close agreement in spectral statistics between real and complex non-Hermitian Gaussian ensembles [Tao and Vu, 2008]. However, the spectral density in the real case involves more intricate expressions, making analogous derivations significantly more tedious. We provide supporting numerical simulations and an additional theoretical discussion of the real case in Appendix B.2.
To summarize, our findings underscore the need to rescale Glorot initialization in order to ensure sta- bility during long-sequence processing, especially in the double- double-scaling regime. Next, we demonstrate the applicability of our findings to the training of linear recurrent networks.
7 Experiments
In addition to our empirical validation of signal propagation at initialization (Figure 1), we evaluate the effectiveness our rescaled initialization on standard long-range sequence modeling benchmarks.
We conduct experiments on three classifica- tion tasks drawn from the Long Range Arena (LRA) benchmark (Taa et al., 2020]: Sequential CIFAR-10, 10, IMDB, and ListOps. In the Sequen- tial CIFAR-10 task, image pixels are flattened into sequences of length 3K. ListOps consists of nested arithmetic expressions over single-digit integers, with a 10-class output and maximum sequence length of 2K. IMDB is a binary senti- ment classification task with input sequences of up to 8K tokens.
Table I: Classification accuracy for dense and di- agonal linear recurrent models. Note the benefit of using our rescaling.
CIFAR-10
IMDB
ListOps
Glorot
X
Glorot/2
76.42 ncos
88.74
46.21
:: ones
Rescale
81.54= b009
87.55 0000
47.51
+ 0001
Diag, Glorot
Diag. Glorot/2
71.6t
86.67 usze
49.01 00844
Diag. Uniform
81.47
86,34 nons
48.93
We compare our rescaled initialization against the standard dense Glorot initialization, and naive non-exploding baseline of Glorot/2 (i.e.,
Diag. Rescaled
79.53+
87.31 0.001
49.49
W AO 1/2n)). Additionally, we implement diagonalized variants of these initializations, which improve computational efficiency in both runtime and memory consumption. These are more closely aligned with the LRU architecture [Orvieto et al., 2023]. To evaluate our method in the diagonal setting while preserving spectral characteristics, we sample a dense matrix and initialize the diagonal using its cigenvalues. Additionally, include circular-law bascline in which diagonal entries are sampled uniformly from the complex unit circle, following prior work [Orvicto et al., 2023].
We release the code with the paper, and provide more details on the experimental setup in Appendix C.
Results. Table summarizes our results. As expected, models initialized with standard Glorot failed to train due to exploding hidden states. Across the remaining baselines, our rescaled initialization achieved the highest accuracy out of 6 settings, outperforming both Glorot/2 and the unit-circle- based diagonal initialization. These results demonstrate that our simple rescaling provides a practical and theoretically grounded improvement over the standard approaches. Although these experiments are conducted on linear RNNs and do not directly compete against state-of -he-thrhtrt LRUs or SSMs, they offer important initial evidence supporting theoretically princepled initialization. Additional comparisons with prior RNN stabilization techniques and an investigation of parameterization effects appear in Appendix D.
Conclusion
We revisited the widely used Glorot initialization the context of linear RNNs and showed that, contrary to common assumptions, it leads to instability when applied to long input sequences. Our findings highlight that initialization schemes derived under the infinite-width, fixed-depth regime may not generalize to the anfinite-input-lent setting characteristic of recurrent architectures. This work

---

contributes to the growing body of theoretical research on the double-scaling limit in deep learning. and to the best of our knowledge, is among the first explore this regime in the context of RNNs.
Limitations and Future Work. Our analysis focuses on the analytically tractable setting of complex Gaussian initialization in linear recurrent networks. A key limitation that it provides only lower bound on the hidden state norm, derived under the simplified assumption of uncorrelated Gaussian inputs. Extending this analysis to more realistic scenarios with temporally correlated imputs and refining the bound to an exact characterization -oorll likely require addressing open problems in random matrix theory, particularly regarding the spectral behavior of high powers of random matrices.
Another promising direction is the development of a rigorous theoretical framework for common initialization heuristies used in practice, such those that enforce minimum spectral radius or constrain eigenvalues to lie predominantly along the positive real axis, A deeper understanding of these strategies could yield more stable and effective architectures for long-sequence modeling.
Finally, like many existing works, our analysis is limited to the behavior of networks at random initialization. While this provides foundational insight, a natural next step is to study the optimization dynamics of recurrent models in the coinite-input-ii length regime.
Acknowledgement
N. Bar and R. Giryes thank KLA and The Center for Al & Data Science at Tel Aviv University (TAD) for supporting this research. G. Kutyniok acknowledge support by the gAIn project, which is funded by the Bavarian Ministry of Science and the Arts (StMWK Bayern) and the Saxon Ministry for Science, Culture and Tourism (SMWK Sachsen).
References
S. Alemohammad, Z. Wang, R. Balestriero, and R. Baraniuk. The recurrent neural tangent kernel. In The International Conference on Learning Representations, 2021.
J. Ba, J. R. Kiros, and G. Hinton. Layer normalization. arXiv preprint arXiv.1607.06450 2016.
Z. Bai. Circular law. The Annals of Probability, 25(1):494-529, 1997.
Y. Bengio, P. Simard, and P. Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2) 1994.
K. Biegun, R. Dolga, J. Cunningham, and D. Barber. Rotrnn Modelling long sequences with rotations. arXiv preprint arXiv:2407.0777
2024.
S.-S. Byun and P. J. Forrester. Progress on the study of the ginibre ensembles i: Ginue. arXiv preprint arXiv:2211.1622. 2022.
M. Chen, Pennington, and S. Schoenholz. Dynamical isometry and a mean field theory of mnss: Gating enables signal propagation in recurrent neural networks. In International Conference Machine Learning, pages 873-882, PMLR, 2018.
K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078, 2014.
S. De. S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y. Chen, S. Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint axXiv:2402.19422 2024.
A. Edelman. The probability that random real gaussian matrix haskreal eigenvalues, related distributions, and the circular law, J. Multivar. Anal., 60(2):203-232, Feb. 1997. ISSN 0047-259X. doi: 10. 1006600/j. 1996. 1653, URL http:: //dp::://oi...e org. 10..00///aaa........ 1996. 1653.
A. Edelman, F. Kostlan, and M. Shub. How many eigenvalues of random matrix are real. Journal of the American Mathematical Society, 7:247-267, 1994.
10

---

D. Y. Fu, T. Dao, K. K. Saab, A. W. Thomas, A. Rudra, and C. Re. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh International Conference on Learning Representations, 2023.
S. Ganguli, D. Huh, and H. Sompolinsky. Memory traces in dynamical systems. Proceedings the national academy of sciences, (05(48):18900-1)))))) 2008.
F. A. Gers and E. Schmidhuber. Lstm recurrent networks learn simple context-free and context- sensitive languages. IEEE transactions on neural networks, 1333:1333-1340, 2001.
D. Gilboa, B. Chang, M. Chen, G. Yang, S. S. Schoenholz, E. H. Chi, and J. Pennington. Dynamical isometry and a mean field theory of istms and grus. arXiv preprint arXiv:1901.0895/ 2019.
J. Ginibre. Statistical ensembles complex, quaternion, and real matrices. Journal of Mathematical Physics, 6(3):400-449 1965.
V. L. Girko. Circular law. Theory of Probability Its Applications, 29(4):694-706, 1985.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752, 2023.
A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33:1474-1487, 2020.
A. Gu, Goel, and Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2 11..099,,, 2021.
B. Hanin and M. Nica. Finite depth and width corrections the neural tangent kernel. arXiv preprint arXiv: M90.//999000.000 1999, 2019.
B. Hanin and M. Nica. Products of many large random matrices and gradients in deep neural networks. Communications Mathematical Physics, 376(1):287-322, 2020.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level per- formance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
M. Henaff, A. Szlam, and Y. LeCun. Recurrent orthogonal networks and long-memory memory tasks. In International Conference on Machine Learning, pages 2034-2042. PMLR, 2016.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98):173351188) 1997.
C. Jose, M. Cissé, and Fleuret. Kronecker recurrent units. In International Conference on Machine Learning, pages 3380-2389 PMLR, 2018.
S. Kanai, Y. Fujiwara, and Iwamura. Preventing gradient explosions in gated recurrent units. Advances in neural information processing systems, 30, 2017.
Q. Le, N. Jaitly, and G. E. Hinton. simple way to initialize recurrent networks of rectified linear units. CoRR, abs/ 150/1000400 2015. URL http://arxiv. .org/ ht////////100....4
M. Li, M. Nica, and D. Roy. The future is log- gaussian: Resnets and their infnite-depph-and-weee limit initialization. Advances Neural Information Processing Systems, 34:7852-7864 2021.
T. Mikolov, M. Karafíát, I.. Burget, J. Cernocký, and S. Khudanpur: Recurrent neural network based language model. In Interspeech, volume 2, pages 1045-1048. Makuhari, 2010.
Mikolov, A. Joulin, S. Chopra, M. Mathieu, and M. Ranzato. Learning longer memory in recurrent neural networks. arXiv preprint arXiv:1412.7753, 2014.

---

A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning. pages 26670-26630 PMLR, 2023.
R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318, Pmlr, 2013.
B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances neural information processing systems, 29, 2016.
S. S. Rangapuram, M. W. Seeger, J. Gasthaus, L. Stella, Y. Wang. and T. Januschowski. Deep state space models for time series forecasting, Advances neural information processing systems, 31, 2018.
N. Razin, Y. Alexander, E. Cohen-Karlik, R. Giryes, A. Globerson, and N. Cohen. Implicit bias of policy gradient in linear quadratic control: extrapolation to unseen initial states, arXiv preprint arXiv:2402.0P5 077777 2024.
Rider. A limit theorem the edge of a non-hermitian random matrix ensemble. Journal of Physies A: Mathematical and General, 36(12):3401, 2003.
B. Rider and C. D. Sinclair. Extremal laws for the real ginibre ensemble. The Annals of Applied Probability, pages 1621-1651, 2014.
D. A. Roberts, S. Yaida, and B. Hanin. The principles of deep learning theory, volume 46. Cambridge University Press Cambridge, MA, USA, 2022.
D. Salinas, V. Flunkert, J. Gasthaus, and Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International journal of forecasting, 36(3):1181-1199) 2020.
S. S. Schoenholz, Gilmer, S. Ganguli, and J. Sohl-Dickstein. Deep information propagation. arXiv preprint 1611v:1611.0123: 2016.
M. Seleznova and G. Kutyniok. Neural tangent kernel beyond the infinite-width limit: Effects of depth and initialization. In International Conference on Machine Learning, pages 19522-19560. PMLR, 2022.
M. Sundermeyer, R. Schlüter, and H. Ney. Lstm neural networks for language modeling. In Interspeech, volume 2012, pages 194-197, 2012.
M. Sundermeyer, H. Ney, and R. Schlüter. From feedforward to recurrent Istm neural networks for language modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(3): 517-529, 2015.
T. Tao. Topics in random mairix theory, volume 132. American Mathematical Soc., 2012.
T. Tao and V. Vu. Random matrices: the circular law. Communications in Contemporary Mathematics, 10(02):261-307) 2008
W. Tarnowski, P. Warchol, S. Jastrzebski, J. Tabor, and M. Nowak. Dynamical isometry is achieved in residual networks in universal way for any activation function. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2221 -230. PMLR, 2019.
Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler. Long range arena: benchmark for efficient transformers. arXiv preprint arXiv:201 (XXv:2011.04000 040006. 2020.
O. L. White, D. Lee, and H. Sompolinsky. Short-term memory in orthogonal neural networks. Physical review letters, 92(14))148102 2004.
L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington. Dynamical isometry and mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. In International Conference Machine Learning, pages 5393-5402. PMLR, 2018.
12

---

J. Zhang, Q. Lei, and I. Dhillon. Stabilizing gradients for deep neural networks via efficient svd parameterization. In International Conference on Machine Learning, pages 5806-5814 PMLR, 2018.
J. Zhang, T. He, S. Sra, and A. Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. arXiv preprint arXiv:7905. 1881, 2019.
N. Zucchet and A. Orvieto. Recurrent neural networks: vanishing and exploding gradients not the end of the story. In Proceedings of the Thirty-Eighth Annual Conference on Neural Information Processing Systems. NeurIPS, Oct. 2024. URL https: //wxivavvvv////////////a////aaaa org/ abs/2405. 21064.
N. Zucchet, R. Meier, and S. Schug. Minimal implementation of linear recurrent units, 2023. URL https:/ /ttp:////tttbbttt htt..//icc..aaaaaccccte//aiaaaaa LRU. Accessed May 2025.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Throughout the paper, develop the proposed initialization scheme and rigorously analyze the hidden state dynamics, as outlined in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We explicitly discuss the limitations of our work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
13

---

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results are stated with their set of assumptions in Sections 3-6. Complete proofs are provided in Appendix B, and we have carefully verified their correctness.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed descriptions of all experimental settings, including model architectures, initialization procedures, hyperparameters, and evaluation metrics. This information is sufficient reproduce the main experimental results that support our claims. We also include our code files in the supplementary material to facilitate exact reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
14

---

While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include our code and instructions, enabling faithful reproduction of all experiments. All datasets used are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. wttpp.ccppppiiiipeepptpppppppppp htc//ublic/aae weaiicc/gges///eeeee (CodeSubmissionnP)
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide full training and evaluation details. Where possible, adopt hyperparameter settings from prior work; otherwise, we perform grid search. These details are specified in the appendix to ensure clarity and reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report standard deviations over three random seeds for experiments real datasets, and use a large number of seeds for numerical simulations to ensure statistical reliability. Exact details, including the number of runs and error bar definitions, are provided in the main text and figure captions.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the compute resources used for the innerunents-iicluudinn hardware type, memory, and runtime estimates in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https // httpa//pp...p.bbiiitttseeeeeeeee
16

---

Answer: [Yes]
Justification: Our work involves theoretical and empirical analysis without ethical concerns, and adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is theoretical in nature and does not directly raise any foresecable societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve the release of models or data with high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
17

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification; All external assets used (e.g., datasets) are properly credited, and their licenses and terms of use have been respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We do not introduce assets beyond minimal training script used to reproduce our results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
18

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We not use crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: We used LLMs only for standard writing assistance and visualizations, with impact on the core methodology scientific contributions.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
19

---

A Numerical Simulation for the Complex Case
Along with the experiments illustrating the behavior of the hidden state and its summands for real Glorot matrices in Section 5, include numerical simulations for the complex case here. The results are shown in Fig. 2. As in the real case discussed the main text, complex Glorot initialization leads to explosion in the hidden state norm, while proposed rescaling effectively mitigates this.
Eigenvalues Radius
Matrix Powers
Hidden States
139
Glorot
IM All
167
832-
Rescaled
10
5334
11h-11
18
330
52A
168
15
Dadus Inagsitade (Eegearakon
Figure 2: Top bottom: Complex Glorot and rescaled initialization, respectively. Left: Empirical density of spectral radius for 100 independent samples of W E C500x000 with entries drawn i.i.d. according complex Glorot (Eq. (3)). Glorot often yields eigenvalues with magnitudes exceeding one. Middle: Norms of W x 2 as a function of k, with each curve corresponding to different realization of W. Inputs are sampled i.i.d. from (0, Ilsoo). We report the mean and variance over 50 random input samples. Glorot leads to exploding norms; the rescaled variant produces slowly decaying norms. Right: Norms of hidden states |ا in recurrent layer with i.i.d. Gaussian inputs. The mean and variance are computed over 50 input realizations. Glorot initialization results in unstable (exploding) hidden states, while the rescaled initialization maintains stability over time.
B Proofs and Additional Theory
this section, we present formal proofs of the main theoretical results discussed the paper. We also provide additional analysis and discussion related to real Glorot matrices, as detailed in Section B.2.
B.1 Variance Lower Bound (Theorem 6.1)
We provide proof for the lower bound on the growth of [ in the fnite-width and -length setting stated Theorem 6.1.
Theorem (Variance lower bound). Suppose W Caxn is sampled from complex Glorot ini- tialization (Eq. (3)), and x ~ (0, 1n) is independent of W. Then the following holds for any k N:
nn +)
E
(21)
T2 k
n!
Proof. We begin by applying a standard identity involving expectations and traces (sometimes called Hutchinson's trick):
B|| x
||
E
( W
Tr(EExx ([( " E(Th WW)),
where used the independence between and W, and the fact that E[xx J I.. This reduces the problem to estimating the expected trace Tr((
20

---

Next, we analyze this trace via the Schur decomposition. Any complex matrix W admits a decompo- sition of the form:
W UTU
where U unitary and T is upper triangular with the eigenvalues of W on its diagonal. This can be derived from the eigenvalue decomposition via Gram- Sraa-Schmidt orthogonalization of the eigenvectors.
Raising both sides to the k-th power gives:
W UUT-U-
and thus:
Tr((W")''' = Tr((TT)*TT))
Because T is upper triangular, the diagonal of T consists of X5, where X are the eigenvalues of W. Consequently, the trace splits as:
Tr((W")' w
Σ
A
2h
Equality holds when W normal (e.g., Hermitian), so the bound tight.
To compute the expectation of this lower bound, we use the known eigenvalue density pa (A) for non-Hermitian Gaussian matrices:
Pr (A)
1 V
R
1
This expression is due to Ginibre [1965]; see also Proposition 2.2 in Byun and Forrester [2022] for modern treatment. Since we use Glorot initialization (i.e., standard Gaussians scaled by 1/vn), we apply the change of variables À + Vnd to preserve the correct scaling.
Using this density, we compute the expected eigenvalue moments:
BIJA
2k
a d(vTi))
We now evaluate the integral in polar coordinates:
2x
d(vnn)
2(ck+s -n² /nr dl /nr) d8
Jo
2n
Ra d Rm) do (Rn)
2 k+ J
30
T(k s)!
[[k
1)
nk=
n kk=
Plugging this back in:
1
+ s)! (k
(k: n))
E 2
2k
n
s!
n k +
n!
8=0
Finally, summing over all i and plugging into the lower bound, we obtain:
E(Tr((W')
2 nk
1 n
(n + k)!
k+ n!
which completes the proof.
21

---

B.2 Real Glorot Initialization
While our discussion of the spectral radius in Section 4 and the rescaling strategy in Section 5 covered both real and complex Gaussian initialization, the results on hidden state growth in Section were presented only for the complex Glorot case. This restriction is due to the greater analytical tractability of the spectrum in the complex Gaussian setting.
Although camplex-valued initialization is increasingly common in RNNs (e.g., Orvieto et al. (2023]), most classical results for feedforward networks focus on real Glorot initialization (Eq. (2)). this section, we examine how our signal propagation results extend to the real-valued case and outline the specific analytical challenges it presents.
The main difficulty in analyzing the eigenvalue moduli for real Gaussian matrices is the lack of perfect spectral symmetry with respect to the real and imaginary axes. Unlike in the complex case, the cigenvalue distribution for real matrices is slightly biased toward the real line, and the empirical densities of real and non-real eigenvalues require separate treatment [Edelman et al., 1994]. We formalize this distinction in the following two propositions, which give the density expressions for each spectral component.
Proposition B.1 (Real-eigenvalue density [Edelman et al., 1994, Cor. 4.3]). Let W E Ruxn be sampled from real Glorot initialization. Then the density of a real eigenvalue À E R of W
Vn
l(n - -1, I(n-1)
(Vnd)" I²2 7( [1--)) nd²
Pn
real (A)
Creaal
2TT
T(g)2
r( r(m=1)
where 2s) = 12 1--le= dt is the upper-incomplete gamma function, ((, -e t--= &
the lower incomplete gamma function, and we use the shorthand [(s) := [s,, 0), and Creal ((yn) is normalizing constant equal the expected number of real eigenvalues.
Proposition B.2 "Complex-eigenallll density [Edelman, 1997, Thm. 6.2]). Ler W Rexa be sampled from real Glorot initialization. The density of complex eigenvalue X = x + by of W in the
upper half-plane (y > 0) is
2n coomplex V
2 " exp/n (12-22)) orfe 2n
n-2
(n(x² + +²))*
complex (x,y)= Na
Σ
y y> 0,
h!
where erfe(x) == 3 1 e- dt is the complementary ervor function, and C Ccomplex (n) is normalizing constant equal to the expected number of complex eigenvalues.
Note that the above proposition considers only cigenvalues in the upper half-plane, as the complex eigenvalues of real Gaussian matrices occur in conjugate pairs. Consequently, the spectral distribution in the lower half-plane mirrors that of the upper half.
As a result, computing the expectation of the 2k-th moment of the spectrum-analogonn analogous to the approach used in the proof of Theorem 6.1.1 for the complex Gaussian case reduces to the following expression in the real case:
Proposition B.3 (Expected k-tth absolute moment). For any integer 2 0, the expectation of the 2k-th absolute moment of the spectrum of W can be computed as follows:
Creal
Coomplex
Sf y20 (x2 + 52)a pa
A 22
|
real (x) dr + 2k
complex (x, y) da dy
n
real eigenvalues
complex conjugate pairs
Unlike the complex Glorot case, there is no known closed-form expression for the corresponding integral in the real setting. Nevertheless, numerical results indicate that the behavior is remarkably similar across both cases. As shown in Fig. 3, the expected hidden state growth under real and complex initialization closely aligns. While the mean spectral radius for real matrices is slightly lower than that of complex matrices, the real case exhibits a noticeably heavier tail in its distribution.
This suggests that although real Glorot initialization is, on average, less prone to instability (as discussed in Section 4), it can lead to more rapid exponential growth when the spectral radius does exceed one.
22

---

Max [A] (n=500, samples=5000
Real Ginibre
350
Complex Ginibre
250
Frequency
200
100
50
1.00
1.02
1.04
1.06
1.08
1.10
Max jelgenvalue|
Figure 3: We show histograms of the maximal eigenvalue sizes for both the real and the complex Glorot ensembles (i.e Glorot initializations). One can see that the behavior is similar overall, and furthermore that the upper bound in Eq. (9) is indeed satisfied with high probability (90.2 percent in the real case, and 99.2 in the complex case) NB: I'm not sure if the reference is correct. maybe 1- --pi is needed. There are however two notable differences: the mean of the real Glorot matrices is slightly smaller than in the complex case, but nevertheless the right tail of the distribution is longer. While the typical size of the largest eigenvalue is similar both cases, this latter feature results in the expected k-th absolute moment being larger in the real case for large k.
10a
Complex
104
10a
10a
50
75
100
123
150
175
200
k
Figure 4: Norms of hidden states computed over 100 realizations of W and independent inputs. Both real and complex Glorot initializations lead to similar unstable (exploding) hidden states.
|lhulla ||2 in a recurrent layer with i.i.d. Gaussian inputs. The mean
B.3 Infiite-Width--nddLeeee Limit
In this section, formally justify the asymptotic expression for the lower bound of || and [h' in the siihite-wddh--nd-leenttt regime. These result were introduced informally in Section 6.1.2.
We first show that the bound on |[| [[[[*x in (19(19) indeed holds the double-scaling limit by the following lemma:
23

---

Lemma B.4. Suppose n, + 2o with k = a n2 for some fixed a E R. Then we have:
1 (k +)! n
22/2
(22)
n!
This type asymptotic expressions is well-know in mathematical analysis and arises frequently in the study of factorial approximations and exponential limits.
Proof. Let h: av/n for some fixed
0. We begin by rewriting the expression:
(n + k)! 1 +
n
n!
d=1
Taking logarithms and using the Taylor expansion log(1 + x)
=
x
-/2
+
we write:
k
d
d
de
log
1
+
n
nta
d=1
d=1
Since d S avn, we have d//n² S a² nn, so:
log
1+
d'
1
Σ
d
0
0
k
n1
n
Using Ed=
d== d k(++1) 2
we obtain:
ann + ava
a²
1
log 1
+0
+0
2n
vn
2
Vn
Exponentiating both sides yields:
nn k)!
a
)1
(11
1
(²²)
011+00
(1
(1+0
(1+0(2))
)
n!
exp
2
+0
|n
exp
2
Vn
which completes the proof.
We next formally justify the exponential growth of the hidden state variance Eq. (20):
E|[[h'| presented in
Lemma B.5. Fix 0 < E < and set t = nV Define
layal
exp( k2 /2m)
Sp(e,a)
(23)
k=levn
Then
- dr + O (n-1/2)
20
ax
(24)
and the integral admits the asymptotic estimate
2 = 2
= ea²12 (1 1 ((a0))))) + 0,(1),
- Ei( (B
dx
(25)
a²
where Ei denotes the exponential integral. In particular,
12 (1 + (2)) + 0.(1) Oc,o (111/)).
a,
exp
avn.
(26)
2n
24

---

Proof. Let An == =n n/2 and f(x) = 12 12/2,, which is C' on [c, Writing 24 := k///..
aynn
Su(c,a)
Σ
f(x)) An:
(27)
Am keLev/]
ie., left Riemann sum for 1 f(x dæ with mesh A.. By the mean value theorem on each subinterval together with endpoint rounding,
ax(e,) f(z) dx s ((a c) sup
[f(x)++ sup If(z)) An Oe,a(n (n
2E(c,o]
sE(c,o]
(28)
which proves (24).
For (25), substitute y = == 12 (so dy
x dx) to obtain
12 dx
1 1
1 r² e dy
- Ei( -
(29)
2/2 y
where Ei(z) == PV f dt. Using the standard asymptotic expansion Ei(2) ~ ct ()) yields
+
en²/2
2 Ei (
(1+
Ei = Oc(1),
(30)
a²
which gives (25). Replacing a² by f/n in the leading term gives (26).
C Implementation Details
Our implementation is provided anonymously at the following link, and is based on the publicly available minimal-LRU codebase originally developed by Zucchet et al. [2023]. All experiments conducted using the Adam optimizer, with weight decay applied only to non-recurrent parameters. We employ a cosine annealing learning rate schedule, starting from base learning rate of 10-3, with warm-up steps specified in Table 2. All models are trained with six recurrent layers. Model dimensions and additional hyperparameters are detailed in Table 2.
Table Hyperparameters used for each dataset.
Hyperparameter
CIFAR-10
IMDB
ListOps
Warmup End
18
7
D
512
192
256
H
384
256
192
Batch Size
50
32
32
Epochs
180
65
LR Factor
0.025
0.025
0.05
Dropout Probability
0.05
0.05
Compute resources and training time. All experiments were conducted on single NVIDIA GeForce RTX 2080 GPU, with peak memory usage reaching up to 9 GB. Training the full dense RNN CIFAR-10 10 required approximately 26 hours, while training times for other datasets were notably shorter. The diagonal variant was more computationally efficient and trained faster; for instance, training CIFAR- 10 completed in approximately 21 hours. All our experiments limited to IK GPU hours on a single device.
D Additional Experiments
We complement the main results by comparing our approach to two stabilizing strategies recurrent models: (i) Layer Normalization [Ba et al., 2016] and LRU parametrization [Orvicto et al., 2023].
25

---

Layer Normalization (LN) (Table 3), LN is standard normalization technique in classical RNNs that stabilizes training by normalizing hidden activations within each time step [Ba al., 2016]. In our experiments setup in Table 3, we insert I.N after each recurrent matrix multiplication in otherwise linear RNN block. While LN can mitigate optimization instabiliites, it breaks striet linearity in the recurrence, undermining the scan-based parallelism that makes linear models efficient.
We observe that LN enables Glorot to train but at a notable computational overhead; moreover, the final performance does not surpass our rescaled initialization without LN, For example, in the dense setting our rescaled model attains the best accuracy (87.55% while training = 29% faster per epoch (09:08 vs. 12:51). In the diagonal setting, LN brings Glorot to 80.64% still well below our rescaled model without LN (87.31%) and with additional runtime cost. Overall, these experiments suggest that while LN can mitigate exploding states for otherwise unstable inits, it sacrifices efficiency; our simple, theory-guidee rescaling attains higher aceuracy without normalization and preserves the linear-time scan advantages.
Table 3: Accuracy and time per epoch with IMDB dataset.
Accuracy
Training Time [min]
Glorot Dense LN
86.82
12:51
Rescaled Dense LN
86.7
12:51
Glorot Dense
Rescaled Dense
87.55
09:08
Glorot Diag LN
80.64
3:25
Rescaled Diag LN
87.28
3:25
Glorot Diag
Rescaled Diag
87.31
2:58
LRU parametrization (Table 4). We evaluate scheme on the complete LRU architecture [Orvi- eto et al., 2023], which differs from standard linear RNN in both initialization and parametrization. Holding the LRU parametrization fixed but replacing dedicated initialization with our rescaled scheme yields a modest drop in accuracy relative to the original LRU setup. This is expected, as the LRU parametrization and initialization were designed to be used in tandem. Nevertheless, our scheme remains competitive, highlighting that principled rescaling can transfer across parametrizations and providing strong baseline for future co-designed variants.
Table 4: Accuracy results with LRU parametrization with IMDB dataset.
LRU
87.32
Diag. Uniform
85.84
Diag. Rescaled
85.67
26


Reviews:
[{"review_id": "PkKt9bNUxG", "reviewer": "Reviewer_Agai", "summary": "This paper revisits the widely used Glorot initialization scheme in the context of Recurrent Neural Networks (RNNs), focusing on long-range linear recurrence architectures. The authors conduct a theoretical and empirical investigation into the stability of Glorot initialization, especially in deep or long-sequence settings.\n\nThe key insight is that Glorot initialization becomes unstable for sequences longer than √d, where d is the hidden state dimension. While Glorot aims to ensure variance stability and spectral norm convergence to 1, the authors highlight a transient phase during training where the spectral norm exceeds 1, causing unstable dynamics, particularly in deep RNNs.\n\nTo address this, the paper proposes a spectral radius-aware modification to Glorot initialization, explicitly bounding the spectral norm of the weight matrix below 1 to guarantee long-range stability and mitigate hidden state explosion.\n\nThe theoretical claims are validated through empirical results on Long Range Arena (LRA) benchmarks, including sequence classification tasks such as ListOps, seq-CIFAR, and IMDB. These experiments demonstrate that the proposed initialization leads to improved stability and convergence in long-sequence settings.", "questions": "none", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n* Clear theoretical contribution:\n The paper provides a well-motivated and mathematically grounded critique of Glorot initialization when applied to long-range recurrent models. The authors identify a specific limitation - transient spectral norm inflation - and rigorously link it to practical instability in deep or long-sequence architectures.\n\n\n* Novel solution grounded in spectral theory:\n The spectral radius-aware variant of Glorot initialization is simple, well-motivated, and practically useful. It’s presented in a way that makes it readily implementable in standard training pipelines.\n\n\n* Insightful hidden state norm analysis:\n The contrast between stability in the infinite-width, shallow limit and practical deep architectures is clearly articulated, helping to bridge theory and practice.\n\n* Comprehensive empirical validation:\n The paper evaluates the proposed method across a variety of standard long-range sequence tasks in the LRA benchmark. The results consistently show improved stability and, in many cases, better task performance or faster convergence.\n\nWeaknesses and Areas for Improvement:\n\n* Evaluation limited to sequence classification tasks:\nThe experiments focus on classification tasks. It would be beneficial to consider sequence generation or forecasting settings as well, where recurrent stability over long horizons is even more critical.", "was_revised": true, "final_justification": "This paper provides a clear theoretical and empirical analysis of the limitations of Glorot initialization in recurrent neural networks, particularly highlighting instability issues in long-sequence and deep architectures. The proposed spectral radius-aware modification is simple, well-motivated, and demonstrates consistent improvements in stability and convergence across Long Range Arena benchmarks.\n\nThe main limitation is the narrow experimental scope, as evaluation is restricted to sequence classification tasks. While the rebuttal acknowledges this and commits to including sequence generation and forecasting tasks in the camera-ready version, the current evidence remains somewhat limited. Nonetheless, the theoretical contribution is solid and the empirical validation is convincing within its scope.\n\nI view this as a technically sound paper that makes a useful contribution to initialization strategies for recurrent neural networks, even if its impact may be somewhat specialized. I recommend borderline acceptance.", "rebuttal": "We thank the reviewer for their thoughtful feedback and appreciate the positive assessment of our theoretical contributions.\n\nWe agree that recurrent stability plays a particularly important role in sequence generation and forecasting tasks. Due to the limited rebuttal period, we were unable to incorporate new benchmarks into our current experimental setup. However, we will include an empirical evaluation of our initialization strategy on these tasks in the camera-ready version."}, {"review_id": "xEhYCAO3Qc", "reviewer": "Reviewer_rSLb", "summary": "The paper revisits the Glorot initialization scheme for Recurrent Neural Networks (RNNs), specifically in the context of long-sequence processing with linear recurrences. The authors make the key theoretical point that for finite-width matrices, the spectral radius of a Glorot-initialized matrix does not converge to one from below, but rather from above. This means the largest eigenvalue modulus is typically slightly greater than one. While this deviation is small, it gets amplified exponentially over long sequences, leading to exploding hidden states. To address this instability, the authors propose a simple, dimension-aware rescaling of the Glorot variance.  The empirical results on a few tasks from the Long Range Arena (LRA) benchmark, showing that their rescaled initialization is stable and performs better than standard Glorot (which fails to train) and a naive Glorot/2 baseline.", "questions": "The authors may need to justify the significance of their work with more empirical results to make sure this paper fits the NeurIPS community.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper's main strength is its clear and well-articulated theoretical motivation. It correctly identifies a subtle but critical flaw in the common wisdom surrounding Glorot initialization for recurrent models: the convergence of the spectral radius to 1 from above. This is an important detail from random matrix theory that is often overlooked, and the paper does a good job of explaining its severe consequences for long-sequence stability.\n\nWeakness\n\n1. The central contribution is an incremental adjustment to the 15-year-old Glorot/Xavier initialization. While the theoretical motivation is sound, its practical impact in 2025 is questionable. The community has developed more powerful and general techniques to ensure training stability in deep models, most notably normalization layers (e.g., Layer Normalization), which are standard practice in nearly all modern recurrent models and Transformers. Furthermore, the work focuses on vanilla linear RNNs, a class of models that is not state-of-the-art. The paper itself acknowledges that recent advances center on State-Space Models (SSMs) and that its models do not compete with them. Improving an old initialization for a non-competitive model architecture feels like a solution to an already-solved problem, which severely limits the paper's significance.\n2. The empirical validation for the method's downstream performance is extremely thin, consisting of a single small table (Table 1). The lack of results on more complex, gated architectures (like LSTMs/GRUs) or modern architectures (like LRUs or SSMs) where initialization could interact with other components is a major omission. Also, a much more relevant and challenging baseline would be a standard Glorot-initialized model stabilized with a standard technique like Layer Normalization. Without this comparison, it's impossible to know if the proposed rescaling offers any benefit over what is already common practice.\n3. The theoretical analysis, while clean, rests on the simplified setting of linear RNNs with i.i.d. Gaussian inputs. This is a significant departure from real-world sequential data, which exhibits complex temporal dependencies.", "was_revised": true, "final_justification": "I appreciate the authors' response which address my concerns and raise my score to borderline accept.", "rebuttal": "We appreciate the reviewer’s comments and respond to their concerns below.\n\n# 1. Relevance and Significance\n\nThe reviewer raised concerns about the significance of our study of Glorot initialization in linear RNNs, particularly in light of the many alternatives for stabilizing signal propagation. To clarify the relevance and impact of our work, we highlight the following points:\n\n**(1) Recent surge in interest for linear RNNs:** The success of LRU [1] and SSMs [2] for long-range sequence modeling has inspired a recent surge of interest in linear RNNs, due to their competitive performance and *computational efficiency*. The linearity of these models enables the use of efficient parallel scan operations, allowing for fast inference and training. Notably, they avoid the use of LayerNorm or complex nonlinear gating mechanisms (as in LSTMs and GRUs) within the recurrent block, since such components would compromise their efficiency. Thus, the LRU's recurrent block differs from vanilla linear RNNs only in its initialization and parametrization. Given that such architectures are becoming increasingly common in recent literature, and frequently adopt Glorot initialization as a baseline, we believe our contribution is both timely and important for informing best practices in this field.\n\n **(2) Theoretical gaps in high-impact RNN works:** While we fully acknowledge that Glorot initialization is a classical method, we emphasize that misunderstandings about its properties, and lack of theoretical analysis, continue to appear in high-impact recent work. We highlight two such examples:\n\n- **\"Resurrecting Recurrent Neural Networks for Long Sequences\" [1]:** This influential paper introduced the LRU scheme for initializing and parametrizing linear recurrent layers, which has proven highly effective for long-range reasoning.  While the method works well empirically, the (non-rigorous) justification presented in Section 3.2.2 of [1] is based on a misconception: namely, that the eigenvalue moduli of Glorot-initialized matrices remain strictly below one. The authors even illustrate this by Figure 2 and state the *strong circular law* in their Theorem 3.1. However, the Glorot initialization baseline used to verify LRU's benefits is actually *truncated Glorot*, which roughly rescales the spectrum by $0.8$ (see our response to Reviewer qFV8, 1.1). This mismatch between the cited spectral theory and the actual behavior in long-range tasks highlights the need for more precise analysis, which our work provides. Based on our results, the authors could have: (1) directly explained the instabilities they likely encountered prior to applying the truncation, and (2) provided a more accurate foundation for their initialization scheme. While our work does not yet offer a full theoretical explanation for the success of LRUs, such an explanation remains a central and timely open question. We believe our paper takes a necessary first step by clarifying misconceptions about Glorot that appear in [1].\n\n- **\"Efficiently Modeling Long Sequences with Structured State Spaces\" [2]:** This paper introduced the influential S4 model, which also involves a combination of initialization and parametrization for discretized linear SSMs (applying only linear temporal operations). To validate their strategy, the authors compare it against Gaussian initialization. Unlike in [1], they explicitly state in Section 4.4 that they *scale down the Gaussian variance until the result stops exploding*. In doing so, they are empirically addressing the same type of instability that our work analyzes theoretically. Specifically, we consider eigenvalues $\\lambda_i$ of a Gaussian matrix $A$, which translates to the eigenvalues $(1 + \\Delta/2 \\cdot \\lambda_i)/(1 - \\Delta/2 \\cdot \\lambda_i)$ in discretized SSMs. This makes the spectral behavior of Gaussian initializations directly relevant to the paper's experiments setup. We view this as another example of recent impactful work where theoretical understanding of Glorot initialization was lacking, leading to ad hoc empirical fixes rather than principled best practices. Our work aims to fill this gap.\n\n**(3) Broader mathematical context:** Recent advances in deep learning theory have led to significant interest in the behavior of large Gaussian matrices, particularly in the infinite-width limit, where neural networks converge to Gaussian processes, and related developments such as Neural Tangent Kernel (NTK) theory. However, the vast majority of this literature assumes that the network's *depth is fixed*, meaning the number of layers (or matrix multiplications) remains finite. From a mathematical perspective, the setting where both depth and width grow simultaneously is substantially more challenging, as it often involves open questions in random matrix theory. To the best of our knowledge, existing work has only explored this joint limit in the special case of fully-connected networks with independent weight matrices across layers [3,4]. In contrast, our work takes a first step toward understanding the infinite-depth-and-width regime in a more difficult setting of linear RNNs, where the same matrix is repeatedly applied, leading to strong dependencies across layers.\n\nWe hope that these considerations help clarify the significance of our contribution. The points outlined above are currently discussed in Sections 1 and 2 of our submission. In the revision, we will rework the introduction to highlight the motivation and relevance of our results more clearly, incorporating the examples presented here.\n\n# 2. Empirical Validation\nThe reviewer expressed concern that our empirical validation is incomplete, particularly because we do not compare our rescaled Glorot scheme against: (1) vanilla Glorot with LayerNorm, (2) gated architectures such as LSTM or GRU, and (3) modern architectures like LRU or SSMs. We address each point below.\n\n**(1) LayerNorm:** We thank the reviewer for suggesting this experiment and have added results evaluating accuracy and training time for a linear RNN with LayerNorm on the IMDB dataset. The results show that, while LayerNorm prevents divergence, it increases training time and slightly reduces performance compared to RNN with rescaled init but without LN. As noted above, incorporating LayerNorm disrupts the linear recurrent structure that enables the hardware-friendly efficiency of recent SSM and LRU architectures. Moreover, the theoretical behavior of RNNs with LayerNorm remains considerably less well understood than that of vanilla linear RNNs. For these reasons, we believe our approach is both more efficient and more transparent. \n\n|Model|Accuracy|Training Time [min]|\n|---------------------|----------|----------------------|\n|Glorot Dense LN|86.82|12:51|\n|Rescaled Dense LN|86.59|12:51|\n|Glorot Dense|–|–|\n|Rescaled Dense|87.55|09:08|\n|Glorot Diag LN|80.64|3:25|\n|Rescaled Diag LN|87.28|3:25|\n|Glorot Diag|–|–|\n|Rescaled Diag|87.31|2:58|\n\n**(2) Gated architectures:** Since gated architectures have a fundamentally different structure from the linear RNNs studied in our work, we believe they fall outside the intended scope. These models rely on nonlinear gating mechanisms and multiple interacting components, which complicates fair and meaningful empirical comparison.\n\n**(3) Modern architectures:** Our initial evaluation was based on the LRU architecture, using the same overall structure as in [1] but without the LRU-specific parametrization. We agree with the reviewer that including results with the LRU parametrization would make the evaluation more comprehensive. Accordingly, we have added these results to the table below.\n\n|Initialization|Accuracy|\n|---------------------|----------|\n|LRU|87.32|\n|Uniform|85.84|\n|Rescaled|85.67|\n\nOverall, we believe the primary contribution of our work is theoretical, which is why we initially kept the empirical section minimal and focused on validating our analysis. That said, we appreciate the reviewer’s suggestions and will expand the empirical evaluation in the revision to include the additional comparisons shown here.\n\n# 3. Reliance on Gaussian Inputs\nThe reviewer notes that our results assume i.i.d. Gaussian inputs and raises concerns about their applicability to real-world signals. While real signals indeed often exhibit temporal dependencies, we believe this limitation does not significantly impact the relevance of our analysis, especially for the main question we address: whether the signal explodes over time.\n\nSince we study linear RNNs, we can decompose the input as a structured signal component and additive noise: $x_t = s_t + n_t$. Due to linearity, the RNN output at time $t$ becomes a sum of two terms: one driven by the signal, $h_s^t := \\sum_{k=1}^t W^k s_{t-k}$, and one by the noise, $h_n^t := \\sum_{k=1}^t W^k n_{t-k}$. Assuming the noise $n_t$ is i.i.d. Gaussian, our theoretical results show that $h_n^t$ explodes under the specified sequence length scaling. Therefore, even if $h_s^t$ remains bounded, the exploding behavior of $h_n^t$ alone is sufficient to cause the total output $h^t = h_s^t + h_n^t$ to explode in norm.\n\nWe agree that extending the analysis to more general or structured input distributions could yield deeper insights. We see this as an important direction for future work and discuss it in the Limitations section of our paper.\n\n\n# References\n[1] Orvieto et al. \"Resurrecting recurrent neural networks for long sequences.\" ICML (2023).\n\n[2] Gu et al. \"Efficiently modeling long sequences with structured state spaces.\" arXiv preprint arXiv:2111.00396 (2021).\n\n[3] Hanin \\& Nica. \"Products of many large random matrices and gradients in deep neural networks.\" Communications in Mathematical Physics (2020).\n\n[4] Roberts \\& Sho Yaida. \"The principles of deep learning theory.\" Cambridge University Press (2022)."}, {"review_id": "XcUkIN6hw5", "reviewer": "Reviewer_qFV8", "summary": "This paper revisits Glorot initialization used for initialization parameters of a Linear Recurrent Unit (LRU) in the context of long range dependency tasks. It first analyzes the instability caused by Glorot initialization in linear RNNs. It makes an observation that spectral radius of matrices initialized using Glorot initialization, typically converges to one from above, resulting in exponential growth in hidden states over long range sequences. To mitigate this issue, this work proposes a rescaled version of the Glorot initialization using the dimension dependent constant, reducing the probability by one standard deviation below expectation for the fact that spectral radius exceeds one.  For the complex Gaussian matrices, it provides some theoretical justification for the fact that with rescaled Glorot initialization, the hidden states explode with less likely probability compared to the standard Glorot initialization. Numerical experiments have been conducted to validate the efficacy of the proposed Rescaled Glorot Initialization.", "questions": "- In Tab.1, in dense matrix case, for the longest sequence task (IMDB with 8K tokens), Rescale variants of the initialization do not always yield better performance compared to Glorot / 2 or uniform variants. How do you explain this behavior? Since the hidden state explosion is already handled at the initialization by all these schemes. \n- In Fig1, how does the density and hidden states evolve for Glorot/2 and Diagonal Uniform scaling?  \n- In Tab.1, the error bars correspond to how many repetitions for the experiments?\n- Although this work is concerned on long-sequences, does the proposed Glorot Rescaling hurt the performance of linear RNNs on short sequences since eigenvalues are less likely to be beyond 1 ?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Strengths:\n- The proposed Rescaled Glorot Initialization is easy to implement and decreases the probability of having eigenvalues above 1 during initialization compared to Glorot initialization.\n- Theoretical analysis shows that linear RNN initialization with complex Glorot matrix results in hidden state norm explosion when the sequence length is at least of the order of sqrt(n) where n is the hidden state dimension. \n\nWeaknesses:\n- Table 1 shows the performance of linear recurrent models for Long Range Arena benchmark. It’s unclear why the Glorot initialization yields exploding baseline when the original linear RNNs do not have such an issue in these benchmarks?\n- Contribution seems to be a bit lacking since the paper only proposes a simple rescaling of the standard Glorot initialization and shows stability of the initial hidden states in two setups (a) infinite width and finite length, and (b) infinite width and length. The experiments do not reveal a lot of insights into the proposed scaling vs standard Glorot initialization.", "was_revised": true, "final_justification": "I read the author rebuttal and other reviewer responses. The rebuttal clarified some of my concerns, I believe the paper has merits from theoretical understanding of glorot initialization.  I would lean towards acceptance and will reflect the same in my scores.", "rebuttal": "We appreciate the reviewer’s comments. Below, we address the concerns raised and provide clarifications regarding the contributions and empirical findings of our work.\n\n# 1. Response to Weaknesses\n\n## 1.1 Instability of Glorot in Previous Works\nThe reviewer asked why vanilla Glorot initialization leads to exploding behavior in our **Table 1**, while such behavior was not reported in prior works using Gaussian initialization on Long Range Arena benchmarks. The explanation here is straightforward: prior works typically applied additional, implicit rescaling to the Glorot weights (often chosen empirically) to avoid instability, but this detail was rarely highlighted. In particular:\n\n- **\"Resurrecting Recurrent Neural Networks for Long Sequences\" [1]:** This influential paper introduced the LRU initialization scheme and demonstrated its empirical advantages over Glorot initialization. However, the actual implementation (based on the default Vanilla RNN module in Haiku, as stated in Footnote 3 of [1]) used a *truncated normal initialization* over the range $[-2, 2]$. This truncation significantly alters the spectrum of the initialized weight matrices. In particular, it effectively rescales Glorot initialization by a factor of approximately $0.8$, which greatly reduces the probability that any eigenvalue exceeds one in modulus. This implicit rescaling is not discussed in [1], but our results show that it plays a crucial role in preventing signal explosion. This highlights the importance of careful spectral analysis when evaluating initialization schemes.\n- **\"Efficiently Modeling Long Sequences with Structured State Spaces\" [2]:** This work, which introduced the widely used S4 model, was more explicit about their rescaling of Glorot initialization baseline. As stated in Section 4.4, they empirically scaled down the variance of the Gaussian initialization until the model no longer exhibited instability.\n\nThese examples demonstrate that the problem of exploding behavior under Glorot initialization is well-known and relevant in modern RNN literature. However, most previous works addressed it only empirically, by tuning scaling factors, without providing theoretical analysis or principled justification.\n\nWe also offer a more detailed discussion of these two works [1,2], highlighting how the instability of Glorot manifests in their setups, in our response to Reviewer rSLB (**par. 1, item (2): \"Theoretical gaps in high-impact RNN works\"**). We will make this point more explicit in the revision to clarify the motivation behind our theoretical analysis and rescaling approach.\n\n## 1.2 Clarification of Contributions\n\nWe would like to clarify the core contributions of our work, as the reviewer’s summary may not fully capture our results in two important respects:\n\n- **Stability and instability of Glorot initialization:** The reviewer mentioned that we show stability of hidden states under Glorot initialization in both the infinite-width and infinite-width-and-length regimes. In fact, our analysis demonstrates **stability** in the infinite-width regime, but **instability** in the infinite-width-and-length regime. In addition, we (1) identify the precise sequence length scaling threshold $t=\\Theta(\\sqrt{n})$, which is sufficient to cause instability, and (2) provide a more general finite-width result for the hidden state norm, which supports both limiting cases. These results offer a theoretical explanation for why long-sequence tasks pose unique challenges for initialization, and allow us to quantify how long is \"long enough\" for instability to arise.\n- **Empirical validation:** The reviewer raised a concern that our experiments do not clearly show the benefits of our rescaled initialization compared to standard Glorot. However, our experiments in **Table 1** and **Fig. 1** directly illustrate that, consistent with our theory, vanilla Glorot leads to exploding hidden states and training divergence, whereas our rescaled variant avoids this issue and enables stable training.\n\nIn summary, the main contribution of our work is a theoretical understanding of the behavior of Glorot initialization in linear recurrent architectures, especially in the long-sequence regime. Given Glorot’s wide use as a default baseline, we believe that shedding light on its limitations and offering a principled alternative is both timely and impactful. We also discuss the significance of our findings in more detail in our response to Reviewer rSLB (**par. 1, “Relevance and Significance”**).\n\n# 2. Response to Questions\n\n## Q1. Performance Comparison\nWhile it is true that all considered initialization schemes successfully prevent hidden state explosion, preventing instability alone does not guarantee equivalent downstream task performance. Different initializations influence not only the magnitude of hidden states but also the model's training dynamics and convergence behavior. Consequently, subtle differences in initialization can significantly affect generalization properties. This explains why, despite all schemes effectively mitigating hidden state explosion, performance differences between our rescaled Glorot, Glorot/2, and Uniform initialization can still occur.\n\n## Q2. Density and Hidden States for Glorot/2 and Uniform Initialization Schemes\n\n**Eigenvalue modulus density:** Since matrices initialized using Glorot/2 correspond exactly to standard Glorot-initialized matrices scaled by a factor of $1/2$, their eigenvalues are simply those of the standard Glorot matrices divided by two. Thus, the eigenvalue density for Glorot/2 initialization is identical to that shown in the upper row of Fig. 1, but scaled horizontally by a factor of $1/2$. For matrices initialized uniformly, the eigenvalues are uniformly distributed on the complex unit circle. Therefore, the modulus density of these eigenvalues follows a linear profile forming a triangular distribution from $0$ to $1$. This distribution closely resembles the Glorot distribution depicted in Fig. 1 but without the small tail extending beyond modulus one.\n\n**Hidden states behavior:** For both Glorot/2 and Uniform initialization, the norms $||W^k x||$ decrease as $k$ increases, and the corresponding hidden-state norms exhibit logarithmic growth with respect to $k$. This behavior is analogous to what we observe for the rescaled Glorot initialization in the lower row of Fig. 1. However, for Glorot/2 initialization, the decrease in $||W^k x||$ is significantly faster compared to Uniform initialization and our rescaled Glorot variant, resulting in notably smaller hidden-state norms.\n\n## Q3. Error Bars in Table 1\nThe results in Table 1 are averaged over three runs, with error bars representing the standard deviation across these repetitions.\n\n## Q4. Behavior on Short Sequences\nThe eigenvalues of the transition matrix remain unchanged regardless of sequence length, as the same matrix is used at each time step. For shorter sequences, any deviation of the spectral radius from 1 has a limited effect due to the reduced number of recurrent applications. As a result, both the Glorot and our proposed Rescaled Glorot initialization are unlikely to cause either exploding or vanishing hidden states in this regime. In general, short sequences are not considered challenging from a stability perspective, and our method is designed to address long-sequence dynamics without compromising performance on shorter ones.\n\n---\nWe would be happy to address any further questions or provide additional clarification, and hope that the provided responses may support a more positive evaluation of our work.\n\n# References\n[1] Orvieto et al. \"Resurrecting recurrent neural networks for long sequences.\" ICML (2023).\n\n[2] Gu et al. \"Efficiently modeling long sequences with structured state spaces.\" arXiv preprint arXiv:2111.00396 (2021)."}, {"review_id": "hZjEF5pXJx", "reviewer": "Reviewer_z2Qs", "summary": "Linear RNNs can only be trained if the recurrence matrix eigenvalues are in the unit disk.  Glorot initialization guarantees that eigenvalues are in the unit disk if the number of hidden nodes is infinite, but experimental results often show that this result fails to hold for finite numbers of hidden nodes.  This paper proves that the maximum eigenvalue converges in distribution to a Gumbel variable that is quite likely to be greater than one, and that therefore, the probability of such a divergent eigenvalue can be precisely controlled by rescaling the Glorot initialization by an amount that depends on the critical values of the Gumbel distribution.  Numerical simulations show that standard Glorot often leads to divergent state variables, and this divergence is eliminated by the proposed scaling. Tests using simple linear RNNs for toy tasks with very long sequences (1k to 3k time steps) demonstrate that standard Glorot often diverges, but that the proposed method often outperforms recent empirically-motivated solutions (half-Glorot and diagonal-uniform).", "questions": "Could you add a sentence giving a brief explanation for the constants in the RHS of Eq. (20)?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nFrom a practical point of view, the literature has been in desperate need of better study of long-sequence behaviors.  The entire deep learning literature has been limited to short sequence lengths for the past forty years, and this is a big problem in practice: Real-world sequences are not short, so the limitation to short sequences means that essentially we are modeling short extracts of the real world instead of modeling reality.\n\nThis study proposes studying long-sequence problems by studying the distribution of the maximum eigenvalue, rather than studying the distribution of all eigenvalues as Glorot did.  This step is very well motivated by 21st-century results in linear algebra, and in my opinion, the work in this paper is long overdue in the machine learning literature.\n\nThe empirical demonstrations in Fig. 1 are convincing.  Experiments include only numerical simulations and toy problems, which would be a weakness if the theoretical results were less interesting, but is not a weakness in the context of this paper: I treat these as a visualization of the key theoretical results.\n\nWeaknesses:\n\nExplanation of Eq. (18) in Eq. (19) is very useful, but I would have appreciated a similar explanation of the RHS of Eq. (20).  The approximation (1/t)exp(t^2/2n) is obviously true, but the factor of n suggests a more careful degree of precision that is not obvious.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful evaluation and generous score, and are glad to see that our work was found to be valuable. Below, we address the reviewer’s question.\n\n## Request to Explain RHS of Eq. (20)\n\nWe thank the reviewer for their engagement with this result. The RHS of Eq. (20) is intended to show a lower bound on the growth rate of the hidden state's norm. Taking \\$t = \\alpha \\sqrt{n}\\$, we approximate the involved sum using an integral as follows:\n\n$$\n\\sum_{k=\\lfloor \\epsilon \\sqrt{n} \\rfloor}^{\\lfloor \\alpha \\sqrt{n} \\rfloor} \\frac{\\exp(k^2 / 2n)}{k}\n= \\sum_{x \\in X_{[\\epsilon, \\alpha]}} \\frac{\\exp(x^2 / 2)}{x} \\cdot \\frac{1}{\\sqrt{n}}\n\\to_{n \\to \\infty} \\int_{\\epsilon}^{\\alpha} \\frac{\\exp(x^2 / 2)}{x}dx =: I,\n$$\n\nwhere we defined a small $\\epsilon>0$ to avoid the singularity, substituted \\$x := k / \\sqrt{n}\\$, and defined the discretization set $X_{\\left[\\epsilon, \\alpha\\right]} := \\left[\\{ \\frac{k}{\\sqrt{n}} \\}\\right]$ for \\$k \\in \\left\\[ \\lfloor \\epsilon \\sqrt{n} \\rfloor, \\dots, \\lfloor \\alpha \\sqrt{n} \\rfloor \\right]\\$. We then further approximate this integral using the substitution \\$y := x^2 / 2 = k^2 / 2n\\$, which yields:\n\n$$\nI = \\frac{1}{2} \\int_{\\epsilon^2 / 2}^{\\alpha^2 / 2} \\frac{\\exp(y)}{y} dy\n\\approx \\frac{\\exp(\\alpha^2 / 2)}{\\alpha^2}\n= \\frac{n}{t^2} \\exp\\left( \\frac{t^2}{2n} \\right).\n$$\n\nWe thank the reviewer for highlighting this equation, as it led us to identify a minor typo in the current version of the formula: the denominator incorrectly included only $t$ instead of $t^2$. Importantly, this typo does not affect the main conclusion of the section regarding the exponential growth rate and the resulting exploding behavior of the hidden state. We will correct this in the revision and include a more detailed version of the derivation."}]

Author rebuttal:

Meta-review: In the context of Recurrent Neural Networks, the paper analyses the weakness of Glorot initialization when long enough sequences wrt the hidden width are considered. The paper theoretically demonstrate that the assumptions underpinning Glorot initialization, i.e.  infinite-width and fixed-length regime, do not hold for RNNs processing long sequences, and proposes a simple rescaling of Glorot that shifts the spectral radius slightly below one, significantly mitigating the rapid signal explosion or decay. 

The paper owns merits that can be summarised as follows:
- a sound and clear theoretical contribution;
- a novel initialisation for RNNs grounded in spectral theory;
- a significative empirical assessment of the proposed initialisation.

Reviewers, however, identified some weaknesses,  the main ones of which can be summarised as follows:
- limited scope of theoretical results and experimental evaluation;
- incremental contribution.

Rebuttal resolved almost all raised issues, including the issue of significance of the work given more modern architectures for sequence learning on the basis of increased recent attention to linear structured models.

After discussion with SAC, it was decided that the main contribution of the paper can be identified in the theoretical analysis of the Glorot initialization, which makes the paper interesting per se since a flaw in Glorot initialization is clearly identified.

In conclusion, the positive contributions described above outweigh the weaknesses.
Decision: Accept (poster)