Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks
Authors: Yi Xie
Stefan Mihalas
Łukasz Kuśmierz
Venue: NeurIPS 2025 poster ()
Keywords: theoretical neuroscience; recurrent neural networks; heavy-tailed connectivity; finite-size effects; Lyapunov exponents
Primary area: neuroscience_and_cognitive_science
TL;DR: We analytically predict the exact critical transition to chaos in finite-size heavy-tailed RNNs and reveal a tradeoff: their broader edge-of-chaos regime comes at the cost of lower attractor dimensionality compared to Gaussian networks.

Abstract: Growing evidence suggests that synaptic weights in the brain follow heavy-tailed distributions, yet most theoretical analyses of recurrent neural networks (RNNs) assume Gaussian connectivity. We systematically study the activity of RNNs with random weights drawn from biologically plausible Lévy alpha-stable distributions. While mean-field theory for the infinite system predicts that the quiescent state is always unstable---implying ubiquitous chaos---our finite-size analysis reveals a sharp transition between quiescent and chaotic dynamics. We theoretically predict the gain at which the finite system transitions from quiescent to chaotic dynamics, and validate it through simulations. Compared to Gaussian networks, finite heavy-tailed RNNs exhibit a broader gain regime near the edge of chaos, namely, a slow transition to chaos. However, this robustness comes with a tradeoff: heavier tails reduce the Lyapunov dimension of the attractor, indicating lower effective dimensionality. Our results reveal a biologically aligned tradeoff between the robustness of dynamics near the edge of chaos and the richness of high-dimensional neural activity. By analytically characterizing the transition point in finite-size networks---where mean-field theory breaks down---we provide a tractable framework for understanding dynamics in realistically sized, heavy-tailed neural circuits.

Slow Transition to Low-Dimensional Chaos in Heavy-Tailed Recurrent Neural Networks
Eva Yi Xie
Stefan Mihalas'
Lukasz Kušmierz!
Allen Institute, Seattle, WA, USA
Princeton Neuroscience Institute, Princeton University, NJ, USA
evayixie@princetonn edu, {stefanm, lukasz. kusndeerr//hlleeenssiiiuueeee
Abstract
Growing evidence suggests that synaptic weights the brain follow heavy-tailed distributions, yet most theoretical analyses of recurrent neural networks (RNNs) assume Gaussian connectivity. We systematically study the activity of RNNs with random weights drawn from biologically plausible Lévy alpha-stabll stable distributions. While mean-field theory for the infinite system predicts that the quiescent state always cettable-implyiin ubiquitous chaos-our finite-size analysis reveals a sharp transition between quiescent and chaotic dynamics. We theoretically predict the gain at which the finite system transitions from quiescent to chaotic dynamics, and validate it through simulations. Compared to Gaussian networks, finite heavy- tailed RNNs exhibit broader gain regime near the edge of chaos, namely, slow transition to chaos. However, this robustness comes with a tradeoff: heavier tails reduce the Lyapunov dimension of the attractor, indicating lower effective dimen- sionality. Our results reveal a biologically aligned tradeoff between the robustness of dynamics near the edge of chaos and the richness of sigh-dimensional neural activity. By analytically characterizing the transition point finite-size networks- where mean-field theory breaks down- we provide tractable framework for understanding dynamics in realistically sized, heavy-tailed neural circuits
Introduction
Advances in connectomics yield increasingly detailed wiring diagrams of neural circuits across species and brain regions [Dorkenwald
[024] The MICrONS Consortium, 2025). This progress raises fundamental questions: what structural principles govern neural circuits, and how do they support the brain's 's remarkable computational power? One such prominent structural feature the presence of heavy-tailed Fooss et al.] 2011] synaptic weight distributions, consistently observed across the mammalian cortex (Song et al. 2005 Lefort et al. 2009 Dorkenwald et al. [022). mammalian hippocampus [Ikegaya al. 2013 and even in the Drosophila central brain [Scheffer et al.) 2220)]. Notably, this feature stands in sharp contrast to the Gaussian weight assumptions that dominate theoretical neuroscience studies and light-tailed distributions utilized in the standard initialization schemes in modern artificial neural networks [LeCun et al. 2002 Glorot and Bengio 2010 He et al. 2015). One way formally model heavy tails is with the family of Lévy o-stable distributions Feller 1971 Borak et al. 2205). which emerges as a natural generalization of the familiar Gaussian distribution (a 2) via the generalized central limit theorem. The family is parameterized by stability index a, where smaller values of a correspond to heavier tails. For a < 2, these distributions feature heavy, power-law tails. Similarly experimentally measured synaptic weights, samples generated from such distributions consistently contain large outliers that dominate many sample statistics. As we show in this paper, this can strongly affect neural dynamics.
"The codebase is publicly available at http:///ssg..a...ooo/aaaaa/aaaaa
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

One key phenomenon studied in theoretical neuroscience and machine learning is the transition to chaos, which has long been hypothesized to support optimal information flow and computational capacity at the so-called edge of chaos in a wide variety of randomly initialized neural networks. This encompasses recurrent neural networks (RNNs) [Bertschinger et al. 2004 Legenstein and Maass 2007 Toyoizumi and Abboti [011] |Schuecker et al. [018] and feedforward neural networks Schoenholz et al. 2016 Poole et al., [016]. In particular, Schoenholz et al. (2016) shows that the trainability of deep networks depends on initializing near the edge of chaos: the farther from this critical regime, the shallower a network must be to remain trainable. Similarly, [Bertschinger et al. (2004) shows that only near the edge of chaos can RNNs perform complex computations on time series.
In the context of feedforward networks, this effect can be understood in terms of the neural network Gaussian process kernel [Neal] 1996]: outside of the critical point, analogous to the edge of chaos in RNNs, trivial fixed point with kernel constant almost everywhere is approached exponentially fast, limiting the effective depth of information propagation and network trainability [Schoenholz]
(et al. 2016 Lee et al. 2017]. The transition between chaotic and non-chaotic dynamics in RNNs is often discussed in terms of eigenvalues of the weight matrix [Rajan and Abbott 2006 Aljjadeff et al. 2013]. According to the circular law [Girko, 1985 Tao et al.) 2010). its eigenvalues are bounded in a circle of radius proportional to the standard deviation of the distribution of weight entries. If the standard deviation is large enough, some eigenvalues fall outside of the unit circle and the quiescent state becomes unstable, paving the way for chaos to emerge. In contrast to random matrices with light-tailed elements, random matrices with a-stable entries feature unbounded limiting density of eigenvalues [Bordenave et al. 2011]. In infinite networks, this can lead to the lack of transition between quiescent and chaotic states, with any perturbation ultimately expanding in chaotic manner Kušmierz et al., 2020).
Understanding the computational implications of heavy-tailed recurrent connectivity is especially timely as RNNs have become central to neuroscience modeling. They are used to reproduce latent trajectories from neural recordings [Sussillo and Abbott 2009 Rajan et al. 2016 Pandarinath et al.
2018 Keshtkaran et al. 2022], to simulate circuit mechanisms of cognitive tasks Yang et al. 2019 Driscoll et al.] [2024], and to complement experiments through hypothesis generation [Pinto et al. Pagan et al. 2025). In NeuroAI, RNNs have been embedded into deep reinforcement learning
agents to recapitulate late biological navigation codes of grid cells [Banino et al. 2018). Yet, despite their prevalence, theoretical understanding of RNNs with biologically realistic, heavy-tailed weights remains limited.
To that end, the contribution of this paper is four-fold:
We reveal that finite-size heavy-tailed RNNs exhibit sharp transition from quiescence to chaos, in contrast to the mean-field prediction of ubiquitous chaos in infinite networks with tanh-like activation functions [Kusmierz et al. 2020
We derive theoretical predictions for the critical gain at which this transition occurs function of network size, and validate them through simulations.
We show numerically that heavier-tailed RNNs exhibit a slower transition to chaos, sustain- ing edge-of-chaos dynamics over a broader gain regime and offering greater robustness to gain variation; we show this can translate to improved information processing, as evidenced by the superior performance of heavy-tailed RNNs on a simple reservoir- conerviinn task.
We quantify attractor dimensionality as a function of tail heaviness, uncovering a tradeoff between robustness and dynamical complexity: heavier tails compress activity onto lower- dimensional manifolds.
2 Related Works
In his seminal work, Neal (196) examined Bayesian inference in neural networks and demonstrated that, in the infinite-width limit, shallow feedforward networks with standard Gaussian weight initial- izations converge to Gaussian processes. He noted that this convergence breaks down when weights are drawn from Lévy o-stable distributions, hypothesizing that such heavy-tailed initializations give rise to a richer class of priors beyond the representational capacity of Gaussian process kernels. This insight has since been extended and formalized by recent series of theoretical works that rigorously

---

characterize the infinite-widd limit of feedforward networks, showing convergence to a-stable pro- cesses [Favaro et al. 2020 Jung et al. 2021 Bordino et al., 2023] Favaro et al. 2023 Additionally. in [Favaro et al. 2024]. the training dynamics of shallow feedforward networks with heavy-tailed distributions of weights are characterized through the neural tangent kernel [Jacot et al. 2018]. While these studies focus on feedforward architectures, our work complements them by uncovering and characterizing a distinct transition in heavy-tailed feedforward networks via an annealed analysis, an effect not previously reported. We then extend this investigation to recurrent networks.
The critical behavior of heavy-tailed networks has also been examined in both RNNs and feedforward settings. Wardak and Gong (2022) report an extended critical regime in teavy-tailed RNNs, while Qu al.] [022] demonstrate that similar extended critical regime emerges in heavy-tailed feedforward neural networks, with training via stochastic gradient descent being most efficient in the region of the parameter space corresponding to the critical regime. Our findings are consistent with these observations and advance them by: (a) explaining the extended critical regime in terms of the behavior of the maximal Lyapunov exponent, (b) showing that the location of the transition depends on the network size, and (c) identifying tradcoff between the size of the critical regime and the dimensionality of the neural manifold in the critical regime.
Additionally, a mean-field theory of Cauchy RNNs (i.e., with weights following a Lévy o-stable distribution where a 1), is presented in Kušmierz et al. 2020). Specifically, they show that Cauchy RNNs with a binary activation function exhibit transition to chaos and generate scale-free avalanches, similarly observed in biological neural recordings (Beggs and Plenz 2003] and often presented as evidence supporting the critical brain hypothesis [Muñoz| 2018). Notably, Kuśmierz et al. (2020) note that Cauchy networks with wide class of activation functions, including tanh studied in our work, are always chaotic in the infinite-size limit, and, as such, do not exhibit transition to chaos. In
contrast, our results reveal that this observation is no longer true in the finite networks, highlighting the importance of finite-size effects.
Finally, our work complements recent studies on brain-like learning with exponentiated gradients [ormford et al. 2024), which showed that such updates naturally give rise to log-normal connectivity distributions. Within this broader context, our results offer a theoretical perspective that elucidates the dynamical consequences of these heavy-tailed structures.
3 Methods
3.1 Setup of recurrent neural network
We study recurrent neural networks that evolve in discrete time according to the update rule
s; (1 + 1)= =
Σ /=1
W (0),)((+++0))
where d = tanh the activation function, It(t) is the external input to neuron i time t, and N is the number of neurons. The synaptic weights W., are independently drawn from a symmetric Lévy a-stable distribution Feller 1971 Borak et al. 2005), i.e. Wij ~ La (a) with characteristic function
LL. (k) exp (=)a)
(2)
and with scale parameter a = //N where the gain g acts as the control parameter in our analysis. The stability parameter a E (0, 2] affects the tails of the distribution. For 2, the corresponding density function features heavy, power-law tails, i.e. (1)) x | when [ar] > 1. The remaining case of a corresponds the familiar Gaussian distribution with light tails.
We perform analyses on both autonomous (zero-input) and stimulus-driven RNNs. In the latter case (see Appendix EE), inputs each time step are sampled i.i.d. from Gaussian distribution with zero mean and variance 0.01. This enables us to study how stochastic drive interacts with heavy-tailed synaptic weight distributions modulate network stability.
3.2 Setup of feedforward networks
Although the weight matrix remains fixed during RNN evolution, in our mathematical analysis, we assume that We is redrawn at each time step. With such annealed approximation [Derrida and

---

Pomeau [986], evolving the RNN for T steps effectively corresponds to passing an input (initial condition) through feedforward network of T layers. In this case, we can reformulate the update equation as
Ni
((+1) 24 =
Σ
w
(2) 15 5 + 1100
where W:(t) is the N411 N₂ weight matrix at layer t. In this case, the initial condition x; is interpreted the input, and activity at T as the output of T-layer network. Additional inputs could also be passed directly to each layer via 1(") Note that we assumed that each layer may have a different width N₁. The case when V,N, = N corresponds to the annealed approximation of I We use (+)x x to denote the expected value with respect to a random variable X.
3.3 Computation of Lyapunov exponents
To quantify the dynamical stability of RNNs, we compute their Lyapunov exponents across range of weight scales (gains) g. This also provides an estimate of the maximum Lyapunov exponent (MIE,, Amax), which measures the average exponential rate at which nearby trajectories diverge in phase space. positive Amax indicates chaotic dynamics, while negative value implies convergence to stable fixed point or limit cycle. When Amax & 0, the system operates at the edge of chaos, a critical regime where perturbations neither grow nor decay rapidly.
We adopt the standard QR-based algorithm Von Bremen et al. 1997 described in Vogt et al. 2022 (detailed in Appendix C) compute the Lyapumov spectrum. For each input sequence, we track how infinitesimal perturbations evolve under the hidden-state Jacobians. These perturbations are orthonormalized via QR decomposition each step. and the logarithms of the diagonal entries the R matrix are accumulated to estimate the exponents. To avoid transient effects, we include short warmup period during which the network state evolves but Lyapunov exponents are not accumulated. The MLE is then averaged over multiple random input sequences to obtain a robust estimate.
3.4 Participation ratio and Lyapunov dimension
Two notions of dimensionality We analyze the dimensionality of RNN dynamics from two complementary perspectives. The first, based on Lyapunov exponents, quantifies how many directions exhibit local expansion under small perturbations; this is captured by the Lyapunov (Kaplan-Yer Korken dimension DKY, derived from the leading part of the Lyapunov spectrum. The second, based on the participation ratio (PR), measures how many orthogonal directions the network activity spans at steady state, using second-order statistics of the hidden states. Intuitively, PR is a linear method that approximates the manifold by an ellipsoid, and as such it may significantly overestimate the dimensionality of highly nonlinear manifold. In contrast, DKY is a nonlinear measure that, for typical systems, correctly estimates the information (fractal) dimension of a chaotic attractor [Oti] 2002
Lyapunov dimension To measure intrinsic dynamical complexity, we compute the full Lyapunov spectrum using the standard QR method (see Section 3.3 Let A
2 X2 2 2 A the ordered A 2 0. Then the Lyapunov defined as:
Lyapunov exponents. Define k as the largest index such that
dimension [Frederickson al. 1983 Farmer al., 1983 Oit 2002
Σ A |
DKY = k +
Near the edge of chaos, all positive Lyapunov exponents are close to 0 and perturbations along the corresponding directions expand with slow timescales. As result, in this regime, higher DKY indicates that the system evolves on digees-dimensional slow manifold (Krishnamurthy et al.) 2022), with more modes contributing to long-term variability and slow divergence-implyine a greater capacity to support rich, temporally extended computations. We track all orthogonal directions and update them with QR decomposition at each step after fixed warmup. We examine how DRY evolves with gain across all dynamical regimes.

---

Participation ratio Let 22 ER
be the hidden state of the RNN at time t, recorded over the final
K steps of length-T trajectory fixed gain g with N, after discarding the initial T K
warmup steps. We compute the empirical covariance matrix S = ビー (a(() æ) (z( - z)
where ī
+
1 z(()) Let denote the eigenvalues of S. The participation ratio is defined as
Kramer and MacKinnon
1993
Gao et al. 2017 Recanatesi al. 2022
(Σ,^)² Σ X2
PR
PR ranges from 1 (all variance in mode) to N (uniform variance), and quantifies how many orthogonal directions carry substantial variance regardless of stability. It has been widely used to characterize neural dimensionality in biological and artificial circuits [Gao al. 2017 Recanatesi
et al. 2022]. We compute PR across all types of regimes for the post-warmup steady-state trajectories.
4 Results
4.1
Finite heavy-tailed networks exhibit predictable quiescent-fo-chaod transition
4.1.1
Information propagation in feedforward networks
We study networks without external inputs (I = 0). Since (0)) = 0, the quiescent state is fixed point of both (T) and (3). In our mathematical derivation, we focus on the simpler case of annealed
dynamics. To study the stability of the quiescent state, expand (3) around (x) 0 and obtain
linear equation
(++ = (1)
(6)
where we used (0(0) = 1. Since sequences of weights at successive layers are generated i.i.d., (6) corresponds the Kesten process [Kesten [973). When t -> cc, the Kesten process may either converge to a limiting distribution or diverge. In our case, the width of the distribution of entries of (+t) acts as parameter that controls the transition between these two qualitatively distinct behaviors.
A detailed analysis in Appendix A shows that the critical width of the distribution is given by
9"' exp (- (x)
(7a)
N
1 N
E..o In
|z
(7b)
with zy T. (1). Let first show that this formula is consistent with the known results in the Gaussian case. Noting that in notation for 2 we have (22) 2, we can take the limit N + and obtain E In v2. This leads to g" 1/v2 and Lz(g") ~ (O, 1). Hence, recover the well-known transition at (W?) 1/N (Sompolinsky al. 1988 Molgedey et al. 1992 Toyoizumi and Abboti 2011]. Our formula, however, is more general and applics to any finite width of the network. It predicts that g", for any fixed is decreasing function of N (Fig. [A]. In the Gaussian case, it quickly reaches its asymptotic value consistent with the mean-field prediction. In heavy-tailed networks, however, the decay is slow and is clearly visible across four orders of magnitude shown in Fig. [A. Our theory predicts that this decay is logarithmic with an a-dependent exponent, i.c., g" x (In for a < 2, see Appendix for the derivation.
We also confirm our theoretical predictions in simulations, by passing a random initial vector through T' 100 steps (layers) of linearized network with weights redrawn at each step from a fixed distribution. We fix a = 1 and vary g. Below (above) the transition, we expect the components of the final state be close to (far from) zero with high probability. Thus, we construct a simple order parameter fze defined as the number of components of the final state E(T) that are within from As shown Fig. B the network goes through sharp transition between fer = and fc, = O, and the location of the transition is consistent with our theoretical prediction. The transition is rather sharp even for small networks (Fig. DC. increasing number of steps T. Our theoretical result becomes exact in the limit of T -
100), and is expected to become even sharper with
5

---

A
a=1.0
1.00
quenched
1.00
1.0
1.5
0.75
annealed
0.75
a
2.0
theory
0.50
0.50
0.5
0.25
0.25
0.00
0.00
0.0
10
10a
102
10a 10*
0.0
0.2
0.4
0.0
0.2
0.4
N
9
Figure 1: (A): Transition point g" predicted by our theory as a function of network size for various a. The transition point of Gaussian networks rapidly converges the mean-field limit (dashed line). In contrast, the transition point of heavy-tailed networks decays slowly towards zero. (B): The fraction of small
0.1) final state components linear networks with a 1 and N 3000 evolved for I" = 100 steps from random initial conditions as a function of g. In the annealed case, we observe a sharp transition at the location predicted by the theory. In the quenched case, each individual realization exhibits sharp transition (thin blue lines), but its location varies between different realizations of the weight matrix. Thus, when averaged over the realizations (thick blue line and dots; shaded region shows the +3 standard error), the transition looks smoother than in the annealed case. Nonetheless, its location approximately predicted by the theory. (C): Same as but with N = 100. As predicted by the theory, the transition point shifts to the right with decreasing N. Moreover, the location of the transition in the quenched case varies more in smaller networks.
4.1.2 Quenched disorder and recurrent neural networks
In contrast our annealed analysis of feedforward networks, the weights of the RNN remain constant throughout the evolution. Since we are interested in finite-sized networks, we can expect the location of the transition vary between the realizations of the weight matrix. A mathematical analysis of this phenomenon is beyond the scope of this work. We expect, however, that the random fluctuations of g" in networks with quenched random weights should concentrated around the annealed prediction and should decrease with N. Moreover, the typical values of g" should decrease with N as predicted by the annealed theory.
To test this hypothesis, we simulate a quenched version of 66 in which v, ( W. In order to observe the transition, we first fix the random seed (i.c., draw random components of the weight matrix from L. (=11//(/)) and then rescale its components by various values of g. We focus on a as representative example. As shown in Fig. [B, evolution of each realization of the weight matrix goes through a very sharp transition, but the location of this transition varies significantly between the realizations. Nonetheless, they are concentrated around the point predicted by the annealed theory. Moreover, the location of the transition shifts to the right and fluctuations increase with decreasing N and (Fig. []C) These results suggest that the location of the transition in the quenched case approaches the annealed prediction with increasing N. We provide further analysis on the behavior of the quanched transition point function of network size N in Appendix J
Note that our theoretical analysis does not specify the nature of the dynamics above the transition. Our simulations indicate that the network hovers around the edge of chaos in a significant range of values of g and, for 2 1, ultimately enters chaotic regime (see Fig. 2 In contrast, the dynamics of networks with < 1 show non-monotonic behavior of the MLE: after staying near the edge of chaos at intermediate values of y, the dynamics seem to ultimately settle stable, non-chaotic regime at larger values of g (see Appendix D). Thus, for 2 1, our annealed prediction g gives the approximate location of the transition to chaos in RNNs. Although our analysis focused on autonomous dynamics, similar to the Gaussian case, expect noise to shift, but not completely remove, the transition [Molgedey et al. 1992 Rajan et al. 2010).
While our analysis focuses on the tanh activation, it generalizes to any function satisfying ø(0) = 0 and admitting local expansion Φ(z) = ar + o(x). In this regime, the existence of the transition follows directly from the linear stability of the quiescent fixed point. For unbounded activations such as ReLU, however, bounded dynamics are no longer guaranteed, and divergence may occur large g. Although the transition itself persists for a broad class of activations, the qualitative behavior above can differ substantially. Beyond the transition, the ensuing dynamics depend sensitively on the nonlinearity: linear or ReLU activations typically diverge for large g. whereas sublinear, saturating

---

nonlinearities constrain activity and preserve stability. We therefore expect our results to hold for any smooth, saturating activation function, while unbounded ones likely produce more complex. divergent dynamics. The framework can also be extended to cases with x(0) * 0 by expanding around the corresponding non-quiescent fixed point. Here, the fixed point's location may vary with the order parameter g, but the overall nature of the transition should remain unchanged.
4.2 Heavier-tailed RNNs exhibit a slower, more robust transition to chaos
Having established the existence of a finite-size transition between quiescent and chaotic dynamics in RNNs with heavy-tailed synaptic weights (Section 4.1). we next examine how the nature of this transition differs across tail indices a. Our simulations of autonomous RNNs (Fig. [ similar results for noisy stimulus-driven RNNs shown in Fig. 55) reveal that although networks with a 2 1 exhibit transition to chaos as predicted, the sharpness and location of the transition vary substantially with o, in which a lower value corresponds to a heavier-tailee distribution.
In networks with Gaussian connectivity (a = 2.0), the maximal Lyapunov exponent (MLE) increases steeply with gain g. indicating rapid onset of chaos. In contrast, RNNs with heavier-tailed weights (lower a) exhibit a slower rise in the MILE as increases near the transition (when the MLE is near zero). This gradual transition implies that these networks remain closer to the edge of chaos over a wider range of gain values, consistent with previous observations of an extended, critical like region [Wardak and Gong, 2022 Qu et al.] 2022]. Such extended critical-like behavior can offer a form of robustness with respect to changes in network parameters, which can be an important property that benefits biological networks in non-stationary environments, allowing the network to maintain sensitive high-capacity dynamics [[Brtschinger et al. 2004] Legenstein and Maass] 2007| Toyoizumi and Abboti [011]] without the requirement of precise parameter adjustment. In our analysis of reservoir-computng computing networks on a delayed XOR task (Appendix D). heavy-tailed networks maintained strong task performance across broader gain regime than Gaussian networks. This provides a concrete proof-of-concept that the extended critical regime enhances robustness and performance without fine-tuning, which may benefit both machine learning applications and neural computation. Future studies could extend this analysis to trained recurrent networks and more complex temporal tasks to further elucidate how heavy-tailed connectivity shapes information processing and learning dynamics.
Moreover, shown in Fig. (with N increases from left to right panels), the locations of the transition (MLE 2 0) shift to the left (lower g) as N increases. Notably, this shift is more pronounced in tailed networks. This finding echoes our theoretical prediction that the critical gain g" slowly decreases with increasing N in the heavy-tailed regime due to the finite-size effect (Fig. [A].
Together, these simulation results provide empirical evidence that while infinite- width nean-field theory predicts ubiquitous chaos for Lévy RNNs, finite-size networks can operate near a well-defined, robust transition point whose properties depend systematically on the tail index and network size N.
This behavior may be particularly relevant in biological systems, where recent experimental evidence suggests synaptic weights follow heavy-tailed statistics, and where robustness parameter variation is essential. Our findings imply that heavy-tailed connectivity may naturally support computations at the edge of chaos in finite-size neural circuits without requiring fine-tuning.
4.3 tailed RNNs compress the chaotic attractor into a lower-dimensional slow manifold
The robustness of transition to chaos in heavy-tailed RNNs raises a natural question: does the structure of the underlying dynamical landscape also vary systematically with respect to the tail index a?
To address this, we first examine the full Lyapunov spectrum of networks near the transition to chaos, then we further characterize the effective dimensionality of the network's 's dynamics using two complementary metrics: the Lyapunov dimension (DKY). which estimates how many directions in phase space are locally expanding or marginally stable [Frederickson et al., 1983 Farmer et al.) 1983, Ott 2002], and the participation ratio (PR), which captures how variance is distributed across neural population activity and is commonly used neuroscience (Kramer and MacKinnon, 1993 Gao et al. 2017 Recanatesi et al. (022). We find that although heavy-tailed networks benefit from robustness near the edge of chaos, this comes with a key tradeoff: the dynamics are compressed into a lowerr-imennioonll slow manifold.
7

---

N=1000
N=3000
N=10000
1
1
1
Amax
Amax
Amox
1.0
0.74
00.15
1.5 -024)
-0.6)
-0552
-4
2-200
-0.911
2-2.0
-0.33)
4-20 -0.7%)
10-2
10
10
10
10
10
10
10
10
10
10
Figure 2: Maximum Lyapunov exponent (Amax) as a function of gain g for autonomous recurrent networks with different tail indices a, shown for: (A) N = 1000, (B) N = 3000, and (C) N 10000. Curves show mean across 10 trials; shaded regions denote +1 SD. We let the networks evolve for T = 3000 steps, among which the Lyapunov exponents are accumulated over the last K 100 steps. See results under noisy stimulus and ablation studies in Appendices E G Heavier- tailed networks (lower a) exhibit slower, more gradual increase in Amax near the transition (where Amax 0), resulting broader edge-of-chaos regime with respect to g. Dashed lines and legend mark the average critical gain g" at which Amax first crosses zero. As N increases, this transition shifts leftward, especially for lower a, in line with our theoretical predictions on finite-size effects.
4.3.1 Lyapunov spectrum shows compressed slow manifold in heavy-tailed RNNs
To probe the structure of the dynamical landscape near the transition chaos, examine the full Lyapunov spectrum of the networks. The spectrum provides a detailed view of local stability across phase space, with each Lyapunov exponent characterizing the growth or decay of perturbations along a particular direction in the network's state space. particular, the density the exponents near zero reflects the presence of slow activity manifold where the network evolves in the steady state. The slow manifold contains marginally stable modes along which input-driven perturbations expand or shrink slowly. Thus, in the absence of other memory mechanisms, slow modes endow RNNs with a crucial capacity to integrate information across long timescales Krrihhnauuttyy et al. 2022].
In Fig. AA we show the Lyapunov spectra for Gaussian and heavy-tailed networks near their respective estimated critical gain when the MILE first exceeds zero. The average critical gain (g*) is estimated through ten realizations in Fig. 2/ and the histograms
averaged across runs with the same (g"). hence they contain positive Lyapunov exponents (see Appendix F for individual realizations across input conditions, and additional discussion on the overestimation of (g")). The distribution of Lyapunov exponents differs markedly between these two types of network connectivity. Gaussian networks show dense band of exponents concentrated near zero, indicating broad, slow manifold. In contrast, as a decreases (i.e., the heaviness of the tail increases), the number of Lyapunov exponents near decreases, revealing compression of the slow manifold.
This suggests a tradeoff between the robustness of the edge of chaos and the dimensionality of the slow manifold. Following this observation, we next quantitatively characterize the attractor dimensionality.
4.3.2 Lyapunov dimensions and participation ratio further characterize low attractor dimensionality in heavy-tailed RNNs
To further characterize the tradeoff introduced by heavy-tailed connectivity, we quantify the dimen- sionality of the dynamical attractor using two complementary metrics (detailed in Section (34).
First, compute the Lyapunov dimension (DKy), which estimates the effective number of directions
in phase space that exhibit local expansion (Frederickson et al. 1983 Farmer et al, 1983 Ou 2002). This measure reflects the intrinsic complexity of the system's attractor. As shown in Fig.
recurrent networks with heavier-tailed synaptic weights (lower a:) exhibit a significantly lower DKY than their Gaussian counterparts across the near-chaotic regime (characterized in Fig. 2). This confirms that despite their robustness to chaos, heavy-tailed networks operate on lower-dimensional attractors.
Second, we evaluate the participation ratio (PR), widely used metric for gauging the effective dimensionality of neural population activity. It has been leveraged to quantify task-relevant low- dimensional subspaces and other properties of multi-unit neuronal recordings in behaving animals
8

---

[Gao et al. 2017], and summarize the collective modes visited by recurrent spiking networks and reveal how these modes depend on local connectivity motifs (Recanatesi al. 2019]. PR measures how variance in population activity is distributed across the eigenmodes of the covariance matrix, providing a compact read-out of the number of degrees of freedom the network explores [Kramer and MacKinnon, T993]. As shown in Fig. p PR declines as decreases, although the drop is shallower than that of DKY. This difference is expected: PR is a second-order statistic that is sensitive to how variance is spread across modes, whereas DKY is quantity set by local expansion rates of the flow. Consequently, PR can remain relatively high even when only a few directions in phase space are truly unstable, highlighting complementary information provided by these two dimensionality measures.
We hypothesize that the large disparity in Lyapunov dimensions between Gaussian and heavy-tailed networks arises from the broader dispersion of Lyapunov exponents in the latter as shown in Fig. BA. Intuitively, only a small subset of leading exponents becomes positive near the edge of chaos in heavy-tailed networks, resulting in lower overall Lyapunov dimension. This effect likely reflects the more heterogeneous eigenvalue distribution of the underlying weight matrix. However, the precise mapping between the weight matrix spectrum and the Jacobian's Lyapunov spectrum remains nontrivial and warrants further analysis.
Together, these metrics reveal that the slow manifold in heavy-tailed RNNs is both more contractive (lower Dxy) and narrower (lower PR), supporting the view that these networks "prioritize" robustness over dynamical richness. This tradeoff is biologically aligned with observations in animal studies, where low-dimensional neural representations are often found relative the nigh-dimensional ambient space of neural recordings, even in complex behaviors (Nieh et al. 2021 Cueva et al. 2020 We return to this point the Discussion.
Chaudhuri et al. 2019 Yoon al. 2013).
A
Lyap. exp. distribution at
B
Lyapunov Dimension
C
Participation Ratio
Probabiity Density
20
-=1211"')-071
レуарunov Dimension
Participation Ratio
wm1.15 11:1:11011-077
300
15
a- 004))
200
10-
200
100
100
0
-0.40
0.00
10-1
10P
10
10
10
10
A
g
g
Figure 3: Heavy-tailed networks exhibit lower-dimensional attractors near the edge of chaos. Curves show mean across 10 trials for networks of size N = 1000; shaded regions denote +1 SD. See Appendix E for results under noisy stimuli. The implementation details and ablation studies provided in Appendices [ (A) Distributions of top 100 Lyapunov exponents for varying a show fewer exponents near zero heavier-tailed networks at estimated (g") obtained in Fig 2A, indicating a lower-dimensional slow manifold. x-axis truncated at the left to omit near-zero tails for clarity. (B) The Lyapunov dimension is smaller for heavier-tailed networks near the regime of edge of chaos, reflecting fewer directions of local expansion in phase space. (C) The participation ratio dimension is similarly smaller with lower a near the edge of chaos, showing reduced variance homogeneity across neural modes. Together, these results indicate that while heavy-tailed networks maintain robustness to neural gain near chaos, they compress dynamics into a lower-dimensional attractor.
5 Discussion
Critically, our findings are with respect to finite-size networks and depend on network size. In the width limit, mean-field theory predicts that Lévy networks are always chaotic (Section 4.1.1. However, our results show that finite-size networks exhibit a clear quiescent- -oocccent-t--choiicc transition, with the critical gain g" shifting systematically with both network size N and tail index a (Eqn. 7 and Fig. 2b. This highlights that mean-field approximations may miss important structure in biologi- cally sized circuits, and that finite-size corrections offer more accurate theoretical framework for understanding real neural systems that are finite in size.
Further, as demonstrated in Fig. heavy-tailed weight distributions make RNNs more robust to changes in gain, a parameter that may correspond biologically to either the width of synaptic weight distributions or neural gain modulated by neuromodulatory systems (Wateroouse et al. 1988 Shine

---

et al., 2018]. Specifically, we observe that networks with heavier-tailed synaptic weights remain near the edge of chaos over a much wider range of gain values than those with Gaussian connectivity, which is commonly assumed in theoretical studies. This property may be especially valuable for biological systems that operate across multiple states (e-g., sleep and waking (Chaudhuri et al.) 2019) or in non-stationary environments. Such robustness could help explain empirical findings that similar neural activity patterns can arise from vastly different underlying circuit parameters in healthy brains (Prinz al. 2004, Marder 2011]. Meanwhile, as shown in Fig. 3 heavier tails reduce both the Lyapunov dimension and the participation ratio, indicating that the slow manifold supporting long- lasting activity becomes lower-dimensionas
Our further analysis show that handful of "mega--ynapses drives the dynamics. implying the robustness and con-dimensionality largely stem from extreme outliers (Appendix K.. Together, these effects imply a tradeoff: heavy-tailed networks are more stable to perturbations but require more neurons to achieve the same computational capacity, such as memory or temporal integration, compared to Gaussian networks.
Notably, a common empirical observation in neuroscience is that neural population activity tends to evolve within low-dimensional manifold relative to the large number of neurons recorded. This phenomenon has been observed across cortical and subcortical regions, and is often behav- iorally meaningful (Nich et al. 2021 Bondanelli et al., 2021]. Theoretical work suggests that cww-dimensoonaltty can arise from constraints imposed by circuit connectivity [Mastrogiuseppee and Ostojic| [018] or task demands [Gao et al. 2017). Our finding that heavier-tailed RNNs yield lower-dimensional attractors biologically aligns with this widespread phenomenon and provides evidence that anatomical connectivity might constrain the expressive capacity of population activity.
The observed couusteess-dmeenionaalii tradeoff also offers predictions for which tasks heavy-tailed circuits can be best suited. Tasks requiring only rank dynamics for reliable integration pattern generation (e.g. binary decision-making [Brunton et al. [0033] or working memory (Panichello and Buschman 2021] ) may benefit from the extended edge-of-chaos regime provided by heavy-tailes weights. In contrast, tasks that rely on high-dimensional dynamics- as representing multiple
independent memories or generating complex trajectories (e.g. virtual reality navigation [Busch
ct al. [2024)) -my require larger networks or connectivity distributions closer to Gaussian. These
predictions can be tested using emerging connectomic (Dorkenwald et al., 2024 The MICrONS
Consortium 2025] and large-scale recording datasets [Bondy et al. [024]] (002eeaaaleyy Manley et al. [2024). which
can jointly measure synaptic weight distributions and task-related activity dimensionality.
Our framework can be naturally extended mixture setting, improving biological plausibility. For instance, neurons could be homogeneous and each draw weights randomly from one of multiple
heavy-tailed distributions or form interacting subpopulations with distinct a values and connectivity
motifs. Such extensions may capture diversity across neuronal cell types [Jin et al. 2025 Zeng 2022 and offer a promising direction for future work.
We acknowledge several limitations: our study used untrained rate- based networks with homogeneous units. Including more biologically realistic features such as spiking dynamics [Kim et al. 2019).
Dale's law [Dale] [935]. cell-type diversity [Zeng, 2022, Yao et al., 2023). and synaptic plasticity (Citri and Malenka 2008) could modify or refine the observed effects. Furthermore, while our results focused on untrained dynamics, a key next step is to study how learning algorithms interact with the broad critical regime and how trained or reservoir computing heavy-tailed networks perform across a range of tasks (Yang al. 2019 Driscoll et al. 2024). Such a study would help us to understand and predict, based on connectivity alone, what kinds of computations a brain-like circuit suited to perform-an important goal as we seek to interpret rich new connectomic datasets and understand how synaptic connectivity ties to function [Garner et al., 2024 Seung 2024). Another valuable next step is to extend this work toward direct comparison with neural recordings. For example, future studies could estimate Lyapumov spectra or related dynamical signatures from long, high-resolution neural activity trajectories. While such analyses are technically challenging and require stable, extended recordings, they would offer a powerful bridge between theory and experiment.
In summary, finite-size recurrent networks with previously understudied Lévy distrriitr weights reveal a clear rule: heavier-tailed synaptic connectivity widens the regime of stable, edge-of-chaos dynamics but reduces the dimensionality of the resulting activity. This tradeoff links connectivity statistics, network size, and functional capacity, offering a principled, biologically plausible frame- work for interpreting both biological data and designing more parameter- artificial systems.
10

---

Acknowledgements
Eva Yi Xie was supported by a NeurIPS Foundation travel award for the NeurIPS conference presen- tation. Support for Stefan Mihalas was provided in part by NSF (2223725) and NIH (RFIDA05566)); support for Lukasz Kušmierz was provided part by NSF (2223725). We are grateful to the col- leagues at the Allen Institute and the COSYNE 2025 community for feedback on an early version of this work.
References
Sven Dorkenwald, Arie Matsliah, Amy Sterling, Philipp Schlegel, Szi-Chich Yu, Claire McKellar, Albert Lin, Marta Costa, Katharina Eichler, Yijie Yin, et al. Neuronal wiring diagram of an adult brain. Nature, 634(8032):124- 138, 2024.
The MICrONS Consortium. Functional connectomics spanning multiple areas of mouse visual cortex. Nature, 640(8058):435 447, 2025.
Sergey Foss, Dmitry Korshunov, Stan Zachary, et al. An introduction heavy-tailed and subexpo- nential distributions, volume 6. Springer, 2011.
Sen Song, Per Jesper Sjöström, Markus Reigl, Sacha Nelson, and Dmitri B Chklovskii. Highly nonrandom features of synaptic connectivity in local cortical circuits. PLoS biology, 3(3)=68,, 2005.
Sandrine Lefort, Christian Tomm, J-C Floyd Sarria, and Carl CH Petersen. The excitatory neuronal network of the c2 barrel column mouse primary somatosensory cortex. Neuron, 62):3001333, 2009.
Sven Dorkenwald, Nicholas L Turner, Thomas Macrina, Kisuk Lee, Ran Lu, Jingpeng Wu, Agnes L Bodor, Adam Bleckert, Derrick Brittain, Nico Kemnitz, et al. Binary and analog variation of synapses between cortical pyramidal neurons. Elife, 11:e76120, 2022.
Yuji Ikegaya, Takuya Sasaki, Daisuke Ishikawa, Naoko Honma, Kentaro Tao, Naoya Takahashi, Genki Minamisawa, Sakiko Ujita, and Norio Matsuki. Interpyramid spike transmission stabilizes the sparseness of recurrent network activity. Cerebral Cortex, 23(2): 20133
Louis K Scheffer, C Shan Xu, Michal Januszewski, Zhiyuan Lu, Shin-ya Takemura, Kenneth J Hayworth, Gary B Huang, Kazunori Shinomiya, Jeremy Maitlin-Shepard, Stuart Berg, et al. A connectome and analysis of the adult drosophila central brain. elife, 9:e57443, 2020.
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. Neural networks: Tricks of the trade, pages 9-50. Springer, 2002.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256 JMLR Workshop and Conference Proceedings. 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
William Feller. An introduction to probability theory and its applications. Vol. II. Second edition. John Wiley & Sons Inc., New York, 1971.
Szymon Borak, Wolfgang Härdle, and Rafal Weron. Stable distributions. Springer, 2005.
Nils Bertschinger, Thomas Natschläger, and Robert Legenstein. At the edge of chaos: Real-time computations and self-organized criticality in recurrent neural networks. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004.
Robert Legenstein and Wolfgang Maass. Edge chaos and prediction of computational performance for neural circuit models. Neural networks, 20(3):323-334, 2007.

---

Taro Toyoizumi and Larry F Abbott. Beyond the edge of chaos: Amplification and temporal integration by recurrent networks in the chaotic regime. Physical Review E Statistical, Nonlinear, and Soft Malter Physics, 84(5):551908, 2011.
Jannis Schuecker, Sven Goedeke, and Moritz Helias. Optimal sequence memory in driven random networks. Physical Review X, 844):04 2018.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations, 2016.
Ben Poole. Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
Radford M. Neal. Bayesian Learning for Neural Networks. Spriner-VVerlas Berlin, Heidelberg, 1996. ISBN 0387947248.
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint vrXiv:/711.00100 2017.
Kanaka Rajan and Larry Abbott. Eigenvalue spectra of random matrices for neural networks. Physical review letters, (1118)):881144 2006.
Johnatan Aljadeff, Merav Stern, and Tatyana Sharpee. Transition to chaos in random networks with cell-iype-specific connectivity. Physical Review Letters, 1188)0008100,, 2015.
Vyacheslav L Girko. Circular law. Theory of Probability & Its Applications, 29(4):694-706, 1985.
Terence Tao, Van Vu, and Manjunath Krishnapur. Random matrices: Universality of esds and the circular law. The Annals of Probability, 2010.
Charles Bordenave, Pietro Caputo, and Djalil Chafai. Spectrum of non-hermitian heavy tailed random matrices. Communications mathematical physics, 307:513-560, 2011.
Lukasz Kuśmierz, Shun Ogawa, and Taro Toyoizumi. Edge of chaos and avalanches in neural networks with heavy-tailed synaptic weight distribution. Physical Review Letters, 125(2):028101, 2020.
David Sussillo and Larry Abbott. Generating coherent patterns of activity from chaotic neural networks. Neuron, 63(4):544-555) 2009.
Kanaka Rajan, Christopher D Harvey, and David W Tank. Recurrent network models of sequence generation and memory. Neuron, 90(1):128-1)) 122- 2016.
Chethan Pandarinath, Daniel J O' O'Shea, Jasmine Collins, Rafal Jozefowicz, Sergey Stavisky, Jonathan C Kao, Eric M Trautmann, Matthew T Kaufman, Stephen I Ryu, Leigh R Hochberg, et al. Inferring single-trial trial neural population dynamics using sequential auto-encoders. Nature methods, 15(10):005-8155 2018
Mohammad Reza Keshtkaran, Andrew R Sedler, Raeed H Chowdhury, Raghav Tandon, Diya Basrai, Sarah L Nguyen, Hansem Sohn, Mehrdad Jazayeri, Lee E Miller, and Chethan Pandarinath. large-scale neural network training framework for generalized estimation of single-trial population dynamics. Nature Methods, 19(12):172-1577, 2022.
Guangyu Robert Yang, Madhura R Joglekar, Francis Song, William Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22(2):297-306, 2019.
Laura N Driscoll, Krishna Shenoy, and David Sussillo. Flexible multitask computation in recurrent networks utilizes shared dynamical motifs. Nature Neuroscience, 2777):149-1133, 2024.
Lucas Pinto, Kanaka Rajan, Brian DePasquale, Stephan Y Thiberge, David W Tank, and Carlos D Brody. Task-dependent changes in the large-seale dynamics and necessity of cortical regions. Neuron, 104(4):810-82) 824, 2019.
12

---

Marino Pagan, Vincent D Tang, Mikio C Aoi, Jonathan W Pillow, Valerio Mante, David Sussillo, and Carlos Brody. Individual variability of neural computations underlying flexible decisions. Nature, 639(8054):421 429, 2025.
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin Chadwick, Thomas Degris, Joseph Modayil, et al. fector-based navigation using wrkd-like representations in artificial agents. Nature, 557(7705):429 433, 2018.
Stefano Favaro, Sandra Fortini, and Stefano Peluchetti. Stable behaviour of infinitely wide deep neural networks. In International Conference on Artificial Intelligence and Statistics, pages 1137-1146. PMLR, 2020.
Paul Jung, Hoil Lee, Jiho Lee, and Hongseok Yang. alpha-stable convergence of heavy-tailed infinitely-wide neural networks. arXiv preprint arXiv:2106.11064 2021.
Alberto Bordino, Stefano Favaro, and Sandra Fortini. Infinitely wide limits for deep stable neural networks: sub-linear, linear and super-linear activation functions. arXiv preprint arXiv:2304.04005 2023.
Stefano Favaro, Sandra Fortini, and Stefano Peluchetti. Deep stable neural networks: large-width asymptotics and convergence rates. Bernoulli, 29(3):2574-255) 2597, 2023.
Stefano Favaro. Sandra Fortini, Stefano Peluchetti, et al. Largh-width asymptotics and training dynamics of alpha-stable re (I.U} neural networks. Transactions on Machine Learning Research, 2024(11), 2024.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kemel: Convergence and gener- alization in neural networks. In Advances in Neural Information Processing Systems, volume 31, 2018.
Asem Wardak and Pulin Gong. Extended anderson criticality in heavy-tailed neural networks. Physical Review Letters, 129(4):048103, 2022.
Cheng Kevin Qu, Asem Wardak, and Pulin Gong, Extended critical regimes of deep neural networks. arXiv preprini arXiv:2203. 2022.
John M Beggs and Dietmar Plenz. Neuronal avalanches in neocortical circuits. Journal of neuro- science, 23(35):11667-11177 2003.
Miguel A Muñoz. Colloquium: Criticality and dynamical scaling in living systems. Reviews of Modern Physics, 90(3):031001, 2018.
Jonathan Cornford, Roman Pogodin, Amn Ghosh, Kaiwen Sheng, Brendan A Bicknell, Olivier Codol, Beverley A Clark, Guillaume Lajoie, and Blake A Richards. Brain-like learning with exponentiated gradients. bioRxiv, pages 2024-10, 2024.
Bernard Derrida and Yves Pomeau. Random networks of automata: a simple annealed approximation. Europhysics letters, 1(2):45, 1986.
Hubertus F Von Bremen, Firdaus E Udwadia, and Wlodek Proskurowski. An efficient qr based method for the computation of Iyapunov exponents. Physica D: Nonlinear Phenomena, 101()-2): 1-16, 1997.
Ryan Vogt, Maximilian Puelma Touzel, Eli Shlizerman, and Guillaume Lajoie. On lyapunov expo- nents for mns: Understanding information propagation using dynamical systems tools. Frontiers in Applied Mathematics and Statistics, 8:818799, 2022.
Edward Ott. Chaos in dynamical systems. Cambridge university press, 2002.
Paul Frederickson, James L Kaplan, Ellen D Yorke, and James A Yorke. The liapunov dimension of strange attractors. Journal of differential equations, 49(2):185-207, 1983.
J Doyne Farmer, Edward Ott, and James A Yorke. The dimension of chaotic attractors. Physica D: Nonlinear Phenomena, 7(1-3):153-180, 1983.
13

---

Kamesh Krishnamurthy, Tankut Can, and David J Schwab. Theory of gating in recurrent neural networks. Physical Review 12(1):011011, 2022.
Bernhard Kramer and Angus MacKinnon. Localization: theory and experiment. Reports on Progress in Physics, 56(12): 1993.
Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. theory of multineuronal dimensionality, dynamics and measurement. BioRxiv, page 214262, 2017.
Stefano Recanatesi, Serena Bradde, Vijay Balasubramanian. Nicholas A Steinmetz, and Eric Shea- Brown. scale-dependent measure of system dimensionality. Patterns, 3(8), 2022.
Harry Kesten. Random difference equations and renewal theory for products of random matrices. Acta Mathematica, 1973.
Haim Sompolinsky. Andrea Crisanti, and Hans-Jurgen Sommers. Chaos in random neural networks. Physical review letters, (3):225,, 1988.
Lutz Molgedey, Schuchhardt, and Heinz G Schuster. Suppressing chaos in neural networks by noise. Physical review letters, 69(26):3717, 1992.
Kanaka Rajan, LF Abbott, and Haim Sompolinsky. Stimulus Stimulus-dependedt dependent suppression of chaos in recurrent neural networks. Physical review e, 82(1):01 1903, 2010.
Stefano Recanatesi, Gabriel Koch Ocker, Michael Buice, and Eric Shea-Brown. Dimensionality in recurrent spiking networks: Global trends in activity and local origins in connectivity. PLoS computational biology, 100644, 2019.
Edward H Nich, Manuel Schottdorf, Nicolas W Freeman, Ryan Low, Sam Lewallen, Sue Ann Koay, Lucas Pinto, Jeffrey L. Gauthier, Carlos D Brody, and David W Tank. Geometry of abstract learned knowledge in the hippocampus, Nature, 595(7865):80- 2021.
Christopher Cueva, Alex Saez, Encarni Marcos, Aldo Genovesio, Mehrdad Jazayeri, Ranulfo Romo, C Daniel Salzman, Michael N Shadlen, and Stefano Fusi. Low-dimensional dynamics for working memory and time encoding. Proceedings of the National Academy of Sciences, 117(37): 23021-23032, 2020.
Rishidev Chaudhuri, Berk Gerçek, Biraj Pandey, Adrien Peyrache, and lla Fiete. The intrinsic attractor manifold and population dynamics of a canonical cognitive circuit across waking and sleep. Nature neuroscience, 22(9):1512-1520, 2019.
KiJung Yoon, Michael A Buice, Caswell Barry, Robin Hayman, Neil Burgess, and Ila R Fiete. Specific evidence of -im-iieesional continuous attractor dynamics in grid cells. Nature neuroscience, 16 (8):1077-1084, 2013.
Barry D Waterhouse, Francis M Sessler, Cheng Jung-Tung, Donald J Woodward, S Ausim Azizi, and Hylan Moises. New evidence for a gating action norepinephrine in central neuronal circuits of mammalian brain. Brain research bulletin, 21(3): 4322, 1988.
James M Shine, Matthew J Aburn, Michael Breakspear, and Russell A Poldrack. The modulation of neural gain facilitates a transition between functional segregation and integration the brain. Elife, 7:e31 1300, 2018
Astrid A Prinz, Dirk Bucher, and Eve Marder. Similar network activity from disparate circuit parameters. Nature neuroscience, 7(12):1355-1322, 2004.
Eve Marder. Variability, compensation, and modulation in neurons and circuits. Proceedings of the 2011.
National Academy of Sciences, 108issss
Giulio Bondanelli, Thomas Deneux, Brice Bathellier, and Srdjan Ostojic. Network dynamics underlying off responses in the auditory cortex. Elife, 10:e53151, 2021.
Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamies, and computations in low-rank recurrent neural networks. Neuron, 99(3):609-66) 623, 2018.
14

---

Bingni W Brunton, Matthew M Botvinick, and Carlos D Brody. Rats and humans can optimally accumulate evidence for making. Science, 340(6128):95-98, 2013.
Matthew F Panichello and Timothy Buschman. Shared mechanisms underlie the control of working memory and attention. Nature, 592(7855):601-6055 2021.
Alexandra Busch, Megan Roussy. Rogelio Luna, Matthew L Leavitt, Maryam H Mofrad, Roberto A Gulli, Benjamin Corrigan, Ján Mináč, Adam Sachs, Lena Palaniyappan, et al. Neuronal acti- vation sequences in lateral prefrontal cortex encode visuospatial working memory during virtual navigation. Nature Communications, 15(1):4471, 2024.
Adrian G Bondy, Julie A Charlton, Thomas Zhihao Luo, Charles D Kopec, Wynne M Stagnaro, Sarah Jo Venditto, Laura Lynch, Sanjeev Janarthanan, Stefan N Oline, Timothy Harris, et al. Coordinated cross-brain activity during accumulation of sensory evidence and decision commitment. bioRxiv, pages 2024-08, 2024.
Jason Manley, Sihao Lu, Kevin Barber, Jeffrey Demas, Hyewon Kim, David Meyer, Fran- cisca Martínez Traub, and Alipasha Vaziri. Simultaneous, cortex-wide dynamics of up to I million neurons reveal unbounded scaling of dimensionality with neuron number. Neuron, 112 (10):1694-7799 2024.
Kelly Jin, Zizhen Yao, Cindy TJ van Velthoven, Eitan S Kaplan, Katie Glattfelder, Samuel T Barlow, Gabriella Boyer, Daniel Carey, Tamara Casper, Anish Bhaswanth Chakka, et al. Brain-wide cell-type-specifie transcriptomic signatures of healthy ageing in mice. Nature, 638(8049):182-1999 2025.
Hongkui Zeng. What is cell type and how to define it? Cell, 185(15):2739-2755 2022.
Robert Kim, Yinghao Li, and Terrence J Sejnowski. Simple framework for constructing functional spiking recurrent neural networks. Proceedings of the national academy of sciences, 116(45): 22811 11811-2280 122800, 2019.
Henry Dale. Pharmacology and nerve-endings, 1935.
Zizhen Yao, Cindy TJ van Velthoven, Michael Kunst, Meng Zhang, Delissa McMillen, Changkyu Lee, Won Jung, Jeff Goldy, Aliya Abdelhak, Matthew Aitken, al. A high-resolution transcriptomic and spatial atlas of cell types in the whole mouse brain. Nature, 624(7991):317-332, 2023.
Ami Citri and Robert C Malenka. Synaptic plasticity: multiple forms, functions, and mechanisms. Neuropsychopharmacoooss 33(1):18-41, 2008
Dustin Garner, Emil Kind, Jennifer Yuet Ha Lai, Aljoscha Nern, Arthur Zhao, Lucy Houghton, Gizem Sancer, Tanya Wolff, Gerald M Rubin, Mathias F Wernet, et al. Connectomic reconstruction predicts visual features used for navigation. Nature, (34(8032)1181-113 2024.
H Sebastian Seung. Predicting visual function by interpreting a neuronal wiring diagram. Nature, 63448032):113-123 2024.
Adiel Statman, Maya Kaufman, Amir Minerbi, Noam E Ziv, and Naama Brenner. Synaptic size dynamics an effectively stochastic process. PLoS computational biology, 10110)ee003844 2014.
Shai Wiseman and Eytan Domany. Selr-aver,ging, distribution of pseudocritical temperatures, and finite size scaling in critical disordered systems. Physical Review E,, 58(3):2938, 1998.
Dongsung Huh and Terrence J Sejnowski. Gradient descent for spiking neural networks. Advances in neural information processing systems, 31, 2018
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs. /tttps://githi com/ jax-mi) ml/ jax 2018. Version 0.3.13.
15

---

Charles R Harris, K Jarrod Millman, Stéfan Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian TIylor, Sebastian Berg. Nathaniel Smith, al. Array program- ming with numpy. Nature, 58577755:7722235533222 2020.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):211-27,, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitoto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019.
Abadi Martín, Agarwal Ashish, Barham Paul, Brevdo Eugene, Chen Zhifeng, Citro Craig, S Corrado Greg, Davis Andy, Dean Jeffrey, Devin Matthieu, et al. Tensorflow: Large-scale machine learning on heterogeneous systems. Software available from tensorflow. org, 7, 2015.
John Hunter. Matplotlib; 2d graphics environment. Computing in science & engineering, 9(03):
90-95, 2007.
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: It is accurately reflected, with the flow of the paper corresponds to the main claims made in the abstract and introduction mostly in order if not all of them.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are elaborated in the Discussion section. We additionally demonstrate robustness of results in the main text by providing ablation studies in the appendix (which maintain the same conclusion). The assumptions made with regard to the theoretical results are addressed in its corresponding sections (Section 4.1 Appendices A B).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
17

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification: The full set of assumptions and a complete (and correct) proofs are included in Sections 3 4.1 and Appendices A adhering guidelines below.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Method section includes all the details comprehensively, with additional results and implementation details (e.g.. hyperparameters) in the Appendix. The code is provided with corresponding commands to reproduce in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
18

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The code is provided in the supplementary material with the exact commands and environment needed to reproduce; no dataset is involved.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nnis.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nipscco httppppppwwdppe/////ppppppppaaep http://oooSuummssiisseeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: While don't it train models, we have included all the details for the implemen- tation and statistical practice (e.g., number of trials) in both Results and Appendix sections, well as captions of figures. We also provide the full codebase in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We each figure with at least three trials and plot error bars whenever applicable. We also provide visualizations of individual realizations whenever informa- tive/appropriate in the appendix.
Guidelines:
19

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Yes, we provide details of how compute costs scale with relevant hyperparame- ters the appendices when discussing the robustness each figure and the overall compute resources in Appendix M
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics https: http:://weui.peeeee httpriips..cppbbcc/eeeeee cc/ppubbic//tthicciiieeell1111.d
didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer:
[Yes]
Justification:
Our submission conform with the NeurIPS Code of Ethics every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
20

---

Justification:
Our paper is foundational research and does not tie to particular applications or deployments in the near future. However, we discuss relevant research impacts in neuroscience in Section 5
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We listed all packages we used, which are all open-sourced, in Appendix M with details of the licenses, original papers, URLs, whenever available.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
21

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We include our code as part our submission, which includes documentation on how to install the environment and reproduce the results in this paper. We do not use any assets from others other than the open-sourced packages, which are appropriately credited in Appendix M
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
22

---

Justification:
This paper does not involve corwsoucing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
23

---

A Mathematical analysis of the transition in annealed networks
Our goal is to show that networks with a-stable weight distributions exhibit a transition between two regimes, and to find the location of this transition which we denote as g- As in Gaussian networks, the quiescent state is stable and any small perturbation around it shrinks if weights a narrow enough distribution (i.e., g < g Similarly, the quiescent state is unstable if the underlying distribution is wide enough (g g In contrast Gaussian networks, however, this effect can only be observed through the analysis of finite-size effects.
generated from
As described in the main text, we study linear stability of the quiescent fixed point of (6) Since weights are randomly redrawn at each step, the evolution e(t) is a stochastic process. To quantify its behavior we focus our attention on the conditional distribution g(t+1) given e(O), Components of this vector independent due the assumed independence of rows of the weight matrix. The conditional distribution of single component can be characterized in the Fourier space as
N N
(exp (ike!
(ike(++))E())) (E(0))
exp
ik
W
(e) by EJ)
exp
-111_g0
1
w
(8)
where we used W ~ La (a/N!/0)- Thus, for > 1 the perturbation, when conditioned on the previous step, is an a-stable random variable. More specifically, can be written as of ((tt))/e0) (e(0) ~ Lo ()(t+))), where the conditional scale at step t +
1/a
N N₂
y (1+1) =
(9)
(9)
Ø=1
is a deterministic function of state at time 1, which itself is random variable. We can unpack this relation one step backwards by conditioning on g(t-1) instead, with e( [[((())) Le (,(t1). We
utilize the fact that this can also be expressed as
(t-1) (0,,))) (4)
(10)
where (1) depends on the perturbation at time f - 1, and 2 are i.i.d. a-stable variables. This leads to the recursive formula for scalar (()
(1+1) = (0))(1)
(11)
where ( is sequence of independent random variables distributed as 1/a
N.
1
{(t) = N₂
(12)
with i..dd 25 (1). If layers have the same width N, = N, ((t) i.i.d. and (11 scalar multiplicative process with i.i.d. entries. Thus, we have reduced our problem to a simpler special case of purely multiplicative scalar Kesten process. We can easily solve this recursion and rewrite the solution as a sum
In (1+((+))= = In 7 +
Ing(
(13)
where (1) is deterministically specified by the input perturbation (0)) It is known Kesten 1973 Statman al., 2014) that this sum diverges -oo almost surely if (In () and diverges to co
almost surely if (In {) > Accordingly, the sequence (7()) t=1 either converges to 0 or diverges.
Therefore, the critical width of the synaptic weight distribution is given by
g" = exp (-x,,a)))
(14)
where
N
N
In
|z
(15)
N
1=1
with 2j (1).
24

---

B Derivation of the logarithmic decay of g" (N) Here, we estimate the expected value of
N.o In
2j la
(16)
where 2j (1), for large N. We define YN,o Σラ |zy1
and note that the Laplace
transform of N. can be calculated as
(x)a(s) = (e--o = Fiaa
N )
(17)
where
(s) (0)(s))(ee0))) (e-el())) L1)
(18)
According to (17. the large asymptotic of Ev,a is dominated by the behavior Fi.a (s) around
O. This behavior should be similar for all symmetric distributions with the same stability index. For example, take (2(x) -- [||--1o for |x| > and (z()) otherwise. The resulting expansion be found as
8 In + ((*2)
(19)
2m-Pa
where ((a, 8) is the upper incomplete gamma function and y is the Fuler-Mascheroni constant. Thus, the asymptotic expansion of Fi,a(8) must take the form
(s) = A.s (B., ln + O(s²)
(20)
for some irrelevant constants A., Ba. We plug 20 into 17 and arrive at
In (s) = - A (Ba- In + N) O (N-1)
(21)
For N >>> 1, 21 corresponds random variable XN that can be constructed as
XN X1 + Aq In,,
(22)
where
(exp( 1app--xx10 xxx)) exp(- Aas (Ba In s))
(23)
We rewrite the desired expected value
(Ex.)) ~ (In (Xi + Ao ln)
(24)
The distribution of X, is fixed and does not change with N. Thus, for large N the second term dominates, and arrive
9" = =xp(( (EN.a)) x 1
(25)
(In N)
25

---

C Algorithm to compute Lyapunov exponents for RNNs
We leverage the algorithm proposed in Vogt al. 2022
study the dynamics of RNNs, adapted
our setting where we process a single input sequence a time (i.e., batch size 1). Then, running multiple realizations simply means running the same algorithm but with different seed in the beginning; this is equivalent to having batch of inputs shown in the original algorithm in Vogt et al. 2022].
To reduce the influence of transient dynamics, we include a warmup period during which the RNN is evolved forward but Lyapunov exponents
yet accumulated.
In this procedure, 2y is the imput at time step f, ft is the hidden state of the RNN, Q is an orthogonal matrix that evolves track an orthonormal basis tangent space, J dh the Jacobian of the RNN dynamics with respect the hidden state, R is the upper-triangular matrix from the QR decomposition, and "/ accumulates the log- meggnidts of the diagonal entries Ris- The accumulation begins only after the warmup steps, and the final Lyapunov exponent A is computed by normalizing 7% by the number of post-warmup accumulation steps, which is = warmup.
Algorithm 1: Lyapunov Exponents Calculation
Initialize h, Q: for f = to I do xe)
if't warmup then
J - of.
dh: Q+J-Q; Q:
+= log R
A T/TTT warmup)
26

---

D Lack of transition to chaos for a = 0.5
shown in Fig. networks with a 0.5 do not seem to transition chaos. For small values of the MLE increases with g as expected from the stability analysis. Moreover, similarly to other heavy-tailed networks, they hover close to the edge of chaos for wide range of values of g. However, for larger values of the MLE starts decreasing with 9 again and, as a result, usually stays negative all values of 9. This effect seems to persist for noisy inputs (Fig. B and other changes in the parameters of the simulations (Figs. and TO). More work is required to explain the source of this interesting phenomenon. However, since in this study we focus our attention on transition chaos, for clarity exclude from most figures.
N=1000
B
N=3000
N=10000
1
1
1
Amax
Amax
Amax
0.5
e=15 1g"
-0.751
1-15
-0.6)
1.5 ( 0052
4
8=20
-2.21
-4
2=2.0
-0.31)
-4
2=200
-0.00
10
10
109
10
10
10
10"
10
10
10
109
10
Figure Same as Fig.
2 but with an addition of 0.5.
27

---

E Additional results under small noisy input
We replicate our main results under a small i..dd Gaussian noise drive (variance 0.01) sampled at each time step to test the robustness of the quiescent-to-chaotic transition and attractor geometry in more biologically realistic, stimulus-driven settings. Despite the added input variability, which quenches chaos as expected Molgedey et al. 1992]. the trends largely mirror the autonomous case.
Figure 5 shows the maximum Lyapunov exponent (MLE) as function of gain g across network sizes (N = 1000, 3000, 10000) and tail indices a. Heavier tailed networks (a < 2) exhibit a more gradual increase in MLE and extended edge-of-chaos regime, consistent with Fig. The transition point shifts leftward with increasing N, in line with our mathematical finite-size predictions.
Figure 6 characterizes the attractor dimensionality using Lyapunov spectra, Lyapunov dimension (DKy), and participation ratio (PR). While DRY and the spectrum remain consistent with the autonomous case, PR displays U-shaped profile (Fig. (CC) unlike the monotonic rise seen in Fig. B3. This dip likely reflects a shift in the dominant dynamics: at low g. noise drives weak, independent fluctuations across neurons; near the transition, recurrent dynamics compress activity into an elongated low-dimensional manifold; at higher g. chaotic expansion increases PR. Thus, while the robustnes-diimensionalle dimensionality tradeoff holds under noisy input, noise modulates how variance is distributed across neural modes.
N 1000
N-3000
N 1=10000
1-
1
1
Amax
Amax
Amax
0.5
0.5
3.5
0110
3.001
1=100
0..93)
3.2
0049
3.911
2.0000
00.41)
2.0 0.41
10-2
10
100
10
10-2
10
10a
10
10-2
10
10a
10
Figure 5: Effect of network size under small i.i.d noisy input. Maximum Lyapunov exponent
(Amax) as a function of gain g in noisy cemulus-driven recurrent networks with Lévy a-stable weight distributions. Curves show mean across 10 trials; shaded regions denote + SDD Each panel corresponds to a different network size: (A) N = 1000, (B) N = 3000, and (C) N = 10000. Curves show mean across 3 trials; shaded regions denote 1 SD. As in the autonomous case, if transition exists, then heavier-tailed networks exhibit a slower transition and wider critical regime near Amax 0. The critical gain g (dashed line) shifts leftward with increasing N, consistent with finite-size theory.
A
Lyap. exp. distribution
B
Lyapunov Dimension
C
Participation Ratio
Probability Density
20
-=1
Lyapunov Dimension
1.0
Participation Ratio
a.5.5
15
200
400
10-
5-
100
200
0
-0.40
0.00
10-1
10c
101
10-1
100
10a
A
9
g
Figure 6: Attractor geometry under noisy input (N 1000). Curves show mean across 10 trials; shaded regions denote +1 SD. (A) Lyapunov exponent distributions at the estimated transition point g². Heavier-tailed networks exhibit fewer near-zero exponents, indicating a compressed slow manifold. x-axis truncated at the left to omit near-zero tails for clarity. (B) Lyapunov dimension declines with heavier tails, confirming lower attractor dimensionality as found in the autonomous networks. (C) Participation ratio shows distinct dip near transition to chaos before rising, unlike the monotonic profile observed the autonomous case, but it consistently lower in heavier-tailed networks otherwise.
28

---

F Visualizations of multiple realizations of Lyapunov spectrum
To assess the variability across realizations of network connectivity, visualize the full Lyapunov spectrum from three independent trials (different seeds) for networks with N = 1000, across both autonomous and noisy stimulus-driven settings. These spectra are computed near the estimated critical gain g" (obtained in Figs. 2 and 5). where the maximum Lyapunov exponent Amax first crosses zero in each condition. The same value of (g"), computed by averaging g of multiple runs, is used across different seeds in these figures. The actual transition point can vary in each realization. Moreover, due to the finite resolution of the grid of g values used in simulations, g is overestimated in each seed. Thus, in some realizations, the right edge of the histogram may exceed 0, and the average histograms presented in Figs. BA and can feature some positive Lyapunov exponents. more precise estimate could be obtained through a finer-graine or binary search over gain values near the transition point. However, this additional numerical precision would unlikely affect our overall conclusions.
In both autonomous (Fig. 7 and noisy stimulus-driver cases (Fig. 8 Gaussian networks exhibit dense cluster of exponents near zero, indicative of a broad slow manifold. In contrast, heavier-tailed networks (lower a:) show more widely dispersed exponents with fewer near-zero values, consistent with the compression of the slow manifold described in Section 1.3. Despite random initialization, decreases-rennte
the qualitative trend -rretter spectrum spread and fewer marginal directions consistent across seeds. Notably, in the noisy stimulus-driver case (Fig. 8). the exponents tend to shift downward. and their distributions become more skewed, particularly for heavier-tailed networks. These effects likely reflect interactions between stochastic input and the network's 's intrinsic dynamics, where the noise quenches the chaos.
Together, these visualizations reinforce our claim that heavy-tailed connectivity leads to systematically lower- diwer-dimesiooal attractors, regardless of input conditions or initializations.
A
B
Lyap. exp. distribution
Lyap. exp. distribution (g"
Lyap. distribution (g"
32
Probability Density
Probability Density
24
0799
Probability Density
24
==100"")6699
24
==2. 0.913
18
==2000
16
12
12:
6-
0
-0.46
A 0.00
-0.41
0.00
-0.38
0.00
A
A
Figure 7: Full Lyapunov spectra across random initializations in autonomous networks (N 1000). Each panel shows the Lyapunov exponent distributions near estimated g" for an independent seed. Across seeds, heavier-tailed networks (lower a) exhibit a broader spectrum with fewer exponents near zero, indicating reduced slow-manifold dimensionality compared to Gaussian networks.
A
B
C
Lyap. exp. distribution at (g"
Lyap. exp. distribution at ig")
Lyap. distribution at (g"
Probability Density
am1:010") (5005)
Probability Density
==10(0")=05))
Probability Density
(1109)
24
24
24-
a=2.0 0.933 0.93)
a=20 "y 0.93)
2=2.00
18
18
18-
12.
12
12:
6-
6-
-0.51
0.00
-0.51
0.00
-0.46
0.00
A
A
A
Figure 8: Full Lyapunov spectra across random initializations in noisy stimulus-driven networks (N 1000). Each panel shows the Lyapunov exponent distributions at the estimated critical gain g" for an independent seed. Spectra under noise remain qualitatively similar to the autonomous case.
29

---

G
Robustness of results shown in Fig.
Since we only examine the maximum (top-1) Lyapunov exponent in Fig. 2 the number of top exponents computed (denoted k k_LE in the codebase) is irrelevant as long as k LE > 1. Throughout Fig. 2 and this appendix, use the default k_ L_ 100. Additionally, we exclude initial warmup period before accumulating exponents to avoid contamination from transients (Appendix (C. In Fig. 2 the network is run for T 3000 steps, and Lyapunov exponents are accumulated over the final
K = 100 steps.
Note that computational cost increases with network size N, number of exponents k_ LE, accumulation duration K, and total time steps T. Here, we verify that our results Figs. and | are robust to these choices by comparing the default configuration against two more computationally demanding variants, keeping all else fixed:
1. Accumulating exponents over a longer period (K T - warmup = 150);
2. Running the network for longer total time (T 4000 with warmup of 3900, fixing K 100).
The results are shown in Fig. 9 (autonomous) and Fig. 10 (noisy). All curves remain nearly identical across conditions, demonstrating that our findings are not sensitive to the specific accumulation duration simulation length. In practice, using T = 3000 and K 100 strikes good balance between computational efficiency and accuracy, especially for large N. These findings validate that the trends reported in Figs. 2 and Jare robust, and additional compute is not necessary. Note that the effect of network size N has been evaluated in Figs. 2 and in which the critical transition g shifts to the left as N increases due to the finite-size effect.
A
B
C
Original K= 1000. T=3000
Longer K= 1500 fixed T-3000
Fixed K=100, longer T=4000
Amex
Amax
Amax
0.10
33005
2.0
+0.09)
3.3 (g 0.40
--15 1s"
-0.991
3.5
=0.7%)
2.5 ( 0.79)
-4-
-0911
-4
2-2.000"
10
10-1
100
10
10
10
109
10a
10
10
109
10
Figure 9: Robustness of MLE to time horizon and accumulation duration (autonomous,
N 1000). Curves show mean across 3 trials; shaded regions denote +1 SD. (A) Default configu- ration: T 3000, K 100; (B) Longer accumulation: K = 150; (C) Longer sequence: = 4000 with K 100. Results are nearly identical, confirming that the choice of T and K does not affect the reported trends.
A
B
C
Original K- 100. T-3000
Longer K=150, fixed T=3000
Fixed K=100, longer T=4000
1
Amac
Amax
Amax
1=10
e.1.5
15"
3.79
1.5
-0.29)
3.5 (0"
0.299
-4
2.000 -0.22)
2=2.0'0 00.3)
10-2
10-1
100
10a
10-2
10-1
100
10a
10-2
10
109
10a
Figure 10: Robustness of MLE to time horizon and accumulation duration (noisy stimulus-
driven, N = 1000). Curves show mean across 3 trials; shaded regions denote +1 SD. (A) Default configuration: T 3000, K = 100; (B) Longer accumulation: K = 150; (C) Longer sequence: T' 4000 with K = 100. Results remain stable, indicating that stochastic input does not impact the robustness of MLE computation.
30

---

H Robustness of results shown in Fig. 3A
We showed the representative top 100 Lyapunov exponents (k_LE= 100) in Fig. 3A using networks of size N = 1000. As we are primarily interested in the region near À 0, this choice of k_LE is sufficient and larger values do not change the results.
In both Fig. 3A and the visualizations in Appendix networks were evolved for 3000 time steps, with the exponents accumulated over the final K 100 steps, after initial warmup. As with all our experiments, computation becomes more expensive as the network size N, number of exponents k_ LE, accumulation duration K, and total time steps 7 increase. Here test the robustness of our findings in Fig. BA by varying these computational parameters, holding all else fixed:
1. Increasing network size N 3000 (panels A);
2. Accumulating over longer time window K 150 (panels B);
3. Increasing the total simulation length to T 4000 while maintaining K 100 (using longer warmup of 3900, panels C).
The resulting spectra, shown below in both autonomous (Fig. and noisy stimulus-driven networks (Fig. [ are qualitatively the same as the original results. The shape of the Lyapunov spectrum remains consistent: Gaussian networks show dense band near zero, and heavier-tailed networks exhibit broader spectra with fewer exponents near zero. These results confirm that our main finding- compression of the slow manifold with decreasing a-is robust across range of network sizes and simulation settings. For large-scale experiments, using N = 1000, T = 3000, and K = 100 provides reliable and computationally efficient default.
B
Larger N 3000, fixed
Larger K=150, fixed T=3000
Fixed K= 100, longer 4000
60
Probability Density
0g [.0))))
Probability Density
1-11011011-0666
Probability Density
20
c-1.5 1-675
20
1.5 -
40
-- 2.0 (gg00000g") (10]]-0001
e-2.0 1w 0.93)
2.0 [ip") 0.111
10-
10
20
0
-0.25
0.00
-0.30
0.00
-0.30
0.00
Figure 11: Robustness of Lyapunov spectra to simulation and accumulation parameters (au- tonomous). Curves show mean across 3 trials; shaded regions denote +1 SD. Mean Lyapunov spectra near g" under three conditions: (A) larger network size (N 3000): (B) longer accumulation period (K = 150); (C) longer total simulation length (T = 4000). Both (B) and (C) use N = 1000. The compressed spectrum in heavier- networks remains consistent across all conditions.
A
B
Larger N-3000, fixed K. T
Larger K= 150, fixed T 3000
Fixed K=100, longer 4=4000
Probability Density
1.5 08 (797)
0p 1 0911
Probability Density
20-
1-100
-=1.511'1-6770
Probability Density
20
=- 1.0
1.5 Di 0.229
1.05)
40
a-2.0
15
00111
10.
10
20
0
-0.25
0.00
-0.35
0.00
-0.40
0.00
A
A
Figure 12: Robustness of Lyapunov spectra to simulation and accumulation parameters (noisy input). Same conditions as Fig. [I but for noisy stimulus-driven networks. Despite stochastic input, heavier-tailed networks continue to exhibit wider Lyapunov spectrum with fewer marginally stable directions as indicated by the Lyapunov exponents being zero.
31

---

I Robustness of results shown in Fig.3B,C 3B,C
Lyapunov dimension We use the full Lyapunov spectrum to compute the results shown in Fig. 3
due to the definition of Lyapunov dimension (Eqn. 4 hence k_LE = N = 1000 in Fig. 3B. We
simulate the dynamics over total number T 2950 steps, and use the last K 50 steps to compute the Lyapunov dimension.
Participation ratio To ensure well-defined participation ratio (PR, Eqn. 5p. we require K N, where N the network size and K T warmup denotes the number time steps used for computing PR after the network has evolved for a number of warmup steps. This condition guarantees that the empirical covariance matrix S, computed from K samples N-dimensional hidden states, is full-rank and not rank-deficient. When K < N, S becomes singular or ill-conditioned, leading to unreliable estimates of eigenvalue spectrum and thus of the participation ratio. In Fig. BC. we use T = 2900 + N + 50 3950, meaning 2900 warmup steps with an accumulation period over the last 1050 steps.
Note that the computation cost increases as N, k_LE, K, and T increase.
Here show results in Fig. i robust, meaning it is consistent with the computationally demanding case(s) with all else fixed:
1. Bigger network size N = 3000 (panels A);
2. Longer accumulation period K 100 for computing Lyapunov dimension, and longer 1100 for computing participation ratio (panels B);
3. Longer time trajectory T = 3950 for computing Lyapunov dimension and T 4950 for computing participation ratio (panels C).
Larger N=3000, fixed K, T
Longer K-100, fixed T-2950
Fixed K=50, longer T=3950
Lyapunov Dimension
2=1.0
Lyapunov Dimension
Vyapunov Dimension
600
2-2.0
200
200
.=20
400
100
100
0
0
10
10a
10-
10
10"
10a
10
10a
10a
9
g
g
Figure 13: Robustness of Lyapunov dimension simulation parameters (autonomous). (A)
Larger network size N = 3000; (B) Longer accumulation period K = 100; (C) Longer total sequence
3950 with K 50. All trends remain consistent with those Fig. 3B.
A
B
C
Larger N=3000, fixed K,
Longer K=100, fixed T=2950
Fixed K=50, longer T=3950
Lyаpunov Dimension
800
p=10
syapunov Dimension
Syapunov Dimension
a=1.
600
2.0
200
200
2-2
100
200
0
10-1
10"
10-1
100
10a
10-1
10"
101
Figure 14: Robustness of Lyapunov dimension to simulation parameters (noisy), Same settings as Fig. [3 with i.i.d. Gaussian input. Results are stable across conditions, confirming robustness of DRy in noisy networks, consistent with those in Fig. BB.
32

---

B
C
Larger N 3000, fixed T
Longer K=1100, fixed T=3950
Fixed K=1050, longer T=4950
Participation Ralio
1000
Participation Ratio
300
.-2
300
Participation
200
200
100
100
0
10-1
10
10
-1
101
10
10'
Figure 15: Robustness of participation ratio to simulation parameters (autonomous). (A) Larger
network size N = 3000, K 3050; (B) Longer accumulation period K 1100; (C) Longer
sequence 7 = 4950, K = 1050. All curves are consistent with Fig. 3C, confirming stability of PR under varying conditions.
B
C
Larger N 3000, fixed
Longer K=1100, fixed T=3950
Fixed k-1050, longer T-4950
1500
Participation Ralio
Partic pnttcipption Rato
Participation Ratio
.92.0
400
0.2.0
1000
200
200
500
10-1
109
10
-1
101
10-1
101
Figure 16: Robustness of participation ratio to simulation parameters (noisy). Same configura- tions as Fig. [5] but under i.i.d. Gaussian input. The non monotonic profile and overall trends in PR are preserved across all tested conditions, consistent with those in Fig. 6C.
33

---

Behavior of the quenched transition point as a function of N
In finite-sized quenched networks, the location of the transition point fluctuates between realizations of the weight matrix. Since our annealed theory does not offer any insight into the nature of these fluctuations, we resorted to numerical simulations to study how the statistics g" scale with N. The results for the representative case of line with our theoretical predictions (Fig. 17A). The annealed
are shown in Fig. [7] The mean location of the
transition point scales like 1/ In prediction seems to underestimate the true mean over quenched realizations. The standard deviation of g" decreases with N comparable rate as the mean (Fig. [7B). The coefficient of variation of
falls off slowly in the studied range of N (Fig. I7c), suggesting that the location of the transition may be (weakly) self-averaging Wiseman and Domany 1998
this system.
A
B
C
7-
quenched theory
0.15
0.50
6
0.45
1/t
5-
0.10
CVig
0.40
4
0.35
3
0.05
0
0.30
102
10a
102
10a
Figure 17: Statistics of g" in quenched networks as functions of N. Note the logarithmic scale on the x-axis. Each point corresponds to the statistics estimated using 100 independent realizations of the weight matrix. For each value of N, we included three data points that correspond to independent estimates calculated based on different random seeds. (A) Reciprocal of the mean. (B) Standard deviation. (C) Coefficient of variation.
34

---

K The effect of mega-synapses on dynamics
To test whether robustness and low dimensionality arise from global heavy-tailed statistics few extreme "nega-synapses." we pruned recurrent weights in a network of size N = 1000 by absolute magnitude (bottom 95%, top %, top 3%) and report the results averaged across three trials. The slow transition vanished when top outliers were removed, shifting the critical gain g to larger values, whereas pruning weakest 95% had little effect (Fig. 18)
Similarly, removing the bottom percentage of weights has very little effect on the general trend of attractor dimensionality.. However, when top outliers removed, the changes more nuanced: the attractor dimensionality for heavy-tailed weights increases in the chaotic regime, while the transition chaos pushed larger g" when more top outlier weights are pruned as mentioned above (Figs, 19 20). The general ranking of dimensionality by is largely consistent with the main paper for both dimension measures, though the max dimensionality 2 over range of g when top outliers are pruned.
1.5 is comparable to that of
B
Pruned bottom 95.0%
Pruned top 1.0%
Pruned top 3.0%
1
0 Jono
Amax
Amas
Amax
0.5
12"
=..6)
-2212)
+2.44)
2.0
2-2.0 2.000"
.0..1)
2.0 -1.05)
10 2
10
100
10
10
10-1
100
10
10
2
10
10a
10
Figure 18: Effect of pruning on critical gain g². (A) bottom 95% removed. (B) top 1% removed.
(C) top 3% removed.
A
Pruned bottom 95.0%
Pruned top 1.0%
Pruned top 3.0% (LD)
Lyapunov Dimension
Lyapunov Dimension
Lyapunov Dimension
200
200
200
100
100
100
0
10
10"
10-
10
10"
102
10
10"
10a
9
g
g
Figure 19: Effect of pruning on Lyapunov dimension. (A) bottom 95% removed. (B) top 1%
removed. (C) top 3% (LD) removed.
A
B
C
Pruned bottom 95.0% (PR)
Pruned top 1.0% (PR)
Pruned top 3.0% (PR)
Participation Ratio
300
Participation Ratio
300
2,0
1.0
300
o-2.
a-15
a=10
Participation
200
200
100
100
01
0
0
10
100
10
10
g
Figure 20: Effect of pruning participation ratio (PR). (A) bottom 95% removed. (B) top 1%
removed. (C) top 3% removed.
35

---

L Information processing in heavy-tailed recurrent neural networks
To examine whether our results extend structured external imputs (and toward learned settings), we provide croof-of-concept through a reservoir-compee computing experiment on the delayed-memory XOR task (a similar task used in [Huh and Sejnowski, 2018). We use networks of size N 1000 and report average performance across three trials. Specifically, in the XOR task, each trial presents two binary stimulus vectors 81, 82 separated by silent delays; after the second delay, the readout must report XOR(s1, 82), requiring short-term maintenance of both stimuli and nonlinear decision rule.
Across gains g, heavy-tailed reservoirs exhibited broader and more stable operating regime than Gaussian reservoirs: the transition to chaos was slower and more robust (Fig. (2IA), and task performance remained high over a wider range of g (Fig. 21B). These observations suggest that the extended critical regime of heavy-tailed networks can enhance robustness and performance without fine-tuning, with potential benefits for machine learning applications.
A
Max Lyap. under diff, dist. width in autonomous
B
Test accuracy under diff. dist. width XOR
100
Test Test accuracy (%)
Chance accuracy
70
-2
60
-3-
==
Predicted transition of ==1
-4
Predicted transition =
40
109
101
109
10
Figure 21: Delayed-memory XOR with heavy-tailed reservoirs. (A) Dynamics across gain g as measured by maximum Lyapunov exponent for Gaussian (pink) heavy-tailed reservoirs (blue), showing a slower, more robust transition to chaos in the latter. (B) Task accuracy of a linear readout on the same reservoirs, with heavy-tailed networks maintaining strong performance over a broader range of g.
36

---

M
Additional details
M.1
Experiments compute resources
All experiments reported in this paper can be reproduced using CPUs only; no GPUs are required. The only exception Fig. [ for which we strongly recommend using single GPU avoid potential compatibility issues with the JAX package. Jobs were executed on a compute cluster using maximum of 4 CPU cores and 20 GB of memory per task (which is very conservative allocation; for networks of size N 1000. for example, 5 GB is likely sufficient). Each experimental run was allocated up 24 hours of wall-clock time. Most runs completed well within this limit, with small networks N = 1000 usually completed within 5 hours running serially over a grid of 50 gain values, three tail indices a, and over trials. Storage requirements were modest and standard across all runs. While additional preliminary experiments were conducted during development, they did not require significantly more compute and are not reported in the final results.
M.2 Licenses for existing assets
This project makes use of several open-source Python packages. While the main paper does not formally cite each package, we acknowledge their use here and ensure full transparency by providing all code and dependencies in the released repository. Below we list each core package, its version, license, and citation if applicable:
Package
Version
License
URL
Citation
jax, jaxlib
v0.4.38
Apache 2.0
sttps://github
[Bradbury
com/ com/google//1 jax jax
2018
numpy
v1.26.4
Modified BSD
httpa: /umpyy
Harris et al.
2020
org
scipy
v1.15.2
BSD
attps://scipy
Virtanen al.
2020
org/
torch
v2.7.0
Modified BSD
Paszke et al.
2019
org/
tensorfleu, keras
v2.19.0,
v3.9.2
Apache 2.0
nttpa:// uuv.
Martín et al.
2015
tensorflow.co org/
matplotlib
v3.10.1
PSF
https://
Hunter 2007
hatplotlib.org
tqdm
v4.67.1
MIT
https://github com/ hon/tqdn/tqdm
Table 1:
Third-pane Python packages used in this paper, with version numbers, licenses, source URLs, and citations where applicable.
Python versions >=3. 10 and <3. are recommended. All software dependencies are installable via pip using the provided requirements. txt. No proprietary assets were used in this study.
37


Reviews:
[{"review_id": "OSQcqArF6r", "reviewer": "Reviewer_PCeb", "summary": "The paper investigates recurrent neural networks (RNNs) whose synaptic weights are drawn from Lévy $\\alpha$-stable (heavy-tailed) distributions.\nThe authors derive a finite-size theory for the critical gain $g^\\ast$ at which an autonomous network leaves the quiescent fixed point and becomes chaotic.\nThey also confirm the prediction in simulations and demonstrate that heavier tails ($\\alpha<2$) broaden the “edge-of-chaos’’ regime, producing a slower, more robust transition and reveal a trade-off heavier tails compress activity onto lower-dimensional attractors, quantified by smaller Lyapunov and participation-ratio dimensions.", "questions": "1. Can the authors discuss more about how robust the critical-gain prediction is when the annealed assumption is removed? Can the authors bound the variance of $g^\\ast$ across quenched realizations analytically?\n2. Do the authors believe that robustness stems from global heavy-tail statistics or from extreme outliers? An $\\alpha$-stable law has diverging variance, so a handful of ''mega-synapses’’ could be driving the dynamics. Do the authors have a sense of which scenario better accounts for the observed phenomena? One way to probe this would be by removing the top few weights (or clipping them) and seeing whether that shifts the critical gain and attractor dimension.\n3. How easily can this analysis be extended to a setting with mixed $\\alpha$ weights (i.e., a subset of weights with $\\alpha_1$, and another with $\\alpha_2$, etc.)? A full investigation would likely be best for a follow-up work, but it seems like an interesting extension that may be worth commenting on to improve the model's biological plausibility.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\nThis is a well-written paper that addresses an important gap in the literature on heavy-tailed RNNs, which are of significant interest for several reasons, chief among them being modeling biological and neural connectivities.\nThe results are clearly presented and substantiated, and the results of the analysis are carefully interpreted to reveal insights about how heavy tails impact finite-size networks (e.g., clear quiescent-to-chaotic transition).\nIt is a compelling instance showing how mean-field analyses falls short of characterizing an interesting phenomenon.\n\nTo my knowledge, this is indeed the first finite-size theory for heavy-tailed RNNs, and it yields interesting results.\nThe authors provide solid empirical support with extensive sweeps over $g$, $\\alpha$, and $N$ validate theory and expose non-trivial phenomena (e.g.\\ non-monotonic behaviour for $\\alpha<1$).\n\n**Weaknesses**:\nThe analysis relies on an annealed approximation to the true, quenched, setting.\nThere is a significant gap between the behavior of the annealed and quenched settings, as shown in Figure 1, especially for smaller $N$.", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer’s thoughtful feedback and their high opinion on the significance, solidity, and clarity of our work, especially pointing out that, to the best of their knowledge, our work presents the first finite-size theory for heavy-tailed RNNs (along with Reviewer PCeb). \n\nThe reviewer mainly has questions regarding the interpretation of what drives robustness, the setting and assumptions of our theoretical analysis, and an interesting extension of the work to models of mixed heavy-tailed weight distributions. Here, we address each, with additional experiments on probing whether outlier weights are critical to the observed phenomena. **Together with the discussion below, does this sufficiently address the reviewer’s questions?**\n\n> Do the authors believe that robustness stems from global heavy-tail statistics or from extreme outliers? An $\\alpha$-stable law has diverging variance, so a handful of ''mega-synapses’’ could be driving the dynamics. Do the authors have a sense of which scenario better accounts for the observed phenomena? One way to probe this would be by removing the top few weights (or clipping them) and seeing whether that shifts the critical gain and attractor dimension.\n\nWe thank the reviewer for their spot-on insight! To probe this, we ran the simulations in Figs. 2 and 3, with four settings: we removed (i) top 1%, (ii) top 3%, (iii) bottom 90%, (iv) bottom 95% of recurrent weights by the absolute magnitude.\n\nWe found the results largely indicate it is a handful of “mega-synapses” driving the dynamics, implying **the robustness indeed largely stems from *extreme outliers.*** Specifically, we found the slow transition is no longer present in cases (i) and (ii), and the transition points are shifted to larger $g^\\*$ (the more top outliers pruned, the larger $g^\\*$ becomes). The slow transition persists in cases (iii) and (iv) when the outliers remain but the rest of the weights are pruned.\n\nSimilarly, removing the bottom percentage of weights has very little effect on the general trend of attractor dimensionality as presented in the main paper. However, when top outliers are removed, the changes are more nuanced: we observe that the attractor dimensionality for heavy-tailed weights increases in the chaotic regime, while the transition to chaos is pushed to a larger $g^\\*$ when more top outlier weights are pruned as mentioned above. The general ranking of dimensionality by $\\alpha$ is largely consistent with the main paper for both participation ratio and Lyapunov dimension, though the max dimensionality of $\\alpha=1.5$ is comparable to that of $\\alpha=2$ over a range of $g$ when top outliers are pruned.\n\nGiven no external links can be included per the submission rule, we state the results here and will include the figures in the manuscript if accepted.\n\n\n> The analysis relies on an annealed approximation to the true, quenched, setting. There is a significant gap between the behavior of the annealed and quenched settings, as shown in Figure 1, especially for smaller N.\n\n> Can the authors discuss more about how robust the critical-gain prediction is when the annealed assumption is removed? Can the authors bound the variance of across quenched realizations analytically?\n\nThanks for the insights! Inspired by the reviewer’s comment, we ran additional simulations to check how realization-to-realization fluctuations of the transition point change with the size of the network $N$. For simplicity, we focus on $\\alpha=1.0$. In the manuscript, we showed that the annealed theory predicts scaling $g^\\* \\sim 1/\\ln(N)$. In the case of quenched weight disorder, when averaged over realizations of the weight matrix, $\\langle g^\\* \\rangle$ also seems to behave like $\\sim1/\\ln(N)$ and it stays close to the annealed theoretical prediction for all $N$, with a small positive bias that slowly decreases with $N$. The standard deviation also decreases with increasing $N$. \n\nIn order to characterize the relative strength of the fluctuations, we additionally plotted the coefficient of variation as a function of $N$. It showed a significant decreasing trend, indicating that in the studied range ($N$ between $32$ and $3000$), the standard deviation decreases faster than the mean. Similarly, the relative mean squared error (i.e., the MSE between annealed and quenched transition $g$ divided by the squared mean annealed transition $g$) decreased with $N$. These results suggest that the dynamics becomes ‘more self-averaging’ and the annealed approximation becomes better with $N$. However, these empirical observations do not give us certainty that the same qualitative trends continue for even larger networks. Unfortunately, the mathematical analysis of the finite-size effects in the quenched networks is beyond the tools used in this work, and we were not able to obtain mathematical results bounding the variance across quenched realizations. \n\nTo summarize, as expected, the annealed theory is an approximation of the behavior of the quenched system (it is exact for feedforward networks, but not exact for RNNs). But this approximation predicts quite well the average behavior of even small quenched systems and does seem to get better as $N$ increases. \n\nWe plan to include the results of this numerical experiment in the Appendix of the revised version of the manuscript. \n\n> How easily can this analysis be extended to a setting with mixed weights (i.e., a subset of weights with $\\alpha_1$, and another with $\\alpha_2$, etc.)? A full investigation would likely be best for a follow-up work, but it seems like an interesting extension that may be worth commenting on to improve the model's biological plausibility.\n\nWe thank the reviewer for this very insightful and constructive suggestion. We agree that a mixture setting is an interesting avenue and could enhance biological plausibility, and that a full investigation would be best suited for future work. We believe our framework can indeed be extended to this setting, though there are multiple possibilities to consider: e.g., the model can \n\n(i) be homogeneous, where each neuron’s weight is chosen randomly from one of $\\alpha_1$ or $\\alpha_2$ distributions, or \n\n(ii) have two (or more) interacting populations, where each population has a distinct $\\alpha$ and projects differently within or across groups—this could biologically correspond to different cell types with distinct synaptic statistics and connectivity patterns. \n\nWe will include this point in our Discussion to highlight the potential for future exploration."}, {"review_id": "5odCVxr8xZ", "reviewer": "Reviewer_zXGE", "summary": "This paper focuses on chaos in finite-size RNNs with heavy tailed weight distributions. Authors first show that the transition to chaos in finite-size heavy tailed RNNs is sharp and depends on the network size. They relate the empirically observed extended critical regime to the largest Lyapunov exponent of the system. Finally, authors show that heavy-tailed RNNs in the critical regime present lower-dimensional chaotic dynamics the heavier the weight distribution tail is.", "questions": "- How do the results relate to task/information processing in recurrent networks? In particular, how doe s the extended critical regime and lower dimensionality interact with information processing at large, whether for reservoir computing or for a trained network? If the authors can provide a better link to task performance I would reconsider my evaluation score.\n- Can the authors provide support for the claim regarding the tradeoff between dimensionality and robustness? \n- In Figure 3c, the participation ratio increases with \\alpha for strongly chaotic regime (large g). However for small g, near the transition (g<1) it appears the trend is reversed. What is the intuition behind this?\n- Can the authors provide intuition for the definition of the Lyapunov dimension (Eq 4)? In their results it seems like the first term is dominant, is that right?", "limitations": "The authors have well identified the limitations of their work, in particular the open question of how the extended critical regime and lower dimensionality interact with information processing, be it in a reservoir computing framework or any type of trained RNN.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \nThe submission is technically very solid and clearly written. Section 4.1 in particular is an elegant piece of work! While the transition to chaos in heavy tailed networks has been studied previously (Wardak and Gong, PRL 2022), the results on dimensionality via the PR make this work novel and interesting. \n\nTo the best of my knowledge, this sharpness-vs-size phenomenon for heavy-tailed RNNs has not been documented yet.\n\nOne additional contribution of the work (Line 95, point b) is to show the dependence of the transition point with N, which could be added to the list of contributions in line 63. \n\nWeaknesses:\nOverall the major weakness of the paper is the relevance for the NeurIPS community. The study appears to be directed primarily to a statistical physics audience, rather than to the machine learning community. No downstream information-processing or task-performance results are reported.\n\nOther weaknesses include:\n- More references to works in NeurIPS or other ML conferences would be appreciated.\n- Line 64 could be misleading, as the authors reveal the nature of the transition, not the existence of the transition itself.\n- In Figure 2a, it is unclear that the yellow and green dashed lines are overlapping. The authors could use transparencies to make this clear.\n- The discussion, especially lines 345-362, is overstated and could be more precise.", "was_revised": true, "final_justification": "The authors have responded to my technical questions and they have motivated their work better in the context of ML, including a new benchmark on an XOR task, which is a fair compromise for my request to better understand how their heavy tailed weight initializations affect task performance. I have therefore raised my score by a point.", "rebuttal": "We deeply appreciate the reviewer’s detailed feedback and their high opinion on the elegance of Section 4.1 and the general clarity & solidity of our work. Reviewers zXGE & PCeb both point out that we present the first documentation of sharpness-vs-size phenomenon for heavy-tailed RNNs.\n\nThe reviewer’s main concerns are regarding related extension, e.g., how the results tie to info processing, and suggests including more ML literature to reach a broader audience. Here we state the preliminary result on the robustness of heavy-tailed weights benefits reservoir computing performance in a temporal XOR task as a proof-of-concept, along with additional edits and intuition they kindly suggested. **Together with the discussion below, does this sufficiently address the reviewer’s concerns?**\n\n> Overall the major weakness of the paper is the relevance for the NeurIPS community…directed primarily to a statistical physics audience, rather than to the ML community. No downstream information-processing or task-performance results are reported.\n\n> More references to works in NeurIPS or other ML conferences would be appreciated.\n\n> How do the results relate to task/information processing in RNNs?...whether for reservoir computing or for a trained network? If the authors can provide a better link to task performance I would reconsider my evaluation score.\n\nWe sincerely appreciate the constructive feedback aimed at broadening our reach and the opportunity to better situate our work in ML. We would like to respectfully clarify why we think the current work is directly relevant to ML, and how we will strengthen the relevance following the feedback:\n\n**(i) Downstream info-processing**\n\nWhile the core aim is to provide a principled & general characterization of the dynamical regimes induced by heavy-tailed connectivity (common in biological systems but underexplored in ML), we recognize the value of linking these results to info processing.\n\nIf accepted, **we will include our new result on a delayed memory XOR task under a reservoir computing setup** (see task description in response to Reviewer V5Z3). Here, we observe that heavy-tailed networks exhibit a broader gain regime with strong performance compared to Gaussian networks. This aligns with our main results showing heavier-tailed networks exhibit a slower, more robust transition to chaos, which translates to task benefits here. This would provide a concrete proof-of-concept that the extended critical regime can enhance robustness and performance without fine-tuning, beneficial in ML.\n\nWe hope the reviewer would find this addition a good middle ground between an extensive study of info processing (which we think is best-suited for future work) v.s. the current version of the paper, and would reconsider the evaluation score given the proposed change & below. \n\n**(ii) Current relevance to ML and NeurIPS**\n\nWe respectfully argue that our study on the edge-of-chaos regime in untrained networks is highly relevant to ML, with precedents cited: prior work establishes the edge of chaos as a key initialization regime for network trainability and information propagation in both feedforward and recurrent networks (cited in lines 36-41). \n\nIf accepted, we will integrate more references to further emphasize the ML connection, with the existing ones:\n\n* Schoenholz et al. (ICLR, 2017; **cited**) analyzed **untrained** random networks, showing that the trainability of deep networks depends on initializing near the edge of chaos; the farther from this critical regime, the shallower a network must be to remain trainable.\n\n* Bertschinger et al. (NeurIPS, 2004; **cited**) proposed that only near the edge of chaos can RNNs perform complex computations on time series.\n\n* Hayou et al. (ICML, 2019) extended Schoenholz et al. (2017) by tuning initialization and activation functions along the edge of chaos, improving training efficiency and performance.\n\n* Yang & Schoenholz (NeurIPS, 2017), with mean-field theory, show that residual connections enable deep networks to operate near the edge of chaos, thereby preserving input space geometry and improving stability across depth.\n\n* The topics are still highly relevant to ML, e.g., regularizing Lyapunov exponents can significantly enhance the effectiveness of RNN training (Engelken, NeurIPS, 2023); the expressivity and trainability of the Fourier Neural Operator peak at the edge of chaos, as shown via mean-field theory (Koshizukae et al., NeurIPS, 2024).\n\nWe build directly upon these foundational insights. We show that heavy-tailed weight distributions naturally *broaden* the edge-of-chaos regime, an attribute desirable for training (above). While we draw upon tools from statistical physics, we believe the insights are relevant and our practice is fairly common in ML, e.g., mean-field analysis, energy landscapes, & scaling laws.\n\nWe also emphasize the novelty of our contribution: theoretical understanding of heavy-tailed weights in finite-size networks remains limited due to analytical challenges. Our results provide a tractable & general framework for characterizing the dynamical behavior of such networks. We hope this foundational work will spark community interest in further investigations in ML. \n\n> Line 64 could be misleading, as the authors reveal the nature of the transition, not the existence of the transition itself.\n\nWe respectfully clarify that, to our knowledge, the **existence** of the transition in finite-size networks is not a previously known result (lines 102-106) and **is first revealed in this work** (Section 4.1). The transition does not exist for the `tanh` activation function in the infinite-size limit under a mean-field theory (lines 103-104).\n\n> In Fig 3c, the participation ratio increases with \\alpha for strongly chaotic regime (large g). However for small g, near the transition (g<1) it appears the trend is reversed. What is the intuition behind this?\n\nIntuitively, this is due to the early but slow transition of heavy-tailed networks. As shown in Fig. 2, heavier-tailed (smaller $\\alpha$) networks undergo the transition to chaos at lower values of $g$, but the transition is slower. Thus, for small $g$, heavier-tailed networks have entered the weakly chaotic regime, whereas Gaussian networks remain in the quiescent regime. This explains why the attractor dimensionality can be higher for smaller $\\alpha$ at low $g$: the heavier-tailed network has already developed some dynamical richness. \n\n> Can the authors provide support for the claim regarding the tradeoff between dimensionality and robustness?\n\nThe tradeoff between dimensionality & robustness can be supported by interpreting Fig 2 (robustness) and Fig 3 (dimensionality) together, where only the tail-index $\\alpha$ varies. The intention is to show that RNNs of smaller $\\alpha$ (heavier tail) exhibit a slow transition to chaos (more robust to changes in $g$) but have a lower participation ratio & Lyapunov dimension (lower dimensionality). \n\nWe interpret this as a tradeoff because of the opposite effects each is expected to have on information processing. While placing a system closer to the edge of chaos should allow it to integrate information along longer timescales (in RNNs) or across more layers (in feedforward networks), reducing the dimensionality of the attractor is expected to lower the effective dimensionality of the tasks for which these benefits can be observed. This is because information can only be effectively passed along the chaotic attractor, whereas any information encoded within directions associated with very negative Lyapunov exponents is effectively erased due to rapid contraction.  \n\n> One additional contribution of the work (Line 95, point b) is to show the dependence of the transition point with N, which could be added to the list of contributions in line 63.\n\nThanks! We will modify point b to “We reveal that…, *with the locations of the transition depending on network size N.*”\n\n> Can the authors provide intuition for the definition of the Lyapunov dimension (Eq 4)? In their results it seems like the first term is dominant, is that right?\n\nYes, the 1st term is dominant. The Lyapunov dimension (LD) is related to the Kaplan-Yorke conjecture that states, for ‘typical attractors’, Lyapunov exponents (LEs) can be used to estimate the fractional (information) dimension. Intuitively, it gives the number of dimensions needed to describe dynamics on the attractor. The positive (negative) LEs correspond to directions in which a small cloud of points is expanded (contracted). In the definition of the LD, $k$ denotes the largest integer such that the sum of the top LEs is still positive. Thus, if we follow an initial $k+1$-dimensional cloud of points generated around a point on the attractor, we will notice that the overall volume of the cloud tends to contract on the attractor (since the sum of Lyapunov exponents is negative). In contrast, if we instead follow a $k$-dimensional cloud of points associated with the top $k$ LEs, the sum is (usually) still positive, so the cloud tends to expand. The expansion and contraction are balanced when, on average, the number of dimensions is equal to the LD. \n\nThe 2nd term estimates the fractional part of the information dimensions and lies between $0$ and $1$, so the first term indeed dominates in our analysis of high-dimensional chaotic activity.\n\n> In Fig 2a, it is unclear that the yellow and green dashed lines are overlapping. The authors could use transparencies to make this clear.\n\nThanks. We will tune the transparencies to improve visualization.\n\n> The discussion, especially lines 345-362, is overstated and could be more precise.\n\nThanks. We will word the discussion more carefully, e.g.:\n* Instead of saying “supports the idea” (line 351), reword to “provides one piece of evidence”.\n* Instead of saying “are directly testable” (line 360), edit to “offer inspirations for future works”."}, {"review_id": "wsa5jSj9CE", "reviewer": "Reviewer_ga7q", "summary": "This paper investigates recurrent neural networks (RNNs) initialized with heavy-tailed distributions and examines their dynamic properties. Using metrics such as transition points, Lyapunov exponents, Lyapunov dimension, and participation ratio, the authors demonstrate that heavier-tailed synaptic connectivity expands the regime of stable, edge-of-chaos dynamics, while simultaneously reducing the dimensionality of the resulting activity.", "questions": "Please see 'Weakness'.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper provides detailed mathematical proofs and visualizations, along with a comprehensive set of simulations. \n\nMy main concern is that all the networks studied are untrained and rely solely on initialization (as the authors acknowledge). How do the properties of heavy-tailed recurrent neural networks evolve during training? Furthermore, do the weight distributions remain heavy-tailed after training?\n\nAnother point of concern is the use of activation function. What role does the choice of activation function play in the observed dynamics? Would alternative nonlinearities, such as ReLU, significantly alter the findings?\n\nAlthough the model draws inspiration from biological systems, the study lacks a direct comparison with neural data.\n\nGiven that the authors acknowledge these limitations and have nonetheless conducted a thorough analysis of initialization-only networks, my score will be borderline but positive.", "was_revised": true, "final_justification": "This is a well-written paper that addresses a gap on heavy-tailed RNNs. The authors have addressed all my concerns.", "rebuttal": "We deeply appreciate the reviewer’s comprehensive feedback. We sincerely thank the reviewer for acknowledging the thoroughness of our analysis on fixed heavy-tailed networks, and that their main concerns regarding limitations are captured in our discussion section of the submission.\n\nThe reviewer’s main concerns are regarding further results beyond the fixed heavy-tailed networks, e.g., analysis of trained networks, as well as the role of activation functions and comparison to neural data. Here, we address each with our justifications and any related intuition, as well as additional results on reservoir computing networks. **Together with the discussion below, does this sufficiently address the reviewer’s concerns?**\n\n> My main concern is that all the networks studied are untrained and rely solely on initialization (as the authors acknowledge). How do the properties of heavy-tailed recurrent neural networks evolve during training? Furthermore, do the weight distributions remain heavy-tailed after training?\n\nWe appreciate this constructive feedback. Our aim is to establish a clean baseline by analyzing untrained heavy‑tailed RNNs, much as what Schoenholz et al. (ICLR, 2017) did for **untrained** Gaussian deep feedforward networks, where they found trainability hinges on initializing near the edge of chaos. How the heavy‑tailed weight distribution evolves under learning is a rich follow‑up problem whose answer depends on the training regime. \nIn “lazy’’ or neural‑tangent‑kernel limits (Jacot et al., NeurIPS, 2018; Chizat et al., NeurIPS, 2019), weights move only infinitesimally, so we would expect the heavy tails (and the broadened critical region we report) to persist. By contrast, a more sophisticated investigation is required in “rich” / feature‑learning regimes, such as extending our finite‑size theory to the stochastic dynamics of SGD. \n\nExploring how our finite‑size theory interacts with learning dynamics is therefore an exciting direction we leave for future work; we will emphasize this point (lines 363-366) in the revision, if accepted, to highlight the open questions and invite community investigation.\n\nAdditionally, following the reviews, if accepted, we will include in the appendix our new result on a delayed memory XOR task under a reservoir computing setup (see task description in the rebuttal response to Reviewer V5Z3). In this setting, we observe that heavy-tailed networks exhibit a broader gain regime with strong performance compared to Gaussian networks. This aligns with our main results showing heavier-tailed networks exhibit a slower, more robust transition to chaos, which translates to task benefits. This would provide a concrete proof-of-concept that the extended critical regime can enhance robustness and performance without fine-tuning, which is beneficial in ML applications.\n\nWe hope that the reviewer would find this addition a good middle ground between adding an extensive study of training (which, in our opinion, would distract the main narrative and findings of this paper) v.s. the current version of the paper.\n\n> Another point of concern is the use of activation function. What role does the choice of activation function play in the observed dynamics? Would alternative nonlinearities, such as ReLU, significantly alter the findings?\n\nWe thank the reviewer for pointing out this important question. Our analysis is based on the linear stability analysis of the quiescent fixed point, which exists for activation functions that satisfy $\\phi(0)=0$. Moreover, we utilize the linearization, which is valid as long as the activation function has a linear part around the origin, i.e., it admits an expansion of the form $\\phi(x) = ax + o(x)$. The existence of the transition is then assured by the linear stability analysis. \n\nThe analysis can be extended to cases such that $\\phi(0) \\neq 0$, by expanding the update equation around the resulting nonquiescent fixed point. The analysis is complicated by the fact that now the location of the stable fixed point itself may change with the order parameter $g$, but the qualitative nature of the transition should generically remain unchanged.\n\nThe nature of the dynamics above the transition point (i.e., after the fixed point becomes unstable) depends on further details of the activation function. For example, if the activation function is linear, the activity diverges once the fixed point becomes unstable. In general, a sublinear activation function is needed to control the divergence. We expect our results to hold for any smooth saturating nonlinearity with features described above. The case of ReLU seems more complex, and we expect it to behave in a more complex manner, leading to divergent dynamics for large $g$.\n\nWe plan to add a short discussion of this issue in the revised version of the manuscript.\n\n> Although the model draws inspiration from biological systems, the study lacks a direct comparison with neural data.\n\nThank you for raising this thoughtful point. Our goal in the present work is to lay a clear foundation by isolating the consequences of heavy‑tailed connectivity, rather than fitting a specific dataset. We do, however, cite empirical studies that report heavy‑tailed synaptic weight distributions being common in the brain across species (lines 19-25), which motivated our study. We agree that this is a valuable and exciting next step to build on our foundation and compare directly with neural recordings. For instance, future work can estimate Lyapunov spectra or other dynamical fingerprints from long, high‑resolution trajectories. We recognize that such an endeavor is technically demanding, given the need for stable, extended observations, and we will highlight this avenue for future work in the discussion if accepted."}, {"review_id": "4X7BI8CmqZ", "reviewer": "Reviewer_V5Z3", "summary": "This paper analyzes the edge-of-chaos dynamics of the recurrent neural networks (RNN) with heavy-tailed connections. The authors show that comparing to the RNN using Gaussian connections, the heavy-tailed RNN exhibit a broader gain regime near the edge of chaos, while has few dimensionality of the attractor. This result reveals a trade-off of the robustness and the representation ability of the edge-of-chaos state in the RNN.", "questions": "1. Can the main results be recovered when the activation function $\\phi$ is unbounded (ReLU, for example)?\n2. Could the authors provide an intuitive explanation for why, during the mathematical analysis, the weight matrix $W_{ij}$  can be treated as being redrawn at each time step?", "limitations": "Please see the weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. The paper is clearly written and easy to follow.\n2. The main result that the robustness and dimensionality are trade-off is interesting and may be important for deeper understanding of the nonlinear neural dynamics in the brain.\n3. The authors present a systematic set of experiments in both the main paper and the supplementary materials, making their experimental results highly convincing.\n\n### Weaknesses\n\n1. It is not clear whether the results can be extended to the cases that the weights of the RNN are learned instead of randomly sampling, or when the external inputs are structured, limited the impact of the paper.\n2. The variation of the dimensionality and the participation ratio near the edge of chaos is not well explained theoretically. It is better to give more explanation (some intuitive explanations are also helpful).", "was_revised": true, "final_justification": "The rebuttal addresses most of my concerns, and I maintain my postive score.", "rebuttal": "We deeply appreciate the reviewer’s comprehensive feedback. We sincerely thank the reviewer for acknowledging the clarity and soundness of our findings, and pointing out their potential significance to a deeper understanding of the nonlinear neural dynamics in the brain.\n\nThe reviewer’s main concerns are regarding how the results extend to trained RNNs that are task-driven, in addition to seeking clarifications on our analysis. Here, we address each by providing further explanations and intuitions, and state the preliminary result on the robustness of heavy-tailed weights benefits reservoir computing performance in a temporal XOR task (as a proof-of-concept for the robustness benefits in information processing). **Together with the discussion below, does this sufficiently address the reviewer’s concerns?**\n\n> It is not clear whether the results can be extended to the cases that the weights of the RNN are learned instead of randomly sampling, or when the external inputs are structured, limited the impact of the paper.\n\nThank you for this insightful comment. We agree that assessing heavy‑tailed dynamics under learned weights and structured inputs is an exciting next frontier. Because training introduces many additional factors, such as optimizer noise, curriculum effects, and loss‑surface geometry, to name a few, we believe it would be best suited for a dedicated follow‑up study that we hope to pursue. We also warmly welcome community efforts in that direction. \n\nTo offer an initial glimpse and better situate our results in information processing, we will add new reservoir-computing experiments on the delayed memory XOR task as a proof-of-concept for the case of structured external inputs (as suggested by the reviews). \n\n**Task:** In this setting, each trial presents two binary stimulus vectors $s_1, s_2$, each drawn from one of two classes and separated by silent delay periods; after the second delay, the readout must report $\\operatorname{XOR}(s_1, s_2)$. This requires the network to store both stimuli in short‑term memory and apply a nonlinear decision rule. A similar delayed‑XOR task is studied in Huh & Sejnowski (NeurIPS 2018).\n\nWe observe that heavy-tailed networks exhibit a broader gain regime with strong performance compared to Gaussian networks, aligning with our main results showing heavier-tailed networks exhibit a slower, more robust transition to chaos, which can translate to task benefits. This would provide a concrete proof-of-concept that the extended critical regime can enhance robustness and performance without fine-tuning, which is beneficial in ML applications. The revised manuscript, if accepted, will report this result.\n\n> The variation of the dimensionality and the participation ratio near the edge of chaos is not well explained theoretically. It is better to give more explanation (some intuitive explanations are also helpful).\n\nThanks for the constructive feedback. We believe that the qualitative intuition behind the large disparity in Lyapunov dimensions between Gaussian and heavy-tailed networks can be obtained through Fig. 3A: the Lyapunov exponents are more dispersed for networks with heavier tails. Thus, in heavy-tailed networks, only a small number of leading Lyapunov exponents become positive when being close to the edge of chaos, leading to lower values of Lyapunov dimension. This, in turn, is driven by the more heterogeneous distribution of eigenvalues of the weight matrix in heavy-tailed networks. However, the exact correspondence between the eigenvalues of the weight matrix and the Jacobian is nontrivial. Consequently, we were not able to obtain analytical predictions of the Lyapunov and participation ratio dimensions, but we will add this intuition to Section 4.3. \n\n> Can the main results be recovered when the activation function is unbounded (ReLU, for example)?\n\nIf the activation function is unbounded, there is no longer guarantee that the dynamics does not diverge. Therefore, while the existence of the transition can still be assured for a large class of activation functions (see our response to Reviewer ga7q), the nature of the dynamics above the transition may qualitatively change depending on the activation function.\n\n> Could the authors provide an intuitive explanation for why, during the mathematical analysis, the weight matrix W_{ij} can be treated as being redrawn at each time step?\n\nIn the case of feedforward networks, weights at different layers are usually drawn independently at initialization, so our analysis is exact in this case. In the case of RNNs, however, the weights usually stay constant across time steps, so this analysis is an approximation. A priori, there is no obvious reason why this approximation should provide good estimates of the quantities of interest, but in some cases it can become exact in the limit of infinite system size (e.g., in sparsely diluted, asymmetric version of the Hopfield network, see Derrida, Gardner, & Zippelius, A., EPL, 1987). The basic intuition is that the correlations induced by the quenched disorder decrease with the size of the network and become negligible in infinite networks. \n\nAlthough we do not currently know whether the annealed approximation becomes exact in the infinite RNNs with heavy-tailed weights, our numerical experiments indicate that the relative mean squared error (i.e., the MSE between annealed and quenched transition $g$ divided by the squared mean annealed transition $g$) of the estimated location of the transition does significantly decrease with $N$, at least in the range between $N=32$ and $N=3000$."}]

Author rebuttal:

Meta-review: This paper discusses dynamics of RNNs. Given that I have only worked on this area briefly and a long time ago, I do not feel competent to comment. However, since they are all positive, I recommend acceptance.
Decision: Accept (poster)