On Scrambling Phenomena for Randomly Initialized Recurrent Networks 
Authors: Vaggos Chatziafratis
Ioannis Panageas
Clayton Sanford
Stelios Andrew Stavroulakis
Venue: NeurIPS 2022 Accept (2022)
Keywords: RNNs
Scrambling
Jacobian
dynamical systems
trajectories
chaos
initialization
recurrent networks

Abstract: Recurrent Neural Networks (RNNs) frequently exhibit complicated dynamics, and their sensitivity to the initialization process often renders them notoriously hard to train. Recent works have shed light on such phenomena analyzing when exploding or vanishing gradients may occur, either of which is detrimental for training dynamics. In this paper, we point to a formal connection between RNNs and chaotic dynamical systems and prove a qualitatively stronger phenomenon about RNNs than what exploding gradients seem to suggest. Our main result proves that under standard initialization (e.g., He, Xavier etc.), RNNs will exhibit \textit{Li-Yorke chaos} with \textit{constant} probability \textit{independent} of the network's width. This explains the experimentally observed phenomenon of \textit{scrambling}, under which trajectories of nearby points may appear to be arbitrarily close during some timesteps, yet will be far away in future timesteps. In stark contrast to their feedforward counterparts, we show that chaotic behavior in RNNs is preserved under small perturbations and that their expressive power remains exponential in the number of feedback iterations. Our technical arguments rely on viewing RNNs as random walks under non-linear activations, and studying the existence of certain types of higher-order fixed points called \textit{periodic points} in order to establish phase transitions from order to chaos.

On Scrambling Phenomena for Randomly Initialized Recurrent Networks
Vaggos Chatziafratis'
Ioannis Panageas Department of Computer Science
Department of Computer Science and Engineering University of California, Santa Cruz vaggos@ucsc. edu
University of California, Irvine
ipanagealica www.cc.....uccooooooooooo.... .uu.
Clayton Sanford
Stelios Andrew Stavroulakis Department Computer Science University of California, Irvine sstavrou@uci.cm edu
Department of Computer Science Columbia University claytonêcs. cowumbia......mmmc edu
Abstract
Recurrent Neural Networks (RNNs) frequently exhibit complicated dynamics, and their sensitivity to the initialization process often renders them notoriously hard to train. Recent works have shed light on such phenomena analyzing when exploding or vanishing gradients may occur, either of which is detrimental for training dynamics. this paper, we point a formal connection between RNNs and chaotic dynamical systems and prove a qualitatively stronger phenomenon about RNNs than what exploding gradients seem to suggest. Our main result proves that under standard initialization (e.g., He, Xavier etc.), RNNs will exhibit Li- Yorke chaos with constant probability independent of the network's width. This explains the experimentally observed phenomenon of scrambling, under which trajectories nearby points may appear to be arbitrarily close during some timesteps, yet will be far away in future timesteps. In stark contrast to their feedforward counterparts, we show that chaotic behavior RNNs is preserved under small perturbations and that their expressive power remains exponential in the number of feedback iterations. Our technical arguments rely on viewing RNNs as random walks under non-linear activations, and studying the existence of certain types of higher-order fixed points called periodic points that lead to phase transitions from order to chaos.
Introduction
In standard feedforward neural networks (FNNs), computation is performed "from left to right" propagating the input through the hidden units to the output. In contrast, recurrent neural networks (RNNs) form feedback loop, transfering information from their output back their input (e.g.. LSTMs (Hochreiter and Schmidhuber, 1997), GRUs (Cho et al., 2014)). This feedback loop allows RNNs share parameters across time because the weights and biases of each iteration are identical. As result, RNNs can capture long range temporal correlations in the input. For these reasons, they have been very successful in applications in sequence learning domains, such as speech recognition, natural language processing, video understanding, and time-series prediction (Bahdanau et al., 2014; Cho et al., 2014; Chung et al., 2014).
Authors order determined by the output of a randomly initialized recurrent network (operating the chaotic regime).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Unfortunately, their unique ability to share parameters across time comes at a cost: RNNs are sensitive to their initialization processes, which makes them extremely difficult to train and causes complicated evaluation and training dynamics (Le et al., 2015; Laurent and von Brecht, 2016; Miller and Hardt, 2018). Roughly speaking, because the hidden units of an RNN are applied to the input over and over again, the final output can quickly explode or vanish, depending on whether its Jacobian's 's spectral norm is greater or smaller than one respectively. Similar issues also arise during backpropagation that hinder the learning process (Allen-Zhu et al., 2018).
Besides the hurdles with their implementation, recurrent architectures pose significant theoretical challenges. Several basic questions include how to properly initialize RNNs, what is their expressivity power (also known as representation capabilities), and why do they converge or diverge, all of which require further investigations. In this paper, we take closer look at randomly initialized RNNs:
Can we get a better understanding of the behavior of RNNs at imitialization using dynamical systems?
We draw on the extensive dynamical systems literature- wwhih has long asked similar questions about the topologiaal behavior of iterated compositions of functions- to study the properties of RNNs with standard random initializations. We prove that under common initialization strategies, e.g., He or Xavier (He et al., 2015, 2016), 6). RNNs can produce dynamics that are characterized by chaos, even in innocuous settings and even in the absence of external input. Most importantly, chaos arises with constant probability which is independent of the network's 's width. Our theoretical findings explain empirically observed behavior of RNNs from prior works, and are also validated in our experiments.
More broadly, our work builds on recent works that aim at understanding neural networks through the lens of dynamical systems; for example, Chatziafratis et al. (2020b,a) use Sharkovsky's s theorem from discrete dynamical systems to provide depth- wiith tradeoffs for the representation capabilities of neural networks, and Sanford and Chatziafratis (2022) further give more fine- ane-grained lower bounds based on the notion of "chaotic itineraries".
1.1 Two Motivating Behaviors of RNNS
Before stating our main result, we illustrate two concrete behaviors of RNNs that inspired our work. The first example demonstrates that randomly initialized RNNs can lead to what is perhaps most commonly perceived as "chaos", while the second example demonstrates a qualitatively different behavior of RNNs compared to FNNs. Our main result unifies the conclusions drawn from these two.
Scrambling Trajectories at Initialization Prior works have empirically demonstrated that RNNs can behave chaotically when their weights and biases are chosen according to certain scheme. For example, Laurent and von Brecht (2016) consider a simple 4-dimensional RNN with specific parameters in the absence of input data. They plot the trajectories of two nearby points x, y with || - yll S 10 7 as they are propagated through many iterations of the RNN. They observe that the long-term behavior (e.g., after 200 iterations) of the trajectories is highly sensitive to the initial states, because distances may become small, then large again and so on. We ask:
Are RNNs (provably) chaotic even under standard heuristics for random initialization?
We answer this question in the affirmative both experimentally and theoretically. This question is helpful to understand for multiple reasons. First, it informs us about the behavior of most RNNs. as we start with a random setting of the parameters. Second, proving that a system is chaotic is qualitatively much stronger than simply knowing its gradient to be exploding; this will become evident below, where we describe the phenomenon of scrambling from dynamical systems. Finally, understanding why and how often an RNN is chaotic can lead to even better methods for initialization.
To begin, we empirically verify the above statement by examining randomly initialized RNNs. Figure I demonstrates that trajectories of different points may be close together during some timesteps and far apart during future timesteps, or vice versa. This phenomenon, which will be rigorously established in later sections, is called scrambling (Li and Yorke, 1975) and emerges as a direct consequence of the existence of higher order fixed points (called periodic points) of the continuous map defined by the random RNN.
'Our code is made publicly available httt::///iihau..cc//eeeottttaae Prr.ooddd..oooobbtatttttttte RNNS ipynb
blob/main/ Depthh R__NNs__aa and_

---

xand
0.8
0.6
0.4
Distnnce
0.2
0.0
60
100
120
140
Ieet alionss
Figure 1: Points 2, with initial distance ||2 1--yl yll S 10-7 and their subsequent [lf (v)l
distances across 1 = 150 iterations of a randomly initialized RNN. The idea behind scrambling is that the trajectories, even though they get arbitrarily close, they also separate later and vice versa.
Persistence of Chaos in RNNs vs FNNs Our second example is related to the expressive power of neural networks. A standard measure used to capture their representation capabilities is the number of linear regions formed by the non-linear activations (Montufar et al., 2014); the higher this number, the more expressive the network is. Counting the maximum possible number of linear regions across different architectures has also been leveraged in order to obtain depth vs width tradeoffs in many works (Telgarsky, 2015; Eldan and Shamir, 2016; Chatziafratis et al., 2020b,a).
Here, we are inspired by remarkable observation of Hanin and Rolnick (2019a), who showed that real-valued FNNs from - R may lose their expressive power if their weights are randomly perturbed or those weights are initialized according to standard methods. Roughly speaking, they show that in such networks the number of linear regions grows only linearly in the total number of neurons. These results contrast with the aforementioned analyses of the theoretical maximum number of regions, which is exponential in depth. We ask the analogous question for RNNs instead of FNNs:
Will randomly initialized or perturbed RNNs lose their high expressivity like FNNs did?
We show a contrast between RNNs and FNNs, by showing that the number of linear regions in random RNNs remains exponential on the number of its feedback iterations. Once again, the fundamental difference lies on the fact that RNNs share parameters across time. Indeed, the analyses of many prior works (Hanin and Rolnick, 2019a,b; Hanin et al., 2021) crucially relies on the "fresh" randomness injected at every layer, e.g., that the weights/biases of each neuron are initialized independently of each other; obviously, this is no longer true in RNNs where the same units are repeatedly used across different iterations. This shared randomness raises new technical challenges for bounding the number of linear regions, but we manage to indirectly bound them by studying fixed points of random RNNs.
2 Our Main Results
Our main contribution is to prove that randomly initialized RNNs can exhibit Li- Yorke chaos (Li and Yorke, 1975) (see definitions below), and to quantify when and how often this type chaos appears as we vary the variance of the weights chosen by the random initialization. To do so, we use discrete dynamical systems which naturally capture the behavior of RNNs: simply start with some shallow NN implementing a continuous map f. and after ł feedback iterations of the RNN, its output will be exactly f (f composed with itself times). We begin with some basic definitions:
Definition 2.1. (Scrambled Set) Let (X, d) be a compact metric space and let f X + X be continuous map. Two points x, y E X are called proximal if:
lim inf ((f"(()) (z), (y)) =0 000
and are called asymptotic if
lim f")) = 0.
A set Y C X is called scrambled Yx, y E Y, + y, the points x; y are proximal, but not asymptotic. Definition 2.2. (Li- Yorke Chaos) The dynamical system (X, is Li- Yorke chaotic if there is an uncountable scrambled set Y X.
3

---

PNN
RMN
Without
After adding
Without
After adding
Figure 2: Left: Recreation of figure from Hanin and Rolnick (2019a) where small Gaussian perturbation (O, 0.1)) added independently on every weight/bias of each layer of a simple FNN. As result, after adding noise, the high expressivity (i.e., number of linear regions) breaks down; having "fresh" noise in each layer was crucial. Right: Here we depict the same network as before but shown as RNN, and we add noise before. Perhaps surprisingly, RNNs exhibit different behavior than FNNs: high expressivity is preserved even after noise. As we show, due to shared randomness across iterations, small perturbations do not "'break" expressivity.
Li- Yorke chaos leads both to scrambling phenomena (Figure 1) and to high expressivity (Right of Figure 2). Interestingly, this not true for FNNs (Left of Figure 2). where the fresh randomness at each neuron leads to concentration of the Jacobian's norm around 1, which is sufficient to avoid chaos. In other words, these definitions capture exactly the fact that trajectories get arbitrarily close, but also move apart infinitely often (as in Fig. 1). Intuitively, when scrambling occurs in RNNs, their #linear regions will large and their input-output Jacobian will have spectral norm larger than one. Definition 2.3 (Simple RNN Model). For k. 0, let RNN o2) be family of recurrent neural networks with the following properties:
Input: The input to the network is
1 1-dimensional, i.e., single number x E [0, I].
Hidden Layer: There is only one hidden layer of width at least k, with ReLU(x)) =
max(x, 0) activations neurons, each of which has its own bias terms by ~ Unif([0. [0. 1]).
Output: The output real number [0, 1] which takes the functional form fx o ) and
clip(E 1= ReRLUU( b,)), where the weights are i.i.d. Gaussians NN(0, & clip() ensures that fx (zz [0, similarly the input (i.e., it "clips" the input so that remains in [0. 1] as it is an RNN).
Feedback Loop: fx(z)) becomes the new input number, after iterations the output is (S& S& o... o fx)(2), i.e., compositions of fk with itself.
As we show, even this innocuous class of RNNs (see Sec. 3 for general model) leads to scrambling. Theorem 2.4 (Li-Yorke Chaos at Initialization). Consider Sa E RNN o²) initialized according to the He normal initialization (set o² 2, so weight variance is 2/k). Then, there exists some constant > (independent of the width) and width kHe such that for sufficiently large k > kHe, fu is Li- -orke chaotic with probability at least 8.
This answers our two questions posed earlier: RNNs may remain chaotic under initialization heuris- ties, and maintain their high expressivity, because Li- Yorke chaos implies exponential #linear regions (Theorem 1.5 in Chatziafratis et al. (2020b)). Next, we focus on threshold phenomena:
Theorem 2.5. [Order-to-Cre -hCoos Transition] For fk E RNN o?), we get the following 3 regimes vary the variance of the weights a4 (O0 ))
(Low variance, order) Let a; N'0, 4e (llogk) Then, the probability that fx. Li- Yorke chaotic is most +

---

(Edge of chaos) Let a, ~ NO, constant (albeit small) probability.
2) (i.e., He initialization). Then, fu is Li- Yorke chaotic with
(High-variance, chaos) Let ax lim;->o Pr [fx is Lorko chaotic]= 1.
NOO, 4) for
((1).
Then,
To the best of our knowledge, we are the first to study Li- Yorke chaos in the context of RNNs and to prove that scrambling phenomena appear with constant probability independent of the width. We emphasize that Li- Yorke chaos is stronger than exploding gradients, as any chaotic system requires the input-output map be non-contracting (as otherwise would converge fixed point).
Techniques Contrary to many prior works (Laurent and von Brecht, 2016; Miller and Hardt, 2018), we do not require differentiability of activations like tanh(-), as we rely on properties of continuous functions. Most importantly, we deviate from past works (Bertschinger and Natschläger, 2004; Poole et al., 2016), as we do not use tools from mean-field field analysis (where the number of neurons, width or depth goes to infinity), since here we are interested on the dependence on the width k. Instead, we study the fixed points and higher-order fixed points of fa E (K,((K, o²) (see Sec. 3), bounding the probability that fa has a certain type of fixed point known to lead chaos.
This is done by viewing RNNs as random walks under non-linear activations. To give some intuition behind our findings, consider a random walk starting from 0 in which we consider increments d, (O, 1). It is quite well-known in the literature of random walks, that the expected number of steps needed for random walk starting from with increments d; NVO0, 1) reach n and come back to 0 is (n²2) iterates. For an RNN to be chaotic, as it turns out, we need to find the probability that it has certain type of fixed points; this can be bounded below by the probability that the random walk reaches point 1 and then goes back to O. As result, if we rescale appropriately and the increments d; ~ N N(0 }). then we need ((k) iterates. Roughly speaking, plays the role of the size of the width (d; are the weights). In reality, the random walk we analyze is more complicated as the (random) biases affect the length of each increment the random walk.
Experiments Finally, we validate our theory by extensive simulations on a variety of randomly initialized RNN architectures. Our goal is twofold: Firstly, to demonstrate that Li- Yorke chaos is indeed present across different models and different initialization heuristics, and not just an artifact of the specificities of our RNN family. Secondly, to estimate the value of the aforementioned constant probability of chaos 8, in the Edge of Chaos regime, which is known to be the most interesting (Bertschinger and Natschläger, 2004; Yang and Schoenholz, 2017; Hanin, 2018), yielding a comparison for which initializations are more likely to be Li- Yorke chaotic (see table in Fig. 3).
2.1 Further Related Works
Several works have studied RNNs in the context of chaotic dynamical systems and edge of chaos initialization (Sompolinsky et al., 1988; Bertschinger and Natschläger, 2004; Saxe et al., 2013: Kad- mon and Sompolinsky, 2015; Poole et al., 2016). They study neural networks in the limit of infinitely large widths or depth and derive properties of the dynamics, typically using mean-field analysis. Moreover, the notion of chaos used there is not from Li and Yorke (1975), they do not analyze periodic points in the trajectories of the networks, nor do they establish scrambling phenomena. For example, Bertschinger and Natschläger (2004) observe that the sequence of activations in the neurons of the network exhibit increasingly unpredictable and complicated patterns as the variance increases and use this notion to determine the edge of chaos and threshold phenomena.
3 Preliminaries
Higher-order Fixed Points Let f: [a + [a, 6] be a continuous map. The notion of a periodic point is generalization of a fixed point:
Definition 3.1. We say f contains period has point of period 2 if there exists a point To E [a, b) such that:
(200 zo and f'(z0) # vIstsn
5

---

where it is common to use f (zo) to denote the composition of f with itself n times, evaluated at point g-g In particular, == ffo. (zx), f(f(z))........... f=-1(z)) has n distinct elements (each of which is point of period n) and is called cycle (or orbit) with period n.
Period 3 implics Chaos Of particular interest, is the case functions f that contain period 3. In 1975, a seminal paper by Li and Yorke (1975), introduced the term "chaos" as used in mathematics nowadays, and proved connections with periodic functions as defined above.
Fact 1. Let f: R + be continuous. If f has period point then two properties hold: First, f contains points of any period n E N. Second, f will exhibit scrambling as defined in Def. 2.1,2.2.
Sometimes we say that f' is "chaotic the sense of Li and Yorke". On a historical note, this turned out to be special case of an older result by Sharkovsky (1964).
In other words, a sufficient condition obtain scrambling phenomena is to show that contains a point of period 3. In general, the uncountable set of chaotic points may, however, be of measure zero, which case the map said have unobservable chaos or unobservable nonperiodicity (Collet and Eckmann, 2009). Our Theorem 2.4 informs us that this is not the case for randomly initialized RNNs.
Other RNN Models Recurrent models have been thoroughly studied because of their success in sequence learning. Following notation from Laurent and von Brecht 6), here is general model for RNNs:
Ue (u W1 W2 WA
where the index N represents the current time, Ut E IRd represents the current state of the system, I the current external input, and W, 's are weight matrices. Regarding training or initialization of weights many tricks have been proposed, e.g., the popular "identity matrix" trick Le et al. (2015) to preserve lengths. Laurent and von Brecht (2016) to simplify the above, they study what is referred to as the dynamical system induced by the RNN, i.e, they consider trajectories of u, in the absence of external imputs:
Ut = ( 0, .,
This is much more tractable and gives us insights into the architecture of the RNN as decouples the influence of the external data 2;, 's (with which any possible response be produced) from the model itself. For more on other simplified RNN variants with specific statistical assumptions on weights/input, and connections to edge of chaos, see Bertschinger and Natschläger (2004). Similarly, our R.N family has no external input; we show that even in 1-dimensional settings, chaos emerges.
Persistence of Chaos and Phase Transitions
In this section we prove our main results. As we alluded to in the previous section, in order to show chaos in the sense of Li and Yorke, we will bound the probability that the family RNN o²) contains point of period 3, since this will serve as a "witness of chaos". We start with some simple observations.
4.1 Triangular Wave with Period
Perhaps the simplest example of a continuous function f that has period 3 is the following triangle wave function: fxx) 2x, for x [0, 1/2), and 2(1 z). for x E (1/2, 1]. Indeed, one can check that the set { 1 $ 5. 2 is periodic orbit of length 3, since f( 3. ( = and finally. f()) = closing the loop. This simple function has played an interesting role obtaining depth width tradeoffs in recent years Telgarsky (2015, 2016); Chatziafratis et al. (2020b).
4.2 Persistence of Chaos
Now consider a function from RNN: fa(x) = clip( a, ReLU(( - b:)) for independent a- ~ (0, o²) and b, Unif([0, 1)).
Theorem 4.1. Fix any 8 E (0, 1). There exists a such that for any sufficiently large k (whose minimum value depends on 8), I& is 3-periodic with probability at least 1 8.
For the theorem need some intermediate lemmas. We begin with sufficient condition for chaos:

---

Lemma 4.2. Let ys- = Ei= ab(b; - b.) so that fb,b,) = clip():). If there exists indices i < l such that y; > 1 and yr then f& has period
Proof. We can assume without loss of generality that bj < <by <... < by. Suppose there are indices
i < as specified in the statement. Since yi > 1, after clipping we know that fa (zi) = 1, and since
#1 < 0, after clipping we know that fx(xz) = 0. Let's check how intervals are mapped under f: since
3; S æi E [0, 1] and because continuity of [s. we have that [=([0,,r]] -r3)) = [o, 1]. and similarly that
fi ([2,, zzl) [0. 1]. By combining this with the intermediate value theorem for fi (3 iterations of
fa applied to itself), we can exhibit a 3-cycle with two elements in (0, xz) and a third in (Iz, Ze). C
Proof Sketch of Theorem 4.1. Roughly, the proof operates by choosing subset of i ordered elements {us, heej C be that meets several key properties such that there exists j < j' such that V/ > 1 and yy < O. Using this and Lemma 4.2 (for = j and & = j'), we will conclude that fa has period 3. However, there are three immediate challenges that make such result challenging to obtain:
1. (Randomness) If the variance parameter o is small, then there will be a substantial constant probability of an yj lying in the interval [o, 1J and thus meeting neither characteristic.
2. (Correlations) Due to the function's 's expression as a linear combination of ReLU neurons with small coefficients, the function fs is smooth, and y in the same vicinity are highly correlated. This makes it more difficult to ensure that sign changes to the yj 's are likely.
3. (Biases) The randomness of the biases b; makes it more difficult to assess the correlations of the y since having small gaps be b; will yield stronger correlations between the two consecutive values of the function.
We mitigate these issues by carefully selecting parameters & and a and the subset (Viy hee( Problems from (1) can be avoided by choosing a to be large enough to ensure that for sufficiently large i relative to k, it's highly unlikely for any Yi, to lie in the interval [0, 1]: if none of these lie in that interval, then is sufficient to find yi, and yi,, that exhibit a positive-to-negativ minimize correlation for (2) between nearby yi, 's, we choose those ij's 's to ensure that each is large multiplicative factor times its predecessor; this relies on key intuition from random walks, where increasing temporal differences between random variables makes them increasingly independent. We avoid the atypical samples of b; of (3) by gradually defining set of "good"" b, 's that exist with high probability and completing the proof while assuming the 6, 's are good.
sign change. To
Proof of Theorem 4.1. A primary goal of our analysis is to bound the variances and covariances of y: 'ss to aid in our selection of a subset of yi 's that are nearly independent. We first assume that the biases b; are held fixed, while the weights 0; are chosen random; we will later prove bounds that hold for all typical biases. We let E, [-] denote such an expectation over random a for fixed b. For
indices i < L, we have that E. [us] and also the following:
i-1
0² 1-1 (b. - 6.)² and E, [yavel 0²
1-1 Σ (b. b.) b.). k
Eg [ = Σ
a
(b, b
For fixed by's, each yi = Ei=l (b. b.) is drawn from Gaussian distribution with mean 0 and variance =² [b= (b, - b.)². . We bound these quantities with high probability over random choices FF
of biases b;: for the remainder of the argument, we assume that the www-orbabillity "bad case" (i.e..
having small gaps by - b,) does not occur and apply those bounds without worry about the effects of the bias terms. By defining b, as the ith smallest element from sample of size k from Unif( [0. 1)). the distribution of each is a Beta distribution: b; ~ Beta(i, k + 1 i) (see Bertsekas and Tsitsiklis (2008)). We recall the low-order moments of these biases in order to bound the moments of the y 's. Assume < i.
i(i +
i(k + - (I: + 1)2 + 2)
E [bj
E
and Var [b.]
1
1+1
(k: + 1)(k 2)
We establish simple lower bound on the summation used to express E. [w:]: E, [v]] < 2F (b:
22 We apply Chebyshev's inequality to lower-bound by and upper-bound bi/2. To do so, note
7

---

that for sufficiently large k, Var [b:] bound.
(1+1)//4 (6+1)=(6+2)
1 2
Assume i 2 13/1,
Then, by Chebyshev's
7i
32
Pr by
< Pr
bb
EE [bl]
8k
8k
vk
By the same calculation, Pr [bs/2 2
5i
32 Then, Pr Em [
e
2
64
Later, this
8E
will be important for showing that My & [0, 1] with high probability with a large enough.
We also upper-bound the covariances. Note that E. [u:ve] S ia
b,be. For i 2 k3/4, the above
I
36 with probability at Ieast Vk" calculations ensure that Es [y:ve] I 2
Now, we choose the elements Vi, that guarantee the needed properties. Let ig == p (assuming that k is divisible by 2r*) for t == and r = (12t) We assume that 2 (2 r4t, which ensures that every iz 2 h///4, since
k3/4 pt
1//4
These choices ensure that the variance of each element is high enough to make belonging to [o, 1] unlikely for any gi while spreading apart successive iterates Vig and Vi to make them nearly
independent. To do so, we analyze the normalized variables zj == V Vare [1
Vi,
for j E [d].
Note that, given fixed b, Iz ~ NO, and with probability
128r
for every j <
E
302
E.g [xyary]
2r 2140037
12
12 1
Vrr 4
Vara [ Vare
ykfrr-3'
Hence, for a sufficiently large r, the Ttys 's are essentially uncorrelated. Note that x: (x1,... ;e) E R (if assume fixed b that satisfies the above high probability events) has ~ (0, E), where Σjj= = 1 and Ej.y' S 2-- The smallest eigenvalue of can be bounded. (Similarly for Mmax (E.) see Lemma B.I.) Given the lack of correlation between the random variables for sufficiently large we bound the probability that the signs of the elements of æ follow certain pattern. We basically need show that has a + sign that precedes by- 2 slope). Using this, we can estimate the probability of period 3 emerging and this completes the proof. (Due space limitations, some last steps are in Appendix B.)
sign (like triangle waves have +2 slope followed
Corollary 4.3. For weights drawn o, -V(O, ))) the lime- Pr [fr is 3-periodic] 1.
Combining the fact that f will have period with some constant probability 6, with the work of Chatziafratis et al. (2020b) on the expressivity of networks as measured by #linear regions we get:
Corollary 4.4. With the same probability 6, the number of linear regions of J& E RNN is exponen- tial in the number t of feedback iterations, with growth rate of 1+v5
Theorem 4.5. Consider f& initialized uccording to the He initialization (i.e. S& from Thm, 4. with
2). For sufficiently large k, fk is 3-periodic w.p. at least some constant. (Proof in App. B.2.)
Theorem 4.6. Consider Sy initialized with 0; ~ N(, Ik b- Then, the probability that Sk has no fixed points (therefore exhibits no chaotic behavior) is 2 1 }. (Proof in App. B.3.)
5
Experiments
Our goal to evaluate how robust our findings are to specifications of the RNN model, and to empirically estimate some bounds on how often scrambling phenomena appear".
"Our code is made publicly available here: http:://gw.......e...e.oootttraa
blob/main/Du Depth_ J_NNN____a Saadd_hhos_Peee Laanos_Period___ Probabbiitttrtrrrrrrrree RNNs ipynb

---

Robustness across Different Models We try different architectures and initialization techniques from the literature (He et al., 2015; Glorot and Bengio, 2010), truncated Gaussians, and others, for setting the weights and the biases. Our table in Fig. 3 summarizes the results across 10000 runs of each experiment. For each experiment, the first layer has width 2, where the weights and biases of each neuron are initialized according to each line, whereas the second layer has width with weight and bias as specified in each line. Notice that the last column contains the probability that the recurrent model is chaotic, having period 3. The way we classify whether the output of an RNN is chaotic or not done by plotting one iteration of the RNN and three iterations of the RNN and then checking whether there more fixed points in the latter, implying period 3 (for pictorial representation see Figure 7 in Appendix C). The purpose of this experiment was to quantify the probability for
Layer
Layer
Pr[period 3]
weight
bias
activation
weight
bias b
activation
1 He He
He
ReLU
He
He
1
13.77% %
Uniform
ReLU
He
Uniform
ReLU
7.49%
He
ReLU
He
He
ReLU
4.51 %
N (o. }) Glorot
N/ (0,
1-1,,
ReLU
(o, }) N (o, 1) [-1,1]
ReLU
4.45%
Glorot
ReLU
Glorot
Glorot
ReLU
2.28%
Figure 3: The rightmost column has the estimates for the probability that the RNN exhibits period
We ran the experiment for 10000 times and checked whether the random RNN has period (see Fig. 7). Each line specifies the type of initialization activation unit used.
having period one-dimensional randomly initialized RNNs, as higher dimensions there not clean mathematical statement for checking periods 3. This validates theoretical findings that chaos (in the sense of Li- Yorke) persists, in contrast to FNNs (see Figure 2).
RNNs transitions in Higher Dimensions In higher dimensions, we used an MNIST dataset as input to a 64-dimensional RNN with hidden layer, width 64, fully connected, with ReLU activation functions and He initialization. Note that we did train the RNN as this was not our purpose. We simply initialized it and observed (Fig. 4) how the values of the output neurons fluctuate with respect to (the number of compositions of the RNN with itself) for various a. Since high dimensions do not have clean characterizations of chaos such as period 3, we instead detect chaos by relying on necessary condition: Jacobian's spectral norm being more than I (See Fig. 5).
1e8
0.15
0.16
= 0.17
15
Velues
-5
-10
-10
80
100
20
60 80
100
Compositions of RNN
Compesitions of RMN
Compositions of RNN
Figure 4: The y-axis has values for 4 different output neurons, while x-axis is number of RNN compositions. From left-to-right, as we vary a, RNN becomes unstable similarly ID case in Fig. 1.
Additional Experiments and Remarks. Regarding the experiments reported in Figure 3, the reported numbers were without the Clipping operation, so that the architecture is more faithful to practice. We believe an interesting direction further understand the chaotic properties of neural networks at initialization and exactly quantify both experimentally and theoretically the probability of chaos. Of course, for the simpler one-dimensional case, we were able to use the period 3 property which is a simple to state and simple to verify condition. For higher dimensional RNNs, that are closer to those being used in practice nowadays, it still remains challenging problem to characterize when chaos emerges. Other notions of chaos like Devanay's 's chaos (Huang and Ye, 2002) could also

---

% of randomy inttiaized ANNS whose jacobian spectrol nomm is creater thai
1.0
Normal
0.8
0.4
0.2
0.003
0.025
0.100 0.125 0.150 0.175 0.200 Standard Deviation at each weight
Figure 5: The y-axis is the of RNNs (out of 1000 runs) having large Jacobian norm, and x-axis is varying values of a. Echoing our main theorems 2.4, 2.5, we experimentally determine for two popular initialization schemes the threshold where the input-output Jacobian has spectral norm >
be useful when exploring chaotic properties of neural nets (Devaney's chaos is stronger notion of chaos compared Li-Yorke's 's chaos).
For completeness, we also performed some additional experiments with the Clipping operation and the numbers still ranging from 2%%9% for the probability of chaos. Specifically, following the setup Fig. 3, out 10000 runs, we got (with Clipping) that the chaos frequency reported in the last column is 3.6, 5.8, 9.7, 5.7, 2.2 respectively for the different initializations. We also performed experiments with other types of activations units and higher values for the width and the depth and still observe that these variants
exhibit period 3. For example, with tanh activations, the last column would be 9.7, 11, 17.6, 2.6, 4.6 respectively. we instead change the width to be equal to the numbers become 13.1,8. 8.1, 14.09, 0.8, and in another setting where the depth equals the numbers become 6.5, 3.1, 5, 0.34, 1.14.
Conclusion
Using tools from discrete dynamical systems, we show simple models for randomly initialized recurrent neural networks exhibiting scrambling phenomena and we study the associated threshold phenomena for the emergence Li- Yorke chaos. Our theory and experiments explain observed RNN behaviors, highlighting the difference between FNNs and RNNs in simple setting. More broadly, we believe this connection can fruitful to obtain a better understanding of RNNs at initialization adds to a recent thread of works that give interesting new perspectives on neural networks, borrowing ideas from dynamical systems.
Acknowledgments
The authors would like thank the anonymous reviewers for their time and their useful feedback that improved the presentation of our paper. VC would like to acknowledge support from a start-up grant at UC Santa Cruz. IP would like to acknowledge support from start-up grant at UC Irvine. Part of this work was done while the authors VC, IP, SS were visiting UC Berkeley and the Simons Institute for the Theory Computing, while VC was FODSI fellow at MIT and Northeastern. CS was supported by NSF GRFP and NSF grants CCF-1814873 and I18-1838154.
10

---

References
Allen-Zhu, Z., Li, Y., and Song. Z. (2018). On the convergence rate of training recurrent neural networks. arXiv preprint arXiv:1810.12065.
Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:
Bertschinger, N. and Natschläger, T. (2004). Real-time computation at the edge of chaos in recurrent neural networks. Neural computation, 16(7):1413-1466
Bertsekas, D. and Tsitsiklis, J. N. (2008). Introduction to probability, volume 1. Athena Scientific.
Chatziafratis, V., Nagarajan, S. G., and Panageas, I. (2020a). Better depth-width trade-offs for neural networks through the lens of dynamical systems. In International Conference Machine Learning, pages 1478. PMLR.
Chatziafratis, V., Nagarajan, S. G., Panageas, I., and Wang, X. (2020h). Depth-width trade-offs for relu networks via sharkovsky's theorem. International Conference Learning Representations, Addis Ababa, Africa.
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint 11X6v:1406.110 1078.
Chung, J., Gulcehre, C., Cho, K., and Bengio, Y. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint urXiv:1412.3552
Collet, P. and Eckmann, J.-P. (2009). Iserated maps on the interval as dynamical systems. Springer Science & Business Media.
Eldan, R. and Shamir, O. (2016). The power of depth for feedforward neural networks. In Conference on learning theory, pages 907-940.
Glorot, X. and Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings.
Hanin, B. (2018). Which neural net architectures give rise to exploding and vanishing gradients? arXiv preprint arXiv: 01001100/777
Hanin, B., Jeong, R., and Rolnick, D. (2021). Deep relu networks preserve expected length. arXiv preprint arXiv:2102.10492.
Hanin, B. and Rolnick, D. (2019a). Complexity of linear regions in deep networks. In International Conference on Machine Learning, pages 2596-2604. PMLR.
Hanin, B. and Rolnick, D. (2019b). Deep relu networks have surprisingly few activation patterns, In Advances in Neural Information Processing Systems, pages 359-368.
He, K., Zhang, X., Ren, S., and Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEFE international conference on computer vision, pages 1026-1034.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778.
Hochreiter, and Schmidhuber, J. (1997). Long short-term memory. Neural computation, 98): 1780.
Huang, W. and Ye, X. (2002). Devaney's chaos or 2-scattering implies li-yorke's chaos. Topology and its Applications, 117(3):259227 272..
Kadmon, J. and Sompolinsky, H. (2015). Transition to chaos random neuronal networks. Physical Review X. 504)0041030

---

Laurent, T. and von Brecht, J. (2016). A recurrent neural network without chaos. ICLR.
Le. Q. V., Jaitly, N., and Hinton, G. E. (2015), A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv: 1504.00941.
Li, T.-Y. and Yorke, A. (1975). Period three implies chaos. The American Mathematical Monthly, 82(10):985-992.
Miller, J. and Hardt, M. (2018). Stable recurrent models. arXiv preprint arXiv: 10369.
Montufar, G. F., Pascanu, R., Cho, K., and Bengio, Y. (2014). On the number of linear regions of deep neural networks. In Advances in neural information processing systems, pages 2924-209 2932.
Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., and Ganguli, S. (2016). Exponential expressivity in deep neural networks through transient chaos. Advances neural information processing systems, pages 3360-3368.
Sanford, C. H. and Chatziafratis, V. (2022). Expressivity of neural networks via chaotic itineraries beyond sharkovsky's is theorem. International Conference on Artificial Intelligence and Statistics, pages 9505 9549. PMLR.
Saxe, A. M., McClelland, I.., and Ganguli, S. (2013). Exact solutions the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120
Sharkovsky, O. (1964), Coexistence of the cycles of continuous mapping of the line into itself. Ukrainskij matematicheskij zhurnal, 16(01):61-71.
Sompolinsky, H., Crisanti, A., and Sommers, H.-J (1988). Chaos in random neural networks. Physical review letters, (3):225.
Telgarsky, M. (2015). 1rXiv:1509.08101.
Representation benefits of deep feedforward networks. arXiv preprint
Telgarsky, M. (2016). Benefits of depth in neural networks. In Conference on Learning Theory, pages 1517-1539.
Yang, G. and Schoenholz, S. (2017). Mean field residual networks: On the edge of chaos, arXiv preprint 1rXiv:1712.08969.
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] We just used MNIST dataset. Our code is made publicly available here: https://gi htggttthuub... cot./saaooottttttooloooooooooaa. iostavroulkkiss Chaos hatiie//lobbmaati RNNs ipynb
2 RNNs and_ Chwos_aaarraaaee P
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts your work? [N/A] We develop theory to understand observed behavior of RNNs.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Some of the proofs are in the Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] As URL:Our code is made publicly available here: http:s///ggaabb..ccaeeeeettttrre wwwp/staliottarrllaaasss/ee////e
https://github.
https//ChhhootRlleooooooaaaaaaaa httts//bllo/mmm///ppttttt RNNS and_ Chaos Liciosseeriod___ 3at.333aoaaaaaayaaa__.. ipynb
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "znzJU4ZE9m", "reviewer": "Reviewer_YhQb", "summary": "The paper explores how randomly initialized RNNs exhibit \"Li-Yorke chaos\" with a certain probability and which leads to exploding gradients and other undesirable phenomena. By using the fact that functions containing a period three point it exhibits chaos, some probabilistic bounds on the emergence of chaos in RNNs are derived. Finally, the theoretical results are evaluated with two toy examples.", "questions": "- Is the resulting chaos actually desirable or should it be avoided when possible?\n- If it should be avoided, what alternative initialization procedure could be used?\n- What would be the theoretical bounds for the Probabilities given in Figure 3?", "limitations": "Not applicable IMO", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Stunning maths\n- Interesting little experiments with code attached\n\nWeaknesses:\n- Short conclusion, no immediately apparent benefit other than a new name for the phenomenon.\n\nSuspected Errors:\n- Figure 1: A legend would be helpful -  what is the blue line? Typo in caption: \"adabehind\" \n- Figure 2: Unclear how spikes in the plots correspond to linear regions\n- Lemma 4.2.: Some y's should be x's?\n- Line 156 end: something missing here\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZiimKtUyLpK", "reviewer": "Reviewer_EXTU", "summary": "The authors establish non-asymptotic probability bounds that a random (randomly initialized) recurrent neural network is chaotic.\nThe analysis is performed as follows\n\n1. Hidden state $x_t$ is scalar, and the state space is compact interval $x_t \\in [a,b] \\subset \\mathbb{R}$\n\n2. ReLU is used as the activation function, though the analysis only requires continuity rather than differentiability\n\n3. Using the specific definition of chaos due to Li and Yorke for interval maps, the authors need to bound the probability that the RNN $f$ has period three fixed points, i.e. $f^3(x^*) = x^*$\n\n4. The probability of such an event is then expressed in terms of the number of neurons $k$, the variance parameter $\\sigma^2$ of a Gaussian distribution from which the 'weight matrix' is sampled.", "questions": "> small perturbations do not break expressivity\nI am not overwhelmingly familiar with symbolic dynamics, but if the activation were smooth, then the persistence should be guaranteed due to hyperbolicity. I wonder if the authors could offer some comments on the hyperbolicity and robustness.\n\nLines 190-193:\nCould the authors comment further on cases when chaos is unobservable?\n\nFigure 5 and accompanying code:\nIt appears the figure is produced by calculating the input-output Jacobian of the 2 layer NN specified in the paragraphs 'RNNs transitions in Higher Dimensions'. I would argue that evaluating either input-output Jacobian of the iterated map or its Lyapunov exponent would provide a more convincing evidence for chaos.\nOn a related note, I do believe that use of the function `complexe_modulo` is erroneous-- stability of the system should depend only on the real part of the eigenvalues.", "limitations": "N/A", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The submission offers a novel, rigorous analysis that quantifies how commonly is chaos found in RNNs with finitely many neurons.\nUnlike the mean-field approaches, $\\lim{k \\to \\infty}$ isn't necessary.\n\nThis allows to establish chaos more rigorously than the use of Lyapunov exponents usually allows for.\n\nOverall, I believe this to be an extremely interesting study, with excellent presentation.\n\nI would like to offer some comments, presented in chronological order, that might help improve the manuscript further:\n\n1. I found Figure 2 somewhat hard to understand, at least initially. It might be worthwhile displaying the equation for the tent map in the figure. The graph added little intuition, partly because nodes and edges aren't explained. It might also help to mention that the FNN graph is simply 'unrolled'. Perhaps it would also help to indicate where the small Gaussian perturbation is supplied? What overall effect on the dynamics do stochastic perturbations to $b_i$'s have? \n\n2. Li-Yorke chaos is introduced, but never contrasted with other definitions e.g. one due to Devaney. This could also give an avenue to explicit all the consequences of LY chaos for interval maps, such as sensitive dependence on initial conditions/exploding gradients.\n\nSection 2.1\nMost of the cited references analyze their respective problems using ergodic theory. The juxtaposition with LY chaos is entirely warranted, but the phrasing of the sentence gives the impression that non-reliance on LY is somehow 'bad'.\n\nParagraph 'RNNs transitions in Higher Dimensions':\nInvoking LY chaos is warranted for interval maps. For higher dimensional systems period-3 fixed points do not provide sufficient nor necessary conditions for chaos. Similarly, requiring the existence of a scrambled set as in Definition 2.1, can produce dynamics that do not have the sensitive dependence on initial conditions. \nI believe this should be acknowledged. In a similar vein, paragraph 'Techniques' (lines 143-159) gives an impression that the analysis is more general than the mean-field one, rather than that it lies at the \"opposite end of the spectrum\". I believe it should be stated more clearly in the sections of the submission that technical arguments apply only to RNNs with a scalar hidden state $x_t$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Xq2e5BqDuLT", "reviewer": "Reviewer_j5Sz", "summary": "This paper shows that randomly initialized RNNs can display a type of chaos called Li-Yorke chaos and the number of linear regions they have remains exponential with depth (unlike in feedforward nets).", "questions": "In experiments reported in Figure 3, do the authors use a standard RNN or an RNN with a clip operation as the one used for the proofs?", "limitations": "RNNs are not a very commonly used architecture in machine learning any more, so I doubt that these results will be of much interest to the NeurIPS audience. This paper in addition studies a peculiar type of RNN with a clip operation after the nonlinearity that calls into question even more the relevance of these findings for even standard RNN architectures that used to be used by the machine learning community.\n\nIt wasn’t really clear to me what the implications of Li-Yorke chaos or exponential number of linear regions are in practical terms beyond things like instability and exploding gradients etc. The results are in general very far removed from the interests and concerns of the overwhelming proportion of machine learning practitioners today. I believe a more specialized venue on dynamical systems may be more appropriate for this work. \n\nFinally in Fig 4, having a Jacobian norm greater than 1 seems like a very weak proxy for what the paper is primarily concerned with, i.e. Li-York chaos.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe work seems technically sound, although I have to admit I did not go through the proofs very carefully.\n\nWeaknesses:\n\nPlease see questions and limitations below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IfQvdI_pKy0", "reviewer": "Reviewer_niom", "summary": "This paper studies the behavior of a specific class of RNNs at initialization using dynamical systems theory. Its main result shows that chaos can arise in RNNs even in the absence of external input and this behavior holds independently of the width. This result explains the scrambling phenomenon observed experimentally. It also shed light on the expressive power of RNNs, in terms of the number of linear regions formed by the nonlinear activations.", "questions": "- Is it possible to clarify the connection between Li-Yorke chaos and exploding gradients slightly more precisely?\n- Will similar chaotic behavior be observed for other variants of RNNs (say with different activations and depth)?\n- For the experiments with MNIST, how does the Pr[period 3] depends on the width of RNN? Would be good to demonstrate some results along this line", "limitations": "In the checklist, it was mentioned that limitations are described, but I could not find any such discussions in the main paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Provide first understanding of RNNs via the lens of Li-Yorke chaos\n- Overall well written\n\nWeaknesses\n- Analysis is limited to scalar inputs and RNNs with ReLU activation (tanh activation is often used in practice)\n- There are no results connecting the behavior of input data to the chaotic behavior of (input-driven) RNNs, which are highly relevant in practice", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept