How to train RNNs on chaotic data?
Authors: Zahra Monfared
Jonas Magdy Mikhaeil
Daniel Durstewitz
Venue: ICLR 2022 Submitted (2022)
Keywords: dynamical systems
back-propagation through time
chaos
recurrent neural networks
LSTM
Lyapunov spectrum
time series

Abstract: Recurrent neural networks (RNNs) are wide-spread machine learning tools for modeling sequential and time series data. They are notoriously hard to train because their loss gradients backpropagated in time tend to saturate or diverge during training. This is known as the exploding and vanishing gradient problem. Previous solutions to this issue either built on rather complicated, purpose-engineered architectures with gated memory buffers, or - more recently - imposed constraints that ensure convergence to a fixed point or restrict (the eigenspectrum of) the recurrence matrix. Such constraints, however, convey severe limitations on the expressivity of the RNN. Essential intrinsic dynamics such as multistability or chaos are disabled. This is inherently at disaccord with the chaotic nature of many, if not most, time series encountered in nature and society. Here we offer a comprehensive theoretical treatment of this problem by relating the loss gradients during RNN training to the Lyapunov spectrum of RNN-generated orbits. We mathematically prove that RNNs producing stable equilibrium or cyclic behavior have bounded gradients, whereas the gradients of RNNs with chaotic dynamics always diverge. Based on these analyses and insights, we offer an effective yet simple training technique for chaotic data and guidance on how to choose relevant hyperparameters according to the Lyapunov spectrum. 


Under review as a conference paper at ICLR 2022
How TO TRAIN RNNS ON CHAOTIC DATA?
Anonymous authors Paper under double-blind review
ABSTRACT
Recurrent neural networks (RNNs) are wide-spread machine learning tools for modeling sequential and time series data. They are notoriously hard to train be- cause their loss gradients backpropagated in time tend to saturate or diverge during training. This is known the exploding and vanishing gradient problem. Previ- ous solutions to this issue either built on rather complicated, purpose-engineered architectures with gated memory buffers, more recently- imposed constraints that ensure convergence to a fixed point or restrict (the eigenspectrum of) the re- currence matrix. Such constraints, however, convey severe limitations on the ex- pressivity of the RNN. Essential intrinsic dynamics such as multistability or chaos are disabled. This is inherently disaccord with the chaotic nature of many, if not most, time series encountered in nature and society. Here we offer compre- hensive theoretical treatment of this problem by relating the loss gradients during RNN training to the Lyapunov spectrum of RNN-generated orbits. We mathe- matically prove that RŇNs producing stable equilibrium or cyclic behavior have bounded gradients, whereas the gradients of RNNs with chaotic dynamics always diverge. Based on these analyses and insights, we adapt the old idea of teacher forcing to yield effective yet simple training technique for chaotic data, and offer guidance on how to choose relevant hyperparameters according to the Lya- punov spectrum.
INTRODUCTION
Recurrent neural networks (RNNs) are widely used across various fields in engineering and science for learning sequential tasks or modeling and predicting time series (Lipton al., 2015). Yet, they struggle when long-term temporal dependencies. very slow, or hugely varying time scales are in- volved (Hochreiter) [991] Bengio al. 1994] [Schmidt al.) [02T et al. 2018 Rusch Mishra, 2021a). Time series sequential data with such properties are, however, very common in fields like climate physics (Thomson 1990), neuroscience (Fusi et al.) 2007] Russo & Durstewitz) 2017). ecology (Turchin & Taylor 1992). or language processing (Cho et al. 2014b). Training RNNs on such data is hard because the loss gradients backpropagated in time easily saturate or diverge in this process. This is commonly referred to the exploding and vanishing gradient problem (EVGP) (Hochreiter) 1991 Bengio et al. 1994 Pascanu et al. 2013).
One solution to the EVGP is based specifically designed RNN architectures with gating mecha- nisms, such as long short-term memory (LSTM) (Hochreiter & Schmidhuber 1997) or gated recur- rent units (GRU) (Cho et al.! 2014a). These architectures allow states at earlier time steps more easily influence activity much later through a kind of protected memory buffer, thus alleviating the EVGP by structural design. like gradient clipping to keep the gradients in check (Pascanu et al., 2013). The relatively complex architectural design of these networks impedes their mathematical analysis and requires reverse en- gineering after training (Maheswaranathan et al. 2019; Monfared & Durstewitz, 2020a b) SShhiddi ctt al. 202I). Partly to forego these complications, a variety other solutions has been proposed re- cently, imposing restrictions on the recurrence matrix to bound the gradients (Arjovsky et al. 2016
practice, such models need to be backed up by further techniques
Chang et al. 2019), 9. or enforcing global stability by design or regularization (Erichson et al. 2021
Kolter & Manek, 2019). Often these procedures dramatically curtail the expressivity of the RNN
(Kerg et al. 2019 Orhan & Pitkow| 20200] Schmidt et al., 202I); in particular, they rule out chaotic
dynamics (see further discussion below).

---

Under review as a conference paper at ICLR 2022
This is at odds with the plethora of chaotic phenomena in nature, engineering, and society. Chaotic dynamics are commonplace, almost default in any complex physical or biological system. This in- cludes scientific areas diverse neuroscience (Durstewitz & Gabriel 2007; van Vreeswijk & Sompolinsky 1996), physiology (Kesmia et al. 2020). geophysics (Sivakumar) 2004), climate sys- tems (Tziperman et al. 1997). astrophysies (I (Laskar & Robutel| 1993), ecology (Duarte et al. chemical reactions (Field & Györgyi| [193). cell (Olsen & Degn, 1977) or population (May 1987) biology. Chaotic phenomena are also crucial for the understanding of societal and epidemiologi- cal processes, such as the spread of diseases (Mangiarotti et al. 2020), or in economics (Faggini 2014]. They are further relevant in purely technical contexts such as electrical engineering (Tchit-) inga et al. 2019 Kamdjeu Kengne et al., 2021] or laser optics (Kantz et al., [993). They have even been suggested to play an up to now largely neglected, but potentially very significant role in speech recognition (Sabanal & Nakagawa, 1996) and natural language processing (Inoue et al.)
2010).
2021]. Hence, in almost any practical setting, chaotic phenomena abound. They cannot, in general, be ignored when devising RNN training algorithms.
Here we offer a comprehensive theoretical treatment of the relation between RNN dynamics and the behavior of the loss gradients during training. We find close connection between an RNN's loss gradients and the largest Lyapumov exponent of its freely generated orbits. We mathematically prove that RNNs producing stable fixed point or cyclic behavior have bounded gradients. Crucially, how- ever, the loss gradients of RNNs producing chaotic dynamics always diverge. Hence, the chaotic nature of many time series data induces a principle problem, and, despite significant efforts in the past to solve the EVGP, training RNNs on such data remains an open issue. We illustrate the impli- cations of our theory for RNN training on several simulated and empirical chaotic time series, and adapt the old idea of sparsely forced BPTT as simple yet effective remedy that enables to learn the underlying dynamics despite exploding gradients.
2 RELATED WORKS
Exploding and vamishing gradients. While 'classical' remedies of the EVGP (Hochreiter 1991 Ben- gio al. 1994 Pascanu al. 2013) rest on purposetailored architectures with gating mechanisms, which safeguard information flow across longer temporal distances (Hochreiter & Schmidhuber, 1997 Cho et al. 2014a), the focus has recently shifted to simpler RNNs that address the EVGP by restricting the recurrence matrix to be orthogonal (Henaff et 2016 [Hllfiich et al. 2018 Jing al. 2019), unitary (Arjovsky et al. 2016), or antisymmetric (Chang et al.) 2019), or by ensur- ing globally stable fixed point solutions (Kag et al. |2020; Kag & Saligrama 202 for example through co-trained Lyapunov functions (Kolter & Manek 2019). However, all these approaches impose strong limitations
the dynamical repertoire of the RNN, enforcing global convergence to fixed points or simple cycles| In doing so, they drastically reduce the expressiveness of these models (Kerg et al. 2019 Orhan & Pitkow (020). To address this problem, |Erichson et al. (2021) somewhat relaxed the constraints on the recurrence matrix by introducing skew-symmetric decom- position combined with a Lipschitz condition on the activation function. Another recent approach discretizes oscillator ODEs to arrive at a stable system of coupled (Rusch & Mishral (02 or independent (Rusch & Mishra 202T6) oscillators which increase the RNN's expressiveness while bounding its gradients. By design (and as acknowledged by the authors), neither of these architec- tures is capable of producing chaotic dynamics, however, as the underlying ODEs do not allow for exponential divergence of close-by trajectories (a prerequisite for chaos). Given these often prin- ciple limitations of parametrically or dynamically strongly constrained models, a fruitful direction may be to modify the training process itself, c.g. through modified auxiliary loss functions (Trinh) ctt al. [2018] (cchmidt et al.] (022), or special procedures for parameter updating (Kag & Saligramal 2021b) or loss truncation (Williams & Zipser 1989; Menick al. 2021). Our empirical evalu- ation will follow up on such ideas, but also highlight that simple loss truncation, windowing, or architectural solutions like LSTMs are not sufficient.
Learning dynamical systems. Surprisingly disconnected from the work on the EVGP and learning long-term dependencies, huge and long-standing literature deals with training RNNs on nonlinear dynamical systems (DS) (Pearlmutter) 1990 Trischler & D O'Eleuterio] 2016 Vlachas et al., 2020). including chaotic systems like the famous Lorenz equations (Lorenz) 1963) or chaotic turbulence in fluid dynamics (Li et al. (02I). Teacher forcing (TF: Williams & Zipser 1989). Pearlmutter 1990).
We make this point more formal in Appx. A.1.6
2

---

Under review as a conference paper at ICLR 2022
Doya 1992), , also Goodfellow et al. (2016)) is one of the earliest techniques introduced to keep RNN trajectories on track while training. The idea behind TF is to simply replace RNN states by observations when available, thereby also efffetively cutting off the gradients. TF essentially derives from ideas in dynamical control theory, and adaptive schemes that increasingly hand over control to the RNN throughout training have been devised (Abarbanel) 2013 Abbarbanel et al., 2018 Bengio ict al. 2015). A related technique from the control theory literature is "multiple shooting" (Voss et al., 2004 Here the whole observed time series is chopped into chunks, and for each chunk of trajectory new initial condition is estimated. Explicit constraints ensure continuity between the separate trajectory bits during optimization. State space models and the Expectation-Maxinne became popular particularly in the 90es for uncovering the latent dynamics underlying set of time series observations (Ghahramani & Roweis, 1999), and remain an important tool until today (Durstewitz, 2017; Koppe et al., 2019). Most recently, approaches based on variational inference and the reparamcterization trick (Kingma & Welling 2014). like sequential variational autoencoders (SVAE), gained in popularity for DS approximation (Hernandez et al. 2020). "Deterministic" RNNs (i.e., with latent states not treated as random variables), like conventional LSTMs (Vlachas et al,, 2018 remain top choices for DS reconstruction, however.
algorithm
Although connections between DS ideas and loss gradients have been drawn early on (Bengio et al,, 1994], far only particular scenarios (like fixed point attractors) have been considered. Closest to our work is recent work by |Schmidt et al.] (2021), where non-divergence of loss gradients is established when RNNs converge to fixed points or cycles. However, this was done only for the particular class of piecewise-linear RNNs (PLRNNs), more restrictive conditions for cycles were imposed than assumed here, and most importantly the chaotic case on which not considered. Another recent study (Engelken et al. 2020) also points out the general connections between Lyapunov exponents and loss gradients that we develop in sect. 3..1 but does not provide any in-depth theoretical treatment, proofs, or empirical evaluation, as we do here. Thus, systematic theoretical framework that relates RNN dynamics more generally, and across a range of different RNN architectures, to the behavior of its training gradients, is still lacking so far.
focus here was
3 THEORETICAL ANALYSIS: RELATION BETWEEN RNN DYNAMICS AND LOSS GRADIENTS
In our analysis, we will cover all major types of system dynamics (fixed points, cycles, chaos, and quasi-periodicitt), and mathematically investigate their implications for the loss gradients. We will do this for all major classes of RNNs, including standard RNNs with largely arbitrary activation function, LSTMs, GRUs, and PLRNNS. The next section will first develop and illustrate the basic intuition behind the relations between RNN dynamics and loss gradients.
3.1 PRELIMINARIES: RNN DYNAMICS AND LOSS GRADIENTS
Formally, all popular RNN architectures, including LSTMs, GRUs, or PLRNNS, are discrete time DS, defined by (firs-order-Markoviian) recursive prescription for the temporal evolution of the latent states 2, E RM of the form
2, = F=
(1)
where 8, E V is the input time and 0 are RNN parameters. For instance, for standard RNNs 2 + Bs, + h), where / is an element- wise activation function
we have Fa P0(=--1,88 8.) = s(W 21-1 like tanh or rectified linear unit (ReL.U).
Assuming we start at some initial value 21 E M and given sequence of external inputs S = {s we can recursively rewrite eq. as
(2), 2T FFF s2) =: F 82).
In DS theory, we characterize the long-term behavior of such sequences by its spectrum of Lyapunov exponents. The Lyapunov exponents estimate the exponential growth rates in different local direc- tions of the system's 's state space, and the largest Lyapunov exponent gives the dominant exponential behavior. Let us denote the system's Jacobian at time by
J, FF St)
dz
(3)
0z,-1
d2,-1
3

---

Under review as a conference paper at ICLR 2022
For instance, for standard RNNs we would have J W diag(f (f (W 21-1 Bst + h)), where diag denotes a diagonal matrix for which the i-th diagonal entry is the derivative of f w.r.t 21,1-1- Then, the maximal Lyapunov exponent along RNN trajectory {*1 22,
"r, } is defined as
T-2
Xmaz
1 T T
lim
log
II 1==
J
(4)
where || || denotes the spectral norm (or any subordinate norm) of a matrix. If Amaz < 0 nearby trajectories will ultimately converge fixed point or cycle, while for Xmar 0 (a necessary condition for chaos) initially nearby trajectories will exponentially separate, i.e. we will have diver- gence along one (or more) directions in state space. This accounts for the sensitive dependence on initial conditions in chaotic systems.
Now let L(W, h)) be some loss function employed for RNN training that decomposes in time as C = Σ Lt. Suppose we fancy BPTT as our training algorithm (similar derivations could be performed for RTRL), recursively develop the loss gradients w.r.t. some RNN parameter 0 in time (i.e., across layers of the RNN unrolled in time) as
T
aL,
dh, 00
Oh. 0z, 0+z,
with
(5)
08
08
Oz 0z, 00 '
and
Oz
dz
Ūz.-1
Ūz,+1
h-r-1
dzi-k
h-r-1 II k=0
II k=0
Ji-k:
(6)
2z,
dz.-1
Oz,
0z.-k-1
where 0- denotes the immediate derivative. Now observe that the behavior of the loss gradients crucially depends on the product series of Jacobians in eqn. (6) If the maximum absolute eigenval-
1/T 1),
ues of the Jacobians Jr will, in the geometric mean, be larger than 1 (i.e.,
IT
gradients will explode as T - oo, while they will saturate if ||T T-2 Jr-r --r
1/T
1. Thus, the key point to note is that the same terms that occur in the definition of the I Lyapunov spectrum, eqn. (4) resurface in the loss gradients, eqn. dynamics and gradients.
6
This accounts for the tight links between system
3.2 FIXED POINTS AND CYCLIC DYNAMICS
Let us start by considering the simplest types of dynamics that can occur in RNNs (or any discrete- time DS): fixed points and cycles. In fact, by far most of the literature global stability in RNNs and on loss gradients focused on just fixed points (Chang et al. 2019 Kolter & Manek 2019
Erichson et al. 2021), with only few authors who recently started to also connect cyclic behavior
to loss gradients (Schmidt et al., 2021 Rusch & Mishra 2021a). Recall that a fixed point of a recursive map 2, F() is defined
point 2n for which we have x" = F(2") Likewise, a k-cycle (k > 1) is a set of temporally consecutive periodic points Pv == [Zr, Stz..... Z., = F(z,),,....... - (x,,)) }} that we obtain from recursive application of the map such that each of the cyclic points zt, Ps is fixed point of the l times iterated map Fk (with & being the smallest positive integer for which this holds). To simplify the subsequent treatment, we will collectively refer to fixed points and cycles as k-cycles (k 2 1). Further recall that fixed point k-cycle is called stable if the maximum absolute eigenvalue of the Jacobian evaluated at that point is smaller than 1, neutrally stable if exactly 1, and unstable otherwise. Although the results we develop in this and the following sections will hold more widely, we will restrict our attention to recursive maps Fe from the class of RNNs R- {s andarrdRNag LSTM, GRU, P PLRNN} (see Appx. A.I for details).
Based on the observations made in the previous sections we can state the following theorem that links RNN dynamics and loss gradients:
Theorem 1. Consider RNN Fg R parameterized by 0, and assume that it converges stable fixed point or k-cycle I'ss (k N 1) with Br, as its basin of attraction 3 Then for every z E Br, (i)
FFrom here on we will suppress the explicit dependence on external inputs 8, notation-wise, see Remark 2
'The basin of attraction defined the of all points from which orbits converge the resp. attractor.
4

---

Under review as a conference paper at ICLR 2022
the Jacobian der dz) exponentially vanishes as T oc; for Tj the tangent vectors der and thus the gradient of the loss function, BCr, will be bounded above, i.e. will not diverge T + oo; and (iti) for the PLRNN 27 both 00
"0 dzr and @Cr will remain bounded for every 21 Br, as
DC.
Proof. (i) Assume that r is a stable k-cycle (h 2 1) denoted by
T& = [2,, 22, 2T, :. [ Ziok- - (R- 1): 2pnk,
Zt (K- 1)
Then, the largest Lyapunov exponent of & is given by
)
Ar.
lim 1000
1 1 In|lj Ji Jzll lim jk
In
II Jr-
(8)
0=0
By assumption of stability of I' we have Ar < 0 and also p( II== Jemk_s) < 1, which implies
lim J J 6000
J
lim 11000
Jet 0.
(9)
Now suppose that Oz, is an orbit of ( converging to Ps, i.e. 21 Br Since Oz, and Is have the same largest Lyapunov exponent, we have
1 In Jr JT ... J2ll dd, < 0,
A = lim T400
(10)
T
and hence for 21 E Br
0zr
lim --000
lim T-x
|| Jr-1 J2ll 0.
(11)
0z1
(ti) & (181) See Appx. A.2.1
Remark 1. The result of Theorem 7 part (1) will be generally true for any fest-order-Markkoian recursive map 4. but the conclusions in part (ii) may hinge on its specific definition.
Remark 2. None the results above and throughout sect. require the dynamics to be autonomous, the theory applies whether there external input or not. In fact, mathematically, non-autonomoos (externally forced) systems can always be rewritten as autonomous dynamical systems (Alligood al. 1996 Perko 2007: Zhang et al. [009)) see Appx. A.I.I details.
The results above ensure that loss gradients will not diverge (explode) as T - Do in RNNs that are "well-behaved" in the sense that they converge to a fixed point or cycle. This is a generalization of the results given in Theorem in (Shhidtt al. (202T), where this was shown only for the specific class of PLRNNs and b) for specific constraints imposed on the eigenvalue spectrum the RNN's Jacobians which were relaxed in our theorem above.
While our treatment above is centered on the "exploding-rradients" case, various architectural mod- ifications regularization techniques bounded from below as well. This was established, for instance, in (Schmidt et al.] (202T) for PLRNNs using 'manifold attractor regularization'. In Appx. [..2.I we show that the results from Theorem 2 from Schhmitt et al. (2021) on doubly bounded (from below and above) loss gradients can indeed be extended the more general case covered by Theorem [ above.
ensure that gradients do not vanish either, i.e. remain
3.3 CHAOTIC DYNAMICS
We will now consider the all-important chaotic case. Let F be a recursive map and Oz,
an orbit of F. The orbit is chaotic if (i) it is not asymptotically periodic and (ii) has at least one positive Lyapunov exponent (Glendinning & Simpson 2021 Meiss] 2007). If the system's 's invariant set is bounded, condition (i) is considered a standard signature of chaos, as in this case two nearby orbits separate exponentially fast, but at the same time their mutual sepa- ration cannot go to infinity so that there are also folds. The following theorem states the sufficient condition for exploding gradients:
{21, ø2, ... }
5

---

Under review as a conference paper at ICLR 2022
Theorem 2. Suppose that RNN Fø E R (parameterized by 0) has chaotic attractor I with Br- as its basin of attraction. Then, for every orbit with 21 E Br-, (i) the Jacobians connecting temporally distal states zr and z, > t), özz, will exponentially explode for T + oc, and (ii) the tangent vector
ăz."
dzz and so the gradients of the loss function, 0Cr, will diverge T cc.
21 Then, denoting by J the Jacobian of (1 at 2j E _ the largest Lyapunov exponent of I is given by
Proof. Let the RNN Fe E have chaotic orbit denoted by I _
lim
In Jr JT-1 Jz
(12)
T-+00 T
Since I* is chaotic, so À > O. Hence, from 12. it is concluded that
0z7 00
lim JT-1 J ... Jyll T-200
lim T T-
T>L L.
(13)
Now, according to Oseledec's multiplicative ergodic Theorem, nearly all the points in the basin of attraction of have the same largest Lyapunov exponent X. Thus, (1 holds for every #1 E Br-.
(ti) See Appx. A.2.2
Remark 3. The first part of Theorem 2 holds for all furstoodeerMarkkovian recursive maps T Note that for LSTMs,
Sur 02L
(=:= = (h, c) denotes the full Jacobian of both hidden and cell states.
We collect some further mathematical results and remarks related to Theorem 2 in Appx. A.3.I
Hence, the essential result is that for all popular RNNs R and activation functions, loss gradients will inevitably diverge if the RNN latent states converge to a chaotic attractor.
3.4 QUASI-PERIODICITY
Quasi-perion periodierid is a long-term behavior which occurs on a torus and, superficially, bears some similarity to chaos in the sense that, strictly speaking, orbits are also aperiodic. That is, as T - oo, trajectories will never close up with themselves. Moreover, every trajectory becomes arbitrarily close to any point on the torus, that is, it is dense. One important difference between quasi-periodis quasi-periodic system, as time passes, two close initial
and chaotic systems is, however, that conditions are linearly diverging, while in a chaotic system the divergence is exponential.
Theorem 3. Assume that an RNN Fo E R. (parameterized by 0) has quasi-periodic attractor with Br as its basin of attraction. Then, for every 21 E Br
0zr
vO<<<1 3To > s.t. VT 2 To = (1 T)) <
<
(1+
(14)
θzi
Proof. See Appx. A.2.3
According to Theorem for every orbit converging to quasi-periodic attractor, the Jacobians @er may diverge vanish as T + but this will not occur exponentially fast as T -> Thus, even for bounded non-chaotic RNNs we may sometimes stumble into the problem of diverging gradients. Although this may be less common scenario. point out it may occur if we train RNNs real data from oscillatory systems with incommensurate frequencies, as for instance encountered in electronic engineering.
oz.
In Appx. A.3.2 we have collected further mathematical results on the connection between RNN dynamics and loss gradients that hold regardless of the RNN's limiting behavior.
4 EMPIRICAL EVALUATION
Our theoretical results imply that chaotic time series pose principle challenge for RNN training that cannot easily be circumvented through specifically designed architectures, constraints, or regu- larization criteria. If the underlying DS we aim to capture is chaotic, loss gradients propagated back in time will inevitably explode. Here we will work out some implications for RNN training and potential remedies empirically.

---

Under review as a conference paper at ICLR 2022
4.1 TRAINING ON SYSTEMS WITH EXPLODING GRADIENTS BY SPARSE TEACHER FORCING
To illustrate the connections between theory and RNN training, we revive the old idea of TF (Williams & Zipser 1989) as mechanism for truncating error gradients while training. However, we would like to do this such that important information about the system dynamics does not get lost, for which Lyapunov theory offers some guidance. Specifically, we should not force the system back onto the true trajectory all or most of the time (as in "classical TF"), but should effectively "re-calibrate" it only at certain time points chosen wisely according to the system's 's local divergence rates. This procedure will be referred to as sparsely forced BPTT in the following.
Assume we want train an RNN with hidden states 2₁ E RM and linear (or affine) output layer on a time-series {æ1, 22," 2T) generated by a chaotic system The linear output layer a; Bz,, B E R XMM maps the RNN hidden states into the observation space. This allows us to modify the
original TF procedure by constructing a control series {21, 22, . from the observations by "inverting" the linear output mapping
(B' B)-BB" Dt.
(15)
The idea to supply this control signal only sparsely, separated by the learning interval r between consecutive forcings. Hence, defining T = {nTT + InENo as the set of all time points at which we force the RNN onto the 'true' values, the RNN updates can be written as
RRNN()) i N
21+1
(16)
else
This forcing is applied after calculation of the loss, such that L, = 1|2, - Bzell2 irrespective of whether is in T or not (and of course it applied only during training, not test time!). Replacing hidden states ze with their teacher- forced signals Σς simply breaks divergence between true and predicted trajectories time points E T, and also cuts off the Jacobians by breaking the temporal contingency (for details see Appx. A.7]. The learning interval r hence controls how many time steps are included in the gradient calculation and has to chosen with care such as to balance the effects of exploding gradients vs. those of loosing relevant time scales and long-term dependencies. While it is general wisdom that an optimal batch size will facilitate training, the point here is thus much more specific: Ideally should be chosen in accordance with the system's 's Lyapunov spectrum, for instance based on the predictability time (Bezruchko & Smirnov 2010)
In 2
Tpoo
(17)
We emphasize that such a simple recipe for addressing the exploding gradient problem is based on modifying the training routine, and is thus in principle applicable any model architecture
4.2 EXAMPLE 1: LORENZ SYSTEM AND EXTERNALLY FORCED DUFFING OSCILLATOR IN CHAOTIC REGIME
Let us illustrate these ideas on two classical textbook examples of chaotic DS, the chaotic Lorenz attractor as an autonomous system, and the chaotically forced Duffing oscillator as an example with explicit external input (see Appx. A.4 for details). Trajectories were repeatedly drawn from these systems, on which we trained a PLRNN, a vanilla RNN with tanh activation function, and a LSTM by stochastic gradient descent (SGD) to minimize the MSE loss between predicted and actual observations. As optimizer we used Adam (Kingma & Ba 2015) from PyTorch (Paszke et al. 2017) with a learning rate of 0.001. For all models, training proceeded solely by sparsely forced BPTT and did not employ gradient clipping or any other technique that may interfere with optimal loss truncation.
In nonlinear DS reconstruction, we are mainly interested in reproducing invariant properties of the underlying system such as the attractor geometry (or topology: Takens [181] Sauer et al. (1911) or
4 'Note that in DS reconstruction one usually considers the data as observations (unsupervised problem); according to Remark 2 however, it is mainly matter of the scientific question addressed whether we include certain variables explicit inputs as observations.
'To ensure invertibility, one could add a regularizer AI to B= B in eqn. (15 as in ridge regression, but we did not find this necessary in any of our examples.
'lll code produced here available [placeholder].
7

---

Under review as a conference paper at ICLR 2022
the frequency composition (i.e., time-averaged properties), while measures like ahead thhead-predicttonn rors are less meaningful especially on chaotic time series (Wood) 2010 Koppe et al. (2019). Thus, in evaluating training performance, here follow follow/Koppe al. (2019) in using Kullback-Leibler di- vergence sttsp to quantify the agreement between observed and generated probability distributions across state-space asses the overlap in attractor geometry (Appx. A.5).. Moreover, employ dimension-wiid wwie frequency correlation measure (PSC) to quantify the agreement of power-spectra of the observed and generated time-series (Appx. A.5
aa)
15
sttpp
10
stop
10
5-
Speai
150
200
Tood
learning interval
learning interval
1.0
LSTM
1.0
RNN
PSC
PLRNN
PSC
0.5
0.5
0.0
Tprad
200
100
Fyeedl
100
200
Learning interval
learning interval
Figure 1: Overlap in attractor geometry (Datap. lower = better) and dimension wise ppwer-spectra correlations (PSC. higher better) against learning interval for (a) the Lorenz and (b) the chaotically forced Duffing oscillator. Continuous lines = sparsely forced BPTT. Dashed lines classical BPTT with gradient clipping. Prediction time indicated vertically in black.
Fig. 1 shows the dependence of the reconstruction quality on the learning interval T for all RNN ar- chitectures on (a) the Lorenz and (b) the externally forced Duffing system. Fig. provides particular examples of reconstructions for chosen too small, too large, about right. For all models find a system-dependent range for the optimal learning interval that agrees well with the predictability time defined in eqn. (17), where estimates for the maximal Lyapunov exponent were taken from the literature (Rosenstein et al. 1993 Gilpin 2021). As a reference, dashed lines represent the recon- struction performance for all architectures when trained with classical BPTT and gradient clipping. The training procedure was the same as for sparsely forced BPTT, except that instead of supplying control-signal, gradients much worse performance, gradient clipping does not effectively address the EVGP, even for LSTMs. As further shown in Fig. in Appx. A.6.4 using the optimal window length but resetting the ini- tial condition to zero (instead of its control value ž,) for each chunk equally destroys performance. This suggests that neither mere gradient normalization nor simple windowing are sufficient, but will wipe out essential information about the dynamics.
normalized to prior each parameter update. As evidenced by the
Appx. A.6 we collect further results on chaotic Rössler attractor (Fig. 5). high-dimensional Mackey-Glass equations (Fig. 7 and the Lorenz attractor with partial observations (Fig. 8).
(a)
(b)
(c)
2
2
0
0.0
-2.5
Figure 2: Lorenz attractor (blue) and example reconstructions by LSTM (orange) trained with learning
interval (a) chosen too small (r = 5), (b) chosen optimally (r = 30), and (c) chosen too large (r = 200).
8

---

Under review as a conference paper at ICLR 2022
4.3 EXAMPLE 2: CHAOTIC WEATHER DATA
As for empirical example, trained all RNNs (vanilla RNN, PLRNN, LSTM) tempera- ture time series recorded at the Weather Station at the Max Planck Institute for Biogeochemistry in Jena, Germany To expose the chaotic behavior and obtain robust estimate of the maximal Lyapunov exponent, trends and yearly cycles were removed, and nonlinear noise-reduction was per- formed (Kantz al.] (1993); Appx. A.4)) The maximal Lyapunov exponent was determined with the TISEAN package (Hegger et al.) 1999). as shown in Figure (a). The value obtained is in close agreement with the literature (Millán et al. 2010).
Figure shows that also for these empirical data the optimal training interval T agrees well with the predictability time, eqn. (17). for all trained RNNs. Furthermore, as was the case for the DS benchmarks, gradient clipping was not able to satisfactorily tackle the EVGP, even when paired with architectures like LSTMs explicitly designed for alleviating this problem. Similar results are reported for another real-world dataset, electroencephalogram (EEG) recordings, in Appx. A 11
0.5
1.0
(log d
15-
-1.0-
stap
RNN LSTM
0.5
10
-1.5
100
Tyrod
1001 learning interval 5
200
Tyeed 100
200
time-steps
learning interval
Figure 3: (a) The maximal Lyapunov exponent was determined as slope of the average log-divergence of nearest neighbors in embedding space (m = embedding dimension). (b) Reconstruction quality assessed by attractor overlap (lower = better) and power-spectrum correlation (higher better). Black vertical lines = Tpedd
5 DISCUSSION AND CONCLUSIONS
In this paper we proved that RNN dynamics and loss gradients are intimately related for all major types of RNNs and activation functions. If the RNN is "well behaved" in the sense that its dynamics converges fixed point or cycle, loss gradients will remain bounded, and established remedies (Hochreiter & Schmidhuber] However, if the dynamics are chaotic, gradients will always explode. This constitutes a principle problem in RNN training that cannot easily be mastered through architectural design or gradient clipping. is furthermore a practically highly relevant one, as most time series we encounter in nature, and many from man-made systems well, are inherently chaotic. While we do not offer a full solution to this problem here, we suggest might be tackled in training by taking a system's local divergence rates measured through the Lyapunov spectrum into account. Hence, rather than conquering the EVGP by structural design or specific constraints or regularization terms, we recom- mend put the focus more on the training process itself. We illustrated this point empirically using sparsely forced BPTT, a training technique that pulls trajectories back on track at times determined by the maximal Lyapunov exponent. Doing so leads to optimal reconstruction results for a variety of simulated and real-world benchmarks, regardless of the specific RNN architecture employed in training. We stress that our goal above all was to provide a mathematically grounded perspective on the problem, with the empirical section focused on elucidating the practical implications of the theoretical results. Empirically, for instance, precise Lyapunov exponents may sometimes be hard to obtain, although our empirical examples confirm that estimates based on fog-divergence plots may work sufficiently well.
et al. [021] can be used to refrain them from vanishing.
ACKNOWLEDGMENTS
ETHICS STATEMENT
The current work deals with inooretccllmathmmattie aspects of RNN training and performs basic research on limitations of training algorithms. As such it has no direct ethical implications. Since many real world applications depend accurate forward predictions of time series, however, this

---

Under review as a conference paper at ICLR 2022
paper may raise awareness for important issue that will also be relevant in practical settings, including sensitive domains like medical time series, weather forecasts, or traffic control.
REPRODUCIBILITY STATEMENT
All theoretical results in this paper were carefully and thoroughly proven, with all proofs and de- tailed derivations available in the Appendix. Likewise, we will make available all code used in the empirical section a way that will allow others easily reproduce the results from this paper. This means we will include everything, starting with the code for benchmark simulations and simulated time series data used for evaluation, code for our model and training algorithms, up to the meta- files that produce the actual figures in this work, our lab github site. All of this will be clearly documented.
REFERENCES
Henry D. Abarbanel. Predicting the Future: Completing Models of Observed Complex Systems. Understanding Complex Systems. Springer-Verlag, New York, 2013. ISBN 978-1-4614-7211
9. doi:
URL https: /ttp://wwwww//.. springer wtt cttp:///cc/////////////////////a
9 9781461472179
Henry D. I. Abarbanel, Paul Rozdeba, and Sasha Shirman. Machine Learning: Deepest Learning as Statistical Data Assimilation Problems. Neural Computation, (14)):2025-2555 August 2018.
ISSN 1530-888X. doi: 10.1 10.1ne2/neco. a001094.
Kathleen T. Alligood, Tim D. Sauer, and James Yorke. Chaos: An Introduction to Dynamical Systems. Springer, New York, NY, 1996.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICML'16, pp. 1220-1228. JMLR.org, 2016.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent Neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 1171-1179, Cambridge, MA, USA, December 2015. MIT Press.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157-166, 1994. doi: 10. 109/ 72.279181.
Boris P. Bezruchko and Dmitry A. Smirnov. Extracting Knowledge From Time Series: An In- troduction Nonlinear Empirical Modeling. Springer Series in Synergetics. Springer-Verlag. doi: 10. (0007998-33-442-1666))) https://www wwwp///wwww/////m wwwrngerr......................r com/ /wt.k/ 0ttook////23422220000
Berlin Heidelberg. 2010. ISBN 978-3-642-1266044
URL
Bo Chang. Minmin Chen, Eldad Haber, and Ed H. Chi. AntisymmetricRNN: A dynamical system view on recurrent neural networks. International Conference on Learning Representations, 2019. /tRL:ttps:://// /ttttp:://opeeevvie. net forum? forum??_=ryxeepooexx
Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decodes approaches. In Proceedings of SSS7-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103-111, Doha, Qatar, October 2014a. Association for Computational Linguistics. doi: 10.31 10.3115/11/11-40011
htRpstts::/ http://aaaaattoooo...o wttphhhhooooo.rr/1111111 4012
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical ma- chine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), 2014b.
Kenji Doya. Bifurcations in the learning of recurrent neural networks. In [Proceedings] 1992 IEEE International Symposium on Circuits and Systems, volume 6, pp. 2777-2780 vol.6, May 1992. doi: 10.1 10.1109/SSCS. 1992.2233299
10

---

Under review as a conference paper at ICLR 2022
Jorge Duarte, Cristina Januário, Nuno Martins, and Josep Sardanyés. Quantifying chaos for ecological stoichiometry. Chaos: An Interdisciplinary Journal of Nonlinear Science, 20(3): 033105, September 2010. ISSN 1054-1500. doi: 10.1063/1.3464222 URL pattps://aip:
Publisher: American Institute of
Physics.
Georg Duffing. Erzwungene schwingungen bei veränderlicher cigenfrequenz, 1918.
Daniel Durstewitz. Advanced Data Analysis in Neuroscience: Integrating Statistical and Compu- fational Models. Bernstein Series Computational Neuroscience. Springer International Pub-
lishing, 2017. ISBN 978-3-31 978-3-319-59974-8 doi: 10. 1000000/978-3331--5997777
URL https:
//www. spr i nger. com www.e//bbooooo 978331 9599748
Daniel Durstewitz and Thomas Gabriel. Dynamical Basis of Irregular Spiking in NMDA-Driven Prefrontal Cortex Neurons. Cerebral Cortex, 17(4):894-908, April 2007. ISSN 1460-2199, 1047-3211. doi: 10. 10.1093//erro URL
oup com cercor/
article wwwcceellon wtt.ii/leooooopooo/oo//oooooao/a cot..///0/0///rr.////////04
Rainer Engelken, Fred Wolf, and Larry Abbott. Lyapunov spectra of chaotic recurrent neural net- works. arXiv:2006.00 02427 [nlin, q-bio]. June 2020. URL http:///aa.....og//aa///////////
02427
arXiv: 2006.02427.
N. Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W. Mahoney. Lipschitz recurrent neural networks. In International Conference on Learning Repre- sentations, 2021. URL httpss://penrevieew net fete//oormmmiiie N
Marisa Faggini. Chaotic time series analysis in economics: Balance and perspectives. Chaos: An Interdisciplinary Journal of Nonlinear Science, 24(4):04210,, December 2014. ISSN 1054-1500. doi: 10. 1063/1 4903797. (tLLhttps:// //ap:::://app scitationnnniccccc httpr//ooo.or//oo//l/////000
1. 1.4903797] Publisher: American Institute of Physics.
Richard J Field and László Györgyi. Chaos in Chemistry and Biochemistryy. WORLD 1993. doi: 10.1 1142/1770. WRLhttps: www. http://wwwww.olaaaaaaaaaaoooooo/ 1142/1706
Stefano Fusi, Wael F. Asaad, Earl K. Miller, and Xiao-Jing Wang. A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales. Neuron, 54(2):319-33 April 2007. ISSN 0896-6273, doi: 10. wwwp./.ooooooo.........
Zoubin Ghahramani and Sam Roweis. Learning nonlinear dynamical systems using an em algo- rithm. In M. Kearns, S. Solla, and D. Cohn (eds.), Advances in Neural Information Process- ing Systems, volume 11. MIT Press, 1999. URL hptp:teepps/pprrrceeiii.g.ggg
neurips....ppp... cc/ http://9999111100000/03333333e33
William Gilpin. Chaos interpretable benchmark for forecasting and data-driven modelling. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL /
http://pppeeeeeeeeeeeeeeeeeeeeee net. www.:neee/ooummideeeeeem hota:///ooooummmeeeoaaaaatata111
L. Glass and M. C. Mackey. Pathological conditions resulting from instabilities in physiological control systems. Annals of the New York Academy of Sciences, 316:214-235, 1979. ISSN 0077-
8923. doi: 10.
Paul A. Glendinning and David J. W. Simpson. A constructive approach to robust chaos using invariant manifolds and expanding cones. Discrete Continuous Dynamical Systems, 41(7): 3367-3387, 2021.
A. L. Goldberger, A. N. Amaral, L. Glass, J. M. Hausdorff, P. Ch. Ivanov, R. G. Mark, J. E. Mietus, G. B. Moody, C.-K. Peng, and H. E. Stanley. Phys- ioBank, Poysiooo and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation, 101(23):e11--2220 2000. Circulation Electronic Pages: 10.1 11116/00..IRR.01.22.222
PMID:1085218; doi:

---

Under review as a conference paper at ICLR 2022
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
// //www. www..deeplazzninpook.ooo orq
Alex Graves, Greg Wayne, Malcolm Reynolds, and al. Hybrid computing using neural network with dynamic external memory. Nature, 538:471-476, 2016. doi: 10. 10.10388nature2010..
Rainer Hegger, Holger Kantz, and Thomas Schreiber. Practical implementation of nonlinear time series methods: The TISEAN package. Chaos: Interdisciplinary Journal of Nonlinear Sci-
ence, 9(2):413 435, June 1999. ISSN 1054-1500. doi: 10. 10.1063/1.166444 1666444. URL. https:
//aip.acitation wttp/o/////o/////eoooooo http:////.ceeta//1..10111111111/
doi
Publisher: American
Institute of Physics.
Kyle Helfrich, Devin Willmott, and Qiang Ye. Orthogonal Recurrent Neural Networks with Scaled Cayley Transform. In International Conference on Machine Learning, pp. 1969-1978.
PMLR, July 2018. URL
www.eeeddiing...........m.. mlr
html ISSN: 2640-3498.
Mikael Henaff, Arthur Szlam, and Yann LeCun, Recurrent Orthogonal Networks and Long-Memory
Tasks. In International Conference on Machine Learning, pp. 2042. PMLR, June 2016. ISSN: 1938-7228.
URL httlhhtt::/// httap://proceeeding.. m wwt.rrppesss//s.ee/.ee
html
Daniel Hernandez, Antonio Khalil Moretti, Ziqiang Wei, Shreya Saxena, John Cunningham, and
Liam Paninski. Nonlinear Evolution via Spatially-Dependde Linear Dynamics for Electrophysi- ology and Calcium Data. arXiv preprint arXiv: 181 1,02459, 2020. URL http:///t//axxi
abs/ 02459
Sepp Hochreiter.
Untersuchungen zu dynamischen neuronalen netzen, 1991.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, doi: 10.1162/nco.1197.9881553 URL wtt../////......//eeeee.... www.::.......oo.o.......oo wwt.ec...9..... 9. 1735
(8):1735-1780, November 1997. ISSN 0899-7667.
Katsuma Inoue, Soh Ohara, Yasuo Kuniyoshi, and Kohei Nakajima. Transient chaos in bert. 11 arXiv:
10X:v:2/06.0318/ [nlin], June 2021. UtL/http://rrxii.orr httpr///a////...011111 2106.03181.
Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin Soljacic, and Yoshua Bengio. Gated Orthogonal Recurrent Units: On Learning to Forget. Neural Computation, 31()): 765-783, April 2019. ISSN 1530-888X. doi: 10.1 10.1162/neco a a_01 014...
Anil Kag and Venkatesh Saligrama. Time Adaptive Recurrent Neural Network. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15144-15153, Nashville, TN, USA, June 2021a. IEEE. ISBN 978- 566444---66444-0--22 doi: 10. 109/CC/P446437200222 .01490. URL https:://eeexpploee leee htt.:///w........oo//ooooeeee//a
Anil Kag and Venkatesh Saligrama. Training recurrent neural networks via forward propagation through time. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp- 5189-5200. PMLR, 18-24 Jul 2021b. kag21a. htmll
Anil Kag, Ziming Zhang, and Venkatesh Saligrama. Rnns incrementally evolving equilib- rium manifold: A panacea for vanishing and exploding gradients? In International Confer- ence on Learning Representations, 2020. URL ARIhhtt.: httpt::/ htttp::://opnrevviee net. forum?id=
Hy #y1pgA4FWS
Léandre Kamdjeu Kengne, Justin R. Mboupda Pone, and Hilaire B. Fotsin. On the dynamics of chaotic circuits based memristive diode-bridge with variable symmetry: A case study.
Chaos, Solitons & Fractals, 145:1,0795, April 2021. ISSN 0960-0779. doi: 10.1016/J.chaos. 2021.110793 URL https: wwwww//wwww//ww/w//////////ww www.sseiieeieeecccccmmm http://cceeectccooccccc////aaaaa S09 5096007792109140
Holger Kantz and Thomas Schreiber. Nonlinear Time Series Analysis. Cambridge University Press, 2 edition, 2003. doi: 10. 10.1017/B807780511555 1755779.
12

---

Under review as a conference paper at ICLR 2022
Holger Kantz, Thomas Schreiber, Ingo Hoffmann, Thorsten Buzug, Gerd Pfister, Leci G. Flepp, Josef Simonet, Remo Badii, and Ernst Brun. Nonlinear noise reduction: A case study on exper- imental data. Physical Review E, -853)11529-1538, August 1993. doi: 10.1103/PhysRevE.48 1529. htwp:httt:////t www.kaapp.... www.pppp..oo......o...... h1tt:////1...11111///e/eeme 48.1229 American Physical Society.
Publisher:
Matthew B. Kennel, Reggie Brown, and Henry D. I. Abarbanel. Determining embedding dimension for phase-space reconstruction using geometrical construction. Phys. Rev. A, 45:3403-3411, Mar 1992. doi: 10.1 10.1105/hyy.ev4.44..44
URL bttps://link... /
L103/PhysRevA.cr 3403
Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel, Gauthier Gidel, Eugene Vorontsov, Yoshua Bengio, and Guillaume Lajoie. Non-normal recurrent neural network (namnn): learning long time dependencies while improving expressivity with transient dynamics. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL 9d7099d8 794 (10.0988885777788888788888888333 Paperr
neur ips pttps/ccc//pppr//1////111
Mounira Kesmia, Soraya Boughaba, and Sabir Jacquir. Control of continuous dynamical systems modeling physiological states. Chaos. Solitons & Fractals, 136: 109805, July 2020. ISSN 0960- 0779. doi: 10. 10.10166j.chaos.20200 109805. URL https: www. http:/::/wwwwsccccceeeeeee.. sciencedirect. com/
sci /ccienee//rrricllee//////0800070
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA. May 7-9, 2015, Conference Track Proceedings, 2015. |RLhttp:
Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes, In Proceedings of the 2nd International Conference Learning Representations, 2014. URL http::/arxiv.c
abs/ abs/1312. 111111111222 6114
Zico Kolter and Gaurav Manek. Learning Stable Deep Dynamics Models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL 0albeeea11762553333333333 (22225538866622555422222222
htwp://aia.ss.u........e../eeeee pdf
Georgia Koppe, Hazem Toutounji, Peter Kirsch, Stefanie Lis, and Daniel Durstewitz. Identifying
nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI. PLOS Computational Biology, 1-35, 08 2019, doi: 10. /joumaalllllpbii..oo...
URL
http:/111111111.1 wrw.r..ggg................. 1371 (t13///////aall.. ournal pcbii. 1007263.
Jacques Laskar and Philippe Robutel. The chaotic obliquity of the planets. Nature, 361(6413): doi: 10.1038/361608aa. URL httpss:/// www.
608-612 February 1993. ISSN 1476-4687.
nature. www.re.com/arre httam/arrico westtiicless636/ 3616 60
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indmn): Building a longer and deeper mn. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5457-5466, 2018, doi: 10.1109/CVPR.2218.007777
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadeneshelii Burigede liu, Kaushik Bhat- tacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL
https::/openreviier wttprreviewwwee//froruuee http:///ww....e.eeoooooooo/ddddp
Zachary C. Lipton, John Berkowitz, and Charles Elkan. A critical review of recurrent neural net- works for sequence learning. arXiv:1506.000 [cs], October 2015. URL http: //arxiv.
00019 arXiv: 1506.00019.
13

---

Under review as a conference paper at ICLR 2022
Edward N. Lorenz. Deterministic nonperiodic flow. Journal of the Armospheric Sciences, 20(2):
33-144, March 1963. ISSN 0022-4928, 1520-0469. doi: 10.11751120004444((96)0000000000
2.0.CO0.. URL https: /ttp://oouuallssa
www.aaa. http///utaaaaaaatttt..r...... wtt.et/t....///////////......... http:////www/////////aaa/a//////
963_020__ 01 11003ddnf aanf2____ __co______0__ co_ 2. xm1]
Publisher: American Me-
teorological Society Section: Journal of the Atmospheric Sciences.
Niru Maheswaranathan,, Alex Williams, Matthew Golub, Surya Ganguli, and David Sussillo. Re- verse engineering recurrent networks for sentiment classification reveals line attractor dynam- ics. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL DETpS://proceedaagg..oo neurips cc/paperr2019/Eie/ 1 1991::2272222222244444200.011000 .dff
Fox, and Gar-
Sylvain Mangiarotti, Matthieu Peyre, Yixiao Zhang, Maciej Huc, Friederike Roger, and Yvonne Kerr. Chaos theory applied to the outhreak of COVID-19: an ancillary approach to decision making in pandemic context. Epidemiology and Infection, 148:e95, May 2020. ISSN 0950- 2688. doi: 10.101755950268200009999 pmc/ amcc/articlee//MM223112
URL httpps://wwt www..nccii... n1mm nih. gov/
Robert M. May. Chaos and the dynamics of biological populations. Proceedings of the Royal Society of London, Series A, Mathematical and Physical Sciences, 413(1844):27-44, 1987. ISSN
00804630. http:tp: //wp: jstor.
James D. Meiss. Differential Dynamical Systems. Society for Industrial and Applied Mathematics, 2007.
Jacob Menick, Erich Elsen, Utku Evci, Simon Osindero, Karen Simonyan, and Alex Graves. Practical real time recurrent learning with sparse approximation. In International Confer- ence on Learning Representations, 2021. URL
net/ forum?id=
q3KSThy2GWB.
Humberto Millán, Behzad Ghanbarian--lavijjeh,, and Ivan Garcfa-Fornaris. Nonlinear dynamics of mean daily temperature and dewpoint time series at Babolsar, Iran, 1961-2005. Armo- spheric Research, 98(1):89-101, October 2010. ISSN 0169-8095 doi: 10.1016/j
2010.06.001. URL
http://www.cinnneirrttttt..cttte com http://cccnne/aaice////////////e htt.ee/weeeeeecc.c.oooo..eoeeaer
S01 69809599511 0001419
Zahra Monfared and Daniel Durstewitz. Existence of n-cycles and border-collision bi- furcations in piecewise-linear continuous maps with applications to recurrent neural net- works. Nonlinear Dynamics, 101(2):1037-100 269X.
July 2020a. ISSN 0924-090X, 1573- 058411x0 URL http:://iinkk..... 1007/s /0t///110111 020 05841-x
doi: 10.
com/1C.
Zahra Monfared and Daniel Durstewitz. Transformation of ReLU-based recurrent neural networks from discrete-time to continuous-time In International Conference on Machine Learning, pp- 6999-7009, PMLR, November 2020b. URL http:::t/prooceedinnngng mlr. mtt.ppress/v19/////
morfaree20a.. html
ISSN: 2640-3498.
Lars Folke Olsen and Hans Degn. Chaos in an enzyme reaction. Nature, 267(5607) May 1977. ISSN 1476-4687. doi: 10.1038/267177a0. URL wetph:tt////t /www www.nantuee.... com/ Articles/2617
Emin Orhan and Xaq Pitkow. Improved memory in recurrent neural networks with sequential non-normal dynamics. In International Conference on Learning Representations, 2020. URL
https: https://cpenrevvew net /
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning Volume 28, ICML' pp. I11-1310-111-316 JMLR.org, 2013.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zach Vito Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
URL https:://///... htttpp:://oppnrevieee net. fert/foormm20 (ttpummmmdBBJJarrrm Paa.eJJssffff..
14

---

Under review as a conference paper at ICLR 2022
Barak Pearlmutter.
Dynamic recurrent neural networks, 1990. URL https://kiithh)
cmu cmu. http:///oououattooooooooooooornr htcp:yyeaaiieeeereeee_e_e _euu_ee
recurrent
networks/6605018/1
Lawrence Perko. Differential Equations and Dynamical Systems, volume Springer, New York,
NY, 2001.
Michael T. Rosenstein, James J. Collins, and Carlo J. De Luca. A practical method for calculating largest Lyapunov exponents from small data sets. Physica D: Nonlinear Phenomena, 65(1): 117-134, May 1993. ISSN 0167-2789.
doi: 10.10066016672277900999000000
URL https:
//ww/www www. www.sciencedirect. wwwpsetttccccccccooicooco http://reet..ceoeeecceeeeeeartia /tt.a//aaaaaaarora/ooa/aaraaaaaa
T. Konstantin Rusch and Siddhartha Mishra. Coupled oscillatory recurrent neural network (coRNN);
An accurate and (gradient) stable architecture for learning long time dependencies. Interna-
tional Conference on Learning Representations, 2021a. URL https::///// /tKKhttpss//peeeeeeeee wwppppeeeee..eee....
Forum?id=P3s6 9xx221
Konstantin Rusch and Siddhartha Mishra. Unicornn: A recurrent model for learning very long time dependencies. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9168 9178. PMLR, 18-24 Jul 2021b. URL BtL.hhtts:///por... Eusch21a html
ngs..ng....... press//139/ mlr...lr... hgs.mm...ress//11133333
Eleonora Russo and Daniel Durstewitz. Cell assemblies at multiple time scales with arbitrary lag constellations. eLife, 6:e19428, January 2017. ISSN 2050-084X. doi: 10.75544eLife.1990 19428. 10. 7554/ htt.:/elif....... 19428 Publisher: eLife Sciences Publica-
URL
tions, Ltd.
Otto E. Rössler. An equation for continuous chaos. Physics Letters A, 57(5):397-398, July 1976. ISSN 0375-9601. doi: 10.00160335-9900111699000-111111 URL https: //ww: sciencedirect. http:///....eeaaaaa.aa//////////
Salvador Sabanal and Masahiro Nakagawa. The fractal properties of vocal sounds and their
application the speech recognition model. Chaos, Solitons & Fractals, 7(11):1825-1843,
November 1996. ISSN 0960-0779. doi: 10. 10.10166099000079996000044444444
URL https:
/ //www. wcw.eciiennn w rect. com /ttp://..eooooiiiccccc.c.arrrrrr
Tim Sauer, James A. Yorke, and Martin Casdagli. Embedology. Journal of Statistical Physics, 65 (3):579-616, November 1991. ISSN 1572-9613. doi: 10.1007/BF01055745. URL https:
//doi. 1010//0111000/0010553333333
Gerwin Schalk, Dennis J. McFarland, Thilo Hinterberger, Niels Birbaumer, and Jonathan R. Wolpaw. BC12000: general-purpose brain-computer interface (BCI) system. IEEE trans- actions on bio-medical engineering, 316)::034-10443 June 2004. ISSN 0018-9294. doi: 10.1109/TBME.22044377777
Dominik Schmidt, Georgia Koppe, Zahra Monfared, Max Beutelspacher, and Daniel Durste- witz. Identifying nonlinear dynamical systems with multiple time scales and long-range de- pendencies. International Conference on Learning Representations, 2021. URL https:
//openreview. wttp://wwww...et.ottmmmmm... Xddxxd==_xxxxx
Bellie Sivakumar. Chaos theory in geophysics: past, present and future. Chaos, Solitons & Fractals, 19(2)
January 2004. ISSN 0960-0779. doi: 10.
1016/509600077990000005555 URL htttts:///// /ww..//wwww..... www.ssieeneeirret.ccomm wwwp:/eetttttcccceeeeeeeecceceee
Atricle/p11/099667770000000
Floris Takens. Detecting strange attractors in turbulence. In David Rand and Lai-Sang Young (eds.), Dynamical Systems and Turbulence, Warwick 1980, pp. 366-381, Berlin, Heidelberg, 1981. Springer Berlin Heidelberg. ISBN 978-3-540338945533
15

---

Under review as a conference paper at ICLR 2022
Robert Tchitnga, B. Anicet Mezatio, Theophile Fonzin Fozin, Romanic Kengne, Patrick H. Louodop Fotso, and Anaclet Fomethe. A novel hyperchaotic three-component oscillator oper- ating at high frequency. Chaos, Solitons & Fractals, 118:166-180. January 2019. ISSN 0960-
0779. doi: 00.1016f.chaos2201811..... URL https:: www.www...........cc.m......ww.w sci encedi rect. com/ sci
David J. Thomson. Time series analysis of Holocene climate data. Philosophical Trunsactions of the Royal Society of London. Series A, Mathematical and Physical Sciences. 330(1615);001-616,
April 1990. doi: 10.1098/rsta. 1990.0041. URL https:
org/doi abs 10. . 1098/ rsta. 1900 0041 Publisher: Royal Society.
Trieu H. Trinh, Andrew M. Dai, Minh-Thang L Luong, and Quoe V. Le. Learning longer-term de-
pendencies in mns with auxiliary losses, 2018. URChittpes/ppenneeriee..ee
id HyDWyPMD
Adam P. Trischler and Gabriele M.T. D'Eleuterio. Synthesis of recurrent neural networks for dy-
namical system simulation. Neural Networks, 80:67-78, 2016. ISSN 08936080. doi: 10.1016/
J.neunet.2016.04.00 URL htttps://likkinghhtt: elseviers htm///r.ooo/eeeeeeee hetm//eerreeeeppiii 508936080 6300314
Peter Turchin and Andrew D. Taylor. Complex dynamics in ecological time series. Ecology, 73(1); 289-305, 1992. ISSN 1939-9170. doi: 10.2307/1938740.
Eli Tziperman, Harvey Scher, Stephen E. Zebiak, and Mark A. Cane. Controlling spatiotemporal chaos in realistic EI Niño prediction model. Physical Review Letters, 79(6):0034-1337 1037, Au- gust 1997. doi: 10.1 10..PPyeeeeeee 79.1144,........ URL httttttt///iiit
aps org//oi/1c.. Publisher: American Physical Society.
11 #I03/PhysReviett.. 79. 1034
Carl van Vreeswijk and Haim Sompolinsky. Chaos in neuronal networks with balanced excita- tory and inhibitory activity. Science, (144529)::724-11777 December 1996. ISSN 0036-8075, 1095-9203. doi: 10. 22112///cciemce.7455592 1724. URL Utthttps:: //sciinceeeeeee sci encemaç. 7777522/3//7
Publisher: American Association for the Advancement of
Science Section:
Reports.
Pantelis R. Vlachas, Wonmin Byeon, Zhong Y. Wan, Themistoklis P. Sapsis, and Petros Koumout- sakos. Data-driven forecasting of high- chaotic systems with long short-term memory networks. Proceedings of the Royal Sociery A: Mathematical, Physical and Engineering Sciences, doi: 10. 20170000 URL httpp:///eoaalocccee.eyaallaaaap httgg//oooo/// 1098/ www.:10099/r.a....0 2017.00 0844
47402213):001008444, 2018. ISSN 1364-5021, 1471-2946.
Pantelis R. Vlachas, Georgios Arampatzis, Caroline Uhler, and Petros Koumoutsakos. Learning the Effective Dynamics of Complex Multiscale Systems. arXiv:2006. 13431 [nlin, physics:
July 2020. URL ttttt:://arx:. org abs 2006. 13431
arXiv: 2006.13431
Henning U. Voss, Jens Timmer, and Jürgen Kurths. Nonlinear dynamical system identi- fication from uncertain and indirect measurements. International Journal of Bifurcation and Chaos, 1406)):905-1933 June 2004. ISSN 0218-1274, 1793-6551. doi: 10.1 cooridscienn intii fic. cww/doi/ab//oo//////////o
$0218117400410345. URL https:: //ttp:://ww.....
1142/502181274040110
Joseph H. M. Wedderburn. Lectures on Matrices. New York: American mathematical society, New York: Dover Publications, 1964.
Ronald Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 12):270-288) June 1989 ISSN 0899-7667, 1530-888X. doi: 10.1162/neco. (tt.:///ccco..9....222 2/270 280/240244 5490
URL
.iit htt.://et.mmiteee//eeeeerrrallll
Simon N. Wood. Statistical inference for noisy nonlinear ecological dynamic systems. Nature, 466(7310), August 2010, ISSN 1476-4687. doi: 10.1038/nauure09319. URL nature.
Huaguang Zhang, Derong Liu, and Zhiliang Wang. Controlling Chaos. Springer, London, 2009.
16

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 THEOREMS: PRELIMINARIES
A.1.1
TRANSFORMING NON- AUTONOOOS INTO AUTONOMOUS DISCRETE-TIME DS
Following (Zhang al. 2009 and based on similar reasoning as for continuous time (ODE-based)
DS Alllioood et al. 1996 Perko 200T), let us consider the non-autonomous discrete-time DS
F++1 F( (), ac
(18)
Defining 2; autonomous system
(m): 1)t and G(zz) (F 1), 1 + 1)", system
18 can be rewritten as the
21+1 G(z)), Ra+!
(19)
Hence, all our theoretical treatment we can confine our attention to systems of the form eqn. [8]
A.1.2 RNN DERIVATIVES
Considering the loss function C -Er of an RNN Fo R parameterized by 0, have
T
UC
(20)
00
00
where
OC.
0L. Ōz.
(21)
00
0z, 00
The tangent vector
dzr has the form 86
dzr
0t2r
+
T-2
Σ
II
Jr-r
0+ 21
(22)
00
00
00
where 0t denotes the immediate partial derivative. Since for an RNN F6 E R the activation function is element-wise, with the m-th element parameter vector 0 (or belonging to the mi-th row of parameter matrix 0), we have
θt 2T
(0
0)
(23)
00
80
For instance, let 0 W be weight matrix, then
ac
ac
ac Swis BC Dhea
Ow
0wi2 ac
BC.
ac OW
Own
(24)
ac
ac Swrrz
ac Owar
Swari
In this case, for the standard RNN we have - ZT
(0
0 2k, Emk(22-
0)T ((m,)) Emb(2T-1) ZT (25)
Dwin
where Emk 12) Mm m
EEe Wmj 2J
M
M Eee Umj 85. + hm). and sum stands for
the derivative of with respect Wo.k-
Therefore, for standard RNNs,
22 becomes
Ozr
T-2
(m,,k) Emb (ZT-1)
2T-1
Jr-r (m, Ema((T---1) 24 21-1-1- (26)
Owmn
1=1
0=0
17

---

Under review as a conference paper at ICLR 2022
A.1.3 PIECEWISE-LINEAL RNN (PLRNN)
The PLRNN has the generic form (Koppe et al. 2019 Schmidt et 2021
zt F(2z-1) A 2 Wo(z-1) + Cs + Ex,
(27)
where ((x(11) max 1, 0) is the element wise rectified linear unit (ReLU) function, 2, E RM the neural state vector, A E RMxx xM is a diagonal matrix auto-regression weights, W E ER MM X matrix of connection weights, h RR- is the bias vector, 8, IR the external input weighted by C' E MxX and E N (0, Σ) a Gaussian noise term with diagonal covariance matrix E.
Equation 27 can be rewritten
2
W ()t- + Cs + E Wn(t-1) #c-1 + Css h + Ex, (28)
where Dz(s)) = diag( (d(()) ) is a diagonal matrix with da(a) = dd..d..... .ds) an indicator vector such that dm (Zm,t) =: dm = 1 whenever 2m,t > 0, and zeros otherwise.
For the PLRNN (28) have
dz:
J. θz,-1
Wa(t-1);
(29)
and wo <
||||| + |
Furthermore, the derivatives
22 for the PLRNN
28 are
Ozr Owmk
II Warr--)) 2r- (30)
((m,))
2T +
Σ
j=2
A.1.4 LONG SHORT- TERM MEMORY (LSTM)
The LSTM defined by the equations
(W,s;" Wichh- + b.)
ft a (Wifst + Whshe-1 by)
9t tanh (Wiss: + Whghts + by)
O, a (Wis. + Whoh + b.)
c ft c c. c Og.
h o2 3 tanh (c.)
(31)
where {s;} is the input sequence, W denotes weight matrices, b bias terms, é,, ft, 9t, Og demon- strate the input, forget, cell, and output gates, h, and c₁ the hidden and cell states at time t respectively, is the sigmoid activation function, and C represents the element-wise (Hadamard) product (see (Hochreiter & Schmidhuber 1997 Graves al. 2016 Vlachas al. 2018) for further information on LSTMs).
Defining 2, (he T. the LSTM (31 can be represented as the first-order recursive map
01 tanh (fr C- + @ gi)
2: =o(1 =
(32)
ft ② C + gt
18

---

Under review as a conference paper at ICLR 2022
The term 8e
DCt
in
20 for some LSTM parameter 8 can written as
dc,
dL,
dhu 0C. Oz. Oz, dzt dz Oz, 00
(33)
00
Th,
A necessary condition for LSTMs have chaotic orbit is given by:
Proposition 1. Let the LSTM given by attraction. Then for every 21 (h1, C1
131
have chaotic attractor T with Br- as its basin of
Br.
Bhy del
8h1
lim
> 1.
(34)
7-000
dcr 8h1
dcr Bel
Proof. The Jacobian matrix of 32 for can be written in the block form
Oh.
0h, dc:
Oz
Th
Jc
(35)
0z.-1
Sc
ūh,-1
Jc.-1
Further, due to the chain rule, we have
8h--1
1h, 8c:-1
8h.
8hi-1
ih, 0c:-1
8h.
0h
0h.
Th. deg
de;
8eg
J-J
dc,
0h
Dc.
0c, 0h
de, 0c: 0
8h
oh.-2
dc 8h.-2
Sh.-1 Uc:-2
ūc.-1
Uc:-2
8h.
8h.
8he-2
0c.-2
(36)
8c,
3c:
8h.
8c,
and by induction obtain
0h
0h.
dz:
0h
0c1
= Jr Jr J.......
(37)
0z1
del
dcl
8h
8c1
Now assume that
32
has chaotic orbit given by
r 2
(38)
According to
37 the largest Lyapunov exponent of I is given by
Oh:
0h
ah
Ar- lim T4x0
In [i Jj-1
J2ll
lim
ln
T
de
ah,
de
Since In is chaotic, so Ar- > 0, which gives
Ohi Th
ah de!
lim T-000
> 1.
(39)
0cp 0h;
ac; doj
Based on Oseledec's multiplicative ergodic Theorem, 339 holds for every 21 E Br.. This com-
pletes the proof.
19

---

Under review as a conference paper at ICLR 2022
A.1.5 GATED RECURRENT UNIT (GRU)
GRU network is defined by the equations
Ze oW + U,h--1 + bz)
F: a (W, 8, + U, + br)
h2 (1 21) tanh (Wn + Un(ri o h(-1) + bn) h2-1
(40)
where r2 represents the reset gate, 2, the update gate, 8, and h, denote the inputs and the hidden state respectively, W.. W,, W₁, E RMMx and U., Ur. Un E R MMM are weight matrices, bg. by. by, E RA are bias vectors, and or is the clement-wise logistic sigmoid function (for more details about GRUs see Cho et al. 42)
A.1.6 UNITARY EVOLUTION RNN (URNN)
The uRNN, proposed in (Arjovsky et al. 2016), is defined the nonlinear DS
= σb W z + Vs(,,
(41)
for which W E U(M) is an unitary matrix, V E CMXN b E RM is the bias parameter, 8, is the real- or complex-valued input of dimension N, and
(|zal + bi) [all
2ss + by 2 0
[oo(2)]; [modRoLL
(42)
if | ba 0
Proposition 2.
The uRNN given by 41 cannot have any chaotic orbit.
Proof. For any arbitrary orbit Oz of 41 we have
T-2 II Dr-+ WT
Jrr J J2ll
(43)
where D, = diag 06 (6((Wz 2.-1 + Vss)).
Since W is unitary and so a norm preserving matrix, it
concluded that
T-2 Dr Dr WT
T-2 II k=
T-2 D 1,
II k=0
<
Dr 1 = II
(44)
k=
which implies
Amar
lim In JT JT J2ll < 0. T-000 T
(45)
This rules out the existence of chaos (since Amaz 0 is necessary condition for Oz, to be chaotic).
Note that, more generally, any RNN which is constrained such to exhibit global convergence to a fixed point or cycle. by definition must have a maximum Lyapunov exponent Xmar S 0 (in accordance with Theorem hence cannot exhibit chaotic behavior by definition.
20

---

Under review as a conference paper at ICLR 2022
A.2 THEOREMS: PROOFS
A.2.1
PROOF OF THEOREM 1 PARTS (II) & (III
Proof.
(it) If is the Jordan normal form of II -p then I Jen- PJP-1
where
J (21)
0
0
(22)
(46)
0
J (Ap-1)
J (Ap)
and m, is the algebraic multiplicity of each eigenvalue Xç- Since pl - Je-k_s) < 1, so the
eigenvalue associated with each Jordan block satisfies [>s] my x my Jordan block has the form
< 1 1, .... .p). Moreover, every
A
1
0
J (2x)
(47)
A
X
Accordingly
k-1
/1
Jenk
J P- S p ||
(48)
in which p P|| P Furthermore, for j E N. J is a block diagonal matrix of the form
J mg (A1)
0
0
J ing (A2)
0
Jd =
(49)
0
Jin (Ap-1)
my (Ap)
in which every m, x m, Jordan block has the form
X
X
2 X
(m X2- m
0
X
2 X
(m/
X/ - m:+2
J JE (2)
:
(50)
0
X
G x
X
In addition, for every block J (A:), we have
| m. (A+)ll z / m4
Jr (2:)ll
m3
[ (di))
9=1
(
m.
- Σ
/
/mb
| 1m:
Σ
9 9-11 |u lm:
q
mi
$ 1 vmm
[1, 1-
2
Ag m:
=: j NA.-
(51)
9=1
21

---

Under review as a conference paper at ICLR 2022
Moreover, for any < iq < Dal there exists some 4 such that m < F7 for j 2 4. This means for j 2 4
| (A:)l - NA, A.P.
(52)
such that Fr F:|Acl <I
Besides, for J J (A1) @ Tin (22)
Jmp (Ap)
JJ
max ISTS
J (2:)1l =: | (A)l
(53)
Hence, from
48
52
and
53
it is deduced that for j 21
II Jemn=)) )'
S PNNAIFAP =: ='r,,
(54)
in which r FF < 1.
Furthermore, let for F&
max 721
| ] =
max
| } = m,
0sesk-1
@+ zT
}
{
g 2iok_
} E,
max
max
T21
00
USa$k-1
00
max T21
|
max 0sask-1
|| al1
= q.
(55)
Hence, defining 2o
0, for this k-cycle
dzt 00
0tzt
T-2
2+ ZT
+
Jj-n -r
00
00
p=0
0+2T
-+ 2T
Σ
J
09
09
)
- q 1 +
Σ
|
(56)
On the other hand, for T kj. from
54 and
55
we have
Σ
i
2k-
Σ
+
r=0
3k
6-11)-1
Σ
Σ
Σ
2k
r=
(=
r=0
r=o
i-11
1
Σ
Σ
II
J
t= (1- 11-1) 1)
r=
< (m m + +... - + Σ
p
m + m
+ m k p
(57)
22

---

Under review as a conference paper at ICLR 2022
Thus, considering (m + m² .... m m)
M, is deduced that
θzr a0
Ōzkj 30
pr(1+ pr(1+ +11++M +M)
lim T-000
lim 3 +00
qE(1 + M+
M 00,
(58)
--
which, by 21 implies
#Cr 00
will be bounded for T + co.
(iit) Consider the PLRNN given by (27). where for simplicity we ignore the external inputs and noise terms. Let (Z Roz zta....) .. be an orbit which converges to r, Hence
lim d(zt.. TA) n-000
(59)
which implies there exists a neighborhood U of I and k sub-sequences {Zezm 1m cm m 1 {Zlem
1.00 of the sequence {zt, 1oc such that all these sub-sequences belong to and
a) Ztem+s
(20s ) 0,1,2, 1,
b) lim Zlemeto = 20 = 0, 1..........
m-+00
c) for every Zt,, E U there is some (0,1,2, ... , 1) such that Zt, E (ztamta Jmml
In this case, for every 2t, E with 2t, E -m= there exists some ñ E such that Ztn Stanta and lim Ztants h Therefore, continuity of F results in
lim F(Zztant.) F (Z) 70000
(60)
and by 288
lim (Wo(teas)) Steaio + h) = Wa Br-t-a + h 8-000
(61)
which implies
lim Wilten)))
Ztyato
Wa(pk-s)) -s
(62)
R0000
Assuming lim W Wi(ttat.)
since 62 holds for every 2i-k-a: substituting 2i-k-, e
(1, ,0) in 62 we can prove that the first column of equals the first column of Wn(ek=ss) Performing the same procedure for 2prt_x = e.. i 2,3, ....... .,, yields
lim Wo(teass) Wa
(63)
According to 59 U contains an infinite number terms of the sequence {2t,, Joc i.c.
NN E N s.t. n N 21, E U.
(64)
Suppose that 2t, E for some n 2 N. Thus, there exists some s E {0, 1,, ... , 1] such that Zt,, E (Ztemto 1m=1- Without loss of generality let 0. Hence, there is some N such that
100
23

---

Under review as a conference paper at ICLR 2022
Ztm. Sien
and lim Z1zn Zi-k. In this case, moving forward in time gives 12-200
St. Meenn (ztn E f 100 m==),
lim 71000
Zign 25rk,
2t,11 Z1pal1 (2 E (zem mm=)),
lim 71-000
Ztan
21-h-15
(Zentz E {Z Ztat2 20xa42
lim Stan42 -2
8-200
2lo+h-1
Sbante-1 (Zini (K E f
lim 2lan+k-1 =
8-000
Ztath
2(g(n+s) (Ztath E (Ztem I m=
lim 2(z(441) = 2ptk, 8-000
2biRR1 Pleca+1)41 (Zen 11l E (2 100
lim 212(n+1441 = 2, 7-000
A1(0-1)E (#tn42e-1 E (Ztmm4e-1
lim 21 = -x-h-(k-1)) 80000
Zla+2k 242a12) (Ztntza E {Zrem 1mm)),
lim 2( = 2 h
(65)
Consequently, for n 2 N and j E N, write
kj-1
W (((w+tg---d)
1=0
k
k
II
(II
W WD
[II Waltamte
1=1
5 k
11
(66)
1= 2m1
On the other hand, in equation (28). there different configurations for matrix Da and hence different forms for matrix Wn(ten+=) this case, the phase space of the system is divided into different sub-regions by some borders; see (Monfared Durstewitz 2020a b) for more details. Also, since the system (28) is linear map in each sub-region, the k periodic points of I' must belong to different sub-regions (at least two different sub-regions). Accordingly, based on (63) and (65) there exists some N N such that for every > N both Ziza+, and z belong to the same sub-region and so the matrices Wn(taa and W (s E f0, ..... k 1}) are
identical. Hence, for 2 N, Ñ and N, equation (66) becomes
kj
j.
k-1
II
W Watt.
II
II W Watt
II W Wowe=-))
(67)
Therefore, similar the part (ii), we can prove for every 21 E Br bounded.
Uer a0
and
GCr 00
will also remain
A.2.2 PROOF OF THEOREM 2, PART (II)
Proof. (ii) Let for every >
= ... Jz
(68)
24

---

Under review as a conference paper at ICLR 2022
(LT)TEN,TSs T>2 T>2 is sequence of matrices Lr I6 UIsus and, due to
13
+co LT = cc. Hence, there is at least one sub-sequence ( JT,EN, T,>2 (for some m, k E (1,2,... M)) such that linn, +00 mn
(2
oc.
On the other hand
dz7
0+ 0+
J
(69)
00
00
00
1=0
Moreover, there exists some N such that (for T N 1)
x+ N N-1
0.
(70)
For as the k-th element of a parameter vector (or belonging the k-thh row of a parameter matrix 0), the term
II J
JT-
(71)
00
P=0
is a vector in which the i-th element is (2) 0 N-1
ik
00
Since limm, -+00 mi = oo, due to 70 limn, +00
(T.))
(T.)) 8 N mm 50
co, which implies Ozi
will
"0
diverge as T + oc. Similarly, by 21. we can prove
OCt
is divergent for 0o.
80
By Oseledec's multiplicative ergodic Theorem, the results also hold for every 21 E Br-.
A.2.3 PROOF OF THEOREM 3
Proof. Let ] (21, ... .r.,,,,,.... be quasi-periodic attractor. Then, the largest Lyapunov exponent of
Ozr
A = lim In Ir J ... J2ll = lim T T
0.
(72)
T-000 T
Ozi
We prove for every 0 <<<
dzr
lim (1 (c)) lim
lim (1 + cT-1
(73)
0zi
For this purpose, we show 04 <0<<<1
(I) limm (1 -)
dzr
and
0z
(II)
limr-x0
0zr
limy +1c
az1
Assume for the sake of contradiction that (I) does not hold. Then there exists some 0 < e < 1 such that
θzt
lim (1 -- > T-000
lim T T+000
(74)
dz,
Therefore
jzt
37, > s.t. VT To (1 11-68 TT-
(75)
0z1
25

---

Under review as a conference paper at ICLR 2022
and
In
dzr 02
37f > 1 s.t. V To
(76)
1-
T
Consequently, due to 722 for T + oo we have In(1 - c) 2 0. This implies e S 0, which is contradiction.
Similarly if we assume (II) is not true, then there exists some < e < 1 such that
dzr 0z1
lim T
2 lim (1 + c)T-
(77)
T-000
Thereby
dzr Oz
37 > s.t VT 2 To
2 (1 + +)--1
(78)
and thus
In
dzr
In(1 + T-
Dz
3o > 1 s.t. VT 2 To
(79)
T-1
T
This means In(1 +) 0 as T 0o, ie. O, which is contradiction.
Therefore (14) holds for r and also, according to Oseledec's multiplicative ergodic Theorem, for every 21 the basin of attraction of r.
A.3 ADDITIONAL RESULTS RELATION BETWEEN DYNAMICS AND GRADIENTS
A.3.1 FURTHER RESULTS AND REMARKS RELATED TO THEOREM 2
The result of Theorem 2 also holds for unstable orbits {21, 22, ...... with positive largest Lyapunov exponent. Trivially, for such orbits that diverge to infinity (unbounded latent states) gradients of the loss function will explode T - 0O.
Remark 4.
Remark 5, For RNNs with ReLU activation functions there are finite compartments the phase space each with different functional form. In such case, define the largest Lyapunov exponent of [" in the proof of Theorem 2 we assume that T never maps to the points of the borders.
Based Theorem 2. ents in standard RNNs with particular activation functions by considering the norms of their recur- rence matrix, for which the following Corollary provides the basis:
also formulate the necessary conditions for chaos and diverging gradi-
Corollary 1. Let for a standard RNN
|ding (J' (W 2 + Bs: h))l s < x
(80)
If the RNN is chaotic, then |[]] y 1.
Proof. Assume for the sake of contradiction that W S 1. From
II 2<1ST
W diagg( (f' (1 Bs, h))
W (f
Bs, + h))l
2C5TT
$ (l) W [W))+)=+ T-
(81)
it is concluded that lim
W diag (f (f' 2,-1 + Bs: h)) < co, which con- ||]]| y > is a necessary condition for the standard RNN to be
tradicts
13 This means
chaotic.
Remark 6. For RNN with the tanh and sigmoid activation functions and respectively. Thus, by Corollary I
the necessary conditions for chaos these two cases are [WI > 1 and [W]| 4, respectively.
26

---

Under review as a conference paper at ICLR 2022
A.3.2 OTHER CONNECTIONS BETWEEN DYNAMICS AND GRADIENTS
As sect. elucidated, there direct link between the norms of the Jacobians of the RNN along trajectories and the EVGP. By observing this link, we can formulate some general conditions that will have implications for the behavior of the gradients regardless of the limiting behavior of the RNN, as collected in the following theorem:
Theorem 4. Let Oz {21, ...... 27, .... be sequence (orbit) generated by an RNN Fg R parameterized by 0, and Pr = JT - I, T = 2.
(i) Assume that Oz is an orbit for which ot S vt. If ET-211Jrl oo, then the aCz will "80
Jacobian dzr, the tangent vector "21"
Szr and thus the gradient of the loss function, 86
be bounded for + co.
(ii If Σr= Pr < then the Jacobian
dzr will neither vanish nor explode as T - co. dz
Proof. Let ||-I| be any matrix norm satisfying ||1 I ||A || ||2ll.
(i) By boundedness of øtxr 80
we have
åzr
0+ ZT
0+ ZT
JT-r
00
00
00
r=0
$ E (1+ + Σ
T-2
1-1
1
55(
Σ r=0
T-2
I
(82)
Moreover,
T-2
E (=1 P=
Σ
اللل|
+
Σ
|ا + Σ
I Jلll J|||
p
p<c
p<ger
T = (1+ Jrll (1+ | ... (1. (22l]) =: II
[1++ (83)
1=2
Since Er Jr converges, according to Wedderbum 1964), the infinite products I (1+
|| in 83 converge finite number K
O. Consequently, by 82 and (83
Øzt
lim T-
s K < oc.
(84)
00
which implies aCr will be bounded for T -
Furthermore
dzr
[|JT]]
8
lim T-+30
<
|
= lim T-20
||r--1l ... |J21|
(1+11
s R,
dz
T=2
T=2
(85)
which completes the proof.
(ii Since 11 < co, due to /Wedderburn 1964) the infinite product
20
II (I+P)) = II T=2
Jr = lim Jr JJ ... J2.
(86)
T0000
7=2
converges to a matrix * O, which implies
0zt
< lim
| < xc.
(87)
T
0z1
27

---

Under review as a conference paper at ICLR 2022
Part (i) of Theorem 4 relaxes some of the conditions required in Theorem for bounded gradients by imposing Lipschitz condition on the immediate derivatives. Part (it) generalizes conditions
satisfied, for instance, in orthogonal (unitary) RNNS (Arjovsky et al. 2016 Henaff et al. 2016 or fully regularized PIRNNS (Schmidt et al. 2021).
Proposition 3. Let 22 = {z1, 22... 27 :.. be an orbit generated by an RNN Fe E R (param- eterized by 0), and || o O, T 2 2. If ET= In Jr diverges to -o0, then the Jacobian dsr vanishes as T tends to infinity.
Proof. For Jrll
# 0, T 2, we have
θzt azi
< |||||| JT || e 1|rrll e JillJr-1ll l Inl]Jull E In Jell
0 <
(88)
Hence ΣT= In ||J|||
- -c, then
dzr O. T-000 θz
lim
(89)
A.4 EMPIRICAL EVALUATION: DATASETS
Lorenz attractor vection, given by
The Lorenz system Lorenz 1963 is simplified model for atmospheric con-
dr
dt ( 2).
dy
x(p 2)
(90)
dt
de ry 3z.
dt
The system is of particular interest for its chaotic regime and studied here for 16, p 45.92 and B = 4. For these parameters the Lorenz system is known to have a maximal Lyapunov exponent Amax 1.5 (Rosenstein et al., 1993). To generate time series, the ODEs integrated with a step size At = 0.01 using scipy integrate. Accordingly, the prediction time is Tpred ln(
46.2.
Ameus
Dutting oscillator The Duffing oscillator (Dufting 1918 is an example of a periodically forced oscillator with nonlinear elasticity
F + d2 + 3x + ox 7 cos(w:t).
(91)
Note that this system is non-autonomous, that is externally forced due the r.h.s. of eqn. 91 The following parameters were chosen to arrive at chaotically forced oscillator: = 1.0. 3 - 1.0, 5 0.1, 7 0.35, and 1.4. For these parameters the Duffing oscillator has maximum Lyapunov exponent of Amaz 0.0995. The dataset used here was created with the code from Gilpin 2021) as a three dimensional embedding with step size 0.17. The prediction time is Tpred 39.28.
Rössler system Another prime textbook example for a chaotic system the Rössler system Rössler 1976) given by:
dr
dt 8 2,
dy + az,
(92)
dt
dz
b + z(x c).
di
28

---

Under review as a conference paper at ICLR 2022
For the parameters 0.15, b 0.2 and 10, the maximal Lyapunov exponent is Amax 0.09 (Rosenstein et al., 1993). To arrive at time series, a step size of At = 0.1 was chosen for integration. This gives prediction time of Tpred 77.0 for this system.
Mackey-Gine Glass equation delay differential equation
The Mackey-Glas equation GGlas & Mackey 1979) is nonlinear time
2p x
Ta with B, 7,7p2 0.
(93)
Here 2p represents the value of the variable x at time ( - (note that strictly, mathematically. this makes the system infiiie-dimmnsional)) Choosing the parameters to be = 1.0, 9.65,
and p = 2.0, leads to chaotic behavior with a maximum Lyapunov exponent Amaz 0.21.
The dataset was created 10-dimensional embedding with the code from (Gilpin) 2021) using At = 0.04. This yields a prediction time of Tpred = 82.2.
Empirical temperature time series This time series was recorded the Weather Station at the Max Planck Institute for Biogeochemistry in Jena, Germany spanning the time period between 2009 and 2016, and reassembled by François Chollet for the book Deep Learning with Python. The data set can be accessed at https://www.kk httpwwwwwwggl..oom www.kk..gll..ppg.ee www.o//anryyssswwwaaaaaaoooooeee wwwtther-archive-jenn
To expose the underlying chaotic dynamies of the time series, trends and yearly cycles were re- moved, and nonlinear noise-reduction was performed (using ghkss from TISEAN, see also Kantz et al. 1993)). Fig. (a) shows a snippet of the temperature data in comparison with the de- noised time-series. High- frequeeyy noise was further reduced through Gaussian kernel smoothing (o = =20), and the resulting time series was sub-sampled (every 5th data point was retained). Fig. 4 (b) clearly reveals fractional dimension of Deff 2.8 for the de-noised and smoothed time-series. This strongly suggests that the dynamics governing the time series are chaotic. We created a time (estimated by the false nearest neighbor technique, see Kennel et al. (1992)) and delay At = 500 (obtained as the first minimum of the mutual information). The first three embedding dimensions are shown in Fig. c).. The maximal Lyapunov exponent of this time series was determined with lyap_r from TISEAN (Hegger et al., 1999) to \max 0.016, see Fig. a).. This value is in close agreement with the literature (Millán et al. 2010 The predictability time of this system is estimated to be Tpred 43.3.
delay embedding (Kantz & Schreiber 2003) with
(a)
(b)
Day 28
cecsoieed
tempeeaturr
Dafe nu)
20
12
0
2300
50:00
7200
10000
2
27
2000
16300
time strpps
1ogg
time stepe
Figure 4: (a) Snippet of the original temperature data de-noised time series. (b) Blue lines show
local slopes of the correlation sums for embedding dimensions m E (5...., 10). The convergence of these estimates in m reveals a fractional dimension indicated by the platcau. (c) First three dimensions of the time- delay embedding series used for training.
All datasets used were standardized (i.e., centered with unit variance) prior to training.
A.5 EMPIRICAL EVALUATION: MEASURES OF RECONSTRUCTION QUALITY
Attractor overlap To asses the geometrical similarity of the chaotic attractor produced by the RNN to the one underlying the observations, we calculate the Kullback-Leible divergence of the ground truth distribution Pune (æ: and the distribution Pgeu (g) 2) generated by RNN simulation. To
do so in practice, we employ a binning approximation (see (Koppe et al.) 2019
Dstsp (Xtrue()) Pgen
Pg), (x z2)
K
Pirue (x) log
(k)
Ptrue
p(nic(x)) (x)
(k)
(k) Pgen (x |z)
k=
29

---

Under review as a conference paper at ICLR 2022
where K is the total number of bins, and Pirue (x) and Pgen (x z) estimates obtained as rela- tive frequencies through sampling trajectories from the observed time-series and the trained RNN, respectively.
((8)
Power-spectral correlations Since in DS reconstruction we aim to capture invariant (time- independent) properties of the underlying system, besides the geometrical agreement, compare the similarity in true and RNN-reconstructed power spectra. To do so, we generate time series from the RNN and calculate its power spectrum using the fast Fourier transform (scipy. fft). To reduce the influence of noise we apply Gaussian kernel smoothing and cut off the long high-frequency tails of the spectra. The dimension-wise correlation between observed and generated spectra are then averaged to give the PSC.
of length 100, 000
A.6 FURTHER EMPIRICAL EVALUATIONS
A.6.1
RECONSTRUCTION: RÖSSLER SYSTEM
15
1.0
Dstap
10
PSC
0.5
7prel
200
400
Tyeel
200
100
learning interval 7
learning interval 7
Figure 5: Overlap in attractor geometry (Datap. lower better) and dimension wise power-spectra correlations (PSC, higher better) against learning interval for the Rössler attractor. Continuous lines sparsely forced BPTT. Dashed lines classical BPTT with gradient clipping. Prediction time indicated vertically in black.
(1)
(b)
(c)
4
5.0
2
2
2.5
0.0
2
2
2
-2
2
2
0
0
-2
-2
-2
Figure 6: The Rössler attractor (blue) and reconstruction LSTM (orange) trained with a learning interval (a) chosen too small (7 5), (b) chosen optimally (T 30), and (c) chosen too large 200).
A.6.2 RECONSTRUCTION: HIGI--DIEENIONAA MACKEY-GLASS SYSTEM
10a
1.0
Detsp
PSC
0.5+
RNN
PLRNN
LSTM
10-7
Tjevo
300
400
Tyeed
200
300
lcarning interval 7
learning interval
Figure 7: Overlap in attractor geometry (Dat.p. lower better) and dimension wise power-spectre correla- tions (PSC, higher better) against learning interval for the 10d Mackey-Glass system. Continuous lines sparsely forced BPTT. Dashed lines classical BPTT with gradient clipping. Prediction time indicated verti- cally in black.
30

---

Under review as a conference paper at ICLR 2022
A.6.3 RECONSTRUCTION. PARTIALLY OBSERVED LORENZ SYSTEM
For this evaluation we trained models only on the variables {y, 2 of the Lorenz system, eqn. 90 In order to compute the attractor overlap (D step)) in the true state space, however, after training the observation matrix B was recomputed by linearly regressing the first 10 latent states onto the first 10 observations from all three Lorenz variables in eqn. 90
15
10
stsp
10
PSC
0.5
RNN PLRNN
LSTM
100 130
200
Tpref
100 150
2011
learning interval 7
learning interval
7
Figure 8: Overlap in attractor geometry (Datay, lower better) and dimension wise power-spectra correlations (PSS, higher better) against learning interval 7 for the partially observed Lorenz system. Continuous lines sparsely forced BPTT. Dashed lines = classical BPTT with gradient clipping. Prediction time indicated vertically in black.
A.6.4 OTHER INITIALIZATION PROCEDURES: STANDARD BATCHING WITH ZERO RESETTING
common procedure in training RNNs is partitioning the time series into chunks (as we did based on the Lyapunov spectrum), but then simply resetting the hidden states at the beginning of each chunk (window) to 0. Formally this would mean that we do not force the trajectory back on track as in our approach, but instead may kick it off the track. To illustrate this, here we trained LSTM on chunks (windows) with a length given by the optimal I (Topt 30 for the Lorenz system), but then initialized the hidden states to 0 the beginning of each window. The performance obtained this way is indicated by the green dashed line in Fig. below.
151
1.0
LSTM
Datap
10-
PSC
0.0
Tprod
100 150 200
2010
400
learning interval т
learning interval 7
Figure 9: Overlap in attractor geometry (D lower better) and dimension-wise power-spectra correla- tions (PSC, higher better) against learning interval for the Lorenz system. Continuous lines sparsely forced BPTT. Dashed-dotted lines windowing without forcing (choosing windows according to the optimal prediction time, but resetting hidden states to zero rather than its TF control value). Prediction time indicated vertically in black.
A.6.5 ELECTROENCHPHALOORRA (EEG) DATA
We used EEG data recorded by Schalk et al. (004) and provided on PhysioNet (Goldberger al., 2000), from which we took the baseline recording of the first patient for our analysis. Preprocessing
was performed as outlined above for the temperature time-series, i.e. we applied nonlinear noise-reduction (see Fig TO (a)) and Gaussian kernel smoothing (a = 30). Fig. TO (b) indicates fractional dimension Defs 1.8 for the de-noised and smoothed times series. We created a time delay embedding with an embedding dimension of
10 and delay time of At 500. The maximal Lyapunov exponent for this time series was determined to be Xmaz = 0.007, see Fig. II (a). To ease training, the time-series sub-san sub-sampled and only every third data point was retained. With this, we obtain a predictability time Tpred 33.01.
31

---

Under review as a conference paper at ICLR 2022
(b)
Daj =
BBG signul
BBG siggno
Dett.
0
2000
5000
7500
IC00O
10:000
10008
time-stepe
logg
time steg3
Figure 10: (a) Snippet the original EEG data and de-noised time series. (h) lines show local slopes of the correlation sums for embedding dimensions m E {5...., 10). The convergence of these estimates in m reveals a fractional dimension indicated by the plateau. (c) First three dimensions of the time-delay embedding series used for training.
Full reconstruction of the dynamics from complex and noisy EEG signals is very ambitious. Here therefore focused on the system's 's short-term behavior and combined multiple shorter trajectory bits (of length = 70) in the calculation our measure for geometrical agreement in state space.
(b)
(log d(e))
13
-2
Dasp
PSC
m=0
m-=00
ENN
0.04
0.007
200
100
GLD
Tyeo
timeesteps
learning interval
leamiing Interval 7
Figure 11: (a) The maximal Lyapunov exponent was determined as the slope of the average log-divergence of nearest neighbors in embedding space (m embedding dimension). Reconstruction quality assessed hy attractor overlap (lower better) and power-spectrum correlation (higher better). Black vertical lines Tped.
A.7 SPARSELY FORCED BPTT
Loss truncation One implicit consequence of the teacher forcing, eqn. (16), is the interruption of the hidden-to-hidden connections at these time points. More specifically, the system is forced time T, then there is connection between 2; and Z that is
0z.+1 ORN
J.+1 dz.
(94)
To see how these vanishing Jacobians truncate the loss gradients w.n. to some parameter 0, let us focus the loss gradients immediately after the forcing,
ac++1
dL++1
++1
sz++1
0t 2k
00
02,-1
Ozh
80
k=1
0L++1
g+ g++1
8z++1
I 0+ z
θ2.-1
00
Oz
08
k=
=0. because of 94
0c++1 0+ #1+1
(95)
02,-1
00
Eqn. 95 shows that sparsely forced BPTT implicitly truncates the loss gradients because it in- terrupts the hidden-to-hiddeen connection from 2, 2 for i T. More generally, defining
7 T= max (l' T 1' < (), the overall loss gradients are truncated to
Oc
ac,
dz θ+ z/
00
Oz
Oz 00
tr.
aL,
0z, 0 zz
(96)
0z
Ozk
08
k=r
32


Reviews:
[{"review_id": "V4IZNt0UnxK", "reviewer": "Reviewer_DWxP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: The authors establish a connection between Lyapunov exponents and behaviors of the dynamics of recurrent neural networks. Some derivations for particular recurrent neural networks are provided.\n\nWeaknesses: In general, Lyapunov exponents are one of the most important and well-known techniques to characterize the dynamics of the chaotic systems. One of the concerns when training with time-varying signals the dynamics becomes essentially time-varying and the results as stated do not hold. Beside long calculations in the appendix, that are mostly straightforward I am not convinced that the approach will be numerically efficient. Lyapunov exponents are known to lead to primarily numerical computations that are very expensive. They also highly depend on initial conditions, and I failed to find any analysis related to this. In addition, the RNN dynamics are mostly bounded stable and show chaotic behaviors only in bounded regions and can be divergent only locally. Finally, I failed to find any comparison results with any other technique to illustrate and show the claimed effectiveness of the technique.\n", "summary_of_the_paper": "The authors provide a study of gradients of the recurrent neural networks using Lyapunov exponents. They claim to offer a simple and effective training for the chaotic data. Some simulation results are provided.", "summary_of_the_review": "Based on what is provided it is difficult for me to believe the claims are valid.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "O3GMEERCcd2", "reviewer": "Reviewer_BxLc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n- The paper provides a fresh look at how learning chaotic dynamics is difficult for established RNN architectures. \n- The paper proposes to fix this issue by relying on existing architectures and techniques (teacher forcing), i.e., in contrast to proposing obscure architectures, which, unfortunately, seems to be standard in the research community nowadays. \n\n**Weakness:**\n- The Lorenz and Rossler systems used in the experiments seem to be deterministic and fully observable, i.e., the initial state + dynamics function defines the entire trajectory. Thus there is no need for using an RNN with a hidden state. One may directly learn a $x_{t} \\mapsto x_{t+1}$ mapping using a feedforward network. What are the advantages of using an RNN, which suffers from gradient issues, compared to feedforward networks? Experiments on partially observable chaotic systems would have been appreciated. \n- What happens in the \"inverting the output mapping step\" of the in teacher forcing when the output dimension is much lower-dimensional than the hidden state, and the output mapping is not uniquely invertible? Please elaborate.\n- No code was provided", "summary_of_the_paper": "The paper looks at the asymptotic behavior of the Jacobian of various RNN variants (standard, LSTM, GRU, PWL-RNN) when realizing stable vs chaotic dynamics. In particular, the paper shows that in the chaotic setting, the gradients asymptotically explode, i.e., when learning chaotic dynamics, the gradients have to explode (asymptotically).\nThe paper proposes to overcome this limitation by truncating the BP length and applying a teacher forcing method that periodically projects the observation onto the hidden state during training.", "summary_of_the_review": "Overall solid paper. Please elaborate on the weaknesses mentioned above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8EL1xHLptC-", "reviewer": "Reviewer_F1yn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros:\n* The paper is very clear and well written\n* I very much like the idea of providing a systematic and general overview to analyse the gradient behavior of RNNs\n* Very thorough literature review\n\n\n## Concerns:\n\n* In reality, an RNN trajectory heavily depends on the external input (forcing). The paper also states in the very end of the discussion that this forcing can significantly change the dynamics. Hence, it would very interesting to see some sensitivity analysis with respect to changes in the input for popular architectures (one with stable fixed points and one with limit-cycle behavior for instance) and a discussion on how this might change the provided theory. \n    \n* My biggest concern is regarding the experiments. Given the cut-off interval $\\tau$, I understand the suggested approach to be equivalent to simply splitting the input sequence into smaller windows of length $\\tau$ (in a mini-batch manner), and setting the initial hidden-state(s) to the pseudo-inverse of the ground-truth of the last entry of the window before. Hence, the only contribution is using specific initial hidden states and a theoretically motivated length for the windows (which in most cases has to be found anyway by hyperparamter optimization, as in many real-world applications the max. Lyapunov exponent cannot be efficiently computed). I would like the paper to acknowledge that the provided approach is in fact equivalent to windowing, which is a very standard practice for training RNNs. Moreover, I would like to see results using the same window length $\\tau$, as in the suggested approach, but using a standard initial hidden state (set to zero). I suspect, that the results will be quite similar to the reported results of the suggested method (given that LSTMs for instance have been very successfully applied on chaotic time series prediction). But I could be mistaken.\n    \n* It feels very natural to choose the predictability time to be the length of the intervals $\\tau$. However, following the reference provided in the paper (Bezruchko \\& Smirnov, 2010), predictability time is a way more complex concept and can even be defined without appealing to the Lyapunov exponent, meaning systems with the same max. Lyapunov exponent can have very different predictability times. Therefore my question: Why was the predictability chosen as $\\tau_{\\text{pred}}=\\ln(2)/\\lambda_{max}$? Was it simply a fit to the provided experiments? In my opinion, more experiments need to be done in order to have a valid empirical support for that, e.g. what about a high-dimensional chaotic systems in say $>50$ dimensions. I suspect that the chosen predictability time won't be an accurate choice anymore.\n    \n* It would be interesting to compare the suggested training method to other approaches suggested in the context of training RNNs (for instance LSTMs) on chaotic systems, e.g. (Vlachas et al., 2018) in the paper.\n    \n* Unfortunately, the empirical investigation is very limited and I would have liked to see results on other tasks, such as NLP tasks, health-care applications and others (as paper claims in the introduction: chaotic behavior can be found in many sequential data sets), to see if this approach actually increases the performance of RNNs in widely-used RNN applications.\n    That being said, language modeling (such as the Penntree bank (PTB) word-level and char-level task) would have been an excellent application, as the training of the models is already based on cutting the very long tokens stream into smaller windows, but in contrast to the suggested method by the authors, the initial hidden states are not pseudo-inverse of the ground-truth but are set to the last hidden state of the window before.\n\n### Minor comments:\n* The pseudo-inverse introduced in eq. (15) needs more discussion: Why can it be constructed like that (i.e. assumption of linearly independent rows)?\n* Please define things like basin of attraction (first used in Theorem 1)\n", "summary_of_the_paper": "The paper connects the gradient behavior of recurrent neural networks with its dynamics through the maximum Lyapunov exponent of a trajectory, which is generated by the respective RNN. In particular, several connections between RNNs which produce specific dynamics (e.g. with stable fixed points or limit-cycles) and the mitigation of the exploding gradient problem are proved. Moreover, the paper shows that RNNs generating a chaotic trajectory always suffer from the exploding gradient problem. \nThat being said, the paper emphasizes that in order to approximate chaotic systems, the RNN has to be able to produce chaotic trajectories, which is ruled out by design by several state-of-the-art RNNs that are constructed to solve the exploding and vanishing gradient problem. However, given that these RNNs - at least the moment the optimizer finds parameters such that the resulting RNN produces chaotic trajectories - will always run into the exploding gradient problem, the paper suggests an algorithmic fix for that, by forcing the hidden-states of the RNN to be the pseudo-inverse of the underlying ground-truth trajectory at equidistantly distributed points in time during training. This cuts off the gradients at these points in time and forces the output states to be close to the ground-truth trajectories. The interval for that is chosen based on the maximum Lyapunov exponent.\nIn the paper, three experiments are provided, where two of them are based on approximating a chaotic dynamical system (Lorenz and Rössler), while one is based on real-world data from temperature forecasting.", "summary_of_the_review": "At this point, I cannot vote for acceptance, as \n\n* the provided theory is helpful to get an overview over the various general ideas of tackling the exploding and vanishing gradient problem, however, it is often prohibitively hard to prove (or it is simply not true) that the RNN exhibits specific dynamics (for instance limit-cycle behavior), due to the role of the external input, for which preferably no prior assumption should be made. That being said, the provided framework might be very limited in its usefulness, when it comes to constructing new RNN architectures which mitigate the exploding and vanishing gradient problem.\n* the suggested training method seems to be a minor deviation of very standard approaches\n* the empirical evidence is not sufficient (more diverse datasets should be considered) and the presented results lack more useful comparisons. \n\nI am happy to increase my rating, if the authors can resolve my concerns.\n\n\n=====POST-REBUTTAL COMMENTS========\n\nIn general, I'm satisfied with how the authors addressed my minor concerns, in particular I appreciate the additional experiments.\\\nHowever, my main concern that the theoretical framework provided is very limited in its usefulness remains.\\\nI raise my score to (very) marginal acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5VQmEEUCQ1p", "reviewer": "Reviewer_rTPw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- Analyzing chaotic behaviour of RNNs (vanilla, GRU, & LSTMs) based on the connection between loss gradients and Lyapunov spectrum is novel.\n\n- Although ideas similar to sparsely forced BPTT exists in the literature, its interesting to see there exists a natural notion of the interval at which the teacher should be forced onto the student. Although its a little disappointing to note that its not easy to find such an interval and would live as a hyper-parameter in the algorithm.\n\n\n**Weaknesses**\n\n- Proposed sparsely forced BPTT provides feedback in the training process at regular intervals and such an idea has been explored in the literature ( see https://arxiv.org/pdf/1803.00144.pdf, https://proceedings.mlr.press/v139/kag21a.html, and references therein. )\n\n- Paper claims that RNNs based on the fixed point idea or the restricted eigenspectrum of the recurrent matrix do not allow chaotic dynamics. No evidence for this point has been provided. \n\n\n- In all the theorems, there have been assumption on what kind of dynamics is being followed by the RNN (be it fixed point, chaotic, quasi-periodic). One has no hold of any quantity that regulates the dynamics. In essence, the theory only characterizes how the gradients will behave knowing the dynamics and essentially under the hood just boils down to the eigenspectrum of the jacobians, which has been explored numerous times in the literature (starting with Bengio et. al. as to why gradients explode or vanish). This work does not help any practictioner to carefully choose one architecture over other given that the time-series dynamics is chaotic. Theoretical anaylsis given the dyanmics maybe one another way to characterise the EVGP and upto a certain extent has novelty in its own right.\n\n\n**Questions for Authors**\n\n- The predictability time for any other problem would indeed be treated as a hyper-parameter. Do you have any sense of what values might be a good place to initialize the search from?\n\n- Sparsely forced BPTT looks a bit close in spirit to some alternative training schemes where auxiliary supervision is provided in the learning scheme (see https://arxiv.org/pdf/1803.00144.pdf, https://proceedings.mlr.press/v139/kag21a.html). Did the authors try out similar training scheme in their experiments? \n\n- Is there any reason behind not using ODE-RNNs as a baseline, since they are designed to be operating with stable transitions?\n\n**Writing Clarity**\n- After Eq.1, For RNNs, you use $h_t$ to indicate the hidden states and the transition function $F_\\theta$ for standard RNN, uses $h$ as the bias. Consider changing the symbol for bias for readability. \n- Missing references\n\t- Practical Real Time Recurrent Learning with a Sparse Approximation ( https://openreview.net/forum?id=q3KSThy2GwB )\n    - RNNs Incrementally Evolving on an Equilibrium Manifold: A Panacea for Vanishing and Exploding Gradients?  (https://openreview.net/forum?id=HylpqA4FwS)\n\t- Time Adaptive RNNs ( https://openaccess.thecvf.com/content/CVPR2021/papers/Kag_Time_Adaptive_Recurrent_Neural_Network_CVPR_2021_paper.pdf )\n\t- An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories ( https://ieeexplore.ieee.org/document/6797135 )\n\t- Learning Longer-term Dependencies in RNNs with Auxiliary Losses ( https://arxiv.org/pdf/1803.00144.pdf )\n\t- Training Recurrent Neural Networks via Forward Propagation Through Time (https://proceedings.mlr.press/v139/kag21a.html)", "summary_of_the_paper": "This paper studies the exploding and vanishing gradient problem (EVGP) in the sequential modelling with chaotic dynamics. Theoretical analysis of the EVGP, based  on the relationship between loss gradients during RNN training and Lyapunov spectrum of RNN-generated orbits, is provided. Inspired by this analysis, an alternative to BPTT training algorithm is proposed, named sparsely forced BPTT, that forces the diverging dynamics to conform to the true trajectory, at regular time interval provided by the  Lyapunov spectrum.", "summary_of_the_review": "Paper has two main contributions: (a) theoretical connections between loss gradients and Lyapunov spectrum that help in analyzing various RNN dynamics including chaos, and (b) sparse forced BPTT. The latter has been explored in the literature and the paper fails to mention these works in the literature review. Theoretical contribution is of some novelty, but it would be another way to explain EVGP, albeit with dynamics in mind. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the gradient behavior of RNNs in terms of the Lyaponuv exponents of its trajectory/orbit, showing that RNNs with cyclic or stable equilibrium dynamics have bounded gradients, but if the dynamics are chaotic the gradients will explode. From these insights, the authors propose an algorithmic remedy for this pathology, which is essentially a teacher-forcing method that periodically projects the observation onto the hidden state during training. A thorough empirical investigation is performed showing the utility of the proposed approach for modeling chaotic data.

The reviewers had split opinions on this paper. Some reviewers found value in the theoretical contributions and the connection between Lyapunov exponents and behaviors of the dynamics of recurrent neural networks, while others thought the theoretical framework may have limited practical utility. Several reviewers found the initial experiments to be lacking, though many of their concerns were alleviated after the substantial additions the authors provided during the discussion phase.

I believe the observation that exploding gradients are unavoidable when modeling chaotic data is important and would be of significant interest to the broader ICLR community. However, the practical implications of this observation have not been thoroughly described or investigated, and without this perspective, the theoretical results by themselves are much less impactful. In practice, it is usually the case that the ground-truth function is not learned exactly, the time horizon is finite, the gradients are noisy, the data-generating process is opaque, etc. Do these caveats have any bearing on the conclusions? The experiments address some of these questions, but only indirectly, and a more explicit discussion of the practical implications would broaden the impact of the paper.

Along the same lines, the practical utility of the theoretical framework could be further supported if there were some analysis of more varied or additional RNN use-cases. As one reviewer mentioned, I think the ICLR community in particular would appreciate any theoretical or algorithmic insights that might yield improvements on a standard baseline task like seqMNIST, which has served as a point-of-comparison for many alternative methods and which would facilitate comparisons to prior work.

Overall, this paper does make some nice and potentially important theoretical insights about training RNNs on chaotic data, and it does include an extensive battery of empirical evaluations, however the practical implications remain largely unconvincing, and I believe the paper falls just short of the bar for acceptance.
Decision: Reject