How to train RNNs on chaotic data?

Recurrent neural networks ( RNNs ) are wide-spread machine learning tools for modeling sequential and time series data . They are notoriously hard to train because their loss gradients backpropagated in time tend to saturate or diverge during training . This is known as the exploding and vanishing gradient problem . Previous solutions to this issue either built on rather complicated , purpose-engineered architectures with gated memory buffers , or - more recently - imposed constraints that ensure convergence to a fixed point or restrict ( the eigenspectrum of ) the recurrence matrix . Such constraints , however , convey severe limitations on the expressivity of the RNN . Essential intrinsic dynamics such as multistability or chaos are disabled . This is inherently at disaccord with the chaotic nature of many , if not most , time series encountered in nature and society . Here we offer a comprehensive theoretical treatment of this problem by relating the loss gradients during RNN training to the Lyapunov spectrum of RNN-generated orbits . We mathematically prove that RNNs producing stable equilibrium or cyclic behavior have bounded gradients , whereas the gradients of RNNs with chaotic dynamics always diverge . Based on these analyses and insights , we adapt the old idea of teacher forcing to yield an effective yet simple training technique for chaotic data , and offer guidance on how to choose relevant hyperparameters according to the Lyapunov spectrum . 1 INTRODUCTION . Recurrent neural networks ( RNNs ) are widely used across various fields in engineering and science for learning sequential tasks or modeling and predicting time series ( Lipton et al. , 2015 ) . Yet , they struggle when long-term temporal dependencies , very slow , or hugely varying time scales are involved ( Hochreiter , 1991 ; Bengio et al. , 1994 ; Schmidt et al. , 2021 ; Li et al. , 2018 ; Rusch & Mishra , 2021a ) . Time series or sequential data with such properties are , however , very common in fields like climate physics ( Thomson , 1990 ) , neuroscience ( Fusi et al. , 2007 ; Russo & Durstewitz , 2017 ) , ecology ( Turchin & Taylor , 1992 ) , or language processing ( Cho et al. , 2014b ) . Training RNNs on such data is hard because the loss gradients backpropagated in time easily saturate or diverge in this process . This is commonly referred to as the exploding and vanishing gradient problem ( EVGP ) ( Hochreiter , 1991 ; Bengio et al. , 1994 ; Pascanu et al. , 2013 ) . One solution to the EVGP is based on specifically designed RNN architectures with gating mechanisms , such as long short-term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) or gated recurrent units ( GRU ) ( Cho et al. , 2014a ) . These architectures allow states at earlier time steps to more easily influence activity much later through a kind of protected memory buffer , thus alleviating the EVGP by structural design . In practice , such models need to be backed up by further techniques like gradient clipping to keep the gradients in check ( Pascanu et al. , 2013 ) . The relatively complex architectural design of these networks impedes their mathematical analysis and requires reverse engineering after training ( Maheswaranathan et al. , 2019 ; Monfared & Durstewitz , 2020a ; b ; Schmidt et al. , 2021 ) . Partly to forego these complications , a variety of other solutions has been proposed recently , imposing restrictions on the recurrence matrix to bound the gradients ( Arjovsky et al. , 2016 ; Chang et al. , 2019 ) , or enforcing global stability by design or regularization ( Erichson et al. , 2021 ; Kolter & Manek , 2019 ) . Often these procedures dramatically curtail the expressivity of the RNN ( Kerg et al. , 2019 ; Orhan & Pitkow , 2020 ; Schmidt et al. , 2021 ) ; in particular , they rule out chaotic dynamics ( see further discussion below ) . This is at odds with the plethora of chaotic phenomena in nature , engineering , and society . Chaotic dynamics are commonplace , almost default in any complex physical or biological system . This includes scientific areas as diverse as neuroscience ( Durstewitz & Gabriel , 2007 ; van Vreeswijk & Sompolinsky , 1996 ) , physiology ( Kesmia et al. , 2020 ) , geophysics ( Sivakumar , 2004 ) , climate systems ( Tziperman et al. , 1997 ) , astrophysics ( Laskar & Robutel , 1993 ) , ecology ( Duarte et al. , 2010 ) , chemical reactions ( Field & Györgyi , 1993 ) , cell ( Olsen & Degn , 1977 ) or population ( May , 1987 ) biology . Chaotic phenomena are also crucial for the understanding of societal and epidemiological processes , such as the spread of diseases ( Mangiarotti et al. , 2020 ) , or in economics ( Faggini , 2014 ) . They are further relevant in purely technical contexts such as electrical engineering ( Tchitnga et al. , 2019 ; Kamdjeu Kengne et al. , 2021 ) or laser optics ( Kantz et al. , 1993 ) . They have even been suggested to play an up to now largely neglected , but potentially very significant role in speech recognition ( Sabanal & Nakagawa , 1996 ) and natural language processing ( Inoue et al. , 2021 ) . Hence , in almost any practical setting , chaotic phenomena abound . They can not , in general , be ignored when devising RNN training algorithms . Here we offer a comprehensive theoretical treatment of the relation between RNN dynamics and the behavior of the loss gradients during training . We find a close connection between an RNN ’ s loss gradients and the largest Lyapunov exponent of its freely generated orbits . We mathematically prove that RNNs producing stable fixed point or cyclic behavior have bounded gradients . Crucially , however , the loss gradients of RNNs producing chaotic dynamics always diverge . Hence , the chaotic nature of many time series data induces a principle problem , and , despite significant efforts in the past to solve the EVGP , training RNNs on such data remains an open issue . We illustrate the implications of our theory for RNN training on several simulated and empirical chaotic time series , and adapt the old idea of sparsely forced BPTT as a simple yet effective remedy that enables to learn the underlying dynamics despite exploding gradients . 2 RELATED WORKS . Exploding and vanishing gradients . While ‘ classical ’ remedies of the EVGP ( Hochreiter , 1991 ; Bengio et al. , 1994 ; Pascanu et al. , 2013 ) rest on purpose-tailored architectures with gating mechanisms , which safeguard information flow across longer temporal distances ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014a ) , the focus has recently shifted to simpler RNNs that address the EVGP by restricting the recurrence matrix to be orthogonal ( Henaff et al. , 2016 ; Helfrich et al. , 2018 ; Jing et al. , 2019 ) , unitary ( Arjovsky et al. , 2016 ) , or antisymmetric ( Chang et al. , 2019 ) , or by ensuring globally stable fixed point solutions ( Kag et al. , 2020 ; Kag & Saligrama , 2021a ) , for example through co-trained Lyapunov functions ( Kolter & Manek , 2019 ) . However , all these approaches impose strong limitations on the dynamical repertoire of the RNN , enforcing global convergence to fixed points or simple cycles.1 In doing so , they drastically reduce the expressiveness of these models ( Kerg et al. , 2019 ; Orhan & Pitkow , 2020 ) . To address this problem , Erichson et al . ( 2021 ) somewhat relaxed the constraints on the recurrence matrix by introducing a skew-symmetric decomposition combined with a Lipschitz condition on the activation function . Another recent approach discretizes oscillator ODEs to arrive at a stable system of coupled ( Rusch & Mishra , 2021a ) or independent ( Rusch & Mishra , 2021b ) oscillators which increase the RNN ’ s expressiveness while bounding its gradients . By design ( and as acknowledged by the authors ) , neither of these architectures is capable of producing chaotic dynamics , however , as the underlying ODEs do not allow for exponential divergence of close-by trajectories ( a prerequisite for chaos ) . Given these often principle limitations of parametrically or dynamically strongly constrained models , a fruitful direction may be to modify the training process itself , e.g . through modified or auxiliary loss functions ( Trinh et al. , 2018 ; Schmidt et al. , 2021 ) , or special procedures for parameter updating ( Kag & Saligrama , 2021b ) or loss truncation ( Williams & Zipser , 1989 ; Menick et al. , 2021 ) . Our empirical evaluation will follow up on such ideas , but also highlight that simple loss truncation , windowing , or architectural solutions like LSTMs are not sufficient . Learning dynamical systems . Surprisingly disconnected from the work on the EVGP and learning long-term dependencies , a huge and long-standing literature deals with training RNNs on nonlinear dynamical systems ( DS ) ( Pearlmutter , 1990 ; Trischler & D ’ Eleuterio , 2016 ; Vlachas et al. , 2020 ) , including chaotic systems like the famous Lorenz equations ( Lorenz , 1963 ) or chaotic turbulence in fluid dynamics ( Li et al. , 2021 ) . Teacher forcing ( TF ; Williams & Zipser ( 1989 ) , Pearlmutter ( 1990 ) , 1We make this point more formal in Appx . A.1.6 . Doya ( 1992 ) , see also Goodfellow et al . ( 2016 ) ) is one of the earliest techniques introduced to keep RNN trajectories on track while training . The idea behind TF is to simply replace RNN states by observations when available , thereby also effectively cutting off the gradients . TF essentially derives from ideas in dynamical control theory , and adaptive schemes that increasingly hand over control to the RNN throughout training have been devised ( Abarbanel , 2013 ; Abarbanel et al. , 2018 ; Bengio et al. , 2015 ) . A related technique from the control theory literature is “ multiple shooting ” ( Voss et al. , 2004 ) : Here the whole observed time series is chopped into chunks , and for each chunk of trajectory a new initial condition is estimated . Explicit constraints ensure continuity between the separate trajectory bits during optimization . State space models and the Expectation-Maximization algorithm became popular particularly in the 90es for uncovering the latent dynamics underlying a set of time series observations ( Ghahramani & Roweis , 1999 ) , and remain an important tool until today ( Durstewitz , 2017 ; Koppe et al. , 2019 ) . Most recently , approaches based on variational inference and the reparameterization trick ( Kingma & Welling , 2014 ) , like sequential variational autoencoders ( SVAE ) , gained in popularity for DS approximation ( Hernandez et al. , 2020 ) . “ Deterministic ” RNNs ( i.e. , with latent states not treated as random variables ) , like conventional LSTMs ( Vlachas et al. , 2018 ) , remain top choices for DS reconstruction , however . Although connections between DS ideas and loss gradients have been drawn early on ( Bengio et al. , 1994 ) , so far only particular scenarios ( like fixed point attractors ) have been considered . Closest to our work is recent work by Schmidt et al . ( 2021 ) , where non-divergence of loss gradients is established when RNNs converge to fixed points or cycles . However , this was done only for the particular class of piecewise-linear RNNs ( PLRNNs ) , more restrictive conditions for cycles were imposed than assumed here , and - most importantly - the chaotic case on which we focus here was not considered . Another recent study ( Engelken et al. , 2020 ) also points out the general connections between Lyapunov exponents and loss gradients that we develop in sect . 3.1 , but does not provide any in-depth theoretical treatment , proofs , or empirical evaluation , as we do here . Thus , a systematic theoretical framework that relates RNN dynamics more generally , and across a range of different RNN architectures , to the behavior of its training gradients , is still lacking so far .

Summary: The paper connects the gradient behavior of recurrent neural networks with its dynamics through the maximum Lyapunov exponent of a trajectory, which is generated by the respective RNN. In particular, several connections between RNNs which produce specific dynamics (e.g. with stable fixed points or limit-cycles) and the mitigation of the exploding gradient problem are proved. Moreover, the paper shows that RNNs generating a chaotic trajectory always suffer from the exploding gradient problem.  That being said, the paper emphasizes that in order to approximate chaotic systems, the RNN has to be able to produce chaotic trajectories, which is ruled out by design by several state-of-the-art RNNs that are constructed to solve the exploding and vanishing gradient problem. However, given that these RNNs - at least the moment the optimizer finds parameters such that the resulting RNN produces chaotic trajectories - will always run into the exploding gradient problem, the paper suggests an algorithmic fix for that, by forcing the hidden-states of the RNN to be the pseudo-inverse of the underlying ground-truth trajectory at equidistantly distributed points in time during training. This cuts off the gradients at these points in time and forces the output states to be close to the ground-truth trajectories. The interval for that is chosen based on the maximum Lyapunov exponent. In the paper, three experiments are provided, where two of them are based on approximating a chaotic dynamical system (Lorenz and Rössler), while one is based on real-world data from temperature forecasting.