How to train RNNs on chaotic data?

Recurrent neural networks ( RNNs ) are wide-spread machine learning tools for modeling sequential and time series data . They are notoriously hard to train because their loss gradients backpropagated in time tend to saturate or diverge during training . This is known as the exploding and vanishing gradient problem . Previous solutions to this issue either built on rather complicated , purpose-engineered architectures with gated memory buffers , or - more recently - imposed constraints that ensure convergence to a fixed point or restrict ( the eigenspectrum of ) the recurrence matrix . Such constraints , however , convey severe limitations on the expressivity of the RNN . Essential intrinsic dynamics such as multistability or chaos are disabled . This is inherently at disaccord with the chaotic nature of many , if not most , time series encountered in nature and society . Here we offer a comprehensive theoretical treatment of this problem by relating the loss gradients during RNN training to the Lyapunov spectrum of RNN-generated orbits . We mathematically prove that RNNs producing stable equilibrium or cyclic behavior have bounded gradients , whereas the gradients of RNNs with chaotic dynamics always diverge . Based on these analyses and insights , we adapt the old idea of teacher forcing to yield an effective yet simple training technique for chaotic data , and offer guidance on how to choose relevant hyperparameters according to the Lyapunov spectrum . 1 INTRODUCTION . Recurrent neural networks ( RNNs ) are widely used across various fields in engineering and science for learning sequential tasks or modeling and predicting time series ( Lipton et al. , 2015 ) . Yet , they struggle when long-term temporal dependencies , very slow , or hugely varying time scales are involved ( Hochreiter , 1991 ; Bengio et al. , 1994 ; Schmidt et al. , 2021 ; Li et al. , 2018 ; Rusch & Mishra , 2021a ) . Time series or sequential data with such properties are , however , very common in fields like climate physics ( Thomson , 1990 ) , neuroscience ( Fusi et al. , 2007 ; Russo & Durstewitz , 2017 ) , ecology ( Turchin & Taylor , 1992 ) , or language processing ( Cho et al. , 2014b ) . Training RNNs on such data is hard because the loss gradients backpropagated in time easily saturate or diverge in this process . This is commonly referred to as the exploding and vanishing gradient problem ( EVGP ) ( Hochreiter , 1991 ; Bengio et al. , 1994 ; Pascanu et al. , 2013 ) . One solution to the EVGP is based on specifically designed RNN architectures with gating mechanisms , such as long short-term memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) or gated recurrent units ( GRU ) ( Cho et al. , 2014a ) . These architectures allow states at earlier time steps to more easily influence activity much later through a kind of protected memory buffer , thus alleviating the EVGP by structural design . In practice , such models need to be backed up by further techniques like gradient clipping to keep the gradients in check ( Pascanu et al. , 2013 ) . The relatively complex architectural design of these networks impedes their mathematical analysis and requires reverse engineering after training ( Maheswaranathan et al. , 2019 ; Monfared & Durstewitz , 2020a ; b ; Schmidt et al. , 2021 ) . Partly to forego these complications , a variety of other solutions has been proposed recently , imposing restrictions on the recurrence matrix to bound the gradients ( Arjovsky et al. , 2016 ; Chang et al. , 2019 ) , or enforcing global stability by design or regularization ( Erichson et al. , 2021 ; Kolter & Manek , 2019 ) . Often these procedures dramatically curtail the expressivity of the RNN ( Kerg et al. , 2019 ; Orhan & Pitkow , 2020 ; Schmidt et al. , 2021 ) ; in particular , they rule out chaotic dynamics ( see further discussion below ) . This is at odds with the plethora of chaotic phenomena in nature , engineering , and society . Chaotic dynamics are commonplace , almost default in any complex physical or biological system . This includes scientific areas as diverse as neuroscience ( Durstewitz & Gabriel , 2007 ; van Vreeswijk & Sompolinsky , 1996 ) , physiology ( Kesmia et al. , 2020 ) , geophysics ( Sivakumar , 2004 ) , climate systems ( Tziperman et al. , 1997 ) , astrophysics ( Laskar & Robutel , 1993 ) , ecology ( Duarte et al. , 2010 ) , chemical reactions ( Field & Györgyi , 1993 ) , cell ( Olsen & Degn , 1977 ) or population ( May , 1987 ) biology . Chaotic phenomena are also crucial for the understanding of societal and epidemiological processes , such as the spread of diseases ( Mangiarotti et al. , 2020 ) , or in economics ( Faggini , 2014 ) . They are further relevant in purely technical contexts such as electrical engineering ( Tchitnga et al. , 2019 ; Kamdjeu Kengne et al. , 2021 ) or laser optics ( Kantz et al. , 1993 ) . They have even been suggested to play an up to now largely neglected , but potentially very significant role in speech recognition ( Sabanal & Nakagawa , 1996 ) and natural language processing ( Inoue et al. , 2021 ) . Hence , in almost any practical setting , chaotic phenomena abound . They can not , in general , be ignored when devising RNN training algorithms . Here we offer a comprehensive theoretical treatment of the relation between RNN dynamics and the behavior of the loss gradients during training . We find a close connection between an RNN ’ s loss gradients and the largest Lyapunov exponent of its freely generated orbits . We mathematically prove that RNNs producing stable fixed point or cyclic behavior have bounded gradients . Crucially , however , the loss gradients of RNNs producing chaotic dynamics always diverge . Hence , the chaotic nature of many time series data induces a principle problem , and , despite significant efforts in the past to solve the EVGP , training RNNs on such data remains an open issue . We illustrate the implications of our theory for RNN training on several simulated and empirical chaotic time series , and adapt the old idea of sparsely forced BPTT as a simple yet effective remedy that enables to learn the underlying dynamics despite exploding gradients . 2 RELATED WORKS . Exploding and vanishing gradients . While ‘ classical ’ remedies of the EVGP ( Hochreiter , 1991 ; Bengio et al. , 1994 ; Pascanu et al. , 2013 ) rest on purpose-tailored architectures with gating mechanisms , which safeguard information flow across longer temporal distances ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014a ) , the focus has recently shifted to simpler RNNs that address the EVGP by restricting the recurrence matrix to be orthogonal ( Henaff et al. , 2016 ; Helfrich et al. , 2018 ; Jing et al. , 2019 ) , unitary ( Arjovsky et al. , 2016 ) , or antisymmetric ( Chang et al. , 2019 ) , or by ensuring globally stable fixed point solutions ( Kag et al. , 2020 ; Kag & Saligrama , 2021a ) , for example through co-trained Lyapunov functions ( Kolter & Manek , 2019 ) . However , all these approaches impose strong limitations on the dynamical repertoire of the RNN , enforcing global convergence to fixed points or simple cycles.1 In doing so , they drastically reduce the expressiveness of these models ( Kerg et al. , 2019 ; Orhan & Pitkow , 2020 ) . To address this problem , Erichson et al . ( 2021 ) somewhat relaxed the constraints on the recurrence matrix by introducing a skew-symmetric decomposition combined with a Lipschitz condition on the activation function . Another recent approach discretizes oscillator ODEs to arrive at a stable system of coupled ( Rusch & Mishra , 2021a ) or independent ( Rusch & Mishra , 2021b ) oscillators which increase the RNN ’ s expressiveness while bounding its gradients . By design ( and as acknowledged by the authors ) , neither of these architectures is capable of producing chaotic dynamics , however , as the underlying ODEs do not allow for exponential divergence of close-by trajectories ( a prerequisite for chaos ) . Given these often principle limitations of parametrically or dynamically strongly constrained models , a fruitful direction may be to modify the training process itself , e.g . through modified or auxiliary loss functions ( Trinh et al. , 2018 ; Schmidt et al. , 2021 ) , or special procedures for parameter updating ( Kag & Saligrama , 2021b ) or loss truncation ( Williams & Zipser , 1989 ; Menick et al. , 2021 ) . Our empirical evaluation will follow up on such ideas , but also highlight that simple loss truncation , windowing , or architectural solutions like LSTMs are not sufficient . Learning dynamical systems . Surprisingly disconnected from the work on the EVGP and learning long-term dependencies , a huge and long-standing literature deals with training RNNs on nonlinear dynamical systems ( DS ) ( Pearlmutter , 1990 ; Trischler & D ’ Eleuterio , 2016 ; Vlachas et al. , 2020 ) , including chaotic systems like the famous Lorenz equations ( Lorenz , 1963 ) or chaotic turbulence in fluid dynamics ( Li et al. , 2021 ) . Teacher forcing ( TF ; Williams & Zipser ( 1989 ) , Pearlmutter ( 1990 ) , 1We make this point more formal in Appx . A.1.6 . Doya ( 1992 ) , see also Goodfellow et al . ( 2016 ) ) is one of the earliest techniques introduced to keep RNN trajectories on track while training . The idea behind TF is to simply replace RNN states by observations when available , thereby also effectively cutting off the gradients . TF essentially derives from ideas in dynamical control theory , and adaptive schemes that increasingly hand over control to the RNN throughout training have been devised ( Abarbanel , 2013 ; Abarbanel et al. , 2018 ; Bengio et al. , 2015 ) . A related technique from the control theory literature is “ multiple shooting ” ( Voss et al. , 2004 ) : Here the whole observed time series is chopped into chunks , and for each chunk of trajectory a new initial condition is estimated . Explicit constraints ensure continuity between the separate trajectory bits during optimization . State space models and the Expectation-Maximization algorithm became popular particularly in the 90es for uncovering the latent dynamics underlying a set of time series observations ( Ghahramani & Roweis , 1999 ) , and remain an important tool until today ( Durstewitz , 2017 ; Koppe et al. , 2019 ) . Most recently , approaches based on variational inference and the reparameterization trick ( Kingma & Welling , 2014 ) , like sequential variational autoencoders ( SVAE ) , gained in popularity for DS approximation ( Hernandez et al. , 2020 ) . “ Deterministic ” RNNs ( i.e. , with latent states not treated as random variables ) , like conventional LSTMs ( Vlachas et al. , 2018 ) , remain top choices for DS reconstruction , however . Although connections between DS ideas and loss gradients have been drawn early on ( Bengio et al. , 1994 ) , so far only particular scenarios ( like fixed point attractors ) have been considered . Closest to our work is recent work by Schmidt et al . ( 2021 ) , where non-divergence of loss gradients is established when RNNs converge to fixed points or cycles . However , this was done only for the particular class of piecewise-linear RNNs ( PLRNNs ) , more restrictive conditions for cycles were imposed than assumed here , and - most importantly - the chaotic case on which we focus here was not considered . Another recent study ( Engelken et al. , 2020 ) also points out the general connections between Lyapunov exponents and loss gradients that we develop in sect . 3.1 , but does not provide any in-depth theoretical treatment , proofs , or empirical evaluation , as we do here . Thus , a systematic theoretical framework that relates RNN dynamics more generally , and across a range of different RNN architectures , to the behavior of its training gradients , is still lacking so far .

Summary: The authors provide a study of gradients of the recurrent neural networks using Lyapunov exponents. They claim to offer a simple and effective training for the chaotic data. Some simulation results are provided.