Inferring Dynamical Systems with Long-Range Dependencies through Line Attractor Regularization

1 INTRODUCTION . Theories of complex systems in biology and physics are often formulated in terms of sets of stochastic differential or difference equations , i.e . as stochastic dynamical systems ( DS ) . A long-standing desire is to retrieve these generating dynamical equations directly from observed time series data ( Kantz & Schreiber , 2004 ) . A variety of machine and deep learning methodologies toward this goal have been introduced in recent years ( Chen et al. , 2017 ; Champion et al. , 2019 ; Jordan et al. , 2019 ; Duncker et al. , 2019 ; Ayed et al. , 2019 ; Durstewitz , 2017 ; Koppe et al. , 2019 ) , many of them based on recurrent neural networks ( RNN ) which can universally approximate any DS ( i.e. , its flow field ) under some mild conditions ( Funahashi & Nakamura , 1993 ; Kimura & Nakano , 1998 ) . However , vanilla RNN as often used in this context are well known for their problems in capturing long-term dependencies and slow time scales in the data ( Hochreiter & Schmidhuber , 1997 ; Bengio et al. , 1994 ) . In DS terms , this is generally due to the fact that flexible information maintenance over long periods requires precise fine-tuning of model parameters toward ’ line attractor ’ configurations ( Fig . 1 ) , a concept first propagated in computational neuroscience for addressing animal performance in parametric working memory tasks ( Seung , 1996 ; Seung et al. , 2000 ; Durstewitz , 2003 ) . Line attractors introduce directions of zero-flow into the model ’ s state space that enable long-term maintenance of arbitrary values ( Fig . 1 ) . Specially designed RNN architectures equipped with gating mechanisms and ( linear ) memory cells have been suggested for solving this issue ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014 ) . However , from a DS perspective , simpler models that can more easily be analyzed and interpreted in DS terms , and for which more efficient inference algorithms exist that emphasize approximation of the true underlying DS would be preferable . Recent solutions to the vanishing vs. exploding gradient problem attempt to retain the simplicity of vanilla RNN by initializing or constraining the recurrent weight matrix to be the identity ( Le et al. , 2015 ) , orthogonal ( Henaff et al. , 2016 ; Helfrich et al. , 2018 ) or unitary ( Arjovsky et al. , 2016 ) . In this way , in a system including piecewise linear ( PL ) components like rectified-linear units ( ReLU ) , line attractor dimensions are established from the start by construction or ensured throughout training by a specifically parameterized matrix decomposition . However , for many DS problems , line attractors instantiated by mere initialization procedures may be unstable and quickly dissolve during training . On the other hand , orthogonal or unitary constraints are too restrictive for reconstructing DS , and more generally from a computational perspective as well ( Kerg et al. , 2019 ) : For instance , neither chaotic behavior ( that requires diverging directions ) nor settings with multiple isolated fixed point or limit cycle attractors are possible . Here we therefore suggest a different solution to the problem , by pushing ( but not strictly enforcing ) ReLU-based , piecewise-linear RNN ( PLRNN ) toward line attractor configurations along some ( but not all ) directions in state space . We achieve this by adding special regularization terms for a subset of RNN units to the loss function that promote such a configuration . We demonstrate that our approach outperforms , or is en par with , LSTM and other , initialization-based , methods on a number of ’ classical ’ machine learning benchmarks ( Hochreiter & Schmidhuber , 1997 ) . More importantly , we demonstrate that while with previous methods it was difficult to capture slow behavior in a DS that exhibits widely different time scales , our new regularization-supported inference efficiently captures all relevant time scales . 2 RELATED WORK . Long-range dependency problems in RNN . Error gradients in vanilla RNN trained by some form of gradient descent , like back-propagation through time ( BPTT , Rumelhart et al . ( 1986 ) ) , tend to either explode or vanish due to the large product of derivative terms that results from recursive application of the chain rule over time steps ( Hochreiter , 1991 ; Bengio et al. , 1994 ; Hochreiter & Schmidhuber , 1997 ) . Formally , RNN zt = Fθ ( zt−1 , st ) are discrete time dynamical systems that tend to either converge , e.g . to fixed point or limit cycle attractors , or diverge ( to infinity or as in chaotic systems ) over time , unless parameters of the system are precisely tuned to create directions of zero-flow in the system ’ s state space ( Fig . 1 ) , called line attractors ( Seung , 1996 ; Seung et al. , 2000 ; Durstewitz , 2003 ) . Convergence of the RNN in general implies vanishing and global divergence exploding gradients . To address this issue , RNN with gated memory cells have been specifically designed ( Hochreiter & Schmidhuber , 1997 ; Cho et al. , 2014 ) , but these are complicated and tedious to analyze from a DS perspective . Recently , Le et al . ( 2015 ) observed that initialization of the recurrent weight matrix W to the identity in ReLU-based RNN may yield performance en par with LSTMs on standard machine learning benchmarks . For a ReLU with activity zt ≥ 0 , zero bias and unit slope , this results in the identity mapping , hence a line attractor configuration . Talathi & Vartak ( 2016 ) expanded on this idea by initializing the recurrence matrix such that its largest absolute eigenvalue is 1 , arguing that this would leave other directions in the system ’ s state space free for computations other than memory maintenance . Later work enforced orthogonal ( Henaff et al. , 2016 ; Helfrich et al. , 2018 ; Jing et al. , 2019 ) or unitary ( Arjovsky et al. , 2016 ) constraints on the recurrent weight matrix during training . While this appears to yield long-term memory performance superior to that of LSTMs , these networks are limited in their computational power ( Kerg et al. , 2019 ) . This may be a consequence of the fact that RNN with orthogonal recurrence matrix are quite restricted in the range of dynamical phenomena they can produce , e.g . chaotic attractors are not possible since diverging eigen-directions are disabled . Our approach therefore is to establish line attractors only along some but not all directions in state space , and to only push the RNN toward these configurations but not strictly enforce them , such that convergence or divergence of RNN dynamics is still possible . We furthermore implement these concepts through regularization terms in the loss functions , such that they are encouraged throughout training unlike when only established through initialization . Dynamical systems reconstruction . From a natural science perspective , the goal of reconstructing the underlying DS fundamentally differs from building a system that ’ merely ’ yields good ahead predictions , as in DS reconstruction we require that the inferred model can freely reproduce ( when no longer guided by the data ) the underlying attractor geometries and state space properties ( see section 3.5 , Fig . S2 ; Kantz & Schreiber ( 2004 ) ) . Earlier work using RNN for DS identification ( Roweis & Ghahramani , 2002 ; Yu et al. , 2006 ) mainly focused on inferring the posterior over latent trajectories Z = { z1 , . . . , zT } given time series data X = { x1 , . . . , xT } , p ( Z|X ) , and on ahead predictions ( Lu et al. , 2017 ) , hence did not show that inferred models can generate the underlying attractor geometries on their own . Others ( Trischler & D ’ Eleuterio , 2016 ; Brunton et al. , 2016 ) attempt to approximate the flow field , obtained e.g . by numerical differentiation , directly through basis expansions or neural networks , but numerical derivatives are problematic for their high variance and other numerical issues ( Raissi , 2018 ; Baydin et al. , 2018 ; Chen et al. , 2017 ) . Some approaches assume the form of the DS equations basically to be given ( Raissi , 2018 ; Gorbach et al. , 2017 ) and focus on estimating the system ’ s latent states and parameters , rather than approximating an unknown DS based on the observed time series information alone . In many biological systems like the brain the intrinsic dynamics are highly stochastic with many noise sources , like probabilistic synaptic release ( Stevens , 2003 ) , such that models that do not explicitly account for dynamical process noise ( Champion et al. , 2019 ; Rudy et al. , 2019 ) may be less suitable . Finally , some fully probabilistic models for DS reconstruction based on GRU ( Fraccaro et al . ( 2016 ) , cf . Jordan et al . ( 2019 ) ) , LSTM ( Zheng et al. , 2017 ) , or radial basis function ( Zhao & Park , 2017 ) networks are not easily interpretable and amenable to DS analysis . Most importantly , none of these previous approaches considers the long-range dependency problem within more easily tractable RNN for DS reconstruction . 3 MODEL FORMULATION AND OPTIMIZATION APPROACHES . 3.1 MODEL AND PRELIMINARIES . Assume we are given two multivariate time series S = { st } and X = { xt } , one we will denote as ’ inputs ’ ( S ) and the other as ’ outputs ’ ( X ) . We will first consider the ’ classical ’ ( supervised ) machine learning setting where we wish to map S on X through a RNN with latent state equation zt = Fθ ( zt−1 , st ) , as for instance in the ’ addition problem ’ ( Hochreiter & Schmidhuber , 1997 ) . In DS reconstruction , in contrast , we usually have a dense time series X from which we wish to infer ( unsupervised ) the underlying DS , where S may provide an additional forcing function or sparse experimental inputs or perturbations . The latent RNN we consider here takes the specific form zt = Azt−1 +Wφ ( zt−1 ) +Cst + h+ εt , εt ∼ N ( 0 , Σ ) , ( 1 ) where zt ∈ RM×1 is the hidden state ( column ) vector of dimensionM , A ∈ RM×M a diagonal and W ∈ RM×M an off-diagonal matrix , st ∈ RK×1 the external input of dimension K , C ∈ RM×K the input mapping , h ∈ RM×1 a bias , and εt a Gaussian noise term with diagonal covariance matrix diag ( Σ ) ∈ RM+ . The nonlinearity φ ( z ) is a ReLU , φ ( z ) i = max ( 0 , zi ) , i ∈ { 1 , . . . , M } . This specific formulation is originally motivated by firing rate ( population ) models in computational neuroscience ( Song et al. , 2016 ; Durstewitz , 2017 ) , where latent states zt may represent membrane voltages or currents , A the neurons ’ passive time constants , W the synaptic coupling among neurons , and φ ( · ) the voltage-to-rate transfer function . However , for a RNN in the form zt = Wφ ( zt−1 ) + h , note that the simple change of variables yt →W−1 ( zt − h ) will yield the more familiar form yt = φ ( Wyt−1 + h ) ( Beer , 2006 ) . Besides its neuroscience motivation , note that by letting A = I , W = 0 , h = 0 , we get a strict line attractor system across the variables ’ whole support which we conjecture will be of advantage for establishing long short-term memory properties . Also we can solve for all of the system ’ s fixed points analytically by solving the equations z∗ = ( I −A−WDΩ ) −1 h , with DΩ as defined in Suppl . 7.1.2 , and can determine their stability from the eigenvalues of matrixA+WDΩ . We could do the same for limit cycles , in principle , which are fixed points of the r-times iterated map F rθ , although practically the number of configurations to consider increases exponentially as 2M ·r . Finally , we remark that a discrete piecewise-linear system can , under certain conditions , be transformed into an equivalent continuous-time ( ODE ) piecewise-linear system ζ̇ = GΩ ( ζ ( t ) , s ( t ) ) ( Suppl . 7.1.2 , Ozaki ( 2012 ) ) , in the sense that if ζ ( t ) = zt , then ζ ( t + ∆t ) = zt+1 after a defined time step ∆t . These are among the properties that make PLRNNs more amenable to rigorous DS analysis than other RNN formulations . We will assume that the latent RNN states zt are coupled to the actual observations xt through a simple observation model of the form xt = Bg ( zt ) + ηt , ηt ∼ N ( 0 , Γ ) ( 2 ) in the case of real-valued observations xt ∈ RN×1 , where B ∈ RN×M is a factor loading matrix and diag ( Γ ) ∈ RN+ the diagonal covariance matrix of the Gaussian observation noise , or p̂i , t : = p̂t ( xi , t = 1 ) = ( eBi , :zt ) N∑ j=1 eBj , :zt −1 , ( 3 ) in the case of multi-categorical observations xi , t ∈ { 0 , 1 } , ∑ i xi , t = 1 .

Summary: This paper proposes a type of regularization for recurrent networks, with the goal of encouraging particular dynamical structures (in this case, line attractors) in the dynamics of the networks. The proposed regularization penalty is only applied to a subset of the recurrent units; the motivation for this is to allow neurons not contained in the subset to learn different structures. The paper applies this regularization method on three example machine learning sequence tasks: an addition task, a multiplication task, and sequential MNIST classification, as well as on learning a 2-D dynamical system model of a bursting neuron with two different timescales.