Near-optimal Offline and Streaming Algorithms for Learning Non-Linear Dynamical Systems

1 Introduction . Non-linear dynamical systems ( NLDS ) are commonly used to model the data in a variety of domains like control theory , time-series analysis , and reinforement learning ( RL ) [ 1–4 ] . Standard NLDS models the data points ( X0 , X1 , . . . , XT ) as : Xt+1 = φ ( A ∗Xt ) + ηt , ( 1 ) where Xt ∈ Rd are the states , ηt ∈ Rd are i.i.d . noise vectors , A∗ ∈ Rd×d and φ : R → R is an increasing function called the ‘ link function ’ . Here , φ is supposed to act component wise over Rd . System identification problem is a foundational problem for NLDS , i.e. , given ( X0 , X1 , . . . , XT ) generated from ( 1 ) , the goal is to estimate A∗ accurately from a single trajectory ( X0 , X1 , . . . , XT ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The system identification problem is heavily studied in control theory [ 5–8 ] as well as time-series analysis [ 9 ] . For instance , the non-linear dynamical system considered here has an application in modeling non-linear distortions in power amplifiers [ 10 ] . The problem is challenging as data points X0 , X1 , . . . , XT are not i.i.d . as usually encountered in machine learning , but form a Markov process . If the mixing time τmix of the process is finite ( τmix < ∞ ) , then we can make the data approximately i.i.d . by considering only the points separated by Õ ( τmix ) time . While this allows using standard techniques for i.i.d . data , it reduces the effective number of samples to O ( Tτmix ) , which typically gives an error of the order O ( τmixT ) . In fact , even the state-of-the-art results have error bounds which are sub-optimal by a factor of τmix . Interestingly , for the special case of linear systems , i.e. , when φ ( x ) = x , the results are significantly stronger . For example , [ 11 , 12 ] showed that the matrix A∗ can be estimated with an error O ( 1/T ) even when the mixing time τmix > T . But these results rely on the fact that for linear systems , the estimation problem reduces to an ordinary least squares ( OLS ) problem for which a closed form expression is available and can be analyzed effectively . On the other hand , NLDS do not admit such closed form expressions . In fact the existing techniques mostly rely on mixing time arguments to induce i.i.d . like behavior in a subset of the points which leads to sub-optimal rates by τmix factor . Similarly , a direct application of uniform convergence results [ 13 ] to show that the minimizer of the empirical risk is close to the population minimizer still gives sub-optimal rates as off-the-shelf concentration inequalities ( cf . [ 14 ] ) incur an additional factor of mixing time . Finally , existing results are mostly focused on offline setting , and do not apply to the case where the data points are streaming which is critical in several practical problems like reinforcement learning ( RL ) and control theory . In this work , we provide algorithms and their corresponding error rates for the NLDS system identification problem in both offline and online setting , assuming the link function to be expansive ( Assumption 1 ) . The main highlight of our results is that the error rates are independent of the mixing time τmix , which to the best of our knowledge is first such result for any non-linear system identification in any setting . In fact , for offline setting , our analysis holds even for systems which do not mix within time T and even for marginally stable systems which do not mix at all . Furthermore , we analyze SGD-Reverse Experience Replay ( SGD-RER ) method , we provide the first streaming method for NLDS identification with error rate that is independent of τmix ( in the leading order term ) while still ensuring small space and time complexity . This algorithm was first discovered in the experimental RL setting in [ 15 ] based on Hippocampal reverse replay observed in biological networks [ 16–18 ] . It was introduced independently in [ 19 ] for the case of linear systems and efficiently unravels the complex dependency structure present in the problem . Finally , through a lower bound for ReLU— a non-expansive function—we provide strong justification for why expansivity might be necessary for a non-trivial result . Instead of mixing time arguments , our proofs for learning NLDS without mixing use a natural exponential martingale of the kind considered in the analysis of self normalized process ( [ 20 , 21 ] ) . For streaming setting , while we do use mixing time arguments ( proof of Theorems 2 and 3 ) , we combine them with a delicate stability analysis of the specific algorithm and the machinery developed in [ 19 ] to obtain strong error bounds . See Section 6 for a description of these techniques . Our Contributions . Key contributions of the paper are summarized below : 1 . Assuming expansive and monotonic link function φ and sub-Gaussian noise , we show that the offline Quasi Newton Method ( Algorithm 1 ) estimates the parameter A∗ with near optimal errors of the order O ( 1/T ) , even when the dynamics does not mix within time T . 2 . Assuming mixing NLDS , finite fourth moment on the noise , and expansive monotonic link function , we show that offline Quasi Newton Method again estimates the parameter A∗ with near-optimal error of O ( 1/T ) , independent of mixing time τmix . 3 . We give a one-pass , streaming algorithm inspired by SGD− RER method by [ 19 ] , and show that it achieves near-optimal error rates under the assumption of sub-Gaussian noise , NLDS stability ( see section 2.1 for the definition ) , uniform expansivity and second differentiability of the link function . 4 . We then show that learning with ReLU link function , which is non-expansive but is known to be easy to learn with if data points are all i.i.d . [ 22 ] , requires exponential ( in d ) many samples . We believe that the techniques developed in this work can be extended to provide efficient algorithms for learning with dependent data in more general settings . Related Works . NLDS has been studied in a variety of domains like time-series and recurrent neural networks ( RNN ) . [ 9 ] studies specific NLDS models from a time series perspective and establishes non-asymptotic convergence bounds for natural estimators ; their error rates suffer from mixing time factor τmix . [ 23 ] considers asymptotic learning of NLDS via neural networks trained using SGD , whereas [ 24 ] shows that overparametrized LSTMs trained with SGD learn to memorize the given data . [ 25–27 ] consider learning dynamical systems of the form ht+1 = φ ( A∗ht +B∗ut ) for states ht and inputs ut ; this setting is different from standard NLDS model we study . [ 28 ] considers the non linear dynamical systems of the form xt+1 = Aφ ( xt , ut ) + ηt which φ is a known non-linearity and matrix A is to be estimated . [ 29 , 30 ] consider essentially linear dynamics but allow for certain non-linearities that can be modeled as process noise . All these again differ from the model we consider . Standard NLDS identification ( 1 ) has received a lot of attention recently , with results by [ 31 , 32 ] being the most relevant . [ 31 ] uses uniform convergence results via . mixing time arguments to obtain parameter estimation error for offline SGD . [ 32 ] obtains similar bounds via . the analysis of the GLMtron algorithm [ 33 ] . However , both these works suffer from sub-optimal dependence on the mixing time . We refer to Table 1 for a comparison of the results . [ 34 ] , which appeared after the initial manuscript of this work , considers the question from a perspective of time series forecasting . This work considers sparsity in A∗ and an unknown link function which is estimated with isotonic regression . Their recovery guarantees eschew the mixing time dependence . However , the setting , assumptions and the error rates are incomparable to our setting . [ 32 ] also obtains within sample prediction error in the case when φ is not uniformly expansive along with parameter recovery bounds when φ is the ReLU function and the driving noise is Gaussian . However , the parameter estimation bounds for ReLU suffer from an exponential dependence on the dimension d and mixing time τmix . In Theorem 4 we establish that indeed we can not improve the exponential dependence in the dimension d for the case of parameter estimation . We note that the exponential dependence arises due to the dynamics present in the system since ReLU regression with isotropic i.i.d . data in well specified case has only a polynomial dependence in d [ 22 ] . Linear system identification ( LSI ) literature has been well studied with strong minimax optimal bounds [ 11 , 35 , 36 ] . These results primarily consider the ( convex ) empirical square loss which has a closed form solution . However , the square loss in the non-linear case is non-convex . Under the assumption that the link function is increasing , we consider a convex proxy loss which is widely used in generalized linear regression literature [ 22 , 33 , 37 ] . Similarly , GLMtron algorithm for learning NLDS in [ 32 ] ( see Equation ( 3 ) ) also considers a similar proxy loss . In [ 38 ] , the authors consider a family of GLMtron-like algorithms call Reflectron under the i.i.d . data setting . But they compare the performance of these algorithms experimentally on an NLDS similar to one considered in this work under low rank assumption on the system matrix . Finally , streaming setting for LSI has been recently studied in different model settings [ 19 , 39 ] . These methods observe that by exploiting techniques like experience replay ( [ 40 ] ) along with squared loss error , one can obtain strong error rates . SGD− RER method studied in this work is inspired by a similar method by [ 19 ] which was primarily studied for the linear case . 2 Problem Statement . Let φ : R → R be an increasing , 1-Lipschitz function such that φ ( 0 ) = 0 . Suppose X0 ∈ Rd is a random variable and A∗ ∈ Rd×d . We consider the following non-linear dynamical system ( NLDS ) : Xt+1 = φ ( A ∗Xt ) + ηt , ( 2 ) where the noise sequence η0 , . . . , ηT is i.i.d random vectors independent of X0 . The noise ηt is such that Eηt = 0 , Eηtη > t = σ2I for some σ > 0 . We will also assume that M4 : = E‖ηt‖4 < ∞ . Let µ be the law of noise η . We denote the model above as NLDS ( A∗ , µ , φ ) . Whenever a stationary distribution exists for the process , we will denote it by π ( A∗ , µ , φ ) or just π when the process is clear from context . We will call the trajectory X0 , X1 , . . . , XT ‘ stationary ’ if X0 is distributed according to the measure π ( A∗ , µ , φ ) . Unless specified otherwise , we take X0 = 0 almost surely . t=0 ‖φ ( AXt ) −Xt+1‖2 . However , when the link function φ is not linear , then this would be non-convex and hard to optimize . Instead , we use a convex proxy loss given by : Lprox ( A ; X ) = 1 T T−1∑ t=0 d∑ i=1 φ̄ ( 〈ai , Xt〉 ) − 〈ei , Xt+1〉〈ai , Xt〉 , ( 3 ) where φ̄ is the indefinite integral of the link function φ and ai is the i-th row of A . Note that the gradient of Lprox ( A ; X ) with respect to A is given by : ∇Lprox ( A ; X ) = 1 T T−1∑ t=0 ( φ ( AXt ) −Xt+1 ) X > t . ( 4 ) When the model is clear from context and the stationary distribution exists , we will denote the second moment matrix under the stationary distribution by G : = E [ XtX > t ] . Note that G E [ ηtη > t ] = σ2I . Also , the empirical second moment matrix is denoted by Ĝ : = 1T ∑T−1 t=0 XtX > t .

Summary: This work has four major contributions: a) authors provide the first offline algorithm that can learn non-linear dynamical systems without the mixing assumption, b) authors significantly improve upon the sample complexity of existing results for mixing systems, c) in the much harder one-pass, streaming setting authors study a SGD with Reverse Experience Replay (SGD − RER) method, and demonstrate that for mixing systems, it achieves the same sample complexity as our offline algorithm, d) authors justify the expansivity assumption by showing that for the popular ReLU link function — a non-expansive but easy to learn link function with i.i.d. samples — any method would require exponentially many samples (with respect to dimension of Xt) from the dynamical system.   This paper is comprehensive with both solid theoretical analysis and simulations.