Near-optimal Offline and Streaming Algorithms for Learning Non-Linear Dynamical Systems

1 Introduction . Non-linear dynamical systems ( NLDS ) are commonly used to model the data in a variety of domains like control theory , time-series analysis , and reinforement learning ( RL ) [ 1–4 ] . Standard NLDS models the data points ( X0 , X1 , . . . , XT ) as : Xt+1 = φ ( A ∗Xt ) + ηt , ( 1 ) where Xt ∈ Rd are the states , ηt ∈ Rd are i.i.d . noise vectors , A∗ ∈ Rd×d and φ : R → R is an increasing function called the ‘ link function ’ . Here , φ is supposed to act component wise over Rd . System identification problem is a foundational problem for NLDS , i.e. , given ( X0 , X1 , . . . , XT ) generated from ( 1 ) , the goal is to estimate A∗ accurately from a single trajectory ( X0 , X1 , . . . , XT ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The system identification problem is heavily studied in control theory [ 5–8 ] as well as time-series analysis [ 9 ] . For instance , the non-linear dynamical system considered here has an application in modeling non-linear distortions in power amplifiers [ 10 ] . The problem is challenging as data points X0 , X1 , . . . , XT are not i.i.d . as usually encountered in machine learning , but form a Markov process . If the mixing time τmix of the process is finite ( τmix < ∞ ) , then we can make the data approximately i.i.d . by considering only the points separated by Õ ( τmix ) time . While this allows using standard techniques for i.i.d . data , it reduces the effective number of samples to O ( Tτmix ) , which typically gives an error of the order O ( τmixT ) . In fact , even the state-of-the-art results have error bounds which are sub-optimal by a factor of τmix . Interestingly , for the special case of linear systems , i.e. , when φ ( x ) = x , the results are significantly stronger . For example , [ 11 , 12 ] showed that the matrix A∗ can be estimated with an error O ( 1/T ) even when the mixing time τmix > T . But these results rely on the fact that for linear systems , the estimation problem reduces to an ordinary least squares ( OLS ) problem for which a closed form expression is available and can be analyzed effectively . On the other hand , NLDS do not admit such closed form expressions . In fact the existing techniques mostly rely on mixing time arguments to induce i.i.d . like behavior in a subset of the points which leads to sub-optimal rates by τmix factor . Similarly , a direct application of uniform convergence results [ 13 ] to show that the minimizer of the empirical risk is close to the population minimizer still gives sub-optimal rates as off-the-shelf concentration inequalities ( cf . [ 14 ] ) incur an additional factor of mixing time . Finally , existing results are mostly focused on offline setting , and do not apply to the case where the data points are streaming which is critical in several practical problems like reinforcement learning ( RL ) and control theory . In this work , we provide algorithms and their corresponding error rates for the NLDS system identification problem in both offline and online setting , assuming the link function to be expansive ( Assumption 1 ) . The main highlight of our results is that the error rates are independent of the mixing time τmix , which to the best of our knowledge is first such result for any non-linear system identification in any setting . In fact , for offline setting , our analysis holds even for systems which do not mix within time T and even for marginally stable systems which do not mix at all . Furthermore , we analyze SGD-Reverse Experience Replay ( SGD-RER ) method , we provide the first streaming method for NLDS identification with error rate that is independent of τmix ( in the leading order term ) while still ensuring small space and time complexity . This algorithm was first discovered in the experimental RL setting in [ 15 ] based on Hippocampal reverse replay observed in biological networks [ 16–18 ] . It was introduced independently in [ 19 ] for the case of linear systems and efficiently unravels the complex dependency structure present in the problem . Finally , through a lower bound for ReLU— a non-expansive function—we provide strong justification for why expansivity might be necessary for a non-trivial result . Instead of mixing time arguments , our proofs for learning NLDS without mixing use a natural exponential martingale of the kind considered in the analysis of self normalized process ( [ 20 , 21 ] ) . For streaming setting , while we do use mixing time arguments ( proof of Theorems 2 and 3 ) , we combine them with a delicate stability analysis of the specific algorithm and the machinery developed in [ 19 ] to obtain strong error bounds . See Section 6 for a description of these techniques . Our Contributions . Key contributions of the paper are summarized below : 1 . Assuming expansive and monotonic link function φ and sub-Gaussian noise , we show that the offline Quasi Newton Method ( Algorithm 1 ) estimates the parameter A∗ with near optimal errors of the order O ( 1/T ) , even when the dynamics does not mix within time T . 2 . Assuming mixing NLDS , finite fourth moment on the noise , and expansive monotonic link function , we show that offline Quasi Newton Method again estimates the parameter A∗ with near-optimal error of O ( 1/T ) , independent of mixing time τmix . 3 . We give a one-pass , streaming algorithm inspired by SGD− RER method by [ 19 ] , and show that it achieves near-optimal error rates under the assumption of sub-Gaussian noise , NLDS stability ( see section 2.1 for the definition ) , uniform expansivity and second differentiability of the link function . 4 . We then show that learning with ReLU link function , which is non-expansive but is known to be easy to learn with if data points are all i.i.d . [ 22 ] , requires exponential ( in d ) many samples . We believe that the techniques developed in this work can be extended to provide efficient algorithms for learning with dependent data in more general settings . Related Works . NLDS has been studied in a variety of domains like time-series and recurrent neural networks ( RNN ) . [ 9 ] studies specific NLDS models from a time series perspective and establishes non-asymptotic convergence bounds for natural estimators ; their error rates suffer from mixing time factor τmix . [ 23 ] considers asymptotic learning of NLDS via neural networks trained using SGD , whereas [ 24 ] shows that overparametrized LSTMs trained with SGD learn to memorize the given data . [ 25–27 ] consider learning dynamical systems of the form ht+1 = φ ( A∗ht +B∗ut ) for states ht and inputs ut ; this setting is different from standard NLDS model we study . [ 28 ] considers the non linear dynamical systems of the form xt+1 = Aφ ( xt , ut ) + ηt which φ is a known non-linearity and matrix A is to be estimated . [ 29 , 30 ] consider essentially linear dynamics but allow for certain non-linearities that can be modeled as process noise . All these again differ from the model we consider . Standard NLDS identification ( 1 ) has received a lot of attention recently , with results by [ 31 , 32 ] being the most relevant . [ 31 ] uses uniform convergence results via . mixing time arguments to obtain parameter estimation error for offline SGD . [ 32 ] obtains similar bounds via . the analysis of the GLMtron algorithm [ 33 ] . However , both these works suffer from sub-optimal dependence on the mixing time . We refer to Table 1 for a comparison of the results . [ 34 ] , which appeared after the initial manuscript of this work , considers the question from a perspective of time series forecasting . This work considers sparsity in A∗ and an unknown link function which is estimated with isotonic regression . Their recovery guarantees eschew the mixing time dependence . However , the setting , assumptions and the error rates are incomparable to our setting . [ 32 ] also obtains within sample prediction error in the case when φ is not uniformly expansive along with parameter recovery bounds when φ is the ReLU function and the driving noise is Gaussian . However , the parameter estimation bounds for ReLU suffer from an exponential dependence on the dimension d and mixing time τmix . In Theorem 4 we establish that indeed we can not improve the exponential dependence in the dimension d for the case of parameter estimation . We note that the exponential dependence arises due to the dynamics present in the system since ReLU regression with isotropic i.i.d . data in well specified case has only a polynomial dependence in d [ 22 ] . Linear system identification ( LSI ) literature has been well studied with strong minimax optimal bounds [ 11 , 35 , 36 ] . These results primarily consider the ( convex ) empirical square loss which has a closed form solution . However , the square loss in the non-linear case is non-convex . Under the assumption that the link function is increasing , we consider a convex proxy loss which is widely used in generalized linear regression literature [ 22 , 33 , 37 ] . Similarly , GLMtron algorithm for learning NLDS in [ 32 ] ( see Equation ( 3 ) ) also considers a similar proxy loss . In [ 38 ] , the authors consider a family of GLMtron-like algorithms call Reflectron under the i.i.d . data setting . But they compare the performance of these algorithms experimentally on an NLDS similar to one considered in this work under low rank assumption on the system matrix . Finally , streaming setting for LSI has been recently studied in different model settings [ 19 , 39 ] . These methods observe that by exploiting techniques like experience replay ( [ 40 ] ) along with squared loss error , one can obtain strong error rates . SGD− RER method studied in this work is inspired by a similar method by [ 19 ] which was primarily studied for the linear case . 2 Problem Statement . Let φ : R → R be an increasing , 1-Lipschitz function such that φ ( 0 ) = 0 . Suppose X0 ∈ Rd is a random variable and A∗ ∈ Rd×d . We consider the following non-linear dynamical system ( NLDS ) : Xt+1 = φ ( A ∗Xt ) + ηt , ( 2 ) where the noise sequence η0 , . . . , ηT is i.i.d random vectors independent of X0 . The noise ηt is such that Eηt = 0 , Eηtη > t = σ2I for some σ > 0 . We will also assume that M4 : = E‖ηt‖4 < ∞ . Let µ be the law of noise η . We denote the model above as NLDS ( A∗ , µ , φ ) . Whenever a stationary distribution exists for the process , we will denote it by π ( A∗ , µ , φ ) or just π when the process is clear from context . We will call the trajectory X0 , X1 , . . . , XT ‘ stationary ’ if X0 is distributed according to the measure π ( A∗ , µ , φ ) . Unless specified otherwise , we take X0 = 0 almost surely . t=0 ‖φ ( AXt ) −Xt+1‖2 . However , when the link function φ is not linear , then this would be non-convex and hard to optimize . Instead , we use a convex proxy loss given by : Lprox ( A ; X ) = 1 T T−1∑ t=0 d∑ i=1 φ̄ ( 〈ai , Xt〉 ) − 〈ei , Xt+1〉〈ai , Xt〉 , ( 3 ) where φ̄ is the indefinite integral of the link function φ and ai is the i-th row of A . Note that the gradient of Lprox ( A ; X ) with respect to A is given by : ∇Lprox ( A ; X ) = 1 T T−1∑ t=0 ( φ ( AXt ) −Xt+1 ) X > t . ( 4 ) When the model is clear from context and the stationary distribution exists , we will denote the second moment matrix under the stationary distribution by G : = E [ XtX > t ] . Note that G E [ ηtη > t ] = σ2I . Also , the empirical second moment matrix is denoted by Ĝ : = 1T ∑T−1 t=0 XtX > t .

Summary: This work considers learning NLDS. Previous works can only learn mixing systems with sub-gaussian noise in the offline setting, with sample complexity depending on the mixing time. This work improves existing results in several ways, assuming the link function is expansive: 1, it's shown that the offline Quasi Newton Method achieves near-optimal convergence rate for non-mixing systems. 2, the offline Quasi Newton Method still achieves near-optimal convergence rate with heavy-tail noise, assuming mixing systems. 3, a new algorithm SGD-RER is introduced, which achieves near-optimal convergence rate in the online setting. 4, an exponential lower bound is proven for the ReLU link function, showing the expansivity assumption is necessary.