Streaming Linear System Identification with Reverse Experience Replay

1 Introduction . In this paper , we study the problem of learning linear-time invariant ( LTI ) systems , where the goal is to estimate the matrix A∗ ∈ Rd×d from the given samples ( X0 , . . . , XT ) that obey : Xτ+1 = A ∗Xτ + ητ , Xτ ∈ Rd , ητ i.i.d.∼ µ , ( 1 ) where µ is an unbiased noise distribution . The problem is central in control theory and reinforcement learning ( RL ) literature [ 1 , 2 ] . It is also equivalent to estimating Vector Autoregressive ( VAR ) model popular in the time-series analysis literature [ 3 ] , where it has been used in several applications like finding gene regulatory information network [ 4 ] . Despite a long line of classical literature for the problem , most of the existing results focus on the offline setting , where all the samples ( X0 , . . . , XT ) are available apriori . In this setting , ordinary 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . least squares ( OLS ) method that estimates A as , Â = argminA ∑T−1 τ=0 ∥Xτ+1 −AXτ∥2 is known to be nearly optimal [ 5 , 6 ] . However , such offline solutions do not apply to the streaming setting – where A∗ needs to be estimated online – that has applications in several domains like RL , large-scale forecasting systems , recommendation systems [ 7 , 8 ] . In this paper , we study the above mentioned problem of learning LTI systems via first order gradient oracle with streaming data . The goal is to design an estimator that provides accurate estimation while ensuring nearly optimal time complexity and space complexity that is nearly independent of T . Note that due to specific form arising in linear regression , the optimal solution to OLS can be estimated in online fashion using Sherman-Morrison-Woodbury formula . But such a solution is limited and does not apply to practically important settings like generalized non-linear dynamical system or when A∗ is high-dimensional and has special structure like low-rank or sparsity [ 9 , 10 ] . So , in this work , we focus on designing Stochastic Gradient Descent ( SGD ) style methods that can work directly with first order gradient oracle , and hence is more widely applicable to the settings mentioned above . In fact , after the first appearance of this manuscript , the algorithm ( SGD− RER ) and the techniques introduced in this paper were used to obtain near-optimal guarantees for learning certain classes of non-linear dynamical systems [ 11 ] as well as in Q-learning tabular MDPs in RL [ 12 ] . We note that prior to [ 11 ] , even optimal offline algorithms were unknown for such non-linear systems . SGD is a popular method for general streaming settings , and has been shown to be optimal for problems like streaming linear regression [ 13 ] . However , when the data has temporal dependencies , as in the estimation of linear dynamical systems , such a naive implementation of SGD may not perform well as observed in [ 14 , 15 ] . In fact , for linear system identification , our experiments suggest that SGD suffers from a non-zero bias ( Section 6 ) . In order to address temporal dependencies in data , practitioners use a heuristic called experience replay , which maintains a buffer of points , and samples points randomly from the buffer . However , for linear system identification , experience replay does not seem to provide an accurate unbiased estimator for reasonable buffer sizes ( see Section 6 ) . In this work , we propose reverse experience replay for linear system identification . Our method maintains a small buffer of points , but instead of random ordering , we replay the points in a reverse order . We show that this algorithm exactly unravels the temporal correlations to obtain a consistent estimator for A∗ . Similar to the standard linear regression problem with i.i.d . samples , we can break the error in two parts : a ) bias : that depends on the initial error ∥A0 −A∗∥ , b ) variance : the steady state error due to noise η . We show that our proposed method , under fairly standard assumptions and with a small buffer size , is able to decrease the bias at fast rate , while the variance error is nearly optimal ( see Theorem 1 ) , matching the information theoretic lower bounds [ 5 , Theorem 2.3 ] . To the best of our knowledge , we provide first non-trivial analysis for a purely streaming SGD-style algorithm with optimal computation complexity and nearly bounded space complexity that is dependent logarithmically on T . We note here that the idea of reverse experience replay was independently discovered in experimental reinforcement learning by [ 16 ] based on reverse replay observed in Hippocampal place cells [ 17 ] in Neurobiology . We also refer to [ 18 ] for more on this connection . In addition to the transition matrix estimation error ∥A−A∗∥ , we also provide analysis of prediction error , i.e. , E [ ∥AX −A∗X∥2 ] ( see Theorem 2 ) . Here again , we bound the bias and the variance part of the error separately . We further derive new lower bounds for prediction error ( see Theorem 4 ) and show that our algorithm is minimax optimal , under standard assumptions on the model . As mentioned earlier , our method work with general first order oracles , hence applies to more general problems like sparse LTI estimation with known sparsity structure and unlike online OLS methods , SGD− RER has nearly optimal time complexity . Finally , we also provide empirical validation of our method on simulated data , and demonstrate that the proposed method is indeed able to provide error rate similar to the OLS method while methods like SGD and standard experience replay , lead to biased estimates . Related Work . Due to applications in RL , recently LTI system identification has been widely studied . In particular , [ 19 ] studied the problem in offline setting under the “ stability '' condition , i.e. , the spectral radius ( ρ ( A∗ ) ) of A∗ is a constant bounded away from 1 . The sequence of papers [ 5 , 6 , 20 , 21 ] provide optimal analyses of the offline OLS estimator beyond assumptions of stability . That is , they show that OLS recovers A∗ near optimally even the process defined by ( 1 ) is stable but does not mix within time T ( when ρ ( A∗ ) is 1− O ( 1/T ) ) or is unstable ( when ρ ( A∗ ) is larger than 1 ) . Further [ 5 , 22 ] provide information theoretic lower bounds for the LTI system identification problem . [ 11 , 23 , 24 ] consider the problem of identifying non-linear dynamical systems of the form Xt+1 = ϕ ( A ∗Xt ) + ηt where ϕ is a one dimensional link function which acts co-ordinate wise . In this setting , however , there is no closed for expressions for the estimator of A∗ . [ 23 , 24 ] give offline algorithms whose error guarantees are worse off by factors of mixing time whereas [ 11 ] obtains near optimal offline and streaming algorithms for this setting . In fact , [ 11 ] uses SGD− RER which was first introduced in this work in order to obtain the streaming algorithm . LTI identification problem has been studied in time series forecasting literature as well . For example , [ 25 ] obtains asymptotic consistency results for system identification problem and [ 26 , 27 ] consider the problem of finite time recovery . Both consider a certain parameterized predictor for a linear system with empirical risk minimization for the parameter and analyzes the deviation from population risk . Similarly , [ 28 ] also studies generalization error guarantees . In contrast , our work is able to provide precise bias and variance ( similar to generalization error ) of the estimator in the streaming setting , and show that the asymptotic error is minimax optimal . [ 29 ] studied SISO systems with observations ( xτ , yτ ) ∈ R2 and a hidden state hτ which is high dimensional , thus their model and applications are significantly different than the LTI system we study . For the SISO system , [ 29 ] analyzes SGD to provide error bounds contain ( a large ) polynomial in the hidden state dimension . Here , the hidden state has an evolution similar to Equation 1 whereas x1 , . . . , xT are drawn i.i.d from some distribution . System identification has been studied in the context of partially observed LTI systems as well . Recent works [ 19 , 30–34 ] focus on identifying a certain Hankel-like matrix of the system . These are not directly comparable to the fully observed setting in this work since the model parameters are identifiable only upto a similarity transformation in the partially observed setting . Recently , there has been an exciting line of work in the related domain of online control ( see [ 35–38 ] and references therein ) . The state equation studied in these papers also contain an additive term of Buτ for some unknown matrix B and a control signal uτ and the noise ητ is either stochastic ( as in [ 35 ] ) or adversarial ( as in [ 36–38 ] ) . The goal is to output control signals uτ after observing X1 , . . . , Xτ , such that the cost ∑ τ cτ ( Xτ , uτ ) is minimized for some sequence of convex costs cτ . We focus on the LTI system identification ( or estimation ) problem while the goal of the above mentioned line of work is to design an online controller . We also note here another line of works [ 32 , 39–44 ] focused on online prediction of both fully observed and partially observed LTI systems , and the similar problem of time series forecasting by regret minimization [ 28 , 45 ] . In particular , the main goal there is to design online prediction algorithms minimizing regret against a certain class ( for instance , against a Kalman filter with knowledge of the system parameters in the case of partially observed LTI systems ) . The situation considered in our work is different in atleast two aspects : 1 ) we focus significantly on parameter recovery or system identification and 2 ) our notion of prediction is prediction at stationarity which can be thought of as one-step regret ( compared to T–step regret for instance in [ 39 , 40 ] ) . Finally , [ 9 ] considers offline sparse linear regression with ℓ1 penalty where the feature vector is derived from an auto regressive model . Similarly , [ 14 ] considers the problem of linear regression where the feature vectors come from a Markov chain . This line of work is different from ours in that we try to estimate the parameters of the Markov process itself . Paper Organization . We provide the problem definition and introduce the notations in the next section . We then present our algorithm and the key intuition behind it in Section 3 . We then present our main result in Section 4 and provide a proof sketch in Section 5 . Finally , we present simulation results in Section 6 .

Summary: The authors consider a rather special case of a linear dynamical system, where there is no hidden state and the transition matrix is "externally stable" (less than 1 in the operator norm), process noise is sub-gaussian, and the stationary distribution has bounded fourth moment. In this case, they suggest running the SGD "backwards" in small buffers, which they call "reverse experience replay". This leads to the independence of the observations, and the unbiased nature of the SGD updates.