Streaming Linear System Identification with Reverse Experience Replay

1 Introduction . In this paper , we study the problem of learning linear-time invariant ( LTI ) systems , where the goal is to estimate the matrix A∗ ∈ Rd×d from the given samples ( X0 , . . . , XT ) that obey : Xτ+1 = A ∗Xτ + ητ , Xτ ∈ Rd , ητ i.i.d.∼ µ , ( 1 ) where µ is an unbiased noise distribution . The problem is central in control theory and reinforcement learning ( RL ) literature [ 1 , 2 ] . It is also equivalent to estimating Vector Autoregressive ( VAR ) model popular in the time-series analysis literature [ 3 ] , where it has been used in several applications like finding gene regulatory information network [ 4 ] . Despite a long line of classical literature for the problem , most of the existing results focus on the offline setting , where all the samples ( X0 , . . . , XT ) are available apriori . In this setting , ordinary 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . least squares ( OLS ) method that estimates A as , Â = argminA ∑T−1 τ=0 ∥Xτ+1 −AXτ∥2 is known to be nearly optimal [ 5 , 6 ] . However , such offline solutions do not apply to the streaming setting – where A∗ needs to be estimated online – that has applications in several domains like RL , large-scale forecasting systems , recommendation systems [ 7 , 8 ] . In this paper , we study the above mentioned problem of learning LTI systems via first order gradient oracle with streaming data . The goal is to design an estimator that provides accurate estimation while ensuring nearly optimal time complexity and space complexity that is nearly independent of T . Note that due to specific form arising in linear regression , the optimal solution to OLS can be estimated in online fashion using Sherman-Morrison-Woodbury formula . But such a solution is limited and does not apply to practically important settings like generalized non-linear dynamical system or when A∗ is high-dimensional and has special structure like low-rank or sparsity [ 9 , 10 ] . So , in this work , we focus on designing Stochastic Gradient Descent ( SGD ) style methods that can work directly with first order gradient oracle , and hence is more widely applicable to the settings mentioned above . In fact , after the first appearance of this manuscript , the algorithm ( SGD− RER ) and the techniques introduced in this paper were used to obtain near-optimal guarantees for learning certain classes of non-linear dynamical systems [ 11 ] as well as in Q-learning tabular MDPs in RL [ 12 ] . We note that prior to [ 11 ] , even optimal offline algorithms were unknown for such non-linear systems . SGD is a popular method for general streaming settings , and has been shown to be optimal for problems like streaming linear regression [ 13 ] . However , when the data has temporal dependencies , as in the estimation of linear dynamical systems , such a naive implementation of SGD may not perform well as observed in [ 14 , 15 ] . In fact , for linear system identification , our experiments suggest that SGD suffers from a non-zero bias ( Section 6 ) . In order to address temporal dependencies in data , practitioners use a heuristic called experience replay , which maintains a buffer of points , and samples points randomly from the buffer . However , for linear system identification , experience replay does not seem to provide an accurate unbiased estimator for reasonable buffer sizes ( see Section 6 ) . In this work , we propose reverse experience replay for linear system identification . Our method maintains a small buffer of points , but instead of random ordering , we replay the points in a reverse order . We show that this algorithm exactly unravels the temporal correlations to obtain a consistent estimator for A∗ . Similar to the standard linear regression problem with i.i.d . samples , we can break the error in two parts : a ) bias : that depends on the initial error ∥A0 −A∗∥ , b ) variance : the steady state error due to noise η . We show that our proposed method , under fairly standard assumptions and with a small buffer size , is able to decrease the bias at fast rate , while the variance error is nearly optimal ( see Theorem 1 ) , matching the information theoretic lower bounds [ 5 , Theorem 2.3 ] . To the best of our knowledge , we provide first non-trivial analysis for a purely streaming SGD-style algorithm with optimal computation complexity and nearly bounded space complexity that is dependent logarithmically on T . We note here that the idea of reverse experience replay was independently discovered in experimental reinforcement learning by [ 16 ] based on reverse replay observed in Hippocampal place cells [ 17 ] in Neurobiology . We also refer to [ 18 ] for more on this connection . In addition to the transition matrix estimation error ∥A−A∗∥ , we also provide analysis of prediction error , i.e. , E [ ∥AX −A∗X∥2 ] ( see Theorem 2 ) . Here again , we bound the bias and the variance part of the error separately . We further derive new lower bounds for prediction error ( see Theorem 4 ) and show that our algorithm is minimax optimal , under standard assumptions on the model . As mentioned earlier , our method work with general first order oracles , hence applies to more general problems like sparse LTI estimation with known sparsity structure and unlike online OLS methods , SGD− RER has nearly optimal time complexity . Finally , we also provide empirical validation of our method on simulated data , and demonstrate that the proposed method is indeed able to provide error rate similar to the OLS method while methods like SGD and standard experience replay , lead to biased estimates . Related Work . Due to applications in RL , recently LTI system identification has been widely studied . In particular , [ 19 ] studied the problem in offline setting under the “ stability '' condition , i.e. , the spectral radius ( ρ ( A∗ ) ) of A∗ is a constant bounded away from 1 . The sequence of papers [ 5 , 6 , 20 , 21 ] provide optimal analyses of the offline OLS estimator beyond assumptions of stability . That is , they show that OLS recovers A∗ near optimally even the process defined by ( 1 ) is stable but does not mix within time T ( when ρ ( A∗ ) is 1− O ( 1/T ) ) or is unstable ( when ρ ( A∗ ) is larger than 1 ) . Further [ 5 , 22 ] provide information theoretic lower bounds for the LTI system identification problem . [ 11 , 23 , 24 ] consider the problem of identifying non-linear dynamical systems of the form Xt+1 = ϕ ( A ∗Xt ) + ηt where ϕ is a one dimensional link function which acts co-ordinate wise . In this setting , however , there is no closed for expressions for the estimator of A∗ . [ 23 , 24 ] give offline algorithms whose error guarantees are worse off by factors of mixing time whereas [ 11 ] obtains near optimal offline and streaming algorithms for this setting . In fact , [ 11 ] uses SGD− RER which was first introduced in this work in order to obtain the streaming algorithm . LTI identification problem has been studied in time series forecasting literature as well . For example , [ 25 ] obtains asymptotic consistency results for system identification problem and [ 26 , 27 ] consider the problem of finite time recovery . Both consider a certain parameterized predictor for a linear system with empirical risk minimization for the parameter and analyzes the deviation from population risk . Similarly , [ 28 ] also studies generalization error guarantees . In contrast , our work is able to provide precise bias and variance ( similar to generalization error ) of the estimator in the streaming setting , and show that the asymptotic error is minimax optimal . [ 29 ] studied SISO systems with observations ( xτ , yτ ) ∈ R2 and a hidden state hτ which is high dimensional , thus their model and applications are significantly different than the LTI system we study . For the SISO system , [ 29 ] analyzes SGD to provide error bounds contain ( a large ) polynomial in the hidden state dimension . Here , the hidden state has an evolution similar to Equation 1 whereas x1 , . . . , xT are drawn i.i.d from some distribution . System identification has been studied in the context of partially observed LTI systems as well . Recent works [ 19 , 30–34 ] focus on identifying a certain Hankel-like matrix of the system . These are not directly comparable to the fully observed setting in this work since the model parameters are identifiable only upto a similarity transformation in the partially observed setting . Recently , there has been an exciting line of work in the related domain of online control ( see [ 35–38 ] and references therein ) . The state equation studied in these papers also contain an additive term of Buτ for some unknown matrix B and a control signal uτ and the noise ητ is either stochastic ( as in [ 35 ] ) or adversarial ( as in [ 36–38 ] ) . The goal is to output control signals uτ after observing X1 , . . . , Xτ , such that the cost ∑ τ cτ ( Xτ , uτ ) is minimized for some sequence of convex costs cτ . We focus on the LTI system identification ( or estimation ) problem while the goal of the above mentioned line of work is to design an online controller . We also note here another line of works [ 32 , 39–44 ] focused on online prediction of both fully observed and partially observed LTI systems , and the similar problem of time series forecasting by regret minimization [ 28 , 45 ] . In particular , the main goal there is to design online prediction algorithms minimizing regret against a certain class ( for instance , against a Kalman filter with knowledge of the system parameters in the case of partially observed LTI systems ) . The situation considered in our work is different in atleast two aspects : 1 ) we focus significantly on parameter recovery or system identification and 2 ) our notion of prediction is prediction at stationarity which can be thought of as one-step regret ( compared to T–step regret for instance in [ 39 , 40 ] ) . Finally , [ 9 ] considers offline sparse linear regression with ℓ1 penalty where the feature vector is derived from an auto regressive model . Similarly , [ 14 ] considers the problem of linear regression where the feature vectors come from a Markov chain . This line of work is different from ours in that we try to estimate the parameters of the Markov process itself . Paper Organization . We provide the problem definition and introduce the notations in the next section . We then present our algorithm and the key intuition behind it in Section 3 . We then present our main result in Section 4 and provide a proof sketch in Section 5 . Finally , we present simulation results in Section 6 .

Summary: The paper proposes an SGD-style algorithm for linear dynamical system identification. The authors claim that the proposed algorithm is the first of its kind to have near-optimal error rates. The key idea behind the algorithm is to run SGD in reverse on streaming segments of the data.