Linear Convergence of Gradient Methods for Estimating Structured Transition Matrices in High-dimensional Vector Autoregressive Models

1 Introduction . Learning the network structure through high-dimensional time series data has been an important focus of research for the past decades . There are lots of application examples ranging from macroeconomic analysis [ 1–3 ] to connectivity measuring among financial firms [ 4 ] , gene regularity network inference [ 5 ] and radar signals processing [ 6 , 7 ] . To perform these tasks , vector autoregressive ( VAR ) models play a critical role in both theory and application . For example , VAR models are widely adopted to characterize the spatially and temporally colored disturbance for multichannel adaptive signal detection in [ 7–11 ] . Under the low-dimensional settings where the dimension of the transition matrix and the number of time series are relatively small , the theory of VAR models is well established , see e.g. , [ 12 ] . However , lots of meaningful applications are under the high-dimensional settings where the problem dimension far exceeds the number of time series and additional structure information of parameters is required to guarantee successful recovery . For the cases where the samples are independent , both theoretical properties and practical algorithms of high-dimensional statistical problems have been studied by considerable literature during the past few years , including but not limited to [ 13–17 ] . For the correlated time series cases , the corresponding results are still developing . By assuming that the spectral norm of the transition matrix is less than 1 , Loh and Wainwright consider VAR models regularized by the l1-norm in [ 18 ] . Under the double asymptotic framework , Han and Liu also consider VAR models under the similar assumption in [ 19 ] . In [ 20 ] , Basu and Michailidis analyze sparse transition matrices estimation of VAR models with a milder stability assumption by ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . introducing the spectral density . Recently , Melnyk and Banerjee extend the analysis to structured VAR models regularized by any suitable norm in [ 21 ] . Compared with the progress on the statistical analysis of VAR models , much less is known about their computational issues . For instance , Basu et al . [ 22 ] make exploration in this direction by proposing the fast network structure learning algorithm ( FNSL ) for the penalized recovery procedure . Their analysis does not take the structure information of the parameter into account and it only establishes a sub-linear convergence rate for the FNSL . In this paper , we first provide the non-asymptotic optimization guarantee of PGD for VAR models with single-structured transition matrices . Our analysis illustrates that the distance between iteration points of PGD and the real transition matrix would converge linearly to the statistical error despite the objective function is not strongly convex under the high-dimensional settings . Our result is sharp in the sense that the minimal requirement of samples to guarantee the linear rate matches the phase transition of the model with independent samples up to a constant factor . On the other hand , considering the parameter to be estimated only has one type of low-dimensional structure or is single-structured might be too oversimplified for messy real applications . So superposition-structured models have received more attention of researchers in last decade . Typical examples include robust PCA [ 23 , 24 ] , multi-task learning [ 25 ] and robust matrix sensing [ 26 ] . For these scenarios , we employ AltPGD to solve related optimization problems and establish the corresponding non-asymptotic optimization guarantee . Our results show that AltPGD also enjoys a linear convergence rate , which is much more efficient than the sub-linear rate in [ 22 ] . At the same time , our analysis avoids a drawback in [ 22 ] that the estimation error does not converge to zero when the number of measurements approaches infinity . Last but not the least , we also illustrate that AltPGD is a practical algorithm to solve the general superposition-structured statistical model in [ 27 ] . Apart from the time series case , our analysis also adapts to multi-task learning and robust PCA . 2 Problem formulation . In this paper , we consider a d-dimensional vector-valued stationary time series { x0 , · · · , xn } generated by a VAR model of lag 1 with serially uncorrelated Gaussian errors . The VAR ( 1 ) model is defined as xt+1 = Γ T ? xt + et+1 , t = 0 , · · · , n− 1 , ( 1 ) where Γ ? ∈ Rd×d is the transition matrix and et iid∼ N ( 0 , Σe ) . This model can be reformulated in the matrix form Y = XΓ ? +E , ( 2 ) where Y = [ x1 , · · · , xn ] T ∈ Rn×d , X = [ x0 , · · · , xn−1 ] T ∈ Rn×d , and E = [ e1 , · · · , en ] T ∈ Rn×d . The goal of the VAR ( 1 ) model is to recover the transition matrix Γ ? from the observation matrix Y and the data matrixX . In the high-dimensional settings with n d2 , tractable recovery is possible when the transition matrix Γ ? is well structured . Thus we introduce a convex regularizer R ( · ) to promote the structure of Γ ? . Then a popular way to estimate Γ ? is to solve the following constrained least square problem min Γ 1 2n ||Y −XΓ||2F , s.t . R ( Γ ) ≤ R ( Γ ? ) , ( 3 ) where || · ||F represents the Frobenius norm of a matrix . When the transition matrix Γ ? is superposition-structured in which Γ ? is the sum of two singlestructured components , i.e. , Γ ? = S ? + L ? , we adopt two convex functions RS ( · ) and RL ( · ) to characterize the structures of its two components and solve the following constrained problem to estimate S ? and L ? min S , L 1 2n ||Y −X ( S +L ) ||2F , s.t . RS ( S ) ≤ RS ( S ? ) , RL ( L ) ≤ RL ( L ? ) . ( 4 ) 3 Single-structured transition matrices estimation via PGD . In this part , we consider the case where the transition matrix to be estimated in ( 1 ) only has one type of low-dimensional structure which is characterized by a convex functionR ( · ) . To estimate the transition matrix Γ ? , we solve the problem ( 3 ) via the PGD update ( summarized in Algorithm 1 ) . By setting fn ( Γ ) = ||Y −XΓ||2F/ ( 2n ) , we could write the iteration of PGD as Γk+1 = PK ( Γk − µ∇fn ( Γk ) ) , ( 5 ) where µ is the step size , K = { Γ | R ( Γ ) ≤ R ( Γ ? ) } is the descent set and PK represents the orthogonal projection onto the set K. We also introduce the concept of the descent cone C = cone ( K − Γ ? ) , where cone ( · ) is the conic hull of a set . Algorithm 1 PGD for single-structured transition matrices estimation Input : Initial point Γ0 , step size µ , iteration number K. for k = 0 to K − 1 do Γk+1 = PK ( Γk − µ∇fn ( Γk ) ) end for Output : ΓK To guarantee consistent estimation , we propose the following stability assumption for the VAR model ( 1 ) , which is also imposed in [ 20 , 22 ] . Assumption 1 ( Stability ) . The characteristic polynomial of the VAR model ( 1 ) satisfies det ( A ( z ) ) 6= 0 on the unit circle of the complex plane { z ∈ C : |z| = 1 } , where A ( z ) = Id×d − ΓT ? z . In the non-asymptotic analysis of the VAR model ( 1 ) , we use the following two quantities M ( fx ) = ess sup θ∈ [ −π , π ] λmax ( fx ( θ ) ) , ( 6 ) m ( fx ) = ess inf θ∈ [ −π , π ] λmin ( fx ( θ ) ) , ( 7 ) where fx ( θ ) is the spectral density function defined as fx ( θ ) : = 1 2π ∞∑ l=−∞ Σx ( l ) e −ilθ , θ ∈ [ −π , π ] . ( 8 ) Here we useΣx ( l ) to represent Σx ( l ) = E [ xtxTt+l ] , t , l ∈ Z . ( 9 ) Specially , we writeΣx ( 0 ) = Σx for simplicity . Compared with the model with independent samples , there is dependency among the rows of data matrix X in the VAR model ( 2 ) , which is the main challenge when deriving the deviation bounds required by the estimation problem . For Gaussian processes , this dependency could be characterized by the covariance matrix Υx = E [ vec ( XT ) vec ( XT ) T ] , where vec ( · ) represents the column-wise vectorization of a matrix . The following lemma indicates that the concept of the spectral density function is a convenient tool to bound the extreme eigenvalues of Υx . Lemma 1 ( Proposition 2.3 in [ 20 ] ) . Donate Υx = E [ vec ( XT ) vec ( XT ) T ] , where X is the data matrix in the VAR model ( 2 ) . We could bound the extreme eigenvalues of Υx as 2πm ( fx ) ≤ λmin ( Υx ) ≤ λmax ( Υx ) ≤ 2πM ( fx ) . ( 10 ) In particular , we also have 2πm ( fx ) ≤ λmin ( Σx ) ≤ λmax ( Σx ) ≤ 2πM ( fx ) . ( 11 ) For stable and invertible ARMA processes which include the model ( 1 ) , the spectral density ( 8 ) has a closed form expression based on the matrix valued polynomials [ 20 , Equation ( 2.4 ) ] . Furthermore , the concrete calculation of the upper bound ofM ( fx ) and the lower bound of m ( fx ) for the model ( 1 ) is provided in [ 20 , Proposition 2.2 ] which indicates m ( fx ) andM ( fx ) could be bounded away from zero and infinity . In this way , we could introduce the quantities κmin and κmax to simplify the expression of our analysis . Assumption 2 ( Boundness ) . Suppose there are positive constants κmin and κmax satisfying 0 < κmin 2π ≤ m ( fx ) ≤M ( fx ) ≤ κmax 2π . ( 12 ) With the above assumption , we could represent the extreme eigenvalues of Υx andΣx in a concise way κmin ≤ λmin ( Υx ) ≤ λmax ( Υx ) ≤ κmax , ( 13 ) κmin ≤ λmin ( Σx ) ≤ λmax ( Σx ) ≤ κmax . ( 14 ) In our analysis , we use the Gaussian width to quantify the size of a set T ω ( T ) : = Esup x∈T 〈g , x〉 , where g ∼ N ( 0 , I ) . We are now ready to present the non-asymptotic optimization guarantee of PGD for the problem ( 3 ) . Theorem 1 . Consider the VAR model ( 1 ) satisfying Assumptions 1 and 2 . Suppose Γ ? is singlestructured and R ( · ) is a convex function . Starting from a point Γ0 satisfying R ( Γ0 ) ≤ R ( Γ ? ) , we solve the optimization problem ( 3 ) via PGD with the step size µ = 1/κmax . If the number of measurements satisfies √ n > 2C κmax κmin ( ω ( C ∩ SF ) + u ) , ( 15 ) then the PGD update ( 5 ) would obey ||Γk+1 − Γ ? ||F ≤ ρ k+1||Γ0 − Γ ? ||F + ξ 1− ρ ( 16 ) with probability at least 1− c exp ( −u2 ) . Here ρ = 1− κmin κmax + C ω ( C ∩ SF ) + u√ n < 1− κmin 2κmax , ( 17 ) ξ = C ′ 1 √ κmax ||Σe|| 1 2 ω ( C ∩ SF ) + u√ n , ( 18 ) ξ 1− ρ = 1 1− ρ · C ′ 1√ κmax ||Σe|| 1 2 ω ( C ∩ SF ) + u√ n < 2C ′ √ κmax κmin ||Σe|| 1 2 ω ( C ∩ SF ) + u√ n , ( 19 ) || · || represents the spectral norm of a matrix , SF represents the sphere with unit Frobenius norm and c , C , C ′ are absolute constants . Remark 1 ( Sharpness ) . Our result demonstrates that PGD can converge linearly to the statistical error despite the objective function fn ( Γ ) is not strongly convex under the high-dimensional settings . And the linear convergence is achieved when the number of measurements is of order ω ( C ∩ SF ) 2 , which matches the phase transition of the model with independent samples [ 15 , 16 ] . Remark 2 ( Impact of correlated samples ) . Our result also provide some insights for the impact of correlated samples . It is not hard to find that the temporal dependency is characterized by κmax and κmin which appear in the convergence rate , the estimation error , and the required number of samples . Let κ = κmax/κmin . Clearly , a smaller κ will lead to a faster convergence rate with smaller required samples and estimation error . Remark 3 ( Comparison with related works ) . The existing works on structured VAR models such as [ 19–21 ] are mainly concerned with establishing the statistical error bounds for different recovery procedures . Our work focuses on algorithmic analysis and reveals how many samples would ensure that the algorithm achieves a fast convergence rate . On the other hand , our work could be regarded as a generalization of the result in [ 17 ] from independent samples to time series . This generalization is nontrivial , since new mathematical tools ( e.g. , two deviation inequalities : Lemmas 4 and 5 ) are required to address time-series settings . To the best of our knowledge , Lemma 5 seems to be the first non-asymptotic result for the VAR models which yields the unified estimation error bounds for both independent and correlated samples . Furthermore , our analysis demonstrates PGD enjoys a linear convergence for structured VAR models , which is much more efficient than the sub-linear convergence rate of FNSL in [ 22 ] . Additionally , the linear convergence rate of PGD is also illustrated in [ 18 ] for the VAR ( 1 ) model with a sparse transition matrix , when the spectral norm of the transition matrix is strictly less than 1 . Compared with the result in [ 18 ] , our analysis adapts to general structured signals with the milder Assumption 1 . Remark 4 ( Extension ) . In Theorem 1 , we set the step size µ = 1/κmax for a concise expression of the result . In fact , any step size satisfying µ ≤ 1/κmax could achieve a linear convergence rate by providing the corresponding number of measurements . In [ 20 , 21 ] , VAR ( d ) models are reformulated as VAR ( 1 ) models . With the same reformulation , our analysis also adapts to VAR ( d ) models . Our analysis for the VAR model ( 1 ) also adapts to the multi-task learning problem with independent samples.1 Different from the time series setting , the measurements Y = XΓ ? + E in this case are generated from xt iid∼ N ( 0 , Σx ) and et iid∼ N ( 0 , Σe ) , for t = 1 , · · · , n , where X = [ x1 , · · · , xn ] T ∈ Rn×d and E = [ e1 , · · · , en ] T ∈ Rn×d are independent . Corollary 1 . Consider the multi-task learning problem with the above conditions . Under Assumption 2 , we solve the optimization problem ( 3 ) via PGD with the step size µ = 1/κmax and a starting point Γ0 satisfyingR ( Γ0 ) ≤ R ( Γ ? ) . If the number of measurements satisfies √ n > 2C κmax κmin ( ω ( C ∩ SF ) + u ) , ( 20 ) then the update ( 5 ) would obey ||Γk+1 − Γ ? ||F < ( 1− κmin 2κmax ) k+1||Γ0 − Γ ? ||F + 2C ′ √ κmax κmin ||Σe|| 1 2 ω ( C ∩ SF ) + u√ n ( 21 ) with probability at least 1− c exp ( −u2 ) , where c , C , C ′ are absolute constants .

Summary: The paper proposes projected gradient descent algorithms for estimating high-dimensional vector auto-regressive models, and conducts convergence analysis following the restricted strong convexity lens.  Specifically, sparse, low-rank and sparse plus low-rank models are of interest.  While many of these topics have been studied "to death", the novelty here seems to be (a) there's limited work in applying sparse plus low-rank approaches to time-series settings, where there is an additional constraint that the transition matrix has to lead to a stable dynamical system (b) Typical convergence analysis is done under iid setting, but for VAR models, samples are naturally dependent with strong auto-correlation.