Perturbation-based Regret Analysis of Predictive Control in Linear Time Varying Systems

1 Introduction . We study the problem of predictive control in a linear time-varying ( LTV ) system , where the dynamics is given by GC+1 = CGC + ⌫CDC + FC . Here , GC is the state , DC is the control action , and FC is the disturbance or exogenous input . At each time step C , the online controller incurs a time-varying state cost 5C ( GC ) and control cost 2C ( DC 1 ) , and then decides its next control action DC . In deciding DC the controller makes use of predictions of the next : future disturbances , cost functions , and dynamical matrices , and seeks to minimize its total cost on a finite horizon ) . Our main results bound the dynamic regret and competitive ratio of predictive controllers in this LTV setting . Recently , a growing literature has sought to design controllers that achieve learning guarantees such as static regret [ 1 , 2 ] , dynamic regret [ 3 , 4 ] , and competitive ratio [ 5 ] . The most relevant †Yiheng Lin , Yang Hu , Haoyuan Sun , Guanya Shi , and Guannan Qu contributed equally to this work . ‡This work was supported by NSF grants CNS-2106403 , NGSDI-2105648 , and AitF-1637598 , with additional support from Amazon AWS , PIMCO , and the Resnick Sustainability Insitute . Yiheng Lin was supported by Kortschak Scholars program . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . line of work concerns predictive control with learning guarantees , which studies how to leverage the prediction window : to reduce the regret and competitive ratio . This line of work has focused on linear time-invariant ( LTI ) systems [ 3 , 4 , 6 , 7 ] . However , linear time-varying ( LTV ) systems have received increasing attention in recent years due to their importance in a variety of emerging applications , despite the challenges associated with analysis . For example , in the problem of power grid frequency regulation , the dynamics is determined by the proportion of renewable energy in total power generation , which is time-varying [ 8 , 9 ] . It is also common to use the LTV systems as an approximation of nonlinear dynamics in predictive control and planning [ 10–13 ] . The current lack of progress toward understanding measures like regret and competitive ratio in LTV settings is due to the need for new techniques to generalize the dynamics from LTI to LTV and the costs from quadratic to well-conditioned functions . Specifically , the proof approaches used in previous studies on regret and competitive ratio of predictive control in LTI dynamics with quadratic costs , e.g. , [ 4 , 6 , 7 ] , require explicitly writing down the cost-to-go function , optimal control actions , and algorithm ’ s actions as functions of the system parameters . This is very difficult , if not impossible , for general cost functions that do not have a quadratic form . A promising approach that does not require such explicit characterizations is to derive results via reductions from optimal control to online convex optimization with multi-step memory , e.g. , [ 1 , 3 , 5 , 14 , 15 ] . However , such reductions usually do not work well for LTV systems due to the need to represent the problem in control canonical form [ 3 , 5 ] , or due to limitations on the policy class and comparisons to static benchmarks [ 1 , 15 ] . Perhaps the most prominent approach for controlling LTV systems is Model Predictive Control ( MPC ) , also known as Receding Horizon Control [ 16 ] . Generally speaking , at each time step , an MPC-style algorithm solves a predictive trajectory for the future : time steps and commit the first control action in this trajectory . MPC-style algorithms are known to work well in practice , even when the dynamics are non-linear and time-varying , e.g. , [ 13 , 17–19 ] . On theoretical side , the asymptotic behaviors of MPC such as stability and convergence have been studied intensively under general assumptions on dynamics and costs [ 20–23 ] . However , non-asymptotic guarantees such as regret and competitive ratio of MPC-style policies have been limited . Despite recent work providing such guarantees in the context of LTI systems with quadratic costs , e.g. , [ 4 , 6 , 7 ] , the derivation of regret and competitive ratio results for MPC in LTV systems remains open . Contributions . We provide the first regret and competitive ratio results for a controller in LTV systems with time-varying costs . Specifically , we show that an MPC-style predictive control algorithm ( Algorithm 1 ) achieves a dynamic regret that decays exponentially with respect to the length of prediction window : in the LTV system ( Theorem 4.2 ) : $ ( _ : ) ) , where the decay rate _ is a positive constant less than 1 . This almost matches the exponential lower bound for improvement from predictions in the LTI setting shown in [ 3 ] in the sense that , to achieve any target regret level , the required length of prediction : shown by our bound differs from the theoretical lower bound by at most a constant factor . With a variation of predictive control ( Algorithm 2 ) , we also show the first competitive bound in LTV systems with time-varying well-conditioned costs ( Theorem 4.3 ) : 1 + $ ( _ : ) , where the decay rate _ is identical with the one in the regret bound . We develop a novel analysis framework based on a perturbation approach . Specifically , instead of solving for the optimal states and control actions like previous analyses in the LTI setting with quadratic costs , e.g. , [ 4 , 7 ] , we bound how much impact an perturbation to the system parameters can have on the optimal solution . This type of perturbation bound ( Theorem 3.3 ) can be shown even when the optimal trajectory can not be written down explicitly , which allows it to be applied in LTV systems with well-conditioned costs . Then , we utilize this perturbation bound to establish results on dynamic regret and the competitive ratio . In addition , we want to emphasize that the perturbation approach we develop is highly modular and extendable . For instance , if a stronger perturbation bound for some specific class of dynamics and/or cost functions can be shown , the dynamic regret of the predictive controller will improve . Similarly , to further generalize the problem setting ( e.g. , to include additional constraints ) , one only needs to establish the corresponding perturbation bounds and the regret result will follow . Another important component of the proof is a novel reduction between LTV control and online optimization . Connections between online optimization and control have received increasing attention in recent years , e.g. , [ 1 , 3 , 5 , 14 , 15 ] . Existing reductions rely on the canonical form , which does not apply to LTV systems , and/or formulations of online optimization with memory of multiple prior time steps , which makes the online problem more challenging . The reduction we present here relies on neither , and is thus a fundamentally different approach to connect control and online optimization . Further , this reduction is not specific to the predictive control algorithm we study , and we expect it to prove useful for other controllers in future work . A limitation of our reduction framework is that it can not handle state/control constraints . This limitation is shared by previous works [ 3 , 4 , 6 , 7 ] , and represents a challenging open question in the literature . 2 Background and Setting . We consider a finite-horizon discrete-time online control problem with linear time-varying ( LTV ) dynamics , time-varying costs , and disturbances , namely min G0 : ) , D0 : ) 1 ) ’ C=1 ( 5C ( GC ) + 2C ( DC 1 ) ) s.t . GC = C 1GC 1 + ⌫C 1DC 1 + FC 1 , C = 1 , . . . , ) , ( 1 ) G0 = G ( 0 ) , where GC 2 R= , DC 2 R < , and FC 2 R= respectively denote the state , the control action , and the disturbance of the system at time steps C = 1 , . . . , ) , and G ( 0 ) 2 R= is a given initial state . By convention , the hitting cost function 5C : R= ! R+ and control cost function 2C : R < ! R+ are assumed to be time-varying and well-conditioned . Define the tuple oC : = ( C , ⌫C , FC , 5C+1 , 2C+1 ) . In the classical setting where no predictions are available , after observing state GC at time step C , the algorithm needs to decide the control action DC before observing oC , which is an unknown random disturbance input . We use the following event sequence to describe this ordering : G0 , D0 , o0 , G1 , D1 , o1 , G2 , . . . , G ) 1 , D ) 1 , o ) 1 , G ) . We assume that the algorithm has access to the exact predictions of disturbances , cost functions and dynamical matrices in the future : time steps ( which are time-varying ) ; i.e. , the event sequence is G0 , o0 , o1 , . . . , o : 1 , D0 , o : , D1 , o : +1 , . . . , D ) : 1 , o ) 1 , D ) : , D ) : +1 , . . . , D ) 1 . Here we assume all predictions are exact , and leave the case of inexact predictions for future work . This prediction model has been used in previous works like [ 4 , 24–26 ] , and is available in many real-world applications such as disturbance estimation in robotics and frequency regulation in power grids . The availability is due to the fact that , in such scenarios as mentioned above , experiments or observations on the dynamics can be conducted repeatedly and consistently , which makes it feasible to train a good predictor based on the data collected from repeated trials . 2.1 Assumptions . As is standard in studies of regret and competitive ratio in linear control problems , we assume the cost functions are well-conditioned . Assumption 2.1 ( Well-conditioned Costs ) . The cost functions satisfy the following constraints : 1 . 5C ( · ) is < 5 -strongly convex for C = 1 , . . . , ) , and ✓ 5 -strongly smooth for C = 1 , . . . , ) 1 . 2 . 2C ( · ) is both < 2-strongly convex and ✓2-strongly smooth for C = 1 , . . . , ) . 3 . 5C ( · ) and 2C ( · ) are twice continuously differentiable for C = 1 , . . . , ) . 4 . 5C ( · ) and 2C ( · ) are non-negative , and 5C ( 0 ) = 2C ( 0 ) = 0 for C = 1 , . . . , ) . Note that assumptions ( 1 ) through ( 3 ) are quite common [ 3 , 5 , 14 , 25 , 27 ] . Assumption ( 4 ) is less common , but can be satisfied via re-parameterization without loss of generality . Specifically , when the minimizers of state cost 5C and control cost 2C are nonzero , we perform the transformation G 0 C GC arg min G 5C ( G ) , D0C DC arg min D 2C+1 ( D ) , F 0 C FC + C arg min G 5C ( G ) + ⌫C arg min D 2C+1 ( D ) . The intuition of this transformation is that , when the minimizer of the cost function for the next step is known , we can always perform a translation in the state and control space to align the minimizer with the origin . We refer the interested readers to Example 2.1 for a more intuitive explanation of the above transformation . Additionally , we need to assume the dynamics are controllable . It is crucial that the dynamical system can be steered from an arbitrary initial state to an arbitrary final state via a finite sequence of admissible control actions . For linear time-invariant ( LTI ) systems , the full-rankness of the controllability matrix completely characterizes the reachability of the state space , which is generally used as a standard assumption for analysis [ 7 , 28 , 29 ] . This can be generalized to parallel assumptions for LTV systems as follows . We begin with a definition . Definition 2.1 . For a dynamical system with linear time-varying dynamics GC = C 1GC 1+⌫C 1DC 1+ FC 1 , C = 1 , . . . , ) , the transition matrix ( C2 , C1 ) 2 R=⇥= ( from time step C1 to C2 ) is defined as ( C2 , C1 ) : = ⇢ C2 1 C2 2 · · · C1 if C2 > C1 if C2 C1 , and the controllability matrix `` ( C , ? ) 2 R=⇥ ( < ? ) is defined as `` ( C , ? ) : = ⇥ ( C + ? , C + 1 ) ⌫C , ( C + ? , C + 2 ) ⌫C+1 , . . . , ( C + ? , C + ? ) ⌫C+ ? ⇤ . The dynamical system is called controllable if there exists a constant 3 2 Z+ , such that the controllability matrix `` ( C , 3 ) is of full row rank for any C = 1 , . . . , ) 3 . The smallest constant 3 with such property is called the controllability index of the system . Given the above definition , we can state the key assumption necessary for the analysis of LTV systems . We use a slightly stronger assumption than being merely controllable , which we refer to as ( 3 , f ) -uniform controllability . It is a natural generalization of its counterpart for LTI systems ( see Assumption 2 in [ 28 ] , where ( 3 , f ) is instead named as ( ✓ , a ) ) . Assumption 2.2 . There exists positive constants 0 , 1 , and 10 , such that k C k 0 , k⌫C k 1 , and k⌫†C k 10 hold for all time steps C = 0 , . . . , ) 1 , where ⌫† C denotes the Moore–Penrose inverse of matrix ⌫C . Furthermore , there exists a positive constant f such that fmin ( `` ( C , 3 ) ) f holds for all time steps C = 0 , . . . , ) 3 , where 3 denotes the controllability index . Note that Assumption 2.2 implies fmin ( `` ( C , ? ) ) f for all ? 3 because appending more columns to a matrix with full row rank will not reduce its minimum singular value . The LTV setting we consider is more general than the settings which existing results on regret and competitive ratio have assumed [ 1 , 3 , 4 , 7 ] . We highlight the implications of this general setting for enabling applications in the following examples . Example 2.1 ( Trajectory tracking in LTV systems with well-conditioned costs ) . Consider a trajectory tracking problem with LTV dynamics and well-conditioned costs , which generalizes the standard linear quadratic tracking problem in [ 4 , 30 ] with LTI dynamics and quadratic costs . We adopt LTV dynamics GC+1 = CGC + ⌫CDC + FC and general well-conditioned cost functions 5C ( · ) , 2C ( · ) ( see Assumption 2.1 ) . With the desired trajectory 31 : ) , we consider a new state G̃C : = GC 3C and a new disturbance F̃C : = FC + C3C 3C+1 . Thus , using the new state and disturbance , the problem naturally fits into our problem setting with : future predictions of ( C , ⌫C , FC , 5C , 2C , 3C+1 ) . Note that predictive control with LTV dynamics is practical in nonlinear systems [ 13 , 31 ] because the nonlinearity could be well approximated by LTV models [ 31 ] . Example 2.2 ( Power grid frequency regulation ) . Consider the frequency regulation problem in [ 8 ] , where state G = [ \ > , l > ] > represent the status of a power plant , and power generation ? in 2 R= is the control action . The continuous-time dynamics is given by §\ §l | { z } §G ( C ) = 0 '' ( C ) 1 ! `` ( C ) 1⇡ | { z } ̂ ( C ) \ l | { z } G ( C ) + 0 '' ( C ) 1 | { z } ⌫̂ ( C ) ? in| { z } D ( C ) . Algorithm 1 Predictive Control ( % ⇠ : ) 1 : for C = 0 , 1 , . . . , ) : 1 do 2 : Observe current state GC and receive predictions oC : C+ : 1 . 3 : Solve and commit control actions DC : = k̃ : C ( GC , FC : C+ : 1 ; ) E0 . 4 : At time step C = ) : , observe current state GC and receive predictions oC : ) 1 . 5 : Solve and commit control actions DC : ) 1 : = k̃ : C ( GC , FC : ) 1 ; 0 ) E0 : : 1 . Here `` ( C ) denotes the rotational inertia matrix , which is time-varying and is determined by the proportion of renewable power in total power generation at time C , and can be accurately predicted in a certain time horizon [ 32 , 33 ] ; ! and ⇡ are known system parameters . Using standard discretization techniques , we can formulate a discrete-time linear time-varying system GC+1 = CGC + ⌫CDC + FC , where C and ⌫C are determined by ̂ ( C ) and ⌫̂ ( C ) . The cost functions are quadratic costs which penalizes frequency deviation [ 8 ] . This setting fits into our predictive control algorithm , since the controllers have accurate predictions of C and ⌫C in the near future due to predictablity of `` ( C ) .

Summary: This paper establishes both dynamic regret and competitive ratio bounds for online control with time-varying linear dynamics and adversarial perturbative noise, thereby extending past work for time-invariant noise-free dynamics. The analysis is facilitated by a general-purpose sensitive analysis for smoothed OCO.   Update: The reviewers adequately addressed my concerns, and I am ammending my review to an 8.