Online Control of Unknown Time-Varying Dynamical Systems

1 Introduction . The control of linear time-invariant ( LTI ) dynamical systems is well-studied and understood . This includes classical methods from optimal control such as LQR and LQG , as well as robust H∞ control . Recent advances study regret minimization and statistical complexity for online linear control , in both stochastic and adversarial perturbation models . Despite this progress , rigorous mathematical guarantees for nonlinear control remain elusive : nonlinear control is both statistically and computationally intractable in general . In the face of these limitations , recent research has begun to study the rich continuum of settings which lie between LTI systems and generic nonlinear ones . The hope is to provide efficient and robust algorithms to solve the most general control problems that are tractable , and at the same time , to characterize precisely at which degree of nonlinearity no further progress can be made . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . This paper studies the control of linear , time-varying ( LTV ) dynamical systems as one such point along this continuum . This is because the first-order Taylor approximation to the dynamics of any smooth nonlinear system about a given trajectory is an LTV system . These approximations are widely popular because they allow for efficient planning , as demonstrated by the success of iLQR and iLQG methods for nonlinear receding horizon control . We study online control of discrete-time LTV systems , with dynamics and time-varying costs xt+1 = Atxt +Btut + wt , ct ( xt , ut ) : ( x , u ) → R. ( 1.1 ) Above , xt is the state of the system , ut the control input , wt the disturbances , and At , Bt the system matrices . Our results extend naturally to partial-state observation , where the controller observes linear projections of the state yt = Ctxt . We focus on the challenges introduced when the system matrices At , Bt and perturbations wt are not known to the learner in advance , and can only be determined by live interaction with the changing systems . In this setting , we find that the overall change in system dynamics across time characterizes the difficulty of controlling the unknown LTV system . We define a measure , called system variability , which quantifies this . We show both statistical and computational lower bounds as well as algorithmic upper bounds in terms of the system variabilility . Surprisingly , system variability does not impede the complexity of control when the dynamics are known [ 16 ] . 1.1 Contributions . We consider the recently popularized nonstochastic model of online control , and study regret bounds with respect to common classes of policies : Disturbance Action ( DAC/SLS [ 44 ] ) , Disturbance Response ( DRC/Youla [ 46 ] ) , and linear feedback policies . Planning over the third class of feedback policies in LTI systems admits efficient convex relaxations via the the first two parametrizations , DAC and DRC . This insight has been the cornerstone of both robust [ 49 , 44 ] and online [ 3 , 39 ] control . Separation of parametrizations . For linear time-varying systems , however , we find that equivalences between linear feedback , DAC and DRC fail to hold : we show that there are cases where any one of the three parametrizations exhibits strictly better control performance than the other two . Regret against convex parametrizations . Our first set of results pertain to DAC and DRC parametrizations , which are convex and admit efficient optimization . We demonstrate that no algorithm can obtain sublinear regret with respect to these classes when faced with unknown , LTV dynamics unless a certain measure of system variability also scales sublinearly in the horizon . This is true even under full observation , controllable dynamics , and fixed control cost . This finding is in direct contrast to recent work which shows sublinear regret is attainable over LTV system dynamics if they are known [ 16 ] . We give an efficient algorithm that attains sublinear regret against these policy classes up to an additive penalty for the aforementioned system variability term found in our lower bound . When the system variability is sufficiently small , our algorithm recovers state-of-the-art results for unknown LTI system dynamics up to logarithmic factors . In fact , our algorithm enjoys sublinear adaptive regret [ 21 ] , a strictly stronger metric than standard regret which is more appropriate for time-varying systems . We also show that the stronger notion of adaptivity called strongly adaptive regret [ 11 ] is out of reach in the partial information setting . Regret against state feedback . Finally , we consider the class of state feedback policies , which are linear feedback with memory length one . We show that full-information optimization over state feedback policies is computationally hard . This suggests that obtaining sublinear regret relative to these policies may be computationally prohibitive , though does not entirely rule out the possibility of improper learning . However , improper learning can not be done via the DRC or DAC relaxations in light of our policy class separation results . Finally , we include an inefficient algorithm which attains sublinear ( albeit nonparametric-rate ) regret against state feedback control policies . Paper Structure Discussion of relevant literature and relation to our work can be found in Section 1.2 . In Section 2 , we formally introduce the setting of LTV nonstochastic control , the policy classes we study and our key result regarding their non-equivalence in the LTV setting ( Theorem 2.1 ) . Motivated by this non-equivalence , the remainder of the paper is split into the study of convex policies ( Section 3 ) and of state feedback policies ( Section 4 ) . In Section 3 , we show that regret against the DAC and DRC classes can not be sublinear unless the metric system variability ( Definition 3.1 ) itself is sublinear ( Theorem 3.1 ) , and also propose Algorithm 2 whose adaptive regret scales at the rate of our lower bound plus a T 2/3 term ( Theorem 3.4 ) . On the other hand , in Section 4 we show sublinear regret against state feedback policies is technically possible ( Theorem 4.1 ) with a computationally inefficient algorithm , but also provide a computational lower bound ( Theorem 4.2 ) for planning which reveals significant difficulties imposed by the LTV dynamics in this scenario as well . Finally , in Section 5 we pose several future directions , concerning both questions in LTV control , as well as the extension to nonlinear control . 1.2 Related Work . Our study of LTV systems is motivated by the widespread practical popularity of iterative linearization for nonlinear receding horizon control ; e.g. , the iLQR [ 40 ] , iLC [ 29 ] , and iLQG [ 41 ] algorithms . Recent research has further demonstrated that near-optimal solutions to LTV approximations of dynamics confer stability guarantees onto the original nonlinear system of interest [ 45 ] . Low-Regret Control : We study algorithms which enjoy sublinear regret for online control of LTV systems ; that is , whose performance tracks a given benchmark of policies up to a term which is vanishing relative to the problem horizon . [ 1 ] initiated the study of online control under the regret benchmark by introducing the online LQR problem : where a learner is faced with an unknown LTI system , fixed costs and i.i.d . Gaussian disturbances , and must attain performance relative to the LQR-optimal policy . Bounds for this setting were later improved and refined in [ 12 , 26 , 10 , 38 ] , and extended to partial-state observation in [ 25 , 24 ] . Our work instead adopts the nonstochastic control setting [ 3 ] , where the adversarially chosen ( i.e . non-Gaussian ) noise is considered to model the drift terms that arise in linearizations of nonlinear terms , and where costs may vary with time . [ 3 ] consider known system dynamics , later extended to unknown systems under both full-state [ 20 ] and partial-state observation [ 39 , 37 ] . The study of nonstochastic control of known LTV dynamics was taken up in [ 16 ] , with parallel work by [ 32 ] considering known LTV dynamics under stochastic noise . Unknown LTV dynamics : Our work is the first to consider online ( low-regret ) control of unknown LTV systems in any model . There is , however , a rich body of classical work on adaptive control of LTV systems [ 28 , 42 ] . These guarantees focus more heavily on error sensitivity and stability ; they only permit dynamical recovery up to error that scales linearly in system noise , and thus guarantee only ( vacuous ) linear-in-horizon regret . More recent work has studied identification ( but not online control ) of an important LTV class called switching systems [ 31 , 35 ] . Online Convex Optimization : We make extensive use of techniques from the field of online convex optimization [ 9 , 18 ] . Most relevant to our work is the literature on adapting to changing environments in online learning , which starts from the works of [ 22 , 6 ] . The notion of adaptive regret was introduced in [ 21 ] and significantly studied since as a metric for adaptive learning in OCO [ 2 , 47 ] . [ 11 ] proposed to strengthen adaptive regret and the stronger metric has been shown to imply results over dynamic regret [ 48 ] . Recent nonlinear control literature : Recent research has also studied provably guarantees in various complementary ( but incomparable ) models : planning regret in nonlinear control [ 4 ] , adaptive nonlinear control under linearly-parameterized uncertainty [ 5 ] , online model-based control with access to non-convex planning oracles [ 23 ] , and control with nonlinear observation models [ 27 , 13 ] . 2 Problem Setting . We study control of a linear time-varying ( LTV ) system Eq . ( 1.1 ) with state xt ∈ Rdx , control input ut ∈ Rdu chosen by the learner , and the external disturbance wt ∈ Rdx chosen by Nature . The system is characterized by time-varying matrices At ∈ Rdx×dx , Bt ∈ Rdx×du . For simplicity , the initial state is x1 = 0 . At each time t , oblivious1 adversary picks the system matrices ( At , Bt ) , disturbances wt and cost functions ct : Rdx × Rdu → R. The dynamics ( At , Bt ) are unknown to the learner : one observes only the next state xt+1 and current cost ct ( · , · ) after playing control ut . 1An oblivious adversary chooses the matrices , costs and perturbations prior to the control trajectory . Adaptive Regret . The goal of the learner is to minimize regret w.r.t . a policy class Π , i.e . the difference between the cumulative cost of the learner and the best policy π ? ∈ Π in hindsight . Formally , the regret of an algorithm A with control inputs u1 : T and corresponding states x1 : T , over an interval I = [ r , s ] ⊆ [ T ] , is defined as RegretI ( A ; Π ) = ∑ t∈I ct ( xt , ut ) − inf π∈Π ∑ t∈I ct ( x π t , u π t ) . ( 2.1 ) Here uπt , x π t indicate the control input and the corresponding state when following policy π . For a randomized algorithmA , we consider the expected regret . In this work , we focus on designing control algorithms that minimize adaptive regret , i.e . guarantee a low regret relative to the best-in-hindsight policy π ? I ∈ Π on any interval I ⊆ [ T ] . This performance metric of adaptive regret is more suitable for control over LTV dynamics given its agility to compete against different local optimal policies π ? I ∈ Π at different times [ 16 ] . Key objects . A central object in our study is the sequence of Nature ’ s x ’ s xnat1 : T that arises from playing zero control input ut = 0 at each t ∈ [ T ] , i.e . xnatt+1 = Atxnatt +wt [ 39 ] . define the following operators for all t , Φ [ 0 ] t = I , ∀h ∈ [ 1 , t ) , Φ [ h ] t = t−h+1∏ k=t Ak , ∀i ∈ [ 0 , t ) , G [ i ] t = Φ [ i ] t Bt−i , where the matrix product ∏r s with s ≥ r is taken in the indicated order k = s , . . . , r. the following identities give an alternative representation for the Nature ’ s x ’ s xnatt and state xt with control input ut in terms of the Markov operator at time t , Gt = [ G [ i ] t ] i≥0 : xnatt+1 = t−1∑ i=0 Φ [ i ] t wt−i , xt+1 = x nat t+1 + t−1∑ i=0 G [ i ] t ut−i . These operators and the alternative representation capture the dynamics by decoupling the disturbance and the control action effects . Assumptions . We make the three basic assumptions : we require from ( i ) the disturbances to not blow up the system with no control input , ( ii ) the system to have decaying effect over time , and ( iii ) the costs to be well-behaved and admit efficient optimization . Formally , these assumptions are : Assumption 1 . For all t ∈ [ T ] , assume ‖xnatt ‖ ≤ Rnat . Assumption 2 . Assume there exist RG ≥ 1 and ρ ∈ ( 0 , 1 ) s.t . for any h ≥ 0 and for all t ∈ [ T ] ∑ i≥h ‖G [ i ] t ‖op ≤ RG · ρh : = ψ ( h ) . Assumption 3 . Assume the costs ct : Rdx × Rdu → R are general convex functions that satisfy the conditions 0 ≤ ct ( x , u ) ≤ Lmax { 1 , ‖x‖2 + ‖u‖2 } , and ‖∇ct ( x , u ) ‖ ≤ Lmax { 1 , ‖x‖+ ‖u‖ } for some constant L > 0 , where ∇ denotes any subgradient [ 7 ] . The conditions in Assumption 3 allow for functions whose values and gradient grow as quickly as quadratics ( e.g . the costs in LQR ) , and the max { 1 , · } term ensures the inclusion of standard bounded and Lipschitz functions as well . Assumptions 1 and 2 arise from the assumption our LTV system is open-loop stable ; Appendix A.2 extends to the case where a nominal stabilizing controller is known , as in prior work [ 3 , 39 ] . While these two assumptions may seem unnatural at first , they can be derived from the basic conditions of disturbance norm bound and sequential stability . Lemma 2.1 . Suppose that there exist C1 ≥ 1 , ρ1 ∈ ( 0 , 1 ) such that ‖Φ [ h ] t ‖op ≤ C1ρh1 for any h ≥ 0 and all t ∈ [ T ] , and suppose that maxt ‖wt‖ ≤ Rw . Then , Assumption 1 holds withRnat = C11−ρ1Rw , and Assumption 2 holds with ρ = ρ1 and RG = max { 1 , maxt ‖Bt‖op · C11−ρ1 } . Note that Assumption 2 implies that ‖Gt‖ ` 1 , op = ∑ i≥0 ‖G [ i ] t ‖op ≤ RG . It also suggests that for a sufficiently large h the effect of iterations before t − h are negligible at round t. This prompts introducing a truncated Markov operator : denote Ḡht = [ G [ i ] t ] i < h to be the h-truncation of the true Markov operator Gt . It follows that their difference is ‖Ḡht − Gt‖ ` 1 , op = ∑ i≥h ‖G [ i ] t ‖op ≤ ψ ( h ) negligible in operator norm for a sufficiently large h. Define the bounded set of h-truncated Markov operators to be G ( h , RG ) = { G = [ G [ i ] ] 0≤i < h : ‖G‖ ` 1 , op ≤ RG } with Ḡht ∈ G ( h , RG ) for all t .

Summary: This paper studies online control in linear time-varying (LTV) systems where the dynamics are unknown. The performance metric is the adaptive regret against three common classes of policies. When the benchmark policy class is DRC/DAC, the authors show a lower bound of the adaptive regret of any online controllers that grows linearly with respect to the length of horizon. This negative result demonstrate that the unknown LTV setting is qualitatively harder than either unknown linear time invariant (LTI) or known LTV, where sublinear regrets are achievable. On the other hand, they also propose an online control algorithm whose adaptive regret matches the lower bound in the same setting.  When the benchmark policy class is static feedback controllers, they show that although sublinear regret is achievable under some restrictive assumptions, planning over this policy class is computational inefficient in general.