Online Control of Unknown Time-Varying Dynamical Systems

1 Introduction . The control of linear time-invariant ( LTI ) dynamical systems is well-studied and understood . This includes classical methods from optimal control such as LQR and LQG , as well as robust H∞ control . Recent advances study regret minimization and statistical complexity for online linear control , in both stochastic and adversarial perturbation models . Despite this progress , rigorous mathematical guarantees for nonlinear control remain elusive : nonlinear control is both statistically and computationally intractable in general . In the face of these limitations , recent research has begun to study the rich continuum of settings which lie between LTI systems and generic nonlinear ones . The hope is to provide efficient and robust algorithms to solve the most general control problems that are tractable , and at the same time , to characterize precisely at which degree of nonlinearity no further progress can be made . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . This paper studies the control of linear , time-varying ( LTV ) dynamical systems as one such point along this continuum . This is because the first-order Taylor approximation to the dynamics of any smooth nonlinear system about a given trajectory is an LTV system . These approximations are widely popular because they allow for efficient planning , as demonstrated by the success of iLQR and iLQG methods for nonlinear receding horizon control . We study online control of discrete-time LTV systems , with dynamics and time-varying costs xt+1 = Atxt +Btut + wt , ct ( xt , ut ) : ( x , u ) → R. ( 1.1 ) Above , xt is the state of the system , ut the control input , wt the disturbances , and At , Bt the system matrices . Our results extend naturally to partial-state observation , where the controller observes linear projections of the state yt = Ctxt . We focus on the challenges introduced when the system matrices At , Bt and perturbations wt are not known to the learner in advance , and can only be determined by live interaction with the changing systems . In this setting , we find that the overall change in system dynamics across time characterizes the difficulty of controlling the unknown LTV system . We define a measure , called system variability , which quantifies this . We show both statistical and computational lower bounds as well as algorithmic upper bounds in terms of the system variabilility . Surprisingly , system variability does not impede the complexity of control when the dynamics are known [ 16 ] . 1.1 Contributions . We consider the recently popularized nonstochastic model of online control , and study regret bounds with respect to common classes of policies : Disturbance Action ( DAC/SLS [ 44 ] ) , Disturbance Response ( DRC/Youla [ 46 ] ) , and linear feedback policies . Planning over the third class of feedback policies in LTI systems admits efficient convex relaxations via the the first two parametrizations , DAC and DRC . This insight has been the cornerstone of both robust [ 49 , 44 ] and online [ 3 , 39 ] control . Separation of parametrizations . For linear time-varying systems , however , we find that equivalences between linear feedback , DAC and DRC fail to hold : we show that there are cases where any one of the three parametrizations exhibits strictly better control performance than the other two . Regret against convex parametrizations . Our first set of results pertain to DAC and DRC parametrizations , which are convex and admit efficient optimization . We demonstrate that no algorithm can obtain sublinear regret with respect to these classes when faced with unknown , LTV dynamics unless a certain measure of system variability also scales sublinearly in the horizon . This is true even under full observation , controllable dynamics , and fixed control cost . This finding is in direct contrast to recent work which shows sublinear regret is attainable over LTV system dynamics if they are known [ 16 ] . We give an efficient algorithm that attains sublinear regret against these policy classes up to an additive penalty for the aforementioned system variability term found in our lower bound . When the system variability is sufficiently small , our algorithm recovers state-of-the-art results for unknown LTI system dynamics up to logarithmic factors . In fact , our algorithm enjoys sublinear adaptive regret [ 21 ] , a strictly stronger metric than standard regret which is more appropriate for time-varying systems . We also show that the stronger notion of adaptivity called strongly adaptive regret [ 11 ] is out of reach in the partial information setting . Regret against state feedback . Finally , we consider the class of state feedback policies , which are linear feedback with memory length one . We show that full-information optimization over state feedback policies is computationally hard . This suggests that obtaining sublinear regret relative to these policies may be computationally prohibitive , though does not entirely rule out the possibility of improper learning . However , improper learning can not be done via the DRC or DAC relaxations in light of our policy class separation results . Finally , we include an inefficient algorithm which attains sublinear ( albeit nonparametric-rate ) regret against state feedback control policies . Paper Structure Discussion of relevant literature and relation to our work can be found in Section 1.2 . In Section 2 , we formally introduce the setting of LTV nonstochastic control , the policy classes we study and our key result regarding their non-equivalence in the LTV setting ( Theorem 2.1 ) . Motivated by this non-equivalence , the remainder of the paper is split into the study of convex policies ( Section 3 ) and of state feedback policies ( Section 4 ) . In Section 3 , we show that regret against the DAC and DRC classes can not be sublinear unless the metric system variability ( Definition 3.1 ) itself is sublinear ( Theorem 3.1 ) , and also propose Algorithm 2 whose adaptive regret scales at the rate of our lower bound plus a T 2/3 term ( Theorem 3.4 ) . On the other hand , in Section 4 we show sublinear regret against state feedback policies is technically possible ( Theorem 4.1 ) with a computationally inefficient algorithm , but also provide a computational lower bound ( Theorem 4.2 ) for planning which reveals significant difficulties imposed by the LTV dynamics in this scenario as well . Finally , in Section 5 we pose several future directions , concerning both questions in LTV control , as well as the extension to nonlinear control . 1.2 Related Work . Our study of LTV systems is motivated by the widespread practical popularity of iterative linearization for nonlinear receding horizon control ; e.g. , the iLQR [ 40 ] , iLC [ 29 ] , and iLQG [ 41 ] algorithms . Recent research has further demonstrated that near-optimal solutions to LTV approximations of dynamics confer stability guarantees onto the original nonlinear system of interest [ 45 ] . Low-Regret Control : We study algorithms which enjoy sublinear regret for online control of LTV systems ; that is , whose performance tracks a given benchmark of policies up to a term which is vanishing relative to the problem horizon . [ 1 ] initiated the study of online control under the regret benchmark by introducing the online LQR problem : where a learner is faced with an unknown LTI system , fixed costs and i.i.d . Gaussian disturbances , and must attain performance relative to the LQR-optimal policy . Bounds for this setting were later improved and refined in [ 12 , 26 , 10 , 38 ] , and extended to partial-state observation in [ 25 , 24 ] . Our work instead adopts the nonstochastic control setting [ 3 ] , where the adversarially chosen ( i.e . non-Gaussian ) noise is considered to model the drift terms that arise in linearizations of nonlinear terms , and where costs may vary with time . [ 3 ] consider known system dynamics , later extended to unknown systems under both full-state [ 20 ] and partial-state observation [ 39 , 37 ] . The study of nonstochastic control of known LTV dynamics was taken up in [ 16 ] , with parallel work by [ 32 ] considering known LTV dynamics under stochastic noise . Unknown LTV dynamics : Our work is the first to consider online ( low-regret ) control of unknown LTV systems in any model . There is , however , a rich body of classical work on adaptive control of LTV systems [ 28 , 42 ] . These guarantees focus more heavily on error sensitivity and stability ; they only permit dynamical recovery up to error that scales linearly in system noise , and thus guarantee only ( vacuous ) linear-in-horizon regret . More recent work has studied identification ( but not online control ) of an important LTV class called switching systems [ 31 , 35 ] . Online Convex Optimization : We make extensive use of techniques from the field of online convex optimization [ 9 , 18 ] . Most relevant to our work is the literature on adapting to changing environments in online learning , which starts from the works of [ 22 , 6 ] . The notion of adaptive regret was introduced in [ 21 ] and significantly studied since as a metric for adaptive learning in OCO [ 2 , 47 ] . [ 11 ] proposed to strengthen adaptive regret and the stronger metric has been shown to imply results over dynamic regret [ 48 ] . Recent nonlinear control literature : Recent research has also studied provably guarantees in various complementary ( but incomparable ) models : planning regret in nonlinear control [ 4 ] , adaptive nonlinear control under linearly-parameterized uncertainty [ 5 ] , online model-based control with access to non-convex planning oracles [ 23 ] , and control with nonlinear observation models [ 27 , 13 ] . 2 Problem Setting . We study control of a linear time-varying ( LTV ) system Eq . ( 1.1 ) with state xt ∈ Rdx , control input ut ∈ Rdu chosen by the learner , and the external disturbance wt ∈ Rdx chosen by Nature . The system is characterized by time-varying matrices At ∈ Rdx×dx , Bt ∈ Rdx×du . For simplicity , the initial state is x1 = 0 . At each time t , oblivious1 adversary picks the system matrices ( At , Bt ) , disturbances wt and cost functions ct : Rdx × Rdu → R. The dynamics ( At , Bt ) are unknown to the learner : one observes only the next state xt+1 and current cost ct ( · , · ) after playing control ut . 1An oblivious adversary chooses the matrices , costs and perturbations prior to the control trajectory . Adaptive Regret . The goal of the learner is to minimize regret w.r.t . a policy class Π , i.e . the difference between the cumulative cost of the learner and the best policy π ? ∈ Π in hindsight . Formally , the regret of an algorithm A with control inputs u1 : T and corresponding states x1 : T , over an interval I = [ r , s ] ⊆ [ T ] , is defined as RegretI ( A ; Π ) = ∑ t∈I ct ( xt , ut ) − inf π∈Π ∑ t∈I ct ( x π t , u π t ) . ( 2.1 ) Here uπt , x π t indicate the control input and the corresponding state when following policy π . For a randomized algorithmA , we consider the expected regret . In this work , we focus on designing control algorithms that minimize adaptive regret , i.e . guarantee a low regret relative to the best-in-hindsight policy π ? I ∈ Π on any interval I ⊆ [ T ] . This performance metric of adaptive regret is more suitable for control over LTV dynamics given its agility to compete against different local optimal policies π ? I ∈ Π at different times [ 16 ] . Key objects . A central object in our study is the sequence of Nature ’ s x ’ s xnat1 : T that arises from playing zero control input ut = 0 at each t ∈ [ T ] , i.e . xnatt+1 = Atxnatt +wt [ 39 ] . define the following operators for all t , Φ [ 0 ] t = I , ∀h ∈ [ 1 , t ) , Φ [ h ] t = t−h+1∏ k=t Ak , ∀i ∈ [ 0 , t ) , G [ i ] t = Φ [ i ] t Bt−i , where the matrix product ∏r s with s ≥ r is taken in the indicated order k = s , . . . , r. the following identities give an alternative representation for the Nature ’ s x ’ s xnatt and state xt with control input ut in terms of the Markov operator at time t , Gt = [ G [ i ] t ] i≥0 : xnatt+1 = t−1∑ i=0 Φ [ i ] t wt−i , xt+1 = x nat t+1 + t−1∑ i=0 G [ i ] t ut−i . These operators and the alternative representation capture the dynamics by decoupling the disturbance and the control action effects . Assumptions . We make the three basic assumptions : we require from ( i ) the disturbances to not blow up the system with no control input , ( ii ) the system to have decaying effect over time , and ( iii ) the costs to be well-behaved and admit efficient optimization . Formally , these assumptions are : Assumption 1 . For all t ∈ [ T ] , assume ‖xnatt ‖ ≤ Rnat . Assumption 2 . Assume there exist RG ≥ 1 and ρ ∈ ( 0 , 1 ) s.t . for any h ≥ 0 and for all t ∈ [ T ] ∑ i≥h ‖G [ i ] t ‖op ≤ RG · ρh : = ψ ( h ) . Assumption 3 . Assume the costs ct : Rdx × Rdu → R are general convex functions that satisfy the conditions 0 ≤ ct ( x , u ) ≤ Lmax { 1 , ‖x‖2 + ‖u‖2 } , and ‖∇ct ( x , u ) ‖ ≤ Lmax { 1 , ‖x‖+ ‖u‖ } for some constant L > 0 , where ∇ denotes any subgradient [ 7 ] . The conditions in Assumption 3 allow for functions whose values and gradient grow as quickly as quadratics ( e.g . the costs in LQR ) , and the max { 1 , · } term ensures the inclusion of standard bounded and Lipschitz functions as well . Assumptions 1 and 2 arise from the assumption our LTV system is open-loop stable ; Appendix A.2 extends to the case where a nominal stabilizing controller is known , as in prior work [ 3 , 39 ] . While these two assumptions may seem unnatural at first , they can be derived from the basic conditions of disturbance norm bound and sequential stability . Lemma 2.1 . Suppose that there exist C1 ≥ 1 , ρ1 ∈ ( 0 , 1 ) such that ‖Φ [ h ] t ‖op ≤ C1ρh1 for any h ≥ 0 and all t ∈ [ T ] , and suppose that maxt ‖wt‖ ≤ Rw . Then , Assumption 1 holds withRnat = C11−ρ1Rw , and Assumption 2 holds with ρ = ρ1 and RG = max { 1 , maxt ‖Bt‖op · C11−ρ1 } . Note that Assumption 2 implies that ‖Gt‖ ` 1 , op = ∑ i≥0 ‖G [ i ] t ‖op ≤ RG . It also suggests that for a sufficiently large h the effect of iterations before t − h are negligible at round t. This prompts introducing a truncated Markov operator : denote Ḡht = [ G [ i ] t ] i < h to be the h-truncation of the true Markov operator Gt . It follows that their difference is ‖Ḡht − Gt‖ ` 1 , op = ∑ i≥h ‖G [ i ] t ‖op ≤ ψ ( h ) negligible in operator norm for a sufficiently large h. Define the bounded set of h-truncated Markov operators to be G ( h , RG ) = { G = [ G [ i ] ] 0≤i < h : ‖G‖ ` 1 , op ≤ RG } with Ḡht ∈ G ( h , RG ) for all t .

Summary: This paper studies LTV systems control, which is a very challenging problem in my opinion. A set of interesting results have been obtained, including the non-equivalence between different control classes, and algorithms with adaptive regret bounds against different controller classes. Computational complexity in achieving sublinear regret against state-feedback controller has also been discussed.   In general, I think this paper makes interesting contributions. However, I have some reservations on the notion of "adaptive regret". Further, I find some places in the presentation that need more discussions and clarifications.   I will be happy to revise the score after rebuttal if these points are clarified.