Title: Online Adaptive Policy Selection in Time-Varying Systems: No-Regret via Contractive Perturbations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Online policy selection

online control

online learning
Abstract: We study online adaptive policy selection in systems with time-varying costs and dynamics. We develop the Gradient-based Adaptive Policy Selection (GAPS) algorithm together with a general analytical framework for online policy selection via online optimization. Under our proposed notion of contractive policy classes, we show that GAPS approximates the behavior of an ideal online gradient descent algorithm on the policy parameters while requiring less information and computation. When convexity holds, our algorithm is the first to achieve optimal policy regret. When convexity does not hold, we provide the first local regret bound for online policy selection. Our numerical experiments show that GAPS can adapt to changing environments more quickly than existing benchmarks.

Online Adaptive Policy Selection in Time-Varying Systems: No-Regret via Contractive Perturbations

Abstract

We study online adaptive policy selection in systems with time-varying costs and dynamics. We develop the Gradient-based Adaptive Policy Selection (GAPS) algorithm together with a general analytical framework for online policy selection via online optimization. Under our proposed notion of contractive policy classes, we show that GAPS approximates the behavior of an ideal online gradient descent algorithm on the policy parameters while requiring less information and computation. When convexity holds, our algorithm is the first to achieve optimal policy regret. When convexity does not hold, we provide the first local regret bound for online policy selection. Our numerical experiments show that GAPS can adapt to changing environments more quickly than existing benchmarks.

1 Introduction

We study the problem of online adaptive policy selection for nonlinear time-varying discrete-time dynamical systems. The dynamics are given by x t +1 = g t ( x t , u t ) , where x t is the state and u t is the control input at time t . The policy class is a time-varying mapping π t from the state x t and a policy parameter θ t to a control input u t . At every time step t , the online policy incurs a stage cost c t = f t ( x t , u t ) that depends on the current state and control input. The goal of policy selection is to pick the parameter θ t online to minimize the total stage costs over a finite horizon T .

Online adaptive policy selection and general online control have received significant attention recently [1-8] because many control tasks require running the policy on a single trajectory, as opposed to restarting the episode to evaluate a different policy from the same initial state. Adaptivity is also important when the dynamics and cost functions are time-varying. For example, in robotics, timevarying dynamics arise when we control an aircraft under changing wind conditions [9].

In this paper, we are interested in developing a unified framework that can leverage a broad suite of theoretical results from online optimization and efficiently translate them to online policy selection, where efficiency includes both preserving the tightness of the guarantees and computational considerations. A central issue is that, in online policy selection, the stage cost c t depends on all previously selected parameters ( θ 0 , . . . , θ t -1 ) via the state x t . Many prior works along this direction have addressed this issue by finite-memory reductions. This approach led to the first regret bound on online policy selection, but the bounds are not tight, the computational cost can be large, and the dynamics and policy classes studied are restrictive [1, 3, 6-8].

Contributions. We propose and analyze the algorithm Gradient-based Adaptive Policy Selection (GAPS, Algorithm 1) to address three limitations of existing results on online policy selection. First, under the assumption that c t is a convex function of ( θ 0 , . . . , θ t ) , prior work left a log T regret gap between OCO and online policy selection. We close this gap by showing that GAPS achieves the optimal regret of O ( √ T ) (Theorem 3.3) 1 . Second, many previous approaches require oracle access to the dynamics/costs and expensive resimulation from imaginary previous states. In contrast, GAPS only requires partial derivatives of the dynamics and costs along the visited trajectory, and computes O (log T ) matrix multiplications at each step. Third, the application of existing regret analysis frameworks is limited to specific policy classes and systems because they require c t to be convex in ( θ 0 , . . . , θ t ) . We address this limitation by showing the first local regret bound for online policy selection when the convexity does not hold. Specifically, GAPS achieves the local regret of O ( √ (1 + V ) T ) , where V is a measure of how much ( g t , f t , π t ) changes over the entire horizon.

To derive these performance guarantees, we develop a novel proof framework based on a general exponentially decaying, or 'contractive', perturbation property (Definition 2.6) on the policy-induced closed-loop dynamics. This generalizes a key property of disturbance-action controllers [e.g. 1, 8] and includes other important policy classes such as model predictive control (MPC) [e.g. 11] and linear feedback controllers [e.g. 12]. Under this property, we prove an approximation error bound (Theorem 3.2), which shows that GAPS can mimic the update of an ideal online gradient descent (OGD) algorithm [13] that has oracle knowledge of how the current policy parameter θ t would have performed if used exclusively over the whole trajectory. This error bound bridges online policy selection and online optimization, which means regret guarantees on OGD for online optimization can be transferred to GAPS for online policy selection.

In numerical experiments, we demonstrate that GAPS can adapt faster than an existing follow-theleader-type baseline in MPC with imperfect disturbance predictions, and outperforms a strong optimal control baseline in a nonlinear system with non-i.i.d. disturbances. The source code for all experiments is published at https://www.github.com/jpreiss/adaptive_policy_selection .

Related Work. Our work is related to online control and adaptive-learning-based control [2, 14-19], especially online control with adversarial disturbances and regret guarantees [1, 6-8, 20, 21]. For example, there is a rich literature on policy regret bounds for time-invariant dynamics [1, 6, 8, 15, 20, 22]. There is also a growing interest in algorithms for time-varying systems with small adaptive regret [7, 21], dynamic regret [23-25], and competitive ratio [26-29]. Many prior works study a specific policy class called disturbance-action controller (DAC) [1, 3, 6-8]. When applied to linear dynamics g t with convex cost functions f t , DAC renders the stage cost c t a convex function in past policy parameters ( θ 0 , . . . , θ t ) . Our work contributes to the literature by proposing a general contractive perturbation property that includes DAC as a special case, and showing local regret bounds that do not require c t to be convex in ( θ 0 , . . . , θ t ) . A recent work also handles nonconvex c t , but it studies an episodic setting and requires c t to be 'nearly convex', which holds under its policy class [30].

In addition to online control, this work is also related to online learning/optimization [13, 31, 32], especially online optimization with memory and/or switching costs, where the cost at each time step depends on past decisions. Specifically, our online adaptive policy selection problem is related to online optimization with memory [29, 33-39]. Our analysis for GAPS provides insight on how to handle indefinite memory when the impact of a past decision decays exponentially with time.

Our contractive perturbation property and the analytical framework based on this property are closely related to prior works on discrete-time incremental stability and contraction theory in nonlinear systems [40-46], as well as works that leverage such properties to derive guarantees for (online) controllers [47-49]. In complicated systems, it may be hard to design policies that provably satisfy

x

0

x

1

x

2

u

0

u

1

θ

0

θ

1

c

0

c

1

c

t

x

t

x

t

+1

u

t

θ

t

π

t

g

t

f

t

these properties. This motivates some recent works to study neural-based approaches that can learn a controller together with its certificate for contraction properties simultaneously [50, 51]. Our work contributes to this field by showing that, when the system satisfies the contractive perturbation property, one can leverage this property to bridge online policy selection with online optimization.

Notation. We use [ t 1 : t 2 ] to denote the sequence ( t 1 , . . . , t 2 ) , a t 1 : t 2 to denote ( a t 1 , a t 1 +1 , . . . , a t 2 ) for t 1 ≤ t 2 , and a × τ for ( a, . . . , a ) with a repeated τ ≥ 0 times. We define q ( x, Q ) = x /latticetop Qx . Symbols 1 and 0 denote the all-one and all-zero vectors/matrices respectively, with dimension implied by context. The Euclidean ball with center 0 and radius R in R n is denoted by B n (0 , R ) . We let ‖ · ‖ denote the (induced) Euclidean norm for vectors (matrices). The diameter of a set Θ is diam ( Θ ) := sup x,y ∈ Θ ‖ x -y ‖ . The projection onto the set Θ is Π Θ ( x ) = arg min y ∈ Θ ‖ y -x ‖ .

2 Preliminaries

We consider online policy selection on a single trajectory. The setting is a discrete-time dynamical system with state x t ∈ R n for time index t ∈ T := [0 : T -1] . At time step t ∈ T , the policy picks a control action u t ∈ R m , and the next state and the incurred cost are given by:

respectively, where g t ( · , · ) is a time-varying dynamics function and f t ( · , · ) is a time-varying stage cost. The goal is to minimize the total cost ∑ T -1 t =0 c t .

We consider parameterized time-varying policies of the form of u t = π t ( x t , θ t ) , where x t is the current state at time step t and θ t ∈ Θ is the current policy parameter. Θ is a closed convex subset of R d . We assume the dynamics, cost, and policy functions { g t , f t , π t } t ∈ T are oblivious, meaning they are fixed before the game begins. The online policy selection algorithm optimizes the total cost by selecting θ t sequentially. We illustrate how the policy parameter sequence θ 0: T -1 affects the trajectory { x t , u t } t ∈ T and per-step costs c 0: T -1 in Figure 1. The online algorithm has access to the partial derivatives of the dynamics f t and cost g t along the visited trajectory , but does not have oracle access to the f t , g t for arbitrary states and actions.

We provide two motivating examples for our setting. Appendix H contains more details and a third example. The first example is MPC with confidence coefficients, a generalization of [38].

Example 2.1 (MPC with Confidence Coefficients) . Consider a linear time-varying (LTV) system g t ( x t , u t ) = A t x t + B t u t + w t , with time-varying costs f t ( x t , u t ) = q ( x t , Q t ) + q ( u t , R t ) . At time t , the policy observes { A t : t + k -1 , B t : t + k -1 , Q t : t + k -1 , R t : t + k -1 , w t : t + k -1 | t } , where w τ | t is a (noisy) prediction of the future disturbance w τ . Then, π t ( x t , θ t ) commits the first entry of

where θ t = ( λ [0] t , λ [1] t , . . . , λ [ k -1] t ) , Θ = [0 , 1] k and ˜ Q is a fixed positive-definite matrix. Intuitively, λ [ i ] t represents our level of confidence in the disturbance prediction i steps into the future at time step t , with entry 1 being fully confident and 0 being not confident at all.

The second example studies a nonlinear control model motivated by [12, 47].

Example 2.2 (Linear Feedback Control in Nonlinear Systems) . Consider a time-varying nonlinear control problem with dynamics g t ( x t , u t ) = Ax t + Bu t + δ t ( x t , u t ) and costs f t ( x t , u t ) = q ( x t , Q )+

q ( u t , R ) . Here, the nonlinear residual δ t comes from linearization and is assumed to be sufficiently small and Lipschitz. Inspired by [12], we construct an online policy based on the optimal controller u t = -¯ Kx t for the linear-quadratic regulator LQR( A,B,Q,R ) . Specifically, we let π t ( x t , θ t ) = -K ( θ t ) x t where K is a mapping from Θ to R n × m such that ∥ ∥ K ( θ t ) -¯ K ∥ ∥ is uniformly bounded.

2.1 Policy Class and Performance Metrics

In our setting, the state x t at time t is uniquely determined by the combination of 1) a state x τ at a previous time τ < t , and 2) the parameter sequence θ τ : t -1 . Similarly, the cost at time t is uniquely determined by x τ and θ τ : t . Since we use these properties often, we introduce the following notation.

Definition 2.3 (Multi-Step Dynamics and Cost) . The multi-step dynamics g t | τ between two time steps τ ≤ t specifies the state x t as a function of the previous state x τ and previous policy parameters θ τ : t -1 . It is defined recursively, with the base case g τ | τ ( x τ ) := x τ and the recursive case

in which z t := g t | τ ( x τ , θ τ : t -1 ) . 2 The multi-step cost f t | τ specifies the cost c t as function of x τ and θ τ : t . It is defined as f t | τ ( x τ , θ τ : t ) := f t ( z t , π t ( z t , θ t )) .

In this paper, we frequently compare the trajectory of our algorithm against the trajectory that would arise from applying a fixed parameter θ since time step 0 , which we denote as ˆ x t ( θ ) := g t | 0 ( x 0 , θ × t ) and ˆ u t ( θ ) := π t (ˆ x t ( θ ) , θ ) . A related concept that is heavily used is the surrogate cost F t , which maps a single policy parameter to a real number.

Definition 2.4 (Surrogate Cost) . The surrogate cost function is defined as F t ( θ ) := f t (ˆ x t ( θ ) , ˆ u t ( θ )) .

Figure 1 shows the overall causal structure, from which these concepts follow.

To measure the performance of an online algorithm, we adopt the objective of adaptive policy regret , which has been used by [7, 52]. It is a stronger benchmark than the static policy regret [1, 6] and is more suited to time-varying environments. We use { x t , u t , θ t } t ∈ T to denote the trajectory of the online algorithm throughout the paper. The adaptive policy regret R A ( T ) is defined as the maximum difference between the cost of the online policy and the cost of the optimal fixed-parameter policy over any sub-interval of the whole horizon T , i.e.,

In contrast, the (static) policy regret defined in [1, 6] restricts the time interval I to be the whole horizon T . Thus, a bound on adaptive regret is strictly stronger than the same bound on static regret. Adaptive regret is particularly useful in time-varying environments like Examples 2.1 and 2.2 because an online algorithm must adapt quickly to compete against a comparator policy parameter that can change indefinitely with every time interval [32, Section 10.2].

In the general case when surrogate costs F 0: T -1 are nonconvex, it is difficult (if not impossible) for online algorithms to achieve meaningful guarantees on classic regret metrics like R A ( T ) or static policy regret because they do not have oracle optimization solvers or even the exact knowledge of the surrogate costs. Therefore, we introduce the metric of local regret , which bounds the sum of squared gradient norms over the whole horizon:

Similar metrics have been adopted by previous works on online nonconvex optimization [53]. Intuitively, R L ( T ) measures how well the online agent chases the (changing) stationary point of the surrogate cost sequence F 0: T -1 . Since the surrogate cost functions are changing over time, the bound on R L ( T ) will depend on how much the system { g t , f t , π t } t ∈ T changes over the whole horizon T . We defer the details to Section 3.3.

2.2 Contractive Perturbation and Stability

In this section, we introduce two key properties needed for our sub-linear regret guarantees in adaptive online policy selection. We define both with respect to trajectories generated by 'slowly' time-varying parameters, which are easier to analyze than arbitrary parameter sequences.

Definition 2.5. We denote the set of policy parameter sequences with ε -constrained step size by

The first property we require is an exponentially decaying, or 'contractive', perturbation property of the closed-loop dynamics of the system with the policy class. We now formalize this property.

Definition 2.6 ( ε -Time-varying Contractive Perturbation) . The ε -time-varying contractive perturbation property holds for R C > 0 , C > 0 , ρ ∈ (0 , 1) , and ε ≥ 0 if, for any θ τ : t -1 ∈ S ε ( τ : t -1) , holds for arbitrary x τ , x ′ τ ∈ B n (0 , R C ) and time steps τ ≤ t .

Intuitively, ε -time-varying contractive perturbation requires two trajectories starting from different states (in a bounded ball) to converge towards each other if they adopt the same slowly time-varying policy parameter sequence. We call the special case of ε = 0 time-invariant contractive perturbation , meaning the policy parameter is fixed. Although it may be difficult to verify the time-varying property directly since it allows the policy parameters to change, we show in Lemma 2.8 that time-invariant contractive perturbation implies that the time-varying version also holds for some small ε > 0 .

The time-invariant contractive perturbation property is closely related to discrete-time incremental stability [e.g. 45] and contraction theory [e.g. 46], which have been studied in control theory. While some specific policies including DAC and MPC satisfy ε -time-varying contractive perturbation globally in linear systems, in other cases it is hard to verify. Our property is local and thus is easier to establish for broader applications in nonlinear systems (e.g., Example 2.2).

Besides contractive perturbation, another important property we need is the stability of the policy class, which requires π 0: T -1 can stabilize the system starting from the zero state as long as the policy parameter varies slowly. This property is stated formally below:

Definition 2.7 ( ε -Time-varying Stability) . The ε -time-varying stability property holds for R S > 0 and ε ≥ 0 if, for any θ τ : t -1 ∈ S ε ( τ : t -1) , ∥ ∥ g t | τ (0 , θ τ : t -1 ) ∥ ∥ ≤ R S holds for any time steps t ≥ τ .

Intuitively, ε -time-varying stability guarantees that the policy class π 0: T -1 can achieve stability if the policy parameters θ 0: T -1 vary slowly. 3 Similarly to contractive perturbation, one only needs to verify time-invariant stability (i.e., ε = 0 and the policy parameter is fixed) to claim time-varying stability holds for some strictly positive ε (see Lemma 2.8). The reason we still use the time-varying contractive perturbation and stability in our assumptions is that they hold for ε = + ∞ in some cases, including DAC and MPC with confidence coefficients. Applying Lemma 2.8 for those systems will lead to a small, overly pessimistic ε .

2.3 Key Assumptions

We make two assumptions about the online policy selection problem to achieve regret guarantees.

Assumption 2.1. The dynamics g 0: T -1 , policies π 0: T -1 , and costs f 0: T -1 are differentiable at every time step and satisfy that, for any convex compact sets X ⊆ R n , U ⊆ R m , one can find Lipschitzness/smoothness constants (can depend on X and U ) such that:

The dynamics g t ( x, u ) is ( L g,x , L g,u ) -Lipschitz and ( /lscript g,x , /lscript g,u ) -smooth in ( x, u ) on X × U .

The stage cost function f t ( x, u ) is ( L f , L f ) -Lipschitz and ( /lscript f,x , /lscript f,u ) -smooth in ( x, u ) on X × U .

The policy function π t ( x, θ ) is ( L π ,x , L π , θ ) -Lipschitz and ( /lscript π ,x , /lscript π , θ ) -smooth in ( x, θ ) on X × Θ .

Assumption 2.1 is general because we only require the Lipschitzness/smoothness of g t and f t to hold for bounded states/actions within X and U , where the coefficients may depend on X and U . Similar assumptions are common in the literature of online control/optimization [25, 29, 47].

Our second assumption is on the contractive perturbation and the stability of the closed-loop dynamics induced by a slowly time-varying policy parameter sequence.

Assumption 2.2. Let G denote the set of all possible dynamics/policy sequences { g t , π t } t ∈ T the environment/policy class may provide. For a fixed ε ∈ R ≥ 0 , the ε -time-varying contractive perturbation (Definition 2.6) holds with ( R C , C, ρ ) for any sequence in G . The ε -time-varying stability

(Definition 2.7) holds with R S < R C for any sequence in G . We assume that the initial state satisfies ‖ x 0 ‖ < ( R C -R S ) /C . Further, we assume that if { g, π } is the dynamics/policy at an intermediate time step of a sequence in G , then the time-invariant sequence { g, π } × T is also in G . 4

Note that Assumption 2.2 is on the joint properties of both the dynamical system and the policy class when composed together in a closed loop. The motivation is to generalize two key properties of linear systems under typical reasonable controllers: 1) the effect of past decisions on the current state decays exponentially fast, and 2) if the system is initialized near the origin, it remains near the origin. We generalize these properties via ε -time-varying contractive perturbation (Definition 2.6) and ε -time-varying stability (Definition 2.7) respectively. Although Assumption 2.2 may seem complicated to understand, it is less restrictive than the assumptions in the most closely related work (e.g., [1, 3, 7]) that focus on linear dynamics.

Compared to other settings where contractive perturbation holds globally [1, 8, 54], Assumption 2.2 only requires it to hold locally in a bounded ball B (0 , R C ) , which becomes important in nonlinear settings. This brings a new challenge because we need to guarantee that the starting state stays within B (0 , R C ) whenever we apply this property in the proof. Therefore, in Assumption 2.2, we assume R C > R S + C ‖ x 0 ‖ . Similarly, to leverage the Lipschitzness/smoothness property, we require X ⊇ B (0 , R x ) where R x ≥ C ( R S + C ‖ x 0 ‖ ) + R S and U = { π ( x, θ ) | x ∈ X , θ ∈ Θ , π ∈ G} . Since the coefficients in Assumption 2.1 depend on X and U , we will set X = B (0 , R x ) and R x = C ( R S + C ‖ x 0 ‖ ) + R S by default when presenting these constants. The goal is to ensure that the controller never leaves the region where contractive perturbation applies, which is critical for our analysis and again generalizes properties found in the literature (e.g., Examples 2.1, 2.2, and H.1).

For some systems, verifying Assumption 2.2 is straightforward (e.g., Example 2.1). In other cases, we can rely on the following lemma, which can convert a time-invariant version of the property to general time-varying one. We defer its proof to Appendix C.

Lemma 2.8. Suppose Assumption 2.2 holds for ε = 0 and ( R C , C, ρ , R S ) , which satisfies R C > ( C +1) R S . Suppose Assumption 2.1 also holds and let X := B (0 , R x ) , where R x = ( C +1) 2 R S . Then, Assumption 2.2 also holds for ˆ ε > 0 , ( ˆ R C , ˆ C, ˆ ρ , ˆ R S ) , and x 0 that satisfies ( ˆ R C -ˆ R S ) /C . Here, ˆ R S , ˆ R C , ˆ ρ are arbitrary constants that satisfies R S < ˆ R S < ˆ R C < R C / ( C +1) and ρ < ˆ ρ < 1 . The positive constants ˆ ε and ˆ C are given detailed expressions in Appendix C.

Remark 2.9. Lemma 2.8 can also be useful when applied to some parameterized controllers for time-invariant nonlinear systems. For example, the well-known 'computed torque control' feedback linearization controllers for robotic manipulators (see, e.g., [55]) renders the closed-loop dynamics exponentially stable about an equilibrium, and the feedback gains can be parameterized. Thus, it satisfies Assumption 2.2 in a neighborhood about the equilibrium, via Lemma 2.8. Even with time-invariant dynamics, the time-varying costs (such as tracking a trajectory determined online) provide a setting where selecting the policy parameters online can be beneficial.

3 Method and Theoretical Results

Our algorithm, Gradient-Based Adaptive Policy Selection (GAPS), is inspired by the classic online gradient descent (OGD) algorithm [32, 56], with a novel approach for approximating the gradient of the surrogate stage cost F t . In the context of online optimization, OGD works as follows. At each time t , the current stage cost describes how good the learner's current decision θ t is. The learner updates its decision by taking a gradient step with respect to this cost. Mapping this intuition to online policy selection, the ideal OGD update rule would be the following.

This is because the surrogate cost F t (Definition 2.4) characterizes how good θ t is for time t if we had applied θ t from the start, i.e., without the impact of other historical policy parameters θ 0: t -1 . However, since the complexity of computing ∇ F t exactly grows proportionally to t , the ideal OGD becomes intractable when the horizon T is large.

Definition 3.1 (Ideal OGD Update) . At time step t , update θ t +1 = ∏ Θ ( θ t -η ∇ F t ( θ t )) .

As outlined in Algorithm 1, GAPS uses G t to approximate ∇ F t ( θ t ) efficiently. To see this, we compare the decompositions, with key differences highlighted in colored text:

GAPS uses two techniques to efficiently approximate ∇ F t ( θ t ) . First, we replace the ideal sequence ( θ t ) × ( t +1) by the actual sequence θ 0: t . This enables computing gradients along the actual trajectory experienced by the online policy without re-simulating the trajectory under θ t . Second, we truncate the whole historical dependence to at most B steps . This bounds the memory used by GAPS. ε -time-varying contractive perturbation is the key to bound the bias of G t : Intuitively, in the first step, although θ τ becomes more different with θ t as τ decreases, its impact on f t | 0 decays more quickly (exponentially); In the second step, the terms that we discard are exponentially small with respect to B . We provide a rigorous bound of the bias in Theorem 3.2 and a proof outline in Appendix D.

Algorithm 1 Gradient-based Adaptive Policy Selection (GAPS)

Require: Learning rate η , buffer length B , initial θ 0 .

1: for t = 0 , . . . , T -1 do

2: Observe the current state x t .

3: Pick the control action u t = π t ( x t , θ t ) .

4: Incur the stage cost c t = f t ( x t , u t ) .

5:

Compute the approximated gradient:

G

t

=

6: Perform the update θ t +1 = ∏ Θ ( θ t -η G t ) . 7: end for

∑

{

B

b

-

=0

t

∂

f

∂θ

t

|

0

-

∣

∣

∣

∣

x

0

,

θ

0:

t

Algorithm 1 presents GAPS in its simplest form. Although the expression of the partial derivatives contains θ 0: t , the time- and space-efficient implementation of GAPS only requires to store B partial derivatives for B previous time steps. Details are given in Algorithm 2 in Appendix B.

Compared to many previous online control algorithms that take a reduction approach based on OCO with Memory, our algorithm can be much more computationally efficient (see Appendix I.4 for an empirical comparison). Specifically, these works [1, 3, 6] take a different finite-memory reduction approach toward reducing the online control problem to OCO with Memory [33] by completely removing the dependence on policy parameters before time step t -B for a fixed memory length B . In the finite-memory reduction, one must 'imaginarily' reset the state at time t -B to be 0 and then use the B -step truncated multi-step cost function f t | t -B ( 0 , θ t -B : t ) in the OGD with Memory algorithm [1]. When applied to our setting, this is equivalent to replacing G t in line 1 of Algorithm 1 by G ′ t = ∑ B -1 b =0 ∂ f t | t -B ∂θ t -b | 0 , ( θ t ) × ( B +1) . However, the estimator G ′ t has limitations compared with G t in GAPS. First, computing G ′ t requires oracle access to the partial derivatives of the dynamics and cost functions for arbitrary state and actions. Second, even if those are available, G ′ t is less computationally efficient than G t in GAPS, especially when the policy is expensive to execute. Taking MPC (Example 2.1) as an example, computing G ′ t at every time step requires solving B MPC optimization problems when re-simulating the system, where B = Ω (log T ) . In contrast, computing G t in GAPS only requires solving one MPC optimization problem and O ( B ) matrix multiplications to update the partial derivatives. One may wonder how significant this improvement is, and if it affects the regret. To address this concern, we compare GAPS with Ideal OGD and OGD with Memory in the setting of MPC with confidence coefficients for a 2D double integrator. The simulation results show that GAPS achieve very similar regret with the two benchmarks, while the improvement on computation efficiency is significant (see Appendix I.4).

3.1 Bounds on Truncation Error

We now present the first part of our main result, which states that the actual stage cost f t ( x t , u t ) incurred by GAPS is close to the ideal surrogate cost F t ( θ t ) , and the approximated gradient G t is close to the ideal gradient ∇ F t ( θ t ) . In other words, GAPS mimics the ideal OGD update (Definition 3.1).

b

min

1

,t

}

.

Theorem 3.2. Suppose Assumptions 2.1 and 2.2 hold. Let { ( x t , u t , θ t ) } t ∈ T denote the trajectory of GAPS (Algorithm 1) with buffer size and learning rate . Then, we have where Ω ( · ) and O ( · ) tion 2.1 and C in contractive perturbation - see details in Appendix D.1.

We defer the proof of Theorem 3.2 to Appendix D.1. Note that this result does not require any convexity assumptions on the surrogate cost F t .

3.2 Regret Bounds for GAPS: Convex Surrogate Cost

The second part of our main result studies the case when the surrogate cost F t is a convex function. This assumption is explicitly required or satisfied by the policy classes and dynamical systems in many prior works on online control and online policy selection [1, 3, 6, 54].

The error bounds in Theorem 3.2 can reduce the problem of GAPS' regret bound in control to the problem of OGD's regret bound in online optimization, where the following result is well known: When the surrogate cost functions F t are convex, the ideal OGD update (Definition 3.1) achieves the regret bound ∑ T -1 t =0 F t ( θ t ) -min θ ∈ Θ ∑ T -1 t =0 F t ( θ ) = O ( √ T ) , when the step size η is of the order 1 / √ T [32]. By taking the biases on the stage costs and the gradients into consideration, we derive the adaptive regret bound in Theorem 3.3. Besides the adaptive regret, one can use a similar reduction approach to 'transfer' other regret guarantees for OGD in online optimization to GAPS in control. We include the derivation of a dynamic regret bound as an example in Appendix E.

Theorem 3.3. Under the same assumptions as Theorem 3.2, if we additionally assume F t is convex for every time t and diam ( Θ ) is bounded by a constant D , then GAPS achieves adaptive regret where O ( · ) hides the same constants as in Theorem 3.2 and D -see details in Appendix D.1.

We discuss how to choose the learning rate and the regret it achieves in the following corollary. Corollary 3.4. Under the same assumptions as Theorem 3.3, suppose the horizon length T /greatermuch 1 1 -ρ and the buffer length B ≥ 1 2 log( T ) / log(1 / ρ ) . If we set η = (1 -ρ ) 5 2 T -1 2 , then GAPS achieves adaptive regret R A ( T ) = O ((1 -ρ ) -5 2 T 1 2 ) .

We defer the proof of Theorem 3.3 to Appendix D.1. Compared to the (static) policy regret bounds of [1, 3], our bound is tighter by a factor of log T . The key observation is that the impact of a past policy parameter θ t -b on the current stage cost c t decays exponentially with respect to b (see Appendix D for details). In comparison, the reduction-based approach first approximates c t with ˆ c t that depends on θ t -B +1: t , and then applies general OCO with memory results on ˆ c t [1, 3]. General OCO with memory cannot distinguish the different magnitudes of the contributions that θ t -B +1: t make to ˆ c t , which leads to the regret gap of B = O (log T ) .

In the more restrictive setting of linear time-invariant dynamics with the DAC policy class, the results of a concurrent work [10] can also be used to close the log T gap on static regret of online policy selection. In comparison, Theorem 3.3 considers more general time-varying dynamics and adopts the stronger metric of adaptive regret. As a practical matter, the follow-the-regularized-leader type of algorithm used by [10] is often (much) less computationally efficient than a gradient-based algorithm like GAPS. Nevertheless, [10] made distinct contributions by allowing the state space to be a general Banach space and providing a lower bound for OCO with unbounded memory.

3.3 Regret Bounds for GAPS: Nonconvex Surrogate Cost

The third part of our main result studies the case when the surrogate cost F t is nonconvex. Before presenting the result, we formally define the variation intensity that measures how much the system changes over the whole horizon.

Definition 3.5 (Variation Intensity) . Let { g t , π t , f t } t ∈ T be a sequence of dynamics/policy/cost functions that the environment provides. The variation intensity V of this sequence is defined as

Variation intensity is used as a measure of hardness for changing environments in the literature of online optimization that often appear in regret upper bounds (see [57] for an overview). Definition 3.5 generalizes one of the standard definitions to online policy selection. Using this definition, we present our main result for GAPS applied to nonconvex surrogate costs using the metric of local regret (3).

Theorem 3.6. Under the same assumptions as Theorem 3.2, if we additionally assume that Θ = R d for some integer d , then GAPS satisfies local regret

where O ( · ) hides the same constants as in Theorem 3.2 - see details in Appendix F.

We discuss how to choose the learning rate and the regret it achieves in the following corollary.

Corollary 3.7. Under the same assumptions as Theorem 3.6, suppose the horizon length T /greatermuch 1 1 -ρ and the buffer length B ≥ 1 2 log( T ) / log(1 / ρ ) . If we set η = (1 -ρ ) 3 2 (1+ V ) 1 2 T -1 2 , GAPS achieves local regret R L ( T ) = O ((1 -ρ ) -9 2 (1 + V ) 1 2 T 1 2 ) .

We defer the proof of Theorem 3.6 to Appendix F. Note that the local regret will be sublinear in T if the variation intensity V = o ( T ) . To derive the local regret guarantee in Theorem 3.6, we address additional challenges compared to the convex case. First, we derive a local regret guarantee for OGD in online nonconvex optimization. We cannot directly apply results from the literature because they do not use ordinary OGD, and it is difficult to apply algorithms like Follow-the-Perturbed-Leader [e.g. 58] to online policy selection due to constraints on information and step size. Then, to transfer the regret bound from online optimization to online policy selection, we show how to convert the measure of variation defined on F 0: T -1 to our variation intensity V defined on { g t , π t , f t } t ∈ T .

A limitation of Theorem 3.6 is that we need to assume Θ is a whole Euclidean space so that GAPS will not converge to a point at the boundary of Θ that is not a stationary point. Example 2.2 and Appendix H.3 show that one can re-parameterize the policy class to satisfy this assumption in some cases. Relaxing this assumption is our future work.

4 Numerical Experiments

In this section we compare GAPS to strong baseline algorithms in settings based on Examples 2.1 and 2.2. Details are deferred to Appendix I due to space limitations. Appendix I also includes a third experiment comparing GAPS to a bandit-based algorithm for selecting the planning horizon in MPC, and a computation time comparison between GAPS and the alternative gradient approximation of [1].

0

100

200

300

400

time

0

.

0

0

.

5

1

.

0

confidence

λ

0

100

200

300

400

time

0

2

4

6

cost

GAPS (ours)

λ

-confident

̂

w

becomes accurate

MPC confidence parameter. We compare GAPS to the follow-the-leader-type method of [11] for tuning a scalar confidence parameter in model-predictive control with noisy disturbance predictions. The setting is close to Example 2.1 but restricted to satisfy the conditions of the theoretical guarantees in [11]. We consider the scalar system x t +1 = 2 x t + u t + w t under non-stochastic disturbances w t with the cost f t ( x t , u t ) = x 2 t + u 2 t . For t = 0 to 100 , the predictions of w t are corrupted by a large amount of noise. After t > 100 , the prediction noise is instantly reduced by a factor of 100 . In this setup, an ideal algorithm should learn to decrease confidence level at first to account for the noise, but then increase to λ ≈ 1 when the predictions become accurate.

Figure 2 shows the values of the confidence parameter λ and the per-timestep cost generated by each algorithm. Both methods are initialized to λ = 1 . The method of [11] rapidly adjusts to an appropriate confidence level at first, while GAPS adjusts more slowly but eventually reaches the same value. However, when the accuracy changes, GAPS adapts more quickly and obtains lower costs towards the end of the simulation. In other words, we see that GAPS behaves essentially like an instance of Ideal OGD with constant step size, which is consistent with our theoretical results (Theorem 3.2).

0

200

400

600

800

1000

time

10

20

30

gain

disturbance = gaussian

0

200

400

600

800

1000

time

disturbance = walk

param

k

p

k

d

controller

ours

LQ

(a) Linear gains

k

p

, k

d

tuned by GAPS compared to LQR optimal.

0

200

400

600

800

1000

time

-

10

k

-

5

k

0

5

k

disturbance

gaussian

walk

(b) Cumulative cost difference.

Linear controller of nonlinear time-varying system. We apply GAPS to tune the gain parameters of a linear feedback controller in a nonlinear inverted pendulum system. Every 100 seconds, the pendulum mass changes. The system reflects the smooth nonlinear dynamics and nonconvex surrogate costs in Example 2.2, although it differs in other details (see Appendices H.3 and I.2). We compare GAPS to a strong and dynamic baseline that deploys the infinite-horizon linear-quadratic regulator (LQR) optimal controller for the linearized dynamics at each mass. We simulate two disturbances: 1) i.i.d. Gaussian, and 2) Ornstein-Uhlenbeck random walk.

Figure 3a shows the controller parameters tuned by GAPS, along with the baseline LQR-optimal gains, for each disturbance type. The derivative gain k d closely follows LQR for i.i.d. disturbances but diverges for random-walk disturbances, where LQR is no longer optimal. This is reflected in the cumulative cost difference between GAPS and LQR, shown in Figure 3b. GAPS nearly matches LQR under i.i.d. disturbances, but significantly outperforms it when the disturbance is a random walk. The results show that GAPS can both 1) adapt to step changes in dynamics on a single trajectory almost as quickly as the comparator that benefits from knowledge of the near-optimal analytic solution, and 2) outperform the comparator in more general settings where the analytic solution no longer applies.

5 Conclusion and Future Directions

In this paper, we study the problem of online adaptive policy selection under a general contractive perturbation property. We propose GAPS, which can be implemented more efficiently and with less information than existing algorithms. Under convexity assumptions, we show that GAPS achieves adaptive policy regret of O ( √ T ) , which closes the log T gap between online control and OCO left open by previous results. When convexity does not hold, we show that GAPS achieves local regret of O ( √ (1 + V ) T ) , where V is the variation intensity of the time-varying system. This is the first local regret bound on online policy selection attained without any convexity assumptions on the surrogate cost functions. Our numerical simulations demonstrate the effectiveness of GAPS, especially for fast adaptation in time-varying settings.

Our work motivates interesting future research directions. For example, a limitation is that GAPS assumes all policy parameters can stabilize the system and satisfy contractive perturbation. A recent work on online policy selection relaxed this assumption by a bandit-based algorithm but requires Θ to be a finite set [59]. An interesting future direction to study is what regret guarantees can be achieved when Θ is a continuous parameter set and not all of the candidate policies satisfy these assumptions.

References

Naman Agarwal, Brian Bullins, Elad Hazan, Sham Kakade, and Karan Singh. Online control with adversarial disturbances. In International Conference on Machine Learning , pages 111-119. PMLR, 2019.

Naman Agarwal, Elad Hazan, and Karan Singh. Logarithmic regret for online control. In Advances in Neural Information Processing Systems , volume 32, 2019.

Elad Hazan, Sham Kakade, and Karan Singh. The nonstochastic control problem. In Algorithmic Learning Theory , pages 408-421. PMLR, 2020.

Dimitar Ho, Hoang Le, John Doyle, and Yisong Yue. Online robust control of nonlinear systems with large uncertainty. In International Conference on Artificial Intelligence and Statistics , pages 3475-3483. PMLR, 2021.

Dheeraj Baby and Yu-Xiang Wang. Optimal dynamic regret in LQR control. In Advances in Neural Information Processing Systems , 2022.

Xinyi Chen and Elad Hazan. Black-box control for linear dynamical systems. In Conference on Learning Theory , pages 1114-1143. PMLR, 2021.

Paula Gradu, Elad Hazan, and Edgar Minasyan. Adaptive regret for control of time-varying dynamics. CoRR , abs/2007.04393, 2020.

Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non-stochastic control. In Conference on Learning Theory , pages 3320-3436. PMLR, 2020.

Michael O'Connell, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics , 7(66):eabm6597, 2022.

Raunak Kumar, Sarah Dean, and Robert Kleinberg. Online convex optimization with unbounded memory. arXiv preprint arXiv:2210.09903 , 2022.

Tongxin Li, Ruixiao Yang, Guannan Qu, Guanya Shi, Chenkai Yu, Adam Wierman, and Steven Low. Robustness and consistency in linear quadratic control with predictions. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 6:1-35, 2021.

Guannan Qu, Chenkai Yu, Steven Low, and Adam Wierman. Exploiting linear models for model-free nonlinear control: A provably convergent policy gradient approach. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 6539-6546. IEEE, 2021.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In International Conference on Machine Learning , pages 928-936, 2003.

Yingying Li, Subhro Das, and Na Li. Online optimal control with affine constraints. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 8527-8537, 2021.

Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Information theoretic regret bounds for online nonlinear control. In Advances in Neural Information Processing Systems , volume 33, pages 15312-15325, 2020.

Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. Regret bounds for robust adaptive control of the linear quadratic regulator. In Advances in Neural Information Processing Systems , volume 31, 2018.

Alon Cohen, Tomer Koren, and Yishay Mansour. Learning linear-quadratic regulators efficiently with only √ t regret. In International Conference on Machine Learning , pages 1300-1309. PMLR, 2019.

Yingying Li, Subhro Das, Jeff S. Shamma, and Na Li. Safe adaptive learning-based control for constrained linear quadratic regulators with regret guarantees. CoRR , abs/2111.00411, 2021.

Jing Yu, Dimitar Ho, and Adam Wierman. Online stabilization of unknown networked systems with communication constraints. arXiv preprint arXiv:2203.02630 , 2022.

Dylan Foster and Max Simchowitz. Logarithmic regret for adversarial online control. In International Conference on Machine Learning , volume 119, pages 3211-3221. PMLR, 2020.

Edgar Minasyan, Paula Gradu, Max Simchowitz, and Elad Hazan. Online control of unknown time-varying dynamical systems. In Advances in Neural Information Processing Systems , volume 34, pages 15934-15945, 2021.

Kim Peter Wabersich and Melanie N. Zeilinger. Performance and safety of bayesian model predictive control: Scalable model-based RL with guarantees. CoRR , abs/2006.03483, 2020.

Chenkai Yu, Guanya Shi, Soon-Jo Chung, Yisong Yue, and Adam Wierman. The power of predictions in online control. In Advances in Neural Information Processing Systems , volume 33, pages 1994-2004, 2020.

Yingying Li, Xin Chen, and Na Li. Online optimal control with linear dynamics and predictions: Algorithms and regret analysis. In Advances in Neural Information Processing Systems , volume 32, 2019.

Yiheng Lin, Yang Hu, Guanya Shi, Haoyuan Sun, Guannan Qu, and Adam Wierman. Perturbation-based regret analysis of predictive control in linear time varying systems. In Advances in Neural Information Processing Systems , volume 34, 2021.

Oron Sabag, Sahin Lale, and Babak Hassibi. Optimal competitive-ratio control. CoRR , abs/2206.01782, 2022.

Chenkai Yu, Guanya Shi, Soon-Jo Chung, Yisong Yue, and Adam Wierman. Competitive control with delayed imperfect information. In 2022 American Control Conference (ACC) , pages 2604-2610. IEEE, 2022.

Gautam Goel and Babak Hassibi. Competitive control. CoRR , abs/2107.13657, 2021.

Guanya Shi, Yiheng Lin, Soon-Jo Chung, Yisong Yue, and Adam Wierman. Online optimization with memory and competitive control. In Advances in Neural Information Processing Systems , volume 33, pages 20636-20647, 2020.

Xinyi Chen, Edgar Minasyan, Jason D Lee, and Elad Hazan. Regret guarantees for online deep control. Proceedings of Machine Learning Research vol XX , 1:34, 2023.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing , 32(1):48-77, 2002.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Oren Anava, Elad Hazan, and Shie Mannor. Online learning for adversaries with memory: price of past mistakes. In Advances in Neural Information Processing Systems , volume 28, 2015.

Chlo´ e Rouyer, Yevgeny Seldin, and Nicol` o Cesa-Bianchi. An algorithm for stochastic and adversarial bandits with switching costs. In International Conference on Machine Learning , pages 9127-9135. PMLR, 2021.

Gautam Goel, Yiheng Lin, Haoyuan Sun, and Adam Wierman. Beyond online balanced descent: An optimal algorithm for smoothed online optimization. In Advances in Neural Information Processing Systems , volume 32, 2019.

Yiheng Lin, Gautam Goel, and Adam Wierman. Online optimization with predictions and non-convex losses. Proceedings of the ACM on Measurement and Analysis of Computing Systems , 4(1):1-32, 2020.

Niangjun Chen, Gautam Goel, and Adam Wierman. Smoothed online convex optimization in high dimensions via online balanced descent. In Conference On Learning Theory , pages 1574-1594. PMLR, 2018.

Yingying Li, Guannan Qu, and Na Li. Online optimization with predictions and switching costs: Fast algorithms and the fundamental limit. IEEE Transactions on Automatic Control , 66(10):4761-4768, 2020.

Raman Arora, Ofer Dekel, and Ambuj Tewari. Online bandit learning against an adaptive adversary: from regret to policy regret. In International Conference on Machine Learning , pages 1747-1754, 2012.

Nicholas M Boffi, Stephen Tu, and Jean-Jacques E Slotine. Regret bounds for adaptive nonlinear control. In Learning for Dynamics and Control , pages 471-483. PMLR, 2021.

Guanya Shi, Kamyar Azizzadenesheli, Michael O'Connell, Soon-Jo Chung, and Yisong Yue. Meta-adaptive nonlinear control: Theory and algorithms. In Advances in Neural Information Processing Systems , volume 34, pages 10013-10025, 2021.

Winfried Lohmiller and Jean-Jacques E Slotine. On contraction analysis for non-linear systems. Automatica , 34(6):683-696, 1998.

David Angeli. A Lyapunov approach to incremental stability properties. IEEE Transactions on Automatic Control , 47(3):410-421, 2002.

Bj¨ orn S R¨ uffer, Nathan Van De Wouw, and Markus Mueller. Convergent systems vs. incremental stability. Systems & Control Letters , 62(3):277-285, 2013.

Florian Bayer, Mathias B¨ urger, and Frank Allg¨ ower. Discrete-time incremental iss: A framework for robust nmpc. In 2013 European Control Conference (ECC) , pages 2068-2073. IEEE, 2013.

Hiroyasu Tsukamoto, Soon-Jo Chung, and Jean-Jaques E Slotine. Contraction theory for nonlinear stability analysis and learning-based control: A tutorial overview. Annual Reviews in Control , 52:135-169, 2021.

Tongxin Li, Ruixiao Yang, Guannan Qu, Yiheng Lin, Adam Wierman, and Steven H. Low. Certifying black-box policies with stability for nonlinear control. IEEE Open Journal of Control Systems , 2:49-62, 2023.

Hiroyasu Tsukamoto, Soon-Jo Chung, and Jean-Jacques Slotine. Learning-based adaptive control using contraction theory. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 2533-2538. IEEE, 2021.

Stephen Tu, Alexander Robey, Tingnan Zhang, and Nikolai Matni. On the sample complexity of stability constrained imitation learning. In Learning for Dynamics and Control Conference , pages 180-191. PMLR, 2022.

Charles Dawson, Sicun Gao, and Chuchu Fan. Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and control. IEEE Transactions on Robotics , pages 1-19, 2023.

Dawei Sun, Susmit Jha, and Chuchu Fan. Learning certified control using contraction metric. In Conference on Robot Learning , pages 1519-1539. PMLR, 2021.

Elad Hazan and Comandur Seshadhri. Adaptive algorithms for online decision problems. In Electronic Colloquium on Computational Complexity (ECCC) , volume 14, 2007.

Elad Hazan, Karan Singh, and Cyril Zhang. Efficient regret minimization in non-convex games. In International Conference on Machine Learning , pages 1433-1441. PMLR, 2017.

Runyu Zhang, Yingying Li, and Na Li. On the regret analysis of online LQR control with predictions. In 2021 American Control Conference (ACC) , pages 697-703. IEEE, 2021.

Jean-Jacques E Slotine, Weiping Li, et al. Applied nonlinear control , volume 199. Prentice hall Englewood Cliffs, NJ, 1991.

Nikhil Bansal and Anupam Gupta. Potential-function proofs for gradient methods. Theory of Computing , 15(1):1-32, 2019.

Aryan Mokhtari, Shahin Shahrampour, Ali Jadbabaie, and Alejandro Ribeiro. Online optimization in dynamic environments: Improved regret rates for strongly convex problems. In 2016 IEEE 55th Conference on Decision and Control (CDC) , pages 7195-7201. IEEE, 2016.

Arun Sai Suggala and Praneeth Netrapalli. Online non-convex learning: Following the perturbed leader is optimal. In Algorithmic Learning Theory , pages 845-861. PMLR, 2020.

Yingying Li, James A Preiss, Na Li, Yiheng Lin, Adam Wierman, and Jeff S Shamma. Online switching control with stability and regret guarantees. In Learning for Dynamics and Control Conference , pages 1138-1151. PMLR, 2023.

Rolf Schneider. Convex Bodies: the Brunn-Minkowski Theory . Cambridge University Press, 2014.

Hongyu Zhou, Zirui Xu, and Vasileios Tzoumas. Efficient online learning with memory via Frank-Wolfe optimization: Algorithms with bounded dynamic regret and applications to control. CoRR , abs/2301.00497, 2023.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization , 23(4):2341-2368, 2013.

Ofer Dekel, Jian Ding, Tomer Koren, and Yuval Peres. Bandits with switching costs: T 2 / 3 regret. In ACM Symposium on the Theory of Computing , pages 459-467, 2014.

Yiheng Lin, Yang Hu, Guannan Qu, Tongxin Li, and Adam Wierman. Bounded-regret MPC via perturbation analysis: Prediction error, constraints, and nonlinearity. In Advances in Neural Information Processing Systems , 2022.

Peer reviews:
comments: SUMMARY:
The paper studies online adaptive policy selection for nonlinear time-varying discrete-time dynamical systems. The algorithm named GAPS is proposed and is shown to achieve optimal regret, which closes the regret gap between online convex optimization (OCO) and online policy selection. En route, a general proof framework based on exponentially decaying perturbation property is developed that connects online policy selection with OCO. Numerical experiments are provided to demonstrate GAPS's superior performance over baselines.

STRENGTHS:
The paper is well-written. The problem and the results seem significant.

WEAKNESSES:
A slight weakness is that the algorithm needs $\Omega(\log T)$ memory instead of a constant complexity to $T$. I did not spot any major weakness.

QUESTIONS:
I hope the authors can provide clarification for two questions:

1.  Can the authors comment on the necessity of assuming Definition 2.6 and Definition 2.7 for achieving optimal regret?

2. Does the variational intensity or a similar quantity also appear in the analysis of Theorem 3.3 (the convex setting)? What is $V$ in that setting?

LIMITATIONS:
Limitations are discussed in Section 3 and the conclusion. In addition to the ones mentioned in the paper, the algorithm also needs the knowledge of problem parameters $\rho$ and $\epsilon$ (or $\rho$ and $V$) to set the learning rate and run optimally.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This paper proposes an algorithm, GAPS, for online adaptive policy selection in time-varying systems. The algorithm is shown to achieve optimal $O(\sqrt{T})$ regret based on the contractive perturbation property of the online policy-induced dynamics. Numerical results are provided to verify the performance of GAPS.

STRENGTHS:
1. The paper is well-written and organized. The insights behind the main results are effectively presented.
2. The optimal $O(\sqrt{T})$ regret can be achieved using partial derivatives of the dynamics and costs.

WEAKNESSES:
1. The results require a slow change of the policy parameter sequences and an $\epsilon$-time varying contractive perturbation, which cannot handle sudden changes.
2. There is an additional assumption on the initial state.

QUESTIONS:
1. Is it possible to relax the projection step in the algorithm?
2. It would be interesting to explore the introduction of switching costs in this problem.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies online adaptive policy selection for nonlinear time-varying discrete-time dynamical systems. At time step $t \\in\\mathcal{T}$, the policy picks a control action $u_t$, and the next state and the incurred cost are given by $x_{t+1}=g_t\\left(x_t, u_t\\right), c_t:=f_t\\left(x_t, u_t\\right)$, where $g_t(\\cdot, \\cdot)$ is a time-varying dynamics function and $f_t(\\cdot, \\cdot)$ is a time-varying stage cost. The goal is to minimize the total cost $\\sum_{t=0}^{T-1} c_t$. The regret definition in (2) is similar to [52]( https://arxiv.org/pdf/1708.00075.pdf). They require two key properties to achieve sub-linear regret in this time varying systems. Definition 2.6 requires that Intuitively, two trajectories starting from different states (in a bounded ball) to converge towards each other if they adopt the same slowly time-varying policy parameter sequence, and definition 2.7 requires that the policy class $\\pi_{0:T-1}$ can achieve stability if the policy parameters $\\theta_{0:T-1}$ vary slowly. Assumption 2.1 is standard (see [52]( https://arxiv.org/pdf/1708.00075.pdf)) while assumption 2.2 ensures starting state stays within an euclidean ball whenever the dynamics changes. Since the complexity of computing $\\nabla F_t$ exactly grows proportionally to $t$, the key difference in their approach is that their algorithm GAPS uses $G_t$ to approximate $\\nabla F_t\\left(\\theta_t\\right)$ over a batch size of $B$. This results in solving only one MPC optimization problem. Finally in their main theorem 3.6 they show that a regret of $R^L(T)=O\\left((1-\\rho)^{-\\frac{9}{2}}(1+V)^{\\frac{1}{2}} T^{\\frac{1}{2}}\\right)$ is possible without any convexity assumption on $F_t$, where $V$ is the variation intensity of the time-varying system. Finally they empirically validate their algorithm.\n\nSTRENGTHS:\n1) The paper analyzes Online Gradient Descent (OGD) algorithm for time varying systems. They propose the GAPS algorithm which uses the approximate gradient $G_t$ to estimate surrogate function $F_t$. This is also computationally faster than previous methids.\n\n2) They theoretically analyze their algorithm and provide a regret bound of $R^L(T)=O\\left((1-\\rho)^{-\\frac{9}{2}}(1+V)^{\\frac{1}{2}} T^{\\frac{1}{2}}\\right)$ is possible without any convexity assumption on $F_t$, where $V$ is the variation intensity of the time-varying system. This improves over the previous bounds in this setting by a $log(T)$ factor.\n\n3) They show empirically that their algorithm is competitive.\n\nWEAKNESSES:\n1) While assumption 2.1 is standard, I think assumption 2.2 is very strong. The implication of the $R_C>R_S+C\\left\\|x_0\\right\\|$ in assumption 2.2 is not clear to me. Moreover $\\mathcal{G}$ is the set of all possible dynamics/policy sequences ${g_t, \\pi_t\\}_{t \\in \\mathcal{T}}$ the environment/policy class may provide and you assume that if $\\{g, \\pi\\}$ is the dynamics/policy at an intermediate time step of a sequence in $\\mathcal{G}$, then the time-invariant sequence $\\{g, \\pi\\} \\times T$ is also in $\\mathcal{G}$. This seems to be very strong assumption. Where do you use it? Are there other works also that require this assumption or is this specific for the time varying system to provide stability?\n\n2) The key novelty in their method lies in using $G_t$ instead of $F_t$ and substituting the ideal sequence by the actual sequence $\\theta_{0: t}$. However, doesn't this approach might introduce additional variance in your estimation of the gradient? How do you control for that? Similarly when you truncate your observation to $B$ timesteps rather than the ideal sequence there must be approximation error creeping into your estimation of $F_t$ through $G_t$. How do you account for that? Also it will be great if you can point out where in the theory you deal with these issues.\n\n3) It is not clear to me how the regret improvement occurs in Theorem 3.3 and Theorem 3.6 that results in a regret of $O(\\sqrt{T})$ (and improves by a factor of $\\log(T)$. The paper has limited discussions on how this happens, and I would like the authors to discuss/clarify this in more details. It will be also great if the authors specifically point out where in their proof the use the assumption 2.1 and 2.2 ti get the improvement. Also why [52]( https://arxiv.org/pdf/1708.00075.pdf) fails to achieve this bound.\n\n4) It makes sense to me that the quantity $V$ occurs in the time-varying system which is similar to the quantity in [Besbes et al.](https://arxiv.org/pdf/1307.5449.pdf). However, it is defined on $f,g$, and policy $\\pi$. Shouldn't this $V$ only depend on the environment dynamics and cost $f,g$? Can you please elaborate on this? Also please point out how it comes up in the proof of Theroem 3.6.\n\nQUESTIONS:\nSee weakness section.\n\nLIMITATIONS:\n1) The writing can be improved. I think you some of the definitions and assumptions can be moved to the section 3. Also the authors need to discuss the results more. It is not clear to me exactly what technical novelty over [52]( https://arxiv.org/pdf/1708.00075.pdf) led to the regret bound of $O(\\sqrt{T})$ which does not include the $\\log T$ factor.\n\n2) The two examples in the main paper seems to be slightly contrived (and i did not see the appendix). Can the authors give moire real life examples where their approach can be used?\n\n3) See weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-18"}

comments: SUMMARY:
The paper studies online adaptive policy selection for nonlinear systems. The algorithm proposed by the authors, GAPS, is a gradient-based algorithm that achieves the first optimal regret bound in the convex case, and the first local regret bound in the case when convexity does not hold. The authors provided numerical experiments.

STRENGTHS:
The novel approach to the online control problem is interesting and closes the $\log T$ gap between the currently established bounds for online nonstochastic control and OGD. The paper is well-organized.

WEAKNESSES:
1. The experiments do not compare the algorithm proposed by the paper to the existing algorithms in online nonstochastic control. It would be interesting to see the comparison against benchmarks in the online control literature including GPC in Algorithm 1, https://arxiv.org/pdf/2211.09619.pdf. 

2. Although the analysis is novel, the algorithm proposed is essentially OGD with approximated gradient. The idea of truncation is also very similar to the gradient-based existing algorithms in online control like GPC. 

3. To compute the gradient estimator in GAPS, do we need access to all $\theta_t$'s, requiring storing all $\theta_t$?

QUESTIONS:
1. Can the authors provide more justifications of Definition 2.7 ($\epsilon$-time-varying stability)? How does this assumption compare with the standard assumptions made in existing literature? 

2. One of the main contributions of the paper is that it closes the $\log T$ gap between OCO-M based control algorithms and the OGD regret guarantee. However, there are also OCO-M based algorithm that achieves $O(\sqrt{T})$ bound such as in https://arxiv.org/pdf/2210.09903.pdf. Can the authors compare GAPS to this work?

LIMITATIONS:
Not applicable.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-21

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new algorithm, Gradient-based Adaptive Policy Selection (GAPS), for online adaptive policy selection with time-varying dynamics and costs. For analysis, it proposes a general analytical framework for online policy selection. Under this framework, by restricting the problem and policy class to have the contractive perturbation property it identified, GAPS is shown to approximate an ideal online gradient descent algorithm. This results in better regret bounds compared to existing results. When convexity holds, GAPS is the first to achieve optimal regret in this setting; when convexity doesn’t hold, it gives the first local regret bound for online policy selection. Empirical results on two examples in the main text also illustrate GAPS’s better adaptivity to changing environments compared to baselines.\n\nSTRENGTHS:\n1. The results seem to be significant. First, when assuming convexity, the regret bound of the proposed algorithm improves over existing work and fills a gap in the literature. It is the first to achieve the optimal regret of $O(\\sqrt{(T)})$ under the discussed setting while requiring less information about the problem. Second, when the cost function is nonconvex, it gives the first local regret bound for online policy selection.\n2. The proposed contractive perturbation property and its corresponding analytical framework are general and subsume an existing class (DAC) as well as some known downstream applications. They may be helpful for future research.\n3. The paper is well-written, and the presentation of empirical results is clear, which is convincing.\n\nWEAKNESSES:\nOverall, the paper seems strong to me, and I have a minor suggestion: If possible, it would be nice to have the result of online gradient descent (OGD) oracle in the numerical experiments. This may help the reader gain more understanding about GAPS’s performance.\n\nQUESTIONS:\n1. Though the proposed contractive perturbation property includes some existing works as special cases, I wonder how likely or hard it is to find a more general property.\n\nTypos\n1. In Line 216, it seems that some part is missing after “satisfies”.\n\nLIMITATIONS:\nAs the paper discussed in the Conclusion and Future Directions section, the major limitation of this work may be that the assumptions on the contractive perturbation property and stability are quite strong. It requires the properties should hold for all policy parameters. But still, this work is quite complete, and relaxing the assumptions can be interesting future directions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

comments: SUMMARY:
The paper studies online adaptive policy selection in systems with time-varying costs and dynamics. 

This paper proposes an algorithm that obtains optimal regret bound in the convex case and a local regret bound in the non-convex setting with four assumptions: (1) the dynamics are contractive starting from a ball near 0 if the policy has small variations across time. (2) the dynamics starting from 0 never goes out of an even smaller ball if the policy has small variations across time. (3) the dynamics will start from a point that would never go out of the ball in assumption (1). (4) smoothness and Lipschitzness for the dynamics, policy function, and cost functions.

The algorithm does not require Oracle access to the dynamics.

STRENGTHS:
The problems under investigation are interesting.

WEAKNESSES:
It seems that the restriction on the dynamics is quite severe.

QUESTIONS:
Could you give some examples of dynamics where the assumptions hold? I am giving a low score mainly because I am not very sure that I understand the dynamics of interest. If there are good examples which show that the assumptions are not as severe as vacuous, then I will change my score.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel algorithm, Gradient-based Adaptive Policy Selection (GAPS), for online policy selection in time-varying systems. The authors introduce a general analytical framework for online policy selection via online optimization. The paper also provides theoretical guarantees for the performance of the GAPS algorithm under some assumptions on the stability of the dynamical system and on the convexity of the surrogate cost function. Complementary local bounds are also give in the case of nonconvex cost.\n\nSTRENGTHS:\nThe paper is well-organized and easy to follow, with clear explanations of the theoretical concepts and practical implementation details. The authors provide detailed proofs of their theoretical results in the appendices, as well as numerical experiments to document the performance of the GAPS algorithm in two concrete example settings. \n\nOn the math side, while I did not check the details of the proof, I find the underlying perturbative idea new and interesting, and on a high level the steps of the proof check out.\n\nWEAKNESSES:\nThe main weakness of the paper is, given its novelty, that it is hard for the reader to understand how restrictive the set of assumptions that are put on the dynamical systrem. This holds for both assumptions of Theorem 3.3: Convexity of $F$ and $\\epsilon$-time varying contractive perturbation/stability. It is good that the authors give examples of systems where these assumptions hold, and describe in Lemma 2.8 how the time-invariant stability can be translated to these conditions, but I still find it a bit hard to understand how restrictive these conditions are. For instance, I imagine that in the case of a multistable dynamical system the contractive perturbation property would not hold?\n\nQUESTIONS:\none main question listed in the weaknesses.\n\nLIMITATIONS:\nSome of the limitations havebeen listed in the conclusions, but perhaps a more extended discussion about the applicability of the assumptions would be informative.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-02"}