Title: Optimal Rates for Bandit Nonstochastic Control
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bandit control

online learning
Abstract: Linear Quadratic Regulator (LQR) and Linear Quadratic Gaussian (LQG) control are foundational and extensively researched problems in optimal control. We investigate LQR and LQG problems with semi-adversarial perturbations and time-varying adversarial bandit loss functions. The best-known sublinear regret algorithm~\cite{gradu2020non} has a $T^{\frac{3}{4}}$ time horizon dependence, and its authors posed an open question about whether a tight rate of $\sqrt{T}$ could be achieved. We answer in the affirmative, giving an algorithm for bandit LQR and LQG which attains optimal regret, up to logarithmic factors. A central component of our method is a new scheme for bandit convex optimization with memory, which is of independent interest.

Optimal Rates for Bandit Nonstochastic Control

Abstract

Linear Quadratic Regulator (LQR) and Linear Quadratic Gaussian (LQG) control are foundational and extensively researched problems in optimal control. We investigate LQR and LQG problems with semi-adversarial perturbations and timevarying adversarial bandit loss functions. The best-known sublinear regret algorithm of Gradu et al. [2020] has a T 3 4 time horizon dependence, and the authors posed an open question about whether a tight rate of √ T could be achieved. We answer in the affirmative, giving an algorithm for bandit LQR and LQG which attains optimal regret (up to logarithmic factors) for both known and unknown systems. A central component of our method is a new scheme for bandit convex optimization with memory, which is of independent interest.

1 Introduction

Linear-Quadratic Regulator (LQR) and the more general Linear-Gaussian (LQG) control problems have been extensively studied in the field of control theory due to their wide range of applications and admittance of analytical solutions by the seminal works of Bellman [1954] and Kalman [1960]. LQR and LQG control problems study the design of a feedback control policy for a linear dynamical system with the goal of minimizing cumulative, possibly time-varying quadratic costs. The discrete version of the problem studies the control of the following linear dynamical system governed by dynamics ( A,B,C ) 1 :

where at time t , x t represents the system's state, u t represents the control exerted on the system, and { w t } T t =1 represents a sequence of i.i.d. centered Gaussian perturbations injected to the system. For a simple example of the model, consider the problem of controlling an elevator to reach the desired floors while minimizing the discomfort (due to acceleration or deceleration) and the energy consumption. In this case, the state x t would be a vector consisting of the elevator's position and velocity at time t . The control input u t is the force applied by the elevator motor at time t . w t represents process noise at time t , caused possibly by unexpected mechanical disturbances. In the generality of LQG, the system's states are not accessible. In the elevator example, one may not have the access to the measurement of the elevator's velocity or the precise position of the elevator. Instead, the algorithm has access to an observation y t , which is a linear transformation of state perturbed by a sequence { e t } T t =1 of i.i.d. centered Gaussian noises, i.e. the elevator's observed position. The cost is

a quadratic function of both the observation and the control exerted. The goal in LQR/LQG problems is to find a control policy π in some policy class Π that minimizes the cumulative cost over a finite time horizon T . With y π t , u π t denoting the observation and control at time t resulted from executing policy π , the objective is formally given by

where Q t , R t are some possibly time-varying state-weighting and control-weighting matrices, respectively. In the elevator example, if our goal is to control the elevator to reach the desired floors while minimizing the discomfort and the energy consumption, the cost function at every time step t may take the form c t ( y t , u t ) = c 1 ‖ y t -z goal ‖ 2 2 + c 2 ‖ u t ‖ 2 2 , where z goal is the target position of the elevator, c 1 , c 2 > 0 .

Variations of the LQR/LQG problem have garnered considerable interest in the machine learning community. In the recent literature of online nonstochastic control , several setting-based generalizations to the aforementioned linear-control framework have been explored, including:

Adversarial cost functions (Cohen et al. [2018], Agarwal et al. [2019a]): In this setting, the cost functions are adversarially chosen in an unpredictable way to the learner. This generalized formulation allows for applications such as building climate control in the presence of time-varying energy costs, as the energy costs may depend on unexpected demand fluctuation (Cohen et al. [2018]). More recently, the model of adversarially chosen cost functions, along with adversarial perturbations in the subsequent paragraph, give rise to building connection between control theory and meta optimization (Chen and Hazan [2023]), where the nonconvex problem of hyperparameter tuning (e.g. learning rate tuning) can be reduced to online nonstochastic control, which often admits a convex formulation.

Adversarial perturbations (Agarwal et al. [2019a], etc.): In this setting, differing from the usual Gaussian assumption on the perturbation distribution ( w t i.i.d. ∼ N (0 , Σ ) ), nonstochastic control often considers arbitrary disturbances w t in the dynamics. The relaxed assumption on the perturbations allows for the control of nonlinear systems under model inaccuracies by approximating them as linear systems (Ghai et al. [2022]).

Bandit feedback : Together with the aforementioned two generalizations, control under bandit feedback further generalizes the control problem and suits many real-world applications. In the bandit setting, the cost functions is unknown to the learner. The learner has access only to the cost incurred by their chosen control. In addition, the learner has access to the system's state ( x t in the fully observed LQR setting) or a noisy, possibly and usually low-dimensional observation ( y t in the partially observed LQG setting). The bandit LQG problem can be illustrated with a motivating example. In the problem of controlling medical device such as a ventilator, the state usually represents a number of health status indicators of the patient, such as heart rate, blood pressure, and other related metrics. However, this state may not be fully observable. After the treatment is provided, the learner can observe the patient's response, e.g. changes in some vital indicators, with some observation noise and a cost function, e.g. deviation from the targeted breathing pattern. In this limited information setting, can a learner hope to learn meaning controls under minimal assumptions? Previously, the work of Gradu et al. [2020] and Cassel and Koren [2020] both studied the bandit nonstochastic control problem of fully observed systems and provided provable guarantees.

Taken together, these settings give rise to a general setting in differentiable reinforcement learning that strictly contains a variety of classical problems in optimal and robust control. Naturally, when adversarial costs and perturbations are considered, an optimal solution is not defined a priori. Instead, the primary performance metric is regret : the difference between the total cost of a control algorithm and that of the best controller from a specific policy class in hindsight.

As mentioned above, this general setting of bandit nonstochastic control was considered in the recent work of Gradu et al. [2020], whose proposed Bandit Perturbation Controller ( BPC ) algorithm has a provable regret guarantee of ˜ O ( T 3 4 ) when compared to a rich policy class called disturbance action controllers (DAC) for fully observed systems. Similar setting has also been studied by Cassel and Koren [2020], who established an optimal regret up to logarithmic factor of ˜ O ( √ T ) for fully

observable systems under stochastic perturbations and adversarially chosen cost functions. Recently Ghai et al. [2023] improves the dimension dependence of regret bounds using exploration in actionspace rather than parameter-space. However, bandit control for partially observable systems (e.g. LQG) is less understood. Thus, these developments in the search for efficient, low-regret bandit online control algorithms leave a central open question (also stated by by Gradu et al. [2020]):

Can we achieve optimal regret O ( √ T ) with bandit LQG and nonstochastic noise ?

Our work answers this question up to logarithmic factors. Our novel Ellipsoidal Bandit Perturbation Controller ( EBPC ) achieves a ˜ O ( √ T ) regret guarantee in the presence of semi-adversarial perturbations in bandit LQG problems with strongly convex cost functions, with the additional generality of possibly unknown system dynamics. By Shamir [2013], this is asymptotically optimal up to logarithmic factors, as bandit optimization over quadratics reduces to bandit control with quadratic losses under A = 0 , B = I . Our work therefore resolves the upper-bound/lower-bound gap for this generalization of LQR/LQG. The following table gives a comprehensive comparison between the regret guarantee of EBPC and existing results in literature.

1.1 Related work

Online Nonstochastic Control and Online LQR. In the last decade, much research has been devoted to the intersection of learning and control. Abbasi-Yadkori and Szepesvári [2011] and Ibrahimi et al. [2012] considered the problem of learning a controller in LQR for known quadratic cost functions and stochastic/martingale difference perturbation sequence when the dynamics of the system is unknown, and achieved ˜ O ( √ T ) -regret in this case. Dean et al. [2018] provided the first provable low-regret, efficient algorithm to solve LQR problems with known cost functions and stochastic perturbations. Cohen et al. [2018] extended this result to changing quadratic costs with stochastic perturbations and provided a regret guarantee of O ( √ T ) . Lale et al. [2021] consider the LQG problem with stochastic noise and unknown systems.

More recently, interest has turned to nonstochastic control , in which the cost functions and the perturbations can be adversarially chosen Agarwal et al. [2019a]. A broad spectrum of control problems were reconsidered from the nonstochastic perspective, and several different generalizations were derived. See Hazan and Singh [2022] for a comprehensive text detailing these results.

Online Bandit Convex Optimization with Memory. Aclassical approach to nonstochastic control of stable/stabilizable linear dynamical systems is to reduce control problems to online convex optimization with bounded memory. This reduction stems from the observation that the effects of past controls decay exponentially in stable/stabilizable systems. In the bandit online convex optimization with bounded memory setting, the learner iteratively plays a decision x t in a convex set K ⊆ R d and suffers an adversarially chosen loss F t ( x t -H +1: t ) , where x t -H +1: t is the sequence of points x t -H +1 , ..., x t . In particular, the loss depends on the last H points played by the algorithm, and the only information revealed to the learner is the scalar loss that they incurred. The goal is to minimize

regret , the difference between the loss actually suffered and the loss suffered under the best single play in hindsight:

Since the loss function is unknown to the learner in the bandit setting, online bandit convex optimization algorithms make use of low-bias estimators of the true gradient or Hessian. Therefore, it is standard to measure the algorithm's performance by regret over the expectation of losses, where the expectation is taken over the randomness injected when creating such estimators.

Online convex optimization with memory in the full information setting, where the loss function is known to the learner, was proposed by Anava et al. [2015]. The work of Agarwal et al. [2019a], was the first to connect this to control, and to give a regret bound for online control with adversarial perturbations.

The ˜ O ( T 3 4 ) regret bound of Gradu et al. [2020] leveraged the bandit convex optimization method of Flaxman et al. [2005]. We focus on the bandit LQR/LQG setting with strongly convex and smooth loss functions. It is thus natural to use the techniques of Hazan and Levy [2014], who obtained a ˜ O ( √ T ) regret guarantee for bandit convex optimization without memory.

Online Learning with Delay. One technical difficulty in extending OCO with memory to the bandit setting arises from the requirement of independence between every play and the noises injected from the recent H steps. We resolve this issue by adapting online learning with delay to the subroutine algorithm used in our BCO algorithm. Online learning with delay was introduced by Quanrud and Khashabi [2015]. In particular, Flaspohler et al. [2021] relates online learning with delay to online learning with optimism and established a sublinear regret guarantee for mirror descent algorithms. A similar delay scheme was seen in [Gradu et al., 2020].

1.2 Notations and organization

Notation. For convenience and succinctness, we denote ¯ H def = H -1 . We use lowercase bold letters (e.g. x , y , u , w , e ) to denote the states, observations, controls, and noises of the dynamical system 2 , and d x , d u , d y to denote their corresponding dimensions. We use S n -1 to denote the unit sphere in R n , as S n -1 ∼ = R n -1 . R + , R ++ denote all non-negative, positive real numbers, respectively. For a differentiable function F : ( R n ) H → R , we denote the gradient of F with respect to its i th argument vector by ∇ i F ( · ) . ρ ( · ) acting on a square matrix measures the spectral radius of the matrix. For a sequence M = ( M [ i ] ) i ∈ I , we use ‖ M ‖ /lscript 1 , op to denote the sum of the operator norm:

‖ M ‖ /lscript 1 , op def = ∑ i ∈ I ‖ M [ i ] ‖ op . σ ( A ) denotes the σ -algebra generated by a random variable A . We use O ( · ) to hide all universal constants, ˜ O ( · ) to hide poly(log T ) terms, and O ( · ) to hide all natural parameters.

Organization. Our method has two main components: a novel algorithm for BCO with memory ( EBCO-M ), and its application to building a novel bandit perturbation controller ( EBPC ). Section 2 describes our problem setting. Section 3 gives EBCO-M and its near-optimal regret guarantee. Section 4 introduces EBPC and its regret guarantees to both known and unknown systems.

2 The Bandit LQG Problem

In this section we provide necessary background and describe and formalize the main problem of interest. We consider control of linear time-invariant dynamical systems of the form

with dynamics matrices A ∈ R d x × d x , B ∈ R d x × d u , C ∈ R d y × d x . Here, consistent with previous notations, x t ∈ R d x is the state of the system at time t , u t ∈ R d u is the control applied at time t , and

w t ∈ R d x , e t ∈ R d y are the system and measurement perturbations. At each timestep, the learner may observe y t ∈ R d y , which usually represents a possibly noisy projection of the state x t onto some (possibly and usually) low-dimensional space.

In the online bandit setting, the learner is asked to perform a control u t at time t . After the control is performed, the adversary chooses a quadratic cost function c t ( y t , u t ) . The learner observes the scalar c t ( y t , u t ) ∈ R + and the signal y t , but no additional information about c t ( · , · ) , x t , w t , or e t is given. The goal of the learner is to minimize expected regret , where regret against the controller class Π is defined as

where u π t is the control exerted at time t by policy π , and y π t is the timet observation that would have occurred against the same costs/noises if the control policy π were carried out from the beginning.

In controlling linear dynamical systems with partial observations, we often make use of the system's counterfactual signal had no controls been performed since the beginning of the instance:

Definition 2.1 (Nature's y ) . Nature's y at time t , denoted by y nat t , is the signal that the system would have generated at time t under u 1: t = 0 . We may compute this as

Critically, this may be calculated via the Markov operator:

Definition 2.2 (Markov operator) . The Markov operator G = [ G [ i ] ] i ∈ N corresponding to a linear system parametrized by ( A,B,C ) as in Eq.(2.1) is a sequence of matrices in R d y × d u such that G [ i ] def = CA i -1 B, G [0] def = 0 d y × d u .

It follows immediately that y nat t may be computed from observations as y nat t = y t -∑ t i =1 G [ i ] u t -i .

2.1 Assumptions

We impose four core assumptions on the problem:

Assumption 2.3 (Stable system) . We assume the system is stable: the spectral radius ρ ( A ) < 1 3 .

This assumption has the following important consequence:

Remark 2.4 (Decay of stable systems) . That the system is stable implies that ∃ P /follows 0 d x × d x , P ∈ Sym( d x ) such that rP /followsequal A /latticetop PA for some 0 ≤ r < 1 , and therefore ∃ κ depending on ‖ B ‖ op , ‖ C ‖ op , σ min ( P ) such that ‖ G [ i ] ‖ op ≤ κ r i -1 . Then with H = O (log T ) , we can assume that ‖ G ‖ /lscript 1 , op = ∑ ∞ i =0 ‖ G [ i ] ‖ op ≤ R G and ψ G ( H ) def = ∑ ∞ i = H ‖ G [ i ] ‖ op ≤ R G T .

Together, Assumption 2.3 and its consequence Remark 2.4 allows modeling the nonstochastic control problem as an online convex optimization problem against adversary with bounded memory, as the effect of past controls decays exponentially. Assumption 2.5-2.7 introduce the assumptions on the adversarial cost functions and perturbations.

Assumption 2.5 (Noise model) . The perturbations { w t , e t } T t =1 are assumed to be semi-adversarial: w t , e t decompose as sums of adversarial and stochastic components w t = w adv t + w stoch t and e t = e adv t + e stoch t . The stochastic components of the perturbations are assumed to come from distributions satisfying E [ w stoch t ] = E [ e stoch t ] = 0 , E [ w stoch t w stoch t /latticetop ] /followsequal σ 2 w I , E [ e stoch t e stoch t /latticetop ] /followsequal σ 2 e I , σ e > 0 . { w t , e t } T t =1 are bounded such that ‖ y nat t ‖ 2 ≤ R nat , ∀ t , for some parameter R nat .

The bound on y nat is implied by bounded noise, which is a standard assumption in literature, and the stability of the system. The semi-adversarial assumption is also seen in prior work [Simchowitz et al., 2020], and is a necessary condition for our analysis: we depend on the regret guarantee of a bandit online convex optimization with memory algorithm which requires the strong convexity of the expected loss functions conditioned on all but the Θ (poly(log T )) most recent steps of history. This assumption is essentially equivalent to the adversarial assumption in applications: in almost all systems, noise is either endemic or may be injected. We also emphasize that this assumption is much weaker than that of previous optimal-rate work: even in the known-state, known-dynamic case, the previous optimal guarantee in the bandit setting depended on no adversarial perturbation (see Table 1).

Assumption 2.6 (Cost model) . The cost functions c t ( · , · ) are assumed to be quadratic, σ c -strongly convex, β c -smooth, i.e. c t ( y , u ) = y /latticetop Q t y + u /latticetop R t u with β c I /followsequal Q t /followsequal σ c I, β c I /followsequal R t /followsequal σ c I ∀ t . They are also assumed to obey the following Lipschitz condition: ∀ ( y , u ) , ( y ′ , u ′ ) ∈ R d y + d u ,

These conditions are relatively standard for bandit convex optimization algorithms, and are needed for the novel BCO-with-memory algorithm which underpins our control algorithm.

Assumption 2.7 (Adversary) . { c t ( · , · ) , w adv t , e adv t } T t =1 is chosen by the adversary ahead of time.

2.2 Disturbance Response Controllers

Regret compares the excess cost from executing our proposed control algorithm with respect to the cost of the best algorithm in hindsight from a given policy class. In particular, low regret against a rich policy class is a very strong near-optimality guarantee. We take the comparator policy class Π to be the set of disturbance response controllers (DRC), formally given by the following definition.

Definition 2.8 (Disturbance Response Controllers) . The disturbance response controller (DRC) and the DRC policy class are defined as:

A disturbance response controller (DRC) is a policy π M parametrized by M = ( M [ j ] ) ¯ H j =0 , where H ∈ Z ++ , a sequence of H matrices in R d u × d y s.t. the control at time t given by π M is u π M t = ∑ ¯ H j =0 M [ j ] y nat t -j . We shorthand u t M def = u π M t .

Previous works have demonstrated the richness of the DRC policy class. In particular, Theorem 1 from Simchowitz et al. [2020] has established that the DRC policy class generalizes the state-of-art benchmark class of stabilizing linear dynamic controllers (LDC) with error e -Θ ( H ) .

The DRC policy class parametrized by H ∈ Z ++ , R ∈ R + is the set of all disturbance response controller with bounded length H and norm R : M ( H,R ) = { M = ( M [ j ] ) ¯ H j =0 | ‖ M ‖ /lscript 1 , op = ∑ ¯ H j =0 ‖ M [ j ] ‖ op ≤ R } .

2.3 Approach and Technical Challenges

The classical approach in online nonstochastic control of stable/stabilizable systems is to reduce to a problem of online convex optimization with memory. This insight relies on the exponentially decaying effect of past states and controls on the present, which allows approximating the cost functions as functions of the most recent controls.

A core technical challenge lies in the bandit convex optimization problem obtained from the bandit control problem. In the bandit setting, no gradient information is given to the learner, and thus the learner needs to construct a low-bias gradient estimator. Previous work uses the classical spherical gradient estimator proposed by Flaxman et al. [2004], but the regret guarantee is suboptimal. We would like to leverage the ellipsoidal gradient estimator proposed by Hazan and Levy [2014]. However, when extending to loss functions with memory, there is no clear mechanism for obtaining a low-bias bound for general convex functions. We exploit the quadratic structure of the LQR/LQG cost functions to build EBCO-M (Algorithm 1), which uses ellipsoidal gradient estimators. We note that even outside of the control applications, EBCO-M may be of independent interests in bandit online learning theory.

3 BCO with Memory: Quadratic and Strongly Convex Functions

As with previous works, our control algorithm will depend crucially on a generic algorithm for bandit convex optimization with memory (BCO-M). We present a new online bandit convex optimization with memory algorithm that explores the structure of quadratic costs to achieve near-optimal regret.

3.1 Setting and working assumptions

In the BCO-M setting with memory length H , we consider an algorithm playing against an adversary. At time t , the algorithm is asked to play its choice of y t in the convex constraint set K . The adversary chooses a loss function F t : K H → R + which takes as input the algorithm's current play as well as its previous ¯ H plays. The algorithm then observes a cost F t ( y t -¯ H , . . . , y t ) (and no other information about F t ( · ) ) before it chooses and plays the next action y t +1 . The goal is to minimize regret with respect to the expected loss, which is the excessive loss incurred by the algorithm compared to the best fixed decision in K :

For notation convenience, we will at times shorthand y t -¯ H : t def = ( y t -¯ H , . . . , y t ) ∈ K H .

3.1.1 BCO-M assumptions

We make the following assumptions on the loss functions { F t } T t = H and the constraint set K .

Assumption 3.1 (Constraint set) . K is convex, closed, and bounded with non-empty interior. diam ( K ) = sup z,z ′ ∈ K ‖ z -z ′ ‖ 2 ≤ D .

Assumption 3.2 (Loss functions) . The loss functions chosen by the adversary obeys the following regularity and curvature assumptions:

F t : K H → R + is quadratic and β -smooth:

-Quadratic: ∃ W t ∈ R nH × nH , b t ∈ R nH , c t ∈ R such that F t ( w ) = w /latticetop W t w + b /latticetop t w + c t , ∀ w ∈ K H .

-Smooth: W t /precedesequal β I nH × nH .

F t : K H → R + is σ -strongly convex in its induced unary form: f t : K → R + with f t ( z ) = F t ( z, . . . , z ) is σ -strongly convex, i.e. f t ( z ) ≥ f t ( z ′ ) + ∇ f t ( z ′ ) /latticetop ( z -z ′ ) + σ 2 ‖ z -z ′ ‖ 2 2 ., ∀ z, z ′ ∈ K .

F t satisfies the following diameter and gradient bound on K : ∃ B,L > 0 such that

In the online control problems, when formulating the cost function c t as a function F t of the most recent H controls played, the function F t itself may depend on the entire history of the algorithm through step t -H . Therefore, it is essential to analyze the regret guarantee of our BCO-M algorithm when playing against an adversary that can be ( t -H ) -adaptive, giving rise to the following assumption.

Assumption 3.3 (Adversarial adaptivity) . The adversary chooses F t independently of the noise u t -¯ H : t which is drawn by the algorithm in the H most recent steps, but possibly not independently of earlier noises.

Note that Assumption 3.3 is minimal for BCO: if this fails, then in the subcase of a delayed loss, the adversary may fully control the agent's observations, resulting in no possibility of learning.

Self-concordant barrier. The algorithm makes use of a self-concordant barrier R ( · ) of K as the regularization function in the updates.

Definition 3.4 (Self-concordant barrier) . A three-time continuously differentiable function R ( · ) over a closed convex set K ⊂ R n with non-empty interior is a ν -self-concordant barrier of K if it satisfies the following two properties:

(Boundary property) For any sequence { x n } n ∈ N ⊂ int( K ) such that lim n →∞ x n = x ∈ ∂ K , lim n →∞ R ( x n ) = ∞ .

(Self-concordant) ∀ x ∈ int( K ) , h ∈ R n ,

(a) | ∇ 3 R ( x )[ h, h, h ] | ≤ 2 | ∇ 2 R ( x )[ h, h ] | 3 / 2 .

(b) | 〈∇ R ( x ) , h 〉 | ≤ √ ν | ∇ 2 R ( x )[ h, h ] | 1 / 2 .

3.2 Algorithm specification and regret guarantee

We present EBCO-M (Algorithm 1) for online bandit convex optimization with memory. The key novelty is the use of an ellipsoidal gradient estimator. It is difficult to establish a low-bias guarantee for ellipsoidal gradient estimator for general convex loss functions. However, thanks to the quadratic structure of the loss functions in LQR/LQG problems, we can show provable low bias for the ellipsoidal gradient estimator, and therefore achieve optimal regret.

Algorithm 1 Ellipsoidal BCO with memory ( EBCO-M )

1: Input: Convex, closed set K ⊆ n with non-empty interior, time horizon T , memory length H , step size η , ν -self-concordant barrier R ( ) over , convexity strength parameter σ .

R · K 2: Initialize x t = arg min x ∈ K R ( x ) , ∀ t = 1 , . . . , H . 3: Compute A t = ( ∇ 2 R ( x t ) + ησ tI ) -1 / 2 , ∀ t = 1 , . . . , H . 4: Sample u 1 , . . . , u H ∼ S n -1 i.i.d. uniformly at random. 5: Set y t = x t + A t u t , ∀ t = 1 , . . . , H . 6: Set g t = 0 , ∀ t = 1 , . . . , ¯ H . 7: Play y 1 , . . . , y ¯ H . 8: for t = H,.. . , T do 9: Play y t , suffer loss F t ( y t -¯ H : t ) . 10: Store g t = nF t ( y t -¯ H : t ) ∑ ¯ H i =0 A -1 t -i u t -i . 11: Set x t +1 = arg min x ∈ K ∑ t s = H ( g /latticetop s -¯ H x + σ 2 ‖ x -x s -¯ H ‖ 2 ) + 1 η R ( x ) . 12: Compute A t +1 = ( ∇ 2 R ( x t +1 ) + ησ ( t +1) I ) -1 / 2 . 13: Sample u t +1 ∼ S n -1 uniformly at random. 14: Set y t +1 = x t +1 + A t +1 u t +1 . 15: end for

Before analyzing the regret, we first make note of two properties of Algorithm 1.

Remark 3.5 (Delayed dependence) . In Algorithm 1, x t is independent of u t -¯ H : t , ∀ t , and therefore A t is independent of u t -¯ H : t as A t is determined by x t .

Remark 3.6 (Correctness) . y t played by Algorithm 1 lies in K : ‖ y t -x t ‖ 2 ∇ 2 R ( x t ) = ‖ A t u t ‖ 2 ∇ 2 R ( x t ) ≤ ‖ u t ‖ 2 2 = 1 , and by Proposition C.1, the Dikin ellipsoid centered at x t is contained in K .

We give the following two theorems on regret guarantees of the EBCO-M algorithm. In particular, Theorem 3.6.B is a slight generalization of Theorem 3.6.A in its relaxation on the curvature assumption on the sequence of loss functions F H : T .

Theorem 3.6.A ( EBCO-M regret with strong convexity) . For any sequence of cost functions { F t } T t = H satisfying Assumption 3.2, constraint set K satisfying Assumption 3.1, adversary satisfying Assumption 3.3, and H = poly (log T ) , Algorithm 1 satisfies the regret bound

where the expectation is taken over the randomness of the exploration noises u 1: T , with ˜ O hiding all natural parameters ( B,D,L ) and logarithmic dependence on T .

Theorem 3.6.B ( EBCO-M regret with conditional strong convexity) . Suppose Algorithm 1 is run on K satisfying Assumption 3.1 against an adversary satisfying Assumption 3.3 with a sequence of random cost functions { F t } T t = H satisfying that

F t is quadratic, convex, β -smooth, has diameter bound B and gradient bound L on K H .

∃ filtration G 1: T such that u 1: T is independent of G T , F t ∈ σ ( u 1: t -H ) ∪ G t , and F t is conditionally σ -strongly convex in its induced unary form: ¯ f t ( z ) def = E [ f t ( z ) | u 1: t -H , G t -H ] is σ -strongly convex.

Then, Algorithm 1 satisfies the same regret bound attained in Theorem 3.6.A, i.e.

where the expectation is taken over the randomness of the exploration noises u 1: T and the random functions F H : T , with ˜ O hiding all natural parameters ( B,D,L ) and logarithmic dependence on T .

4 Bandit Controller: Known and Unknown Systems

We will now use our BCO-with-memory algorithm to find an optimal controller (as in Gradu et al. [2020]), arguing that regret in choice of controller transfers into the setting discussed in the previous section. We first consider the case where the system is known, and then reduce the unknown system case to the known system case.

4.1 Known systems

Applying Algorithm 1 to predict controllers 4 with losses given by control losses, we obtain Algorithm 2.

Algorithm 2 Ellipsoidal Bandit Perturbation Controller ( EBPC )

1: Input: Time horizon T , memory length H , Markov operator G . BCO-M parameters σ , η . Self-concordant barrier R ( · ) over M ( H,R ) ⊂ R H × d u × d y .

2: Initialize M = = M H = arg min R ( M )

3: Compute A i = ( ∇ 2 R ( M i ) + ησ tI ) -1 / 2 , ∀ i = 1 , . . . , H .

1 · · · M ∈ M ( H,R ) .

4: Sample ε 1 , . . . , ε H ∼ S H × d u × d y -1 i.i.d. uniformly at random.

8: Play control u t = u t ( ˜ M t ) = ∑ ¯ H i =0 ˜ M [ i ] t y nat t -i , incur cost c t ( y t , u t ) . 9: Observe y t +1 and compute signal y nat t +1 = y t +1 -∑ t i =1 G [ i ] u t -i . 10: Store g t = d u d y Hc t ( y t , u t ) ∑ ¯ H i =0 A -1 t -i ε t -i .

5: Set ˜ M i = M i + ε i , ∀ i = 1 , . . . , H . Set g i = 0 , ∀ i = 1 , . . . , ¯ H . 6: Play control u i = 0 , incur cost c i ( y i , u i ) , ∀ i = 1 , . . . , ¯ H . 7: for t = H,.. . , T do

11: Update M t +1 = arg min M ∈ M ( H,R ) ∑ t s = H ( 〈 g s -¯ H , M 〉 + σ 2 ‖ M -M s -¯ H ‖ 2 ) + 1 η R ( M ) .

Sample ε t +1 S H × d u × d y -1 uniformly at random. Set M t +1 = M t +1 + A t +1 ε t +1 .

12: Compute A t +1 = ( ∇ 2 R ( M t +1 ) + ησ ( t +1) I ) -1 / 2 .

13: ∼ ˜ 14: end for

Theorem 4.1 (Known system control regret) . Consider a linear dynamical system governed by known dynamics ( A,B,C ) and the interaction model with adversarially chosen cost functions

and perturbations satisfying Assumption 2.3, 2.5, 2.6, 2.7. Then running Algorithm 2 with H = Θ (poly(log T )) , σ = σ c ( σ 2 e + σ w σ min ( C ) 1+ ‖ A ‖ 2 op ) , and η = Θ ( 1 d u d y L c H 3 √ T ) guarantees

where regret is defined as in Eq.(2.2), the expectation is taken over the exploration noises ε 1: T of the algorithm as well as the stochastic components { w stoch t , e stoch t } T t =1 of the perturbations { w t , e t } T t =1 , and ˜ O ( · ) hides all universal constants, natural parameters, and logarithmic dependence on T .

4.2 Unknown systems: control after estimation

Note that EBPC (Algorithm 2) relies on the access to the system's Markov operator G , which is available if and only if the system dynamics ( A,B,C ) are known. When the system dynamics is unknown, we can identify the system using a system estimation algorithm, obtain an estimated Markov operator ˆ G , and run EBPC with G ← ˆ G . Algorithm 3 outlines the estimation method of system dynamics via least squares.

Algorithm 3 System estimation via least squares ( SysEst-LS )

1: Input: estimation sample size N , system length H .

2: Initialize: ˆ G [ t ] = 0 , ∀ t ≥ H .

4: Set ˆ G [0: ¯ H ] = arg min ∑ N t = H ‖ y t -∑ ¯ H i =0 ˆ G [ i ] u t -i ‖ 2 2 . 5: Return ˆ G .

3: Sample and play u t ∼ N (0 , I d u × d u ) , ∀ t = 1 , . . . , N .

Theorem 4.2 (Unknown system control regret) . Consider a linear dynamical system governed by unknown dynamics ( A,B,C ) and the interaction model with adversarially chosen cost functions and perturbations satisfying Assumption 2.3, 2.5, 2.6, 2.7. Suppose we obtain an estimated Markov operator ˆ G from Algorithm 3 with N = /ceilingleft √ T /ceilingright and H = Θ (poly log T ) . Then Algorithm 2 with G ← ˆ G , H ← 3 H , σ = 1 8 σ c σ 2 e , and η = Θ ( 1 d u d y L c H 3 √ T ) guarantees

where regret is defined as in Eq.(2.2), the expectation is taken over the exploration noises ε 1: T in Algorithm 2, the sampled Gaussian controls u 1: N in Algorithm 3, and the stochastic components { w stoch t , e stoch t } T t =1 of the perturbations { w t , e t } T t =1 , and ˜ O ( · ) hides all universal constants, natural parameters, and logarithmic dependence on T .

5 Discussion and conclusion

We solve the open problem put forth by Gradu et al. [2020] on the optimal rate for online bandit control for the case of LQR/LQG control, improving to regret ˜ O ( √ T ) from ˜ O ( T 3 4 ) in the semiadversarial noise model and for strongly convex LQR/LQG cost functions. Our method builds upon recent advancements in bandit convex optimization for quadratic functions, providing the first near-optimal regret algorithm for bandit convex optimization with memory in a nonstochastic setting.

It would be interesting to investigate (1) whether the results can be extended to fully adversarial noise, (2) whether a similar stable controller recovery as seen in [Chen and Hazan, 2021] for fully observable systems can be established for partially observable systems, and whether that can be incorporated to extend our result to stabilizable systems even without access to a stabilizing controller.

References

Yasin Abbasi-Yadkori and Csaba Szepesvári. Regret bounds for the adaptive control of linear quadratic systems. In Proceedings of the 24th Annual Conference on Learning Theory , pages 1-26. JMLR Workshop and Conference Proceedings, 2011.

Jacob Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In 21st Annual Conference on Learning Theory, COLT 2008 , 2008.

Naman Agarwal, Brian Bullins, Elad Hazan, Sham Kakade, and Karan Singh. Online control with adversarial disturbances. In International Conference on Machine Learning , pages 111-119. PMLR, 2019a.

Naman Agarwal, Elad Hazan, and Karan Singh. Logarithmic regret for online control. Advances in Neural Information Processing Systems , 32, 2019b.

Oren Anava, Elad Hazan, and Shie Mannor. Online learning for adversaries with memory: price of past mistakes. Advances in Neural Information Processing Systems , 28, 2015.

Richard Bellman. The theory of dynamic programming. Bulletin of the American Mathematical Society , 60(6):503-515, 1954.

Asaf Cassel and Tomer Koren. Bandit linear control. Advances in Neural Information Processing Systems , 33:8872-8882, 2020.

Xinyi Chen and Elad Hazan. Black-box control for linear dynamical systems. In Conference on Learning Theory , pages 1114-1143. PMLR, 2021.

Xinyi Chen and Elad Hazan. A nonstochastic control approach to optimization. arXiv preprint arXiv:2301.07902 , 2023.

Alon Cohen, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, and Kunal Talwar. Online linear quadratic control. In International Conference on Machine Learning , pages 10291038. PMLR, 2018.

Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. Regret bounds for robust adaptive control of the linear quadratic regulator. Advances in Neural Information Processing Systems , 31, 2018.

Genevieve E Flaspohler, Francesco Orabona, Judah Cohen, Soukayna Mouatadid, Miruna Oprescu, Paulo Orenstein, and Lester Mackey. Online learning with optimism and delay. In International Conference on Machine Learning , pages 3363-3373. PMLR, 2021.

Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. arXiv preprint cs/0408007 , 2004.

Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. In Proceedings of the sixteenth annual ACM-SIAM symposium on Discrete algorithms , pages 385-394, 2005.

Dylan Foster and Max Simchowitz. Logarithmic regret for adversarial online control. In International Conference on Machine Learning , pages 3211-3221. PMLR, 2020.

Udaya Ghai, Xinyi Chen, Elad Hazan, and Alexandre Megretski. Robust online control with model misspecification. In Learning for Dynamics and Control Conference , pages 1163-1175. PMLR, 2022.

Udaya Ghai, Arushi Gupta, Wenhan Xia, Karan Singh, and Elad Hazan. Online nonstochastic model-free reinforcement learning. arXiv preprint arXiv:2305.17552 , 2023.

Paula Gradu, John Hallman, and Elad Hazan. Non-stochastic control with bandit feedback. Advances in Neural Information Processing Systems , 33:10764-10774, 2020.

Paula Gradu, John Hallman, Daniel Suo, Alex Yu, Naman Agarwal, Udaya Ghai, Karan Singh, Cyril Zhang, Anirudha Majumdar, and Elad Hazan. Deluca-a differentiable control library: Environments, methods, and benchmarking. arXiv preprint arXiv:2102.09968 , 2021.

Elad Hazan. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Elad Hazan and Kfir Levy. Bandit convex optimization: Towards tight bounds. Advances in Neural Information Processing Systems , 27, 2014.

Elad Hazan and Karan Singh. Introduction to online nonstochastic control. arXiv preprint arXiv:2211.09619 , 2022.

Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2):169-192, 2007.

Morteza Ibrahimi, Adel Javanmard, and Benjamin Roy. Efficient reinforcement learning for high dimensional linear quadratic systems. Advances in Neural Information Processing Systems , 25, 2012.

Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.

Sahin Lale, Kamyar Azizzadenesheli, Babak Hassibi, and Anima Anandkumar. Adaptive control and regret minimization in linear quadratic gaussian (lqg) setting. In 2021 American Control Conference (ACC) , pages 2517-2522. IEEE, 2021.

Kent Quanrud and Daniel Khashabi. Online learning with adversarial delays. Advances in neural information processing systems , 28, 2015.

Ohad Shamir. On the complexity of bandit and derivative-free stochastic convex optimization. In Conference on Learning Theory , pages 3-24. PMLR, 2013.

Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non-stochastic control. In Conference on Learning Theory , pages 3320-3436. PMLR, 2020.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML-03) , pages 928-936, 2003.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an algorithm fro the LQG problem, in which a linear system has to be controlled in the presence of semi-adversarial perturbations in order to minimise an adversarially-chosen cost, with the controller only having access to a projection of the real dynamics onto a lower-dimensionality subspace. The authors show that their proposed method achieves optimal regret in time horizon of $O(\\sqrt{T})$.\n\nSTRENGTHS:\nThe paper's introduction of the problem is quite readable to a non-specialist, though specialist knowledge is required to understand most of the details and the theorem proof.\n\nWEAKNESSES:\nThe paper is extremely theoretical and deals with a very specialised problem. This problem has connections with more mainstream AI problems (Optimal Control and Reinforcement Learning are closely related), and the paper does present it in a way that makes it understandable to a non-specialist reader. Still, the paper does not present any task-oriented AI-based framework involving the training of a Neural Net. While this might not be enough to disqualify the paper from publication at NeurIPS, it does severely restrict the number of potential readers.\n\nQUESTIONS:\nMy understanding of the paper is too limited to provide any meaningful suggestions for improvement.\n\nLIMITATIONS:\nThe paper is extremely theoretical and can only be understood by a very specialised audience. The relevance of the author's solution for the presented problem (or even of the problem itself) to more mainstream AI research, if any, is not at all discussed.\n\nEdit: In their rebuttal, the authors challenged my assessment of their work's limitations by pointing out a connection between nonstochastic control and hyperparameter tuning in large-scale optimization, which was remarked in “A Nonstochastic Control Approach to Optimization” by Chen & Hazan (2023). While the connection is definitely present and fascinating, I still felt that it was not enough of an established research track for me to change my assessment. I therefore kept my initial score.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies LQG with semi-adversarial perturbations and time varying adversarial bandit loss functions.\nThe best known bound for such problems has a T^{3/4} time horizon dependence.\nThey provide an algorithm for LQR/LQG problems that attains $\\tilde{\\mathcal{O}}(\\sqrt{T})$ regret. \nTheir algorithm goes through even if the underlying system dynamics are unknown. \nAdditionally, their method also provides a new scheme for bandit convex optimization with memory.\n\nSTRENGTHS:\nThis paper addresses an important problem in the field of control and online learning.\n\nWEAKNESSES:\nI must express my concerns about its presentation and structure. The writing is not as clear as it needs to be, making it quite challenging to follow the paper. \nThe argument seems to lack the rigorousness and the self-containment.\n\nA crucial element that seems to be missing is detailed explanations for the conclusions drawn. Without these, it becomes arduous to justify the results. Enhancing the clarity and providing in-depth explanations of the methodologies and results will significantly improve the comprehension and overall impact of the study. I list my comments, questions and suggestions below.\n\nQUESTIONS:\n## Major comments:\n\n1) Can you elaborate more on the DRC policy class? To me it seems like you are concentrating on the causal (non-anticipatory) policies that are linear. Is it possible to show that a linear policy will be optimal for your setting because you are concentrating on the quadratic cost function? I believe that would require to show that the estimation of the state given observations would be linear in observations as in the classical LQG setting. \n\n2) I think nature's y seems like the purified the purified output control method in [R1]. Is that true? If yes can you please provide the relevant reference there? \n\n3) I find it very confusing that $y$ suddenly becomes a decision variable in Section 3.1 while it represents the observations of the state in the previous chapter and also in the subsequent section. This is truly confusing if I did not miss something. Following that I do not understand why $u$ is the noise in Assumption 3.3. \n\n\n4) What does $A_t$ represent in Algorithm 1? It seems completely irrelevant to the state transition matrix $A$ to me. Why do you choose the same letter? I find this very confusing. \n\n\n5) Where is the proof of Theorem 3.7? \n\n6) Line 529, I do not understand why $x_1, \\ldots, x_{2\\bar H +1} = argmin_{x \\in \\mathcal K} R(x)$ because it seems to me $x_1, \\ldots x_{H } = argmin_{x \\in \\mathcal K} R(x)$ but $x_{H+1}, \\ldots, x_{2 \\bar H + 1}  = argmin_{x \\in \\mathcal K} \\sum_{s= H}^t \\frac{\\sigma}{2} \\| x - x_{s- \\bar H}\\|^2) + \\eta^{-1} R(x)$. What am I missing here?\n\n7) Can you comment on if it is possible to remove the logarithmic term even from the regret or is it fundamentally not possible? (in Theorem 4.1 and Theorem 4.2)\n\n8) Why this rate is optimal? $\\sqrt{T}$ regret rate would be optimal as it is tight. By not $\\tilde{\\mathcal{O}}(\\sqrt(T))$. Can you please elaborate on that. \n\n## Minor comments:\n\n1) Typo in Page 4, Line 136: Time step instead of timestep.\n\n2) Please provide references for every claim that is mentioned in the paper without a proof. For example, in Page 4, Line 145, I suggest to provide examples of the papers that make us of the system's counterfactual signal. Same goes for Page 5, Line 159. \nAnother example is in Page 5, Line 181. The sentence there reads \"These conditions are relatively standard for bandit convex optimization algorithms, and are needed for the ..\". Please provide references for the bandit convex optimization algorithms that make these standard assumptions.\nPlease provide references for Line 215 (which previous works). \n\n3) Typo in Page 5, Line 157, \"that that\" should be \"that\".\n\n4) Line 444: \"empirical\" instead of \"emprical\"\n5) $W_t$ is not introduced in Line 235. \n6) Line 197, what is $\\Theta(H)$?\n7) It's mandatory to include the code for the experiment section in Neurlps submissions. Your code seems to be missing. Please rectify this immediately by sharing your code via an anonymous GitHub repository\n8) I'm having trouble understanding the meaning of the paragraph found on Line 196. Could you please clarify? \n9) In Line 510, which spiking? I do not understand the paragraph in Line 509. \n10) Why there is a divergence problem in the third plot of Figure 1? \n11) I must express difficulty in comprehending the conclusion about the initial phase of EBPC possessing lower stability. The relationship with Figure 1 is unclear and requires further explanation. I am also perplexed by the decision to choose an extraordinarily high starting multiplier on ε - a justification is needed for this choice.\nIt's important to stress that the writing in the experiment section does not meet the expected standards for rigor. As it stands, the line of reasoning is opaque, and the process by which the conclusions were reached is indiscernible.\nFurther compounding the issue, the code has not been made available for examination. This, as previously emphasised, hampers any possibility of a comprehensive understanding or verification of the results. I strongly recommend addressing these concerns in a revised submission.\n12) Line 504- Why this is a barely-stable system?\n13) Line 261, if the key point is to use ellipsoidal gradient estimator, where is its definition?\n14) In algorithm 1, $S^{n-1}$ is not defined. \n15) Line 810- There is a typo. \n\n\nReferences: \n\n[R1] A. Ben-Tal, S. Boyd, and A. Nemirovski, Control of Uncertainty-Affected Discrete Time Linear Systems via Convex Programming. Technical Report Minerva Optimization Center, Technion, Haifa,\nIsrael, 2005 [Online]. Available: http://www.stanford.edu/~boyd/papers/pur_out_control.html\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper presents an algorithm to perform control in the semi-adversarial setting. After presenting the gap in the literature they want to fill, the authors present an algorithm that shows optimal regret ($O(\sqrt{T})$) for this setting, considerably improving the previous bound known from the literature ($O(T^{3/4})$, Gradu et al., 2020). The authors perform an experimental campaign to validate their solution in the appendix.

STRENGTHS:
- The gap in the literature that the authors fill is clearly stated.

- The setting and the assumptions are clearly presented.

- The work, in general, is clearly presented (except for the algorithms, see Weaknesses).

- The authors face both the cases of known and unknown systems.

- The bounds presented are tight, given the lower bounds known from the literature.

WEAKNESSES:
- Algorithms 1, 2, and 3 should be explained instead of only presenting the pseudo-code. Some step may not be clear to all the readers only from the pseudo-code.

QUESTIONS:
No questions.

LIMITATIONS:
No limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies LQG with semi-adversarial disturbances and bandit feedback. The authors proposed a novel algorithm with $\\sqrt T$ regret bound, which improves the $T^{3/4}$ regret bound in the literature. The algorithm utilizes the quadratic cost function structure and the ellipsoidal gradient estimator.\n\nSTRENGTHS:\nThe paper follows the classic disturbance-action policy and online nonstochastic control and OCO-M framework. The novel part is the Elllipsoida BCO with Memory algorithm and its regret analysis. The algorithm motivation and the key steps are explained clearly. \n\nThe paper solves an interesting open question that whether $\\sqrt T$ regret can be achieved for LQG under bandit feedback, under the assumption that the disturbances consist of both a large enough i.i.d. term and an adversarial term.\n\nWEAKNESSES:\n1. Simulation. The paper lacks simulation results for verification of the improved technical bound. Though the supplementary material provides some figures, the simulation setups are not clearly explained. See more questions in the next box.\n\n2. Novelty. I have mixed feelings about the novelty of this paper. On the one hand, the paper successfully improves the regret bound order to $\\sqrt T$. On the other hand, the majority of the algorithm design is based on the classic disturbance-action policy and OCO-M formulation, so I am not sure if the algorithm design is novel enough for a neurips publication.\n\n3. Presentation. The paper seems to be written in a rush. For example, the title of the paper is inconsistent with the title on the webpage. The major algorithm design trick is the ellipsoidal gradient estimator, but the paper didn't explain this estimator or provide intuition. The main theorems are merely stated without any explanation.\n\nQUESTIONS:\nIn the main paper,\nQ1: the title is inconsistent with that on the openreview webpage. Please make them consistent.\n\nQ2: Why do the authors consider semi-adversarial disturbances? What are the benefits and limitations of this setting? Why do the authors simulate i.i.d. disturbances instead of semi-adversarial disturbances? \n\nIn the supplementary material, for the simulation results:\n\nQ3: Is the dynamical system known or unknown? If the system is unknown, how to obtain the optimal $K$ to the LQR problem? If the system is known, how does the algorithm perform under unknown dynamics?\n\nQ4: What's the y axis? Why is it 0.1, 0.01? The authors skipped most parts of the simulation setup and referred the reader to Gradu et al. 2020. But the figures look very different from those in Gradu et al. 2020. It is hard to interpret the simulation results without future explanation.\n\nQ5: The x axis scale goes as large as 100k, which is unreasonable for a control task. Is it possible to plot the figures in the same settings as in Gradu et al. 2020, where the horizon is usually within 1k?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the online control problem with a quadratic cost function, adversarially chosen dynamics, and semi-adversarially chosen losses (they are the sum of a zero-mean term and an obliviously chosen adversarial term). The algorithm proposed lowers the previous $T^{3/4}$ algorithm to a $\\sqrt{T}$ algorithm against the class of disturbance response controllers, matching the lower bound. The crux of the analysis is handling the non-independence between plays and noise, which many OCO algorithms assume, and to carefully control the bias of the loss estimators. The authors solve the former with a novel method of using an online-learning with delay framework to bypass and the later by adapting an ellipsoidal gradient estimator.\n\nSTRENGTHS:\nGood organization, really clear presentation and summary of the high-level ideas used in this and previous work. Excellent literature review as well. The overall structure of the paper is well organized, and the algorithm follows naturally from all the components used.\n\nThe significance is high, as this paper really makes progress in closing off a field of online control by matching a lower bound. Much steady progress has been made on this problem over the years, so this paper is a great next step for this community. Tangentially, I think some of the ideas for algorithm 1 will be useful for the bandits-with-memory community.\n\nWEAKNESSES:\nAssumption 3.3 seems unnatural. Can you provide a bit more context on its necessity? More generally, how are ideas from learning with delay present in your algorithms? I did not find this point very clear.\n\nMight be worth mentioning that there are empirical evaluations in the main body. \n\nFinally, I wish the authors provided more intuition for how the proofs of the theorems worked; for example, what are the implications of using $\\hat G$ instead of $G$?\n\nQUESTIONS:\nA few questions are asked above.\n\nLIMITATIONS:\nAs a very theoretical work, I don't think any discussion about negative societal impact is necessary.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-28"}