Title: Tight Rates for Bandit Control Beyond Quadratics
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bandit Problem

Online Control
Abstract: Unlike classical control theory, such as Linear Quadratic Control (LQC), real-world control problems are highly complex. These problems often involve adversarial perturbations, bandit feedback models, and non-quadratic, adversarially chosen cost functions. A fundamental yet unresolved question is whether optimal regret can be achieved for these general control problems. The standard approach to addressing this problem involves a reduction to bandit convex optimization with memory. In the bandit setting, constructing a gradient estimator with low variance is challenging due to the memory structure and non-quadratic loss functions.

In this paper, we provide an affirmative answer to this question. Our main contribution is an algorithm that achieves an $\tilde{O}(\sqrt{T})$ optimal regret for bandit non-stochastic control with strongly-convex and smooth cost functions in the presence of adversarial perturbations, improving the previously known $\tilde{O}(T^{2/3})$ regret bound from \citep{cassel2020bandit}. Our algorithm overcomes the memory issue by reducing the problem to Bandit Convex Optimization (BCO) without memory and addresses general strongly-convex costs using recent advancements in BCO from \citep{suggala2024second}. Along the way, we develop an improved algorithm for BCO with memory, which may be of independent interest.

Tight Rates for Bandit Control Beyond Quadratics

Abstract

Unlike classical control theory, such as Linear Quadratic Control (LQC), real-world control problems are highly complex. These problems often involve adversarial perturbations, bandit feedback models, and non-quadratic, adversarially chosen cost functions. A fundamental yet unresolved question is whether optimal regret can be achieved for these general control problems. The standard approach to addressing this problem involves a reduction to bandit convex optimization with memory. In the bandit setting, constructing a gradient estimator with low variance is challenging due to the memory structure and non-quadratic loss functions.

In this paper, we provide an affirmative answer to this question. Our main contribution is an algorithm that achieves an ˜ O ( √ T ) optimal regret for bandit nonstochastic control with strongly-convex and smooth cost functions in the presence of adversarial perturbations, improving the previously known ˜ O ( T 2 / 3 ) regret bound from (Cassel and Koren, 2020). Our algorithm overcomes the memory issue by reducing the problem to Bandit Convex Optimization (BCO) without memory and addresses general strongly-convex costs using recent advancements in BCO from (Suggala et al., 2024). Along the way, we develop an improved algorithm for BCO with memory, which may be of independent interest.

1 Introduction

Optimal control lies at the heart of engineering and operations research, with applications ranging from launching spacecraft to stabilizing economies. The theory of optimal control is a well-established field with a rich history, dating back to 1868 when James Clerk Maxwell analyzed governors, and flourishing in the mid-20th century with the development of dynamic programming by (Bellman, 1954) and the Kalman filter by (Kalman, 1960).

Classic optimal control theory studies the problem where a controller interacts with an environment, according to a (partially observable) linear time-invariant (LTI) dynamical system:

where A,B,C are the dynamics governing the system, x t , u t , y t , w t , e t represent the system state, control input, observation, perturbation, and observation noise at time t , respectively. At each time t , the controller observes y t and chooses a control u t , incurring a cost c t ( y t , u t ) based on the current observation and control. The system then evolves according to Eq. (1) to reach the next state x t +1 .

The theory of optimal control (e.g., LQC) typically relies on three key assumptions on the setting: the perturbation w t is stochastic, the cost function c t is quadratic and known in advance, and the function c t is observable to the controller. The linear-quadratic regulator (LQR) (Kalman et al., 1960) provides a closed-form optimal solution under these conditions, representing a pinnacle of classical control theory.

However, these assumptions are often too idealistic for practical scenarios. In real-world control problems, the perturbations can be adversarial, the feedback model can be bandit, and the cost

function can be non-quadratic. This is evident in applications such as autonomous vehicle navigation, advertisement placement, and traffic signal control. This discrepancy between theory and practice raises a fundamental question for developing a more general control theory:

Can we devise algorithms with provable guarantees for LTI control problems with adversarial perturbations, bandit feedback models, and non-quadratic cost?

Recent research in online non-stochastic control (see (Hazan and Singh, 2022) for a survey) aims to address this broader goal by relaxing two of the standard assumptions: (1) the cost c t can be time-varying, non-quadratic convex functions unknown to the controller; (2) the perturbation w t and the observation noise e t can be adversarially chosen.

The natural performance metric in this context is regret , defined as the excess cost incurred by the controller compared to the best control policy in a benchmark policy class Π :

where ( y t , u t ) is the observation-control pair reached by executing the controller at time t , and ( y π t , u π t ) is the observation-control pair under the policy π at time t . An optimal ˜ O ( √ T ) regret was obtained by (Agarwal et al., 2019a) with the Gradient Perturbation Controller ( GPC ) algorithm.

Several works in online control have made further progress towards the general question. (Cassel and Koren, 2020; Sun et al., 2024) obtained optimal regret under bandit feedback for strongly convex and smooth cost when the perturbation w t and observation noise e t are semi-adversarial (i.e. they contain an additive stochastic component that admits covariance matrices with least singular value bounded from below). (Cassel and Koren, 2020) also showed a sub-optimal ˜ O ( T 2 / 3 ) regret bound for fully adversarial perturbations. More recently, the advancement of (Suggala et al., 2024) showed for the first time that an optimal regret of ˜ O ( √ T ) is achievable for strongly convex and smooth quadratic costs in the presence of adversarial perturbations and observation noises.

Still, no previous work has simultaneously addressed all three challenges with an optimal regret guarantee, which was left as an open problem by (Suggala et al., 2024). The main challenge lies in how to construct a low-variance gradient estimator under bandit feedback, with the memory structure and non-quadratic cost. Due to the Ω( T 2 / 3 ) regret lower bound for general BCO with memory by (Suggala et al., 2024), exploiting the special affine memory structure in control problems is crucial to achieving optimal regret for general convex cost.

In this work, we provide the first affirmative answer to this general question by devising an algorithm that handles all three challenges with an optimal ˜ O ( √ T ) regret bound. Our approach involves reducing the problem to no-memory BCO, which circumvents the high-dimensional estimator issue for non-quadratic costs. We then leverage a special curvature structure of the loss function induced by general strongly convex and smooth costs to obtain the optimal regret guarantee. Our result serves as a preliminary step toward fully solving the general control problem.

1.1 Technical Overview

Several previous works have addressed the bandit non-stochastic control problem, but none achieved optimal regret across all three generalities due to the following technical challenges:

The necessity of curvature : The first work tackling the bandit non-stochastic control problem was (Cassel and Koren, 2020), which achieved an ˜ O ( T 2 / 3 ) regret bound for smooth convex cost. However, such sub-optimality is arguably inevitable due to the ˜ Ω( T 2 / 3 ) regret lower bound for BCO-M with smooth convex 1 cost (Suggala et al., 2024): all existing results on bandit non-stochastic control relies on reduction to BCO-M! Indeed, the algorithm of (Cassel and Koren, 2020) is based on online gradient descent ( OGD ) and ignores the geometry of cost. This is the reason why we introduce the assumption on strong convexity.

Strong convexity is not enough : Does strong convexity alone become an easy remedy to the control problem? Unfortunately, even if we assume the cost functions c t in the control

√

problem are strongly-convex, the induced loss functions in the BCO-M problem are not necessarily strongly-convex! It was observed by (Suggala et al., 2024) that the induced loss functions satisfy a property called κ -convexity, allowing for low-variance estimation of Hessian matrices, which is a key ingredient in Newton-based second-order update to exploit the curvature.

̸

Going beyond quadratic : However, finding a low-biased first-order estimation of gradients remains a challenge for BCO-M, because the nice property on the unit sphere domain S d -1 := { x ∈ R d | ∥ x ∥ 2 = 1 } of the exploration term is not preserved under Cartesian product: S d -1 × S d -1 = S d × d -1 . To handle this issue, (Suggala et al., 2024) relies on the quadratic cost assumption which admits an unbiased estimator of the divergence of the induced loss function. It's unclear how to handle general κ -convex smooth cost in BCO-M.

Similar to its full-information counterpart, we reduce the bandit non-stochastic control problem to bandit convex optimization with memory (BCO-M). To overcome these challenges, we further reduce the BCO-M problem to a no-memory BCO problem following (Cassel and Koren, 2020), to avoid the issue on the Cartesian product of S d -1 . This allows for the use of a Newton-based BCO-M algorithm similar to (Suggala et al., 2024), which can handle general κ -convex smooth cost because the exploration domain is now S d -1 . Besides this new approach, our method also includes novel algorithmic components and analysis.

1.2 Related Work

Optimal control theory (Bellman, 1954) concerns finding a control for a dynamical system such that some objective function is optimized. The basic form of optimal control is linear quadratic control (Kalman, 1960), in which a quadratic function is minimized in a linear first-order dynamical system with stochastic noise, whose solution is given by LQR.

Online non-stochastic control generalizes classic optimal control by considering adversarial perturbation and cost, with the metric of regret to compete with the best fixed policy in some benchmark class. The first work (Agarwal et al., 2019a) obtained an optimal regret bound for general convex cost by reduction to online convex optimization with memory (Anava et al., 2015), under stability assumptions on the system. For the line of works on the full information feedback setting, see (Hazan and Singh, 2022) for a survey.

Online non-stochastic control with bandit feedback were first considered by (Cassel and Koren, 2020; Gradu et al., 2020). Under a smoothness assumption on cost functions, the two works showed an ˜ O ( T 2 / 3 ) and ˜ O ( T 3 / 4 ) regret bound respectively. When the cost functions are additionally stronglyconvex quadratics, (Sun et al., 2024) obtained an ˜ O ( √ T ) regret, under semi-adversarial perturbations. (Yan et al., 2024) showed sublinear regret for cost functions with heterogeneous curvatures under the

same semi-adversarial perturbations. This restriction on perturbation was later removed by (Suggala et al., 2024) which achieved the same regret for adversarial perturbations.

Recent advancements on BCO (Lattimore and György, 2023; Suggala et al., 2021; Lattimore, 2024) considered Newton step updates to achieve improved regret bounds. In particular, (Suggala et al., 2024) also showed an ˜ Ω( T 2 / 3 ) lower bound for bandit quadratic optimization with memory (BQO-M) without strong convexity. When making use of the affine memory structure, (Suggala et al., 2024) obtained an ˜ O ( √ T ) regret bound for BQO-M.

2 Preliminary

2.1 Online non-stochastic control with bandit feedback

Consider the linear dynamical system in Eq. (1). At time t ∈ N , the learner receives observation y t and outputs control u t , to which the learner receives a scalar instantaneous cost c t ( y t , u t ) depending on both the current observation and the control, and no other information about c t is available. The perturbation w t and observation noise e t can be adversarially chosen. The learner's goal is to minimize regret over a fixed time horizon T ∈ N defined in Eq. (2).

Throughout this paper, all cost functions we consider are assumed to be twice continuously differentiable. Consistent with past literature (Sun et al., 2024; Suggala et al., 2024), we make the following assumptions on the cost functions.

Assumption 1 (Cost curvature and regularity) . Let d y , d u ∈ N denote the dimensions of observation and control, respectively. The control cost function c t : R d y × R d u → R + satisfies that

(Curvature) c t is α c -strongly convex and β c -smooth over some compact set Y ×U ⊂ R d y × R d u for some α c , β c > 0 , i.e., ∇ c t ( x 2 ) ⊤ ( x 1 -x 2 ) + β c 2 ∥ x 1 -x 2 ∥ 2 2 ≥ c t ( x 1 ) -c t ( x 2 ) ≥ ∇ c t ( x 2 ) ⊤ ( x 1 -x 2 ) + α c 2 ∥ x 1 -x 2 ∥ 2 2 for any x 1 , x 2 ∈ Y × U .

(Gradient bounds) There exists some G c > 0 such that the gradients satisfy ∥∇ c t ( y, u ) ∥ 2 ≤ G c ∥ ( y, u ) ∥ 2 for any y ∈ Y , u ∈ U .

We also make stability assumption on the LDS in Eq. (1) as well as norm bound on the perturbation and noise. These assumptions are standard in literature (e.g. (Hazan and Singh, 2022)) and necessary for deriving meaningful regret guarantees.

Assumption 2 (Strong stabilizability and bounded dynamics) . ( A,B,C ) that governs the LDS in Eq. (1) satisfies that max {∥ A ∥ op , ∥ B ∥ op , ∥ C ∥ op } ≤ κ sys for some positive constant κ sys . There exists K ∈ R d u × d y s.t. A + BKC = HL -1 H for some H ≻ 0 and max {∥ K ∥ 2 , ∥ H ∥ 2 , ∥ H -1 ∥ 2 } ≤ κ , ∥ L ∥ 2 ≤ 1 -γ for some κ > 0 , 0 < γ ≤ 1 . Such K is called a ( κ, γ ) -stabilizing linear controller for the LDS.

Assumption 3 (Bounded perturbation and noise) . The perturbation and observation noise sequences satisfy the norm bound max t ∈ [ T ] { max {∥ w t ∥ 2 , ∥ e t ∥ 2 }} ≤ R w,e for some R w,e > 0 .

We assume the adversary chooses the cost functions and noise sequences in an oblivious way, i.e. they are chosen independently of the learner's decisions.

Assumption 4 (Oblivious adversary) . The sequence of cost functions { c t } T t =1 and the noise sequences { w t } T t =1 , { e t } T t =1 are chosen by an oblivious adversary and does not depend on the control played by the learner.

For partially observable systems, the standard comparator class in literature (Simchowitz et al., 2020) is the class of Disturbance Response Controllers (DRC). Essentially, this class considers controllers that choose linear combinations of past signals and observations. The signals are simply the would-be observations had a stabilizing controller K been used from the beginning of the time. Formally, the class of DRC is given by the following definition:

Definition 1 (DRC) . (1) A disturbance response controller (DRC) is a policy π M parametrized by m ∈ N matrices M = M [0: m -1] in R d u × d y such that the control at time t according to π M is

where K is a ( κ, γ ) -stabilizing linear controller, and y t K is the would-be observation had the linear policy K been carried out from the beginning of the time.

(2) A DRC policy class M ( m,R M ) , parameterized by m ∈ N , R M > 0 , is the set of all DRC controller of length m and obeys the norm bound ∥ M ∥ ℓ 1 , op := ∑ m -1 j =0 ∥ M [ j ] ∥ op ≤ R M .

A DRC policy always produces controls that ensure the system remains state bounded, the DRC class is expressive enough to approximate the class of linear policies of past observations (see Theorem 1 in (Simchowitz et al., 2020)).

2.2 Bandit convex optimization with memory (BCO-M)

We start with the general protocol of bandit convex optimization with memory (BCO-M). In the (improper) BCO-M problem, at each time t , the learner is asked to output a decision z t ∈ R d , to which a scalar loss f t ( z t -m +1: t ) , depending on the learner's most recent m ∈ N decisions is revealed. Given a convex compact set K ⊂ R d as the domain of comparators, the regret is measured on the best single point z ∈ K over a time horizon of T ∈ N , formally given by

In non-stochastic control, the learner's past decisions affect future states through an affine structure, therefore we are interested in BCO-M problems with such structure. This leads to the following structural assumption on the induced loss function with memory. We will show in Lemma 9 that the bandit control problems with the standard regularity conditions satisfy Assumption 5.

Assumption 5 (Affine memory structure) . At time t ∈ N , the loss function with memory length m takes the form of f t : ( R d ) m → R + given by the following structure

with parameters B t ∈ R n , Y t -i ∈ R p × d , and G = G [0: m -1] a sequence of m matrices where G [ i ] ∈ R n × p for some n, p ∈ N . 2 We denote G t = ∑ m -1 i =0 G [ i ] Y t -i ∈ R n × d , and H t = G ⊤ t G t ∈ R d × d . There exists some R H > 0 such that max { 1 , ∥ G t ∥ 2 , ∥ Y t ∥ 2 , ∥ H t ∥ 2 } ≤ R H . In addition, we assume that G satisfies positive convolution invertibility-modulus, i.e. κ ( G ) = inf ∑ n ≥ 0 ∥ u n ∥ 2 2 =1 ∑ n ≥ 0 ∥ ∑ n i =0 G [ i ] u n -i ∥ 2 2 = Ω(1) . We assume that the learner receives H t every step after they incurred the loss.

We make two standard curvature and regularity assumptions on each of the instantaneous loss f t , that f t is strongly-convex and smooth, with its subgradient norm bounded by some constants, following the previous work of Suggala et al. (2024).

Assumption 6 (Curvature) . Consider a loss function f t satisfying Assumption 5 and the set

where B t , G, Y t -m +1: t are the parameters, and ℓ t : R n → R + is the function associated with f t in Assumption 5. We assume that ℓ t is α f -strongly-convex and β f -smooth, i.e. α f I n ⪯ ∇ 2 ℓ t ( z ) ⪯ β f I n , over Z t , with 0 < α f ≤ 1 ≤ β f here 3 .

Assumption 7 (Regularity) . For d ∈ N , denote B d := { x ∈ R d | ∥ x ∥ 2 ≤ 1 } as the unit ball in R d . We assume that at time t , the with-memory loss function f t : ( R d ) m → R + with memory parameter m ∈ N obeys the following gradient bounds over K + B d := { x + y | x ∈ K , y ∈ B d } :

Additionally, we assume that K has Euclidean diameter D > 0 .

We consider an oblivious adversary model, given by Assumption 8.

Assumption 8 (Oblivious adversary) . For a BCO-M instance, we assume that the adversary is oblivious. In particular, let z t denote the algorithm's decision at time t , then the loss function f t chosen by the adversary does not depend on z 1: t .

In the definition of regret in BCO-M, we compare the cumulative loss of any algorithm with the best fixed comparator z ∈ K , evaluated on the induced unary form f t ( z, . . . , z ) of the loss f t . Formally, we have the following definition

Definition 2 (Induced unary form) . ∀ t ∈ N , let ¯ f t : R d → R + denote the induced unary form of the loss f t , given by ¯ f t ( z ) = f t ( z, . . . , z ) . Then, for f t satisfying Assumption 5, ¯ f t admits the structure ¯ f t ( z ) = ℓ t ( B t + ∑ m -1 i =0 G [ i ] Y t -i z ) .

We note that for f t satisfying Assumption 6, the induced unary form in Definition 2 satisfies a special curvature called κ -convexity introduced in (Suggala et al., 2024). To avoid confusion with the notations with the bound on the dynamics (Assumption 2), we will call it κ 0 -convexity henceforth.

Definition 3 ( κ 0 -convexity, (Suggala et al., 2024)) . A function f : R d → R is called κ 0 -convex over a domain K ⊆ R d if and only if the following holds: f is convex and twice continuously differentiable, and moreover ∃ c, C > 0 and a PSD matrix 0 ⪯ H ⪯ I s.t. the Hessian of f at any z ∈ K satisfies

The benefit that the loss function exhibits an affine memory dependence is that its induced unary form satisfies κ 0 convexity for some κ 0 > 0 , summarized the following observation.

Observation 4 ( κ 0 -convexity and gradient bound of the unary loss.) . Consider f t : ( R d ) m → R + satisfying Assumption 5, Assumption 6, and Assumption 7. Then, the induced unary form ¯ f t : R d → R + in Definition 2 satisfies the following two properties:

( κ 0 -convexity) ¯ f t is κ 0 -convex with κ 0 = β f /α f , H = H t : α f H t ⪯ ∇ 2 ¯ f t ( z ) ⪯ β f H t , where H t = G ⊤ t G t as in Assumption 5, ∀ z ∈ R d .

(Gradient bound) ∥∇ ¯ f t ( z ) ∥ 2 ≤ G f √ m , ∀ z ∈ Z t .

Proof of Observation 4. By our assumption on the affine structure of f t in Assumption 5, ∀ z ∈ R d , ∇ 2 ¯ f t ( z ) = G ⊤ t ∇ 2 ℓ t ( B t + G t z ) G t . κ 0 -convexity follows from the curvature assumption in Assumption 6. For any z 1 , z 2 ∈ R d , we have that | ¯ f t ( z 1 ) -¯ f t ( z 2 ) | ≤ G f ∥ ( z 1 , · · · , z 1 ) -( z 2 , · · · , z 2 ) ∥ 2 = G f √ m ∥ z 1 -z 2 ∥ 2 .

2.3 Notations

For a positive semidefinite square matrix H ⪰ 0 ∈ R d × d , define the norm ∥ · ∥ H on R d so that ∥ v ∥ 2 H = v ⊤ Hv . We write c t to denote the cost function of the control problem, and f t to denote the loss function for BCO with memory. For two sets A,B , A + B = { a + b : a ∈ A,b ∈ B } . For a set S , | S | denotes the cardinality of S . We use B d , S d -1 to denote the unit ball and unit sphere in R d , respectively. For n vectors v 1 , . . . , v n ∈ R d , we slightly abuse notation and shorthand as v 1: n to denote the concatenated vector ( v 1 , . . . , v n ) ∈ R nd .

3 Improved Bandit Convex Optimization with Memory

In this section, we introduce an improved algorithm for the bandit convex optimization with memory problem. Our algorithm incorporates the occasional update idea from (Cassel and Koren, 2020) and the Newton-based update for κ 0 -convex functions from (Suggala et al., 2024), achieving an optimal ˜ O ( √ T ) regret bound, improving the previously best known ˜ O ( T 2 / 3 ) result from (Cassel and Koren, 2020). Besides the advancement on BCO-M, this result is the key component of our main result in control. First, we define the BCO-M instance.

Definition 5 (BCO-M instance) . Given d ∈ N , a BCO-M instance is parametrized by O = {K , m, { f t } t ≥ m,t ∈ N } , where K ⊂ R d is a convex compact set; m is the memory length; f t : K m → R + measures the instantaneous loss at time t . Given any bandit online learning with memory algorithm A B , the regret of A B on O w.r.t. z ∈ K is given by

where z t = z A B t is the decision according to A B at time t . We say a BCO-M instance O is affine and ( α, β, G, D ) -well-conditioned if O satisfies Assumption 5, Assumption 6 with α f = α, β f = β , Assumption 7 with G f = G,D f = D , and the adversary model satisfies Assumption 8.

3.1 BCO-M algorithm

We describe Algorithm 1 that runs bandit convex optimization for any BCO-M instance O = {K , m, { f t } t ≥ m,t ∈ N } . On a high level, our algorithm employs the occasional update idea from (Cassel and Koren, 2020). The decisions are updated at most once every m steps, ensured by the condition b t ∏ m -1 i =1 (1 -b t -i ) = 1 (line 8 in Algorithm 1), where b t is the Bernoulli random variable (partially) deciding whether the algorithm will make an update at the current time step. We write o as the original ideal prediction, v as the random perturbation vector, and z is the actual prediction (as the perturbed o ) of the algorithm.

Algorithm 1 Improved Bandit Convex Optimization with Affine Memory

Input: convex compact set K ⊂ R d , step size η > 0 , memory parameter m , curvature parameter α , time horizon T .

1: Initialize: o = · · · = o ∈ K , ˜ g = 0 , ˆ A = I , τ = 1

1 m 0: m -1 d 0: m -1 . 2: Sample v t ∼ S d -1 i.i.d. uniformly at random for t = 1 , . . . , m . 3: Set z t = o t + ˆ A -1 2 t -1 v t , t = 1 , . . . , m . 4: Draw b t ∼ Ber ( 1 m ) , t = 1 , . . . , m . 5: for t = m,... , T do 6: Play z t , observe f t ( z t -m +1: t ) , receive H t = G ⊤ t G t . 7: Draw b t ∼ Ber ( 1 m ) . 8: if b t ∏ m -1 i =1 (1 -b t -i ) = 1 then 9: Let s τ = t . 10: Update ˆ A t = ˆ A t -1 + ηα 2 H t . 11: Create gradient estimate: ˜ g t = df t ( z t -m +1: t ) ˆ A 1 2 t -1 v t ∈ R d . 12: Update o t +1 = ∏ ˆ A s τ -1 K [ o t -η ˆ A -1 s τ -1 ˜ g s τ -1 ] . 13: Sample v t +1 ∼ S d -1 uniformly at random, independent of previous steps. 14: Set z t +1 = o t +1 + ˆ A -1 2 t v t +1 . 15: τ ← τ +1 . 16: else 17: Set o t +1 = o t , v t +1 = v t , z t +1 = z t , ˆ A t = ˆ A t -1 , ˜ g t = ˜ g t -1 . 18: end if 19: end for

Such occasional update essentially reduces the BCO-M problem to a new no-memory BCO problem, whose equivalence will be shown later (see Appendix A.2). Consistent with the notation used in (Cassel and Koren, 2020), we denote the following set

̸

to be the set of time steps where the algorithm updates its decision. We readily have | S | ≤ T m . Moreover, we have

since t ∈ S implies b t -m +1 = · · · = b t -1 = 0 . Thus, whenever Algorithm 1 updates, it effectively updates with function value ¯ f t ( z t ) .

The occasional update alone only gives a sub-optimal ˜ O ( T 2 / 3 ) regret as shown in (Cassel and Koren, 2020). To achieve the optimal ˜ O ( √ T ) regret, we use a Newton-based update to exploit the κ 0 -convexity of general strongly-convex smooth functions, recently introduced by (Suggala et al., 2024).

For this improvement we require the knowledge of a Hessian estimator H t as in Assumption Assumption 5. This doesn't hold in BCO in general, but we will show in the next section that the control problem indeed satisfies this assumption: thanks to κ 0 -convexity, in the control problem H t can be constructed from the knowledge of system, instead of knowledge of loss which would typically incur a huge variance term.

The regret guarantee of Algorithm 1 is given by the following theorem.

Theorem 6 (BCO-M regret guarantee) . Given an ( α, β, G, D ) -well-conditioned BCO-M instance O = {K , m, { f t } t ≥ m,t ∈ N } with m = poly(log T ) and G,D = ˜ O (1) (Definition 5), let ( K , η = Θ(1 / √ T ) , m, α, T ) be the input to Algorithm 1. Algorithm 1 guarantees that

where ˜ O ( · ) hides all universal constants and logarithmic dependence in T .

√

Theorem 6 is the first algorithm to achieve the optimal ˜ O ( T ) regret bound for the BCO-M problem with general smooth convex loss. Compared with the ˜ O ( T 2 / 3 ) bound from (Cassel and Koren, 2020), our improvement exploits the new assumptions on the strong convexity of loss and the affine structure of memory. We notice that the ˜ O ( √ T ) regret bound of (Suggala et al., 2024) requires the additional quadratic loss assumption and uses the special structure of quadratic functions to construct low-biased gradient estimators for the unary form of losses. Algorithm 1 and its guarantee in Theorem 6 thus improve upon both of these previous results. The proof of Theorem 6 is lengthy, therefore we leave it to the appendix and include a sketch here.

3.2 Proof Sketch

The theorem is proven via reduction to a no-memory BCO algorithm with Newton-based updates.

Step 1: regret of the base algorithm. We devise a new BCO algorithm with two new ingredients different from standard algorithms. First, our algorithm adopts Newton-based updates which require estimating Hessians. In Assumption 5 we assume "free" access to a Hessian estimator H t , which holds in the control setting by utilizing the κ 0 -convexity property.

Second, we incorporate a new delay mechanism to decorrelate neighboring iterates such that o t is independent of recent perturbation vectors, which plays a crucial role in bounding the expectation of moving cost. Our base BCO algorithm is then shown to have an ˜ O ( √ T ) regret bound (see Lemma 11).

Step 2: reduction to the base algorithm. We show that the regret of Algorithm 1 can be controlled by the regret of the base algorithm plus a moving cost term. By the design of Algorithm 1, it updates a univariate loss function at most once every m steps. If all the z t across these m steps are the same, Regret (Algorithm 1) will be exactly the same as m × Regret (base algorithm). When z t are different, we suffer an additional moving cost E [ ∑ T t = m f t ( z t -m +1: t ) -¯ f t ( z t -m +1 ) ] , which can be bounded by the assumption on the gradient of f t if z t changes slowly.

Step 3: bounding the moving cost. We partition the moving cost f t ( z t -m +1: t ) -¯ f t ( z t -m +1 ) into three terms

Here, (a), (c) can be seen as perturbation loss suffered by the algorithm during exploration. (b) is the moving cost determined by the stability of the algorithm's neighboring iterates.

For the three terms in the above equation, (c) is bounded by Jensen's inequality using the convexity of the unary form of loss. We bound (a) using the curvature assumptions and the affine memory structure of f t , where we also make use of the delayed updates to decorrelate neighboring iterates for technical reasons. (b) is bounded by the Lipschitzness of f t and the distance between neighboring iterates. Theorem 6 is reached by putting the three steps together.

4 Optimal Regret for Bandit Non-stochastic Control

In this section, we show how to achieve optimal regret for the bandit non-stochastic control problem with a partially observable LTI dynamical system as described in Eq. (1) for strongly-convex smooth loss, by a reduction to the BCO-M algorithm (Algorithm 1) we devise in the previous section. Previously, the best known regret bound for this problem was ˜ O ( T 2 / 3 ) from (Cassel and Koren, 2020), which is also rooted in a reduction to BCO-M. We use a similar reduction, obtaining a better bound thanks to the improved regret bound of BCO-M (Theorem 6). We first give the formal definition of the control problem.

Definition 7 (Bandit non-stochastic control) . A bandit non-stochastic control problem of a partially observable LDS is parametrized by a tuple L = ( A,B,C,x 1 , ( w t ) t ∈ N , ( e t ) t ∈ N , ( c t ) t ∈ N , Π) , where A,B,C and ( w t ) t ∈ N , ( e t ) t ∈ N are the dynamics and perturbations in the LDS (Eq. (1)) with initial state x 1 (we assume x 1 =0 without loss of generality henceforth); c t measures the instantaneous cost at time t ; Π is the comparator control policy class. Given any bandit non-stochastic control algorithm A , the regret of A NC on L over a time horizon T ∈ N is given by

where ( y A NC t , u A NC t ) , ( y π t , u π t ) are the observation, control pair at time t following the trajectory of A NC and π , respectively. We say that a bandit non-stochastic control problem L is ( α, β, G, κ sys , κ, γ, R w , e , Y , U ) -well-conditioned if L and Π satisfy Assumption 1 with α c = α, β c = β, G c = G over some centered bounded convex domain Y × U ⊂ R d y + d u , Assumption 2 with κ sys , κ, γ , Assumption 3 with R w , e , and Assumption 4.

The reduction from partially observable bandit non-stochastic control to BCO-M consists of two main steps. The first step is to construct the would-be signal y t ( K ) from the observation y t , where y t ( K ) is used for updating but not directly observed by the controller. y t ( K ) is computed by using the Markov operator of the LDS (Simchowitz et al., 2020). The second step is a standard black-box reduction from control to BCO-M, which uses the strong stability of the system. Reduction is formalized in the following definition.

Definition 8 (Approximation) . A bandit non-stochastic control instance (Definition 7) L with some convex comparator class Π over a time horizon T ∈ N is said to be ε -approximated ( ε > 0 ) by a BCO-M instance (Definition 5) O with domain K = Π , if the existence of a BCO-M algorithm A B implies the existence of a bandit non-stochastic control algorithm A NC satisfying

The formal guarantee of the reduction is given below.

Lemma 9 (Control reduction) . Every instance of ( α c , β c , G c , κ sys , κ, γ, R w , e , Y , U ) -wellconditioned bandit non-stochastic control problem L over the ball Y × U ⊂ R d y + d u of radius R 4 with comparator class Π = M ( m,R M ) (Definition 1) is 2 G f DmT -1 -approximated by a ( α f , β f , G f , D ) -well-conditioned BCO-M instance O with

provided that m = Θ(log T/ log(1 / (1 -γ ))) .

Lemma 9 implies that every well-conditioned bandit non-stochastic control problem can be reduced to a well-conditioned BCO-M instance, whose parameters are polynomial in those of the control problem. As a result, any regret bound for BCO-M will directly transfer to the control problem (at the expense of polynomial dependence on the system parameters). In particular, combining Theorem 6 and Lemma 9, we obtain an optimal ˜ O ( √ T ) regret bound for the bandit non-stochastic control with strongly-convex smooth cost.

Theorem 10 (Bandit non-stochastic control regret guarantee) . Given an ( α, β, G, κ sys , κ, γ, R w , e , Y , U ) -well-conditioned bandit non-stochastic control instance L over the ball R B d y + d u ⊂ R d y + d u for the same R as in Lemma 9, with G,κ sys , κ, γ, R w , e , d x , d y , d u = ˜ O (1) 5 , let (Π , K , η, m, T, G, K, α ) be the input to Algorithm 2 with m = poly(log T ) . Algorithm 2 guarantees that

where ˜ O ( · ) hides all universal constants and logarithmic dependence in T .

Theorem 10 strictly improves Theorem 8 of (Cassel and Koren, 2020) and Theorem 5 of (Suggala et al., 2024), in the sense that our result achieves the optimal regret with fewer assumptions: all three results achieve the same ˜ O ( β α √ T ) regret bound, however (Cassel and Koren, 2020) requires the additional assumption that perturbations are stochastic, while (Suggala et al., 2024) requires the additional assumption that costs are quadratic.

Algorithm 2 Improved Bandit Non-stochastic Control

Input: Step size η > 0 , memory parameter m , DRC policy class M ( m,R M ) , time horizon T , system dynamics, ( κ, γ ) -strongly stabilizing linear policy K , strong convexity parameter α > 0 .

1: Let A B be an instance of Algorithm 1 with inputs ( K = M ( m,R M ) , η, m, α, T ) .

2: Initialize: M [ j ] 1 = · · · = M [ j ] m = 0 d u × d y , ∀ j ∈ [ m ] . ˜ g 0 = · · · = ˜ g m -1 = 0 md u d y , ˆ A 0 = ˆ A m -1 = mI md u d y × md u d y .

3: Initialize A B for t = 1 , · · · , m -1 (lines 1-4 in Algorithm 1).

4: Sample ξ t ∼ S md u d y -1 i.i.d. uniformly at random for t = 1 , . . . , m .

5: Play control u t = Ky t , incur cost c t ( y t , u t ) for t ∈ [ m ] .

6: for t = m,... , T do

7: Play control u M t t = Ky t + ∑ m -1 j =0 M [ j ] t y t -j ( K ) , incur cost c t ( y t , u t ) .

8: Let f t : ( M ( m,R M )) m → R be the induced with-memory loss function via reduction in Lemma 9 and H t be the associated Hessian estimator in Assumption 5.

9: Update M t +1 ←A B ( M t , { f s } t s = m , { H s } t s = m ) .

10: end for

5 Conclusion

In this paper, we devise an algorithm with an ˜ O ( √ T ) regret bound for the bandit non-stochastic control problem with adversarial strongly-convex smooth cost functions. This is the first result with optimal regret that simultaneously breaks the three assumptions of LQC (1) stochastic perturbation (2) full-information feedback (3) quadratic cost. Our control algorithm is built upon an improved algorithm for BCO-M, which may be of independent interest.

As a preliminary step to address the question of a general control theory for LTI, our result comes with limitations and potential future research directions. Currently, the improvement over the ˜ O ( T 2 / 3 regret by (Cassel and Koren, 2020) is made under the additional assumption of strong convexity. It's unclear whether this assumption is necessary for an ˜ O ( √ T ) regret, we leave determining the minimal assumption on the cost functions for optimal regret as an open question.

References

Agarwal, N., Bullins, B., Hazan, E., Kakade, S., and Singh, K. (2019a). Online control with adversarial disturbances. In International Conference on Machine Learning , pages 111-119. PMLR.

Agarwal, N., Hazan, E., and Singh, K. (2019b). Logarithmic regret for online control. Advances in Neural Information Processing Systems , 32.

Anava, O., Hazan, E., and Mannor, S. (2015). Online learning for adversaries with memory: price of past mistakes. Advances in Neural Information Processing Systems , 28.

Bellman, R. (1954). The theory of dynamic programming. Bulletin of the American Mathematical Society , 60(6):503-515.

Cassel, A. and Koren, T. (2020). Bandit linear control. Advances in Neural Information Processing Systems , 33:8872-8882.

Gradu, P., Hallman, J., and Hazan, E. (2020). Non-stochastic control with bandit feedback. Advances in Neural Information Processing Systems , 33:10764-10774.

Hazan, E., Agarwal, A., and Kale, S. (2007). Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2):169-192.

Hazan, E. and Singh, K. (2022). Introduction to online non-stochastic control. arXiv preprint arXiv:2211.09619 .

Kalman, R. E. (1960). A new approach to linear filtering and prediction problems.

Kalman, R. E. et al. (1960). Contributions to the theory of optimal control. Bol. soc. mat. mexicana , 5(2):102-119.

Lattimore, T. (2024). Bandit convex optimisation. arXiv preprint arXiv:2402.06535 .

Lattimore, T. and György, A. (2023). A second-order method for stochastic bandit convex optimisation. In The Thirty Sixth Annual Conference on Learning Theory , pages 2067-2094. PMLR.

Simchowitz, M. (2020). Making non-stochastic control (almost) as easy as stochastic. Advances in Neural Information Processing Systems , 33:18318-18329.

Simchowitz, M., Singh, K., and Hazan, E. (2020). Improper learning for non-stochastic control. In Conference on Learning Theory , pages 3320-3436. PMLR.

Suggala, A., Sun, Y. J., Netrapalli, P., and Hazan, E. (2024). Second order methods for bandit optimization and control. arXiv preprint arXiv:2402.08929 .

Suggala, A. S., Ravikumar, P., and Netrapalli, P. (2021). Efficient bandit convex optimization: Beyond linear losses. In Conference on Learning Theory , pages 4008-4067. PMLR.

Sun, Y. J., Newman, S., and Hazan, E. (2024). Optimal rates for bandit non-stochastic control. Advances in Neural Information Processing Systems , 36.

Yan, Y.-H., Wang, J., and Zhao, P. (2024). Handling heterogeneous curvatures in bandit lqr control. In Forty-first International Conference on Machine Learning .

Contents

A Proof of Theorem 6

The proof is reduction-based. It consists of three parts

We first consider an algorithm for the no-memory BCO problem, which uses Newton-based updates and a novel delay mechanism. This algorithm will serve as the base algorithm.

We then show that the regret of our main algorithm 1 can be bounded by the regret of the base algorithm plus a moving cost, via an improved analysis on κ 0 -convexity.

Finally, we bound the moving cost and then put every pieces together.

A.1 Base No-Memory BCO Algorithm and Guarantees

In this section, we prove Lemma 11, which establishes the regret guarantee for the base algorithm used by Algorithm 1. The base no-memory algorithm is a variant of the Newton-based (improper) BCO algorithm considered in Algorithm 1 of (Suggala et al., 2024). The main differences are:

Algorithm 1 in (Suggala et al., 2024) constructs Hessian estimators from the bandit feedback. Algorithm 3 doesn't require Hessian estimators, since the H t determining κ 0 -convexity is given by the system instead of loss in bandit control problems, and thus it's computable by the learner given the system parameters.

Algorithm 3 introduces an additional delay mechanism (specified by the delay parameter d 0 ∈ N ) to decorrelate neighboring iterates, a necessary ingredient to later analysis.

Algorithm 3 Simple BCO-with-delay

Input: convex compact set K ⊂ R d , step size η > 0 , delay parameter d 0 ∈ N , curvature parameter α > 0 , time horizon T ∈ N .

1: Initialize: o 1 ∈ K , ˆ A 0 = I d × d . ˆ A t = 0 d × d , ∀ t < 0 . ˜ g t = 0 , ∀ t ≤ 0 .

2: Sample v 1 ∼ S d -1 uniformly at random. Set z 1 = o 1 + ˆ A -1 2 0 v 1 .

3: for t = 1 , . . . , T do

4: Play z t , observe ¯ f t ( z t ) , receive H t .

5: Update ˆ A t = ˆ A t -1 + ηα 2 H t .

6: Create gradient estimate: ˜ g t = d ¯ f t ( z t ) ˆ A 1 2 t -1 v t ∈ R d .

7: Update o t +1 = ∏ ˆ A t -d 0 +1 K [ o t -η ˆ A -1 t -d 0 +1 ˜ g t -d 0 +1 ] .

8: Sample v t +1 ∼ S d -1 uniformly at random, independent of previous steps.

9: Set z t +1 = o t +1 + ˆ A -1 2 t v t +1 .

10: end for

Lemma 11 (Base BCO regret guarantee) . Suppose that the sequence of loss functions { ¯ f t } T t =1 and the convex compact set K satisfy the conditions in Assumption 7, Assumption 8, and the properties in Observation 4, and max { 1 , ∥ H t ∥ 2 } ≤ R H , ∀ t . With α = α f in the first condition in Observation 4, ( η, d 0 ) satisfying d 0 ≤ 2 / ( ηαR H ) , Algorithm 3 run with inputs ( K , η, d 0 , α, T ) satisfies the following regret guarantee: ∀ o ∈ K ,

In particular, by choosing η = Θ(1 / √ T ) and d 0 = ˜ Θ(1) , the above regret is of order ˜ O ( β α √ T ) .

Proof of Lemma 11. First, note that by the condition on η and d 0 , we have that ˆ A t ⪯ 2 ˆ A t -i , ∀ i ≤ d 0 . To see this, it is equivalent to proving any PSD matrix H with λ max ( H ) ≤ 1 satisfies that H ⪯ I , which follows from the fact that x ⊤ ( I -H ) x ≥ ∥ x ∥ 2 2 -λ max ∥ x ∥ 2 2 ≥ 0 , ∀ x .

By the convexity and curvature assumption on ¯ f t described by the conditions in Observation 4, we have

where the first-order term equals 0 since v t is drawn independently of o t , ¯ f t , ˆ A t -1 , and E [ v t ] = 0 . We can further bound the second order term by

where the first step follows from E [ v ⊤ Av | A ] = A · E [ vv ⊤ ] and the cyclic property of trace, and the last step follows from the definition of ˆ A t , the stability condition that ˆ A t ⪯ 2 ˆ A t -1 , and α = α f . Using standard inequalities on log determinant in Newton step analysis (Hazan et al., 2007), we have

Moreover, the curvature assumption on ¯ f t also implies

where the second step follows from E [˜ g t | o t ] = ∇ ¯ f t ( o t ) by Stoke's theorem. By the projection step in Line 7 of Algorithm 3, we have

where the last inequality follows from that ∀ H ⪰ 0 , ∥ x + y ∥ 2 H ≤ 2( ∥ x ∥ 2 H + ∥ y ∥ 2 H ) . Rearranging, we have

Note that ∀ t ,

where the second inequality in the second bound follows from the update rule in Line 7 of Algorithm 3, which gives ∥ o t -o t -1 ∥ 2 ˆ A t -d 0 ≤ ∥ η ˆ A -1 t -d 0 ˜ g t -d 0 ∥ 2 ˆ A t -d 0 = η 2 ∥ ˜ g t -d 0 ∥ 2 ˆ A -1 t -d 0 . Thus, we can further bound

Combining both and using that ˆ A t ⪯ 2 ˆ A t -i , ∀ i ≤ d 0 , we have that

As a result,

Combining, we have

A.2 Reduction

In this section, we prove Lemma 12, which states that the regret of BCO-M algorithm (Algorithm 1) can be related to the regret guarantee of the base no-memory BCO algorithm (Algorithm 3).

We follow the proof idea of (Cassel and Koren, 2020). In the previous section we have proved an ˜ O ( √ T ) regret bound for a 'no-memory' base BCO algorithm. Our main Algorithm 1, however, has memory dependence. When z t is changing slowly, Algorithm 1 is approximately a 'no-memory' algorithm, and we can do the following reduction: if all the z t across these steps are the same, Regret (Algorithm 1) will be exactly the same as m × Regret (base algorithm).

In this part, we show that when z t are different, we only suffer an additional moving cost

which we will bound in Appendix A.3.

Lemma 12 (Lemma 11, (Cassel and Koren, 2020)) . Let ( K , η, m, T ) be the input for Algorithm 1. Suppose that the loss functions { f t } T t = m and the convex compact set K satisfy Assumption 5, Assumption 6, Assumption 7, and Assumption 8. Then, the regret of Algorithm 1 with respect to any o ∈ K is upper bounded by

where R Algorithm 3 ( T m ) is the regret upper bound obtained in Lemma 11 with time horizon T m and d 0 = 2 .

Proof of Lemma 12. First, note that by assumption, the conditions in Lemma 11 are satisfied for the induced unary forms { ¯ f t } T t = m . The argument follows by reducing Algorithm 1 to Algorithm 3.

Denote χ t = b t ∏ m -1 i =1 (1 -b t -i ) as the indicator of whether the algorithm gets updated during round t . Recall the definition of S = { t ∈ [ T ] : χ t = 1 } in Eq. (4). The algorithm updates during round t if t ∈ S . For any t ∈ S , we have that χ t -1 = · · · = χ t -m +1 = 0 . Therefore, we have z t -m +1 = · · · =

z t by design of Algorithm 1. Thus, we have that f t ( z t -m +1: t ) = ¯ f t ( z t -m +1 ) = ¯ f t ( z t ) . Therefore, constrained to the time steps t ∈ S , Algorithm 1 is essentially running Algorithm 3 with a delay parameter d 0 = 2 (since in Line 12 of Algorithm 1, we update with the gradient information at time s τ -1 during round t ). Note that since | S | ≤ T m (Algorithm 1 updates at most once every m steps), we have that ∀ o ∈ K ,

It is left to relate the quantity on the left hand side of the above expression to the regret of Algorithm 1. Since χ t D = χ m (the two random variables equal in distribution), ∀ t , we have E [ χ t ] = E [ χ m ] . For any fixed o ∈ K , we have

For t ∈ N , denote F t to be the σ -algebra generated by the randomness of Algorithm 1 through sampling b s , v s up to time t , i.e. F t = σ ( { b s , v s } t s =1 ) . Since z t = z t -m +1 whenever χ t = 1 , b t -m +1: t are drawn independently of z t -m +1 and χ t is independent of F t -m by definition of χ t , we have

Together, Eq. (7) and Eq. (8) give that

Therefore, we have that

where the last inequality follows from

A.3 Bounding Moving Cost

Lemma 11 and Lemma 12 almost give the regret guarantee in Theorem 6. We are left with bounding the moving cost term in Lemma 12

(Cassel and Koren, 2020) bounded each of the summands by E [ δ t + ν t ] , where δ t = ∥ o t +1 -o t ∥ 2 , and ν t = ∥ z t -o t ∥ 2 . Making use of the κ 0 -convexity induced by the affine memory structure of non-stochastic control problems, we can establish a tighter bound that is necessary to obtain optimal regret in Theorem 6.

Lemma 13 (Moving cost) . Let ( K , η, m, T ) be the input for Algorithm 1. Suppose that the loss functions { f t } T t = m and the convex compact set K satisfy Assumption 5, Assumption 6, Assumption 7, and Assumption 8. Suppose m ≤ 2 / ( ηαR H ) . Then, the iterates output by Algorithm 1 satisfy

In particular, with η = Θ( √ T ) , m = poly(log T ) and κ ( G ) = Ω(1) by Assumption 5, we have that the above bound is of order ˜ O ( √ T ) .

Proof of Lemma 13. We can decompose the moving cost into three terms:

and we will bound each of the terms separately. In particular, ( a ) , ( c ) can be seen as perturbation loss suffered by the algorithm during exploration. ( b ) is the moving cost determined by the stability of the algorithm's neighboring iterates. We start with establishing bounds on ( a ) , ( c ) .

Perturbation loss. As before, we denote F t = σ ( { b s , v s } t s =1 ) to be the σ -algebra generated by the randomness of Algorithm 1 through sampling b s , v s up to time t . First, ( c ) ≤ 0 by Jensen's inequality for conditional expectations. In particular, recall that χ t = b t Π m -1 i =1 (1 -b t -i ) denotes whether the decision is updated at time t . For every t ∈ N , denote as T ( t ) := max { s < t | χ s = 1 } the last time that the algorithm updates its decision. It naturally holds that o t = o T ( t )+1 , v t = v T ( t )+1 by design of Algorithm 1. Therefore, we have

By the sampling rule in Line 13 of Algorithm 1, if the algorithm updates its decision at time t ( χ t = 1 ), then v t +1 is independent drawn from previous steps. On the other hand, o t +1 is measurable w.r.t. F t . Therefore, we have by Jensen's inequality for conditional expectations that

Summing up over t , we get that ( c ) ≤ 0 . We move on to bound ( a ) . We start with a (conditional) independence argument.

Independence argument. Fix t . Denote t 1 = T ( t ) to be the most recent time when the algorithm makes an update. Additionally, denote t 2 = T ( T ( t )) to be the second most recent time when the algorithm makes an update. We already have o t = o t 1 +1 . By the delayed update rule in Line 12 of Algorithm 1, we have that o t 1 +1 is updated with o t 1 = o t 2 +1 and gradient information ˜ g t 2 and thus is F t 2 -measurable. On the other hand, consider the sequence of random vectors v t -m +1 , . . . , v t . We have that v s = v T ( s )+1 for every t -m +1 ≤ s ≤ t . Since the algorithm updates at most once every m steps, we have that T ( s ) ≥ t 2 for all t -m +1 ≤ s ≤ t . Therefore, v t -m +1 , . . . , v t is independent of F t 2 . This observation de-correlates o t with v t -m +1 , . . . , v t conditioning on F t 2 .

For notation simplicity, for matrices A 1 , . . . , A n ∈ R d × d and vectors v 1 , . . . , v n ∈ R d , we slightly abuse notation and denote as A 1: n v 1: n = ( A 1 v 1 , . . . , A n v n ) ∈ R nd to be the concatenated matrixvector product of the two sequences.

We apply Taylor's theorem to the function f t and obtain that

where q t -m +1: t is some point that lies on the line segment connecting o t -m +1: t and z t -m +1: t . By the conditional independence argument, we have that the first order term vanishes: let t 2 = T ( T ( t )) ,

= 0

.

The above sum thus reduces to the sum of second-order terms:

where the first inequality follows from ˆ A s ⪯ ˆ A t for all s < t ; the second inequality follows from taking maximum over all concatenated unit vectors in R d ; the third inequality follows from the inequality between trace and spectral norm.

From this point on, the bound follows almost identically to the proof of Proposition 21 in (Suggala et al., 2024). We will reiterate the proof in a concise manner for completeness.

Note that by the assumption of affine memory structure (Assumption 5), we can write the following expression for the Hessian matrix of f t evaluated at q t -m +1: t as the following:

where q = B t + ∑ m -1 i =0 G [ i ] Y t -i q t -i , and W t -m + i = G [ m -i ] Y t -m + i , ∀ 1 ≤ i ≤ m . By the curvature assumption on ℓ t (Assumption 6), we can further bound the sum of diagonal blocks by

Therefore, we can further bound

Consider γ = ⌊ √ T ⌋ and endpoints k j = γ ( j -1) + m for j = 1 , . . . , J , and J = ⌊ T -m γ ⌋ . Using the fact that Trace ( AC ) ≤ Trace ( BC ) for any PSD matrices A,B,C with A ⪯ B , we can further decompose and bound the sum in the above expression by

where the first inequality follows by applying the radius bounds on Y t for the last T -Jγ terms and using the fact that ˆ A s ⪯ ˆ A t for any s < t ; the last inequality follows by applying Proposition 4.8 in (Simchowitz, 2020). Therefore, it suffices to bound the sum in the first term in expectation. First, recall that ∀ t , χ t = b t Π m -1 i =1 (1 -b i ) denotes whether Algorithm 1 updates during round t , and S = { m ≤ t ≤ T : χ t = 1 } denotes the set of all rounds where Algorithm 1 updates. To bound the following, we make use of the facts that (1) H t is oblivious, and (2) χ t is independent of ˆ A -1 k for any k ≤ t -m . Then, we have that

Using this and that ˆ A t ⪯ 2 ˆ A t -m , we have that

where the first inequality follows from ˆ A k j +1 -1 ⪯ max { 2 , ηαR H γ } ˆ A k j , and rest follows from the standard inequalities used in Newton-step analysis (Hazan et al., 2007).

Combining all the bounds, we have that

Movement cost. By design, the algorithm updates at most once in every m iterations. Therefore,

where s is the Euclidean distance between neighboring iterates in Algorithm 3. By analysis in Lemma 11, we have s ≤ ηdGD . By Lipschitz assumption on f t , we have

Combining, we have that the total moving cost is bounded by

A.4 Proof of Theorem 6

Combining result from Lemma 11, Lemma 12, and Lemma 13, we have the regret of Algorithm 1 w.r.t. to any z ∈ K is bounded by

and by choosing η = Θ ( 1 α √ T ) , we have the regret above is bounded by ˜ O ( β α GD √ T ) .

B Proof of Lemma 9

In this section, we prove Lemma 9. We begin by defining the Markov operator (Simchowitz et al., 2020).

Definition 14 (Markov operator) . Given a partially observable LDS instance parametrized by dynamics A ∈ R d x × d x , B ∈ R d x × d u , C ∈ R d y × d x satisfying Assumption 2 with ( κ, γ ) -strongly stabilizing K , define the Markov operator to be a sequence of matrices G = { G [ i ] } i ≥ 0 such that G 0 = [0; I d u ] and ∀ i > 0 , G [ i ] is given by

The next observation (Observation 15) relates y t ( K ) , the signals used by the DRC policies (Definition 1), to the observations and controls along the learner's trajectory. This justifies the accessibility of the signals.

Observation 15. Given a partially observable LDS instance with ( κ, γ ) -strongly stabilizing K , let G be the Markov operator in Definition 14. For t, m ∈ N , let M 1 , . . . , M t -1 ∈ R m × d u × d y be ( t -1) DRC matrices (Definition 1). Let ( y t , u t ) be the observation-control pair reached by playing M 1 , . . . , M t -1 for time step t = 1 , . . . , t -1 . Let u t ( K ) be the control at time t by executing the linear policy K , i.e. y t ( K ) = Ku t ( K ) , and y t ( K ) be the would-be observation had K been executed from the beginning of the time. Then, ( y t , u t ) and ( y t ( K ) , u t ( K )) can be related by the following equality:

In particular, Eq. (9) implies that y t ( K ) can be computed by the learner through access to the Markov operator G and the observations along the learner's own trajectory.

With Eq. (9), we are almost ready to reduce the control instance to the BCO-M problem. One delicate detail is that the BCO-M problem is defined for vector-valued decisions, and the the space M ( m,R M ) is a space of sequences of matrices. For clarity of presentation, we define the following embedding operators.

Definition 16 (Embedding operators) . For m,d y , d u ∈ N , denote d = md y d u . The embedding operator e : ( R ( d u × d y ) ) m → R d is the natural embedding of a DRC controller M = M [0: m -1] (Definition 1) in R d . In particular, ∀ k ∈ [ m -1] , i ∈ [ d u ] , j ∈ [ d y ] ,

Let e y : ( R d y ) m → R d u × d be given by ∀ y t -m +1 , . . . , y t ∈ R d y , ∀ i ∈ [ d u ] , k ∈ [ m ] ,

Proof of Lemma 9. Let B t = ( y t ( K ) , Ky t ( K )) + ∑ t i = m G [ i ] Y t -i e ( M t -i ) ∈ R d u + d y , ∀ t . Let Y t = e y ( y t -m +1: t ( K )) . Then, by Eq. (9), we have

We can't directly define our f t as this function because the ∑ t i = m G [ i ] Y t -i e ( M t -i ) term in B t depends on historical steps, which will lead to an unbounded memory.

To this end, we define

which is independent of M t -m +1: t , and Y t is independent of M 1: T . Note that by the loss function with memory has an affine memory structure. Moreover, G [ i ] and Y t -i are computable by the learner given system parameters. By the choice of m = Θ(log T ) , the Lipschitzness of c t , and the norm-decaying property of G [ i ] due to the stability of the system, we can bound the distance

In particular, we can choose m such that this term is O ( 1 T 2 ) , then any regret bound on f t in the BCO-M setting directly transfers to the same regret bound on c t in the control setting, at a negligible o (1) cost which will be subsumed by the approximation error term.

We are left with two steps to conclude this lemma. First, we need to show that f t indeed satisfies the conditions in Definition 5 and specify the constants. Next, because f t is not equal to c t while we only have access to c t , the gradient estimator for f t constructed from c t is biased. We need to bound this error and show that it has negligible impact on the regret bound.

Step 1: We denote the unary form to be ¯ f t as in Definition 2 . Let O = {M ( m,R M ) , m, { f t } t ≥ m,t ∈ N } be the associated BCO-M instance (Definition 5).

By construction, f t satisfies Assumption 8 and Assumption 5 other than the assumption on positive convolution invertibility-modulus if the truncated vector ( y t ( K ) , Ky t ( K ))+ ∑ m -1 i =0 G [ i ] Y t -i e ( M t -i ) always lives in the R B d y + d u and the matrix parameters are bounded. For the positive convolution invertibility-modulus, Lemma 3.1 in (Simchowitz, 2020) proves that the G [0] , . . . , G [ m -1] induced by the Markov operator in Definition 14 satisfies the positive convolution invertibility-modulus, i.e. κ ( G ) = Ω(1) . By assumption on c t , the curvature assumption in Assumption 6 is also satisfied. It is left to check the diameter bounds in Assumption 5 and Assumption 7.

First, we establish a diameter bound on our learning comparator set K = M ( m,R M ) .

Algorithm 2 essentially calls the improper BCO-M algorithm in Algorithm 1. With the diameter bound on M ( m,R M ) , we know that the matrices governing the controls picked by Algorithm 2 during each round lives in the set M ( m,R M + m ) . For simplicity, we may assume that R M ≥ m and thus the decisions made by Algorithm 2 live in M ( m, 2 R M ) .

Now, we are ready to prove the gradient bound on f t and the bounds on the observation-control pairs produced by Algorithm 2.

Gradient bound of f t . First, we bound the sum of operator norms for the Markov operator. By Assumption 2 and the fact that ∀ M ∈ R n × n , ∥ M ∥ op ≤ √ n ∥ M ∥ max ≤ √ n ∥ M ∥ 2 , we have that

The signals y t ( K ) has the following norm bound:

By the unfolding of ( y t , u t ) in Eq. (9), the observation and control pair ( y t , u t ) has the following norm bound: ∀ t ,

The above bound holds similarly for the truncated (ˆ y t , ˆ u t ) := ( y t ( K ) , Ky t ( K )) + ∑ m -1 i =0 G [ i ] Y t -i e ( M t -i ) in Eq. (10), which shows that (ˆ y t , ˆ u t ) ∈ R B d y + d u . This also completes checking Assumption 5.

By assumption, c t obeys the Lipschitz bounds

The gradient bound on f is given by

Step 2: Clearly the gradient estimator obtained for f t is biased. We denote this error as a t which has a norm bound ∥ a t ∥ 2 = O ( 1 T ) with the same reasoning on bounding | c t -f t | . Now, we only need to show that Algorithm 1 can tolerate such perturbation with no loss in regret.

It's known that the OGD algorithm can tolerate per-round adaptive gradient perturbation with norm bounded by O ( 1 T ) , with no cost in regret. We will prove a similar argument for Algorithm 3. Suppose in line 6, the unbiased gradient estimator ˜ g t is replaced by a biased estimator ˆ g t = ˜ g t + a t , then we want to extend the regret bound for Algorithm 3 in Lemma 11 in this setting with an additional term depending on the sum of magnitude of a t . The analysis in the proof of Lemma 11 remains unchanged except when we bound the regret for the sequence { o t } T t =1 in Eq. (6), the inequality becomes

The rest of proof after Eq. (6) is indentical, except that we replace each appearance of ˜ g t by ˆ g t .

Therefore, the regret incurred by using ˆ g t instead of ˜ g t as the gradient estimator is bounded by the regret upper bound in Lemma 11 and the additional term ∑ T t =1 E [ a ⊤ t ( o t -o )] ≤ aDT , where a = max t ∈ [ T ] ∥ a t ∥ 2 . Since a = O ( 1 T ) , the additional regret is bounded by O (1) .

Concluding the lemma: Denote

We have that by Assumption 4 that

where we have

Recall that

Combining, we have

which concludes the claim that O 2 G f DmT -1 -approximates L .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: we make sure all the claims are accurate.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: we discuss the limitations and future directions for improvement in the Discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: we provide the full set of assumptions for all theoretical results. We provide a complete (and correct) proof to for all theoretical results, among which the simpler ones are left to the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: the paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: the paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: the paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: the paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: we carefully follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: there is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: the paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: the paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studys online control with adversarial pertubations, bandit feedback and adversarial strongly-convex smooth cost functions. This setting is more general than previous works and the authors successfully achieve $O(\sqrt{T})$ regret by leveraging occasional update and Newton-based update.

STRENGTHS:
1. This paper generalizes previous settings and get the optimal regret.

2. This paper is well-written and the intuition behind algorithm is explained clearly.

WEAKNESSES:
The technical contribution seems not strong, the analysis of Algorithm 1 (reduce to no-memory BCO, bound regret for base algorithm and moving cost) follows the proof sketch of [1]. The main change is replacing the base algorithm with Newton-based updates in Algorithm 2 of [2], ensuring a tighter bound by utilizing the $\kappa$-convexity and affine memory.


[1] Cassel, A. and Koren, T. (2020). Bandit linear control. Advances in Neural Information Processing Systems, 33:8872–8882.

[2] Suggala, A., Sun, Y. J., Netrapalli, P., and Hazan, E. (2024). Second order methods for bandit
optimization and control. arXiv preprint arXiv:2402.08929.

QUESTIONS:
What is the main techinical challenge when combining and adapting the proof in [1] and [2]?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

comments: SUMMARY:
This paper considers the problem of online non-stochastic control, focusing specifically on scenarios where the loss function is characterized by bandit feedback, strong convexity, and smoothness, and the noise is adversarial. Prior research has typically managed to achieve $O(\sqrt{T})$ regret under assumptions such as quadratic loss functions, full information feedback, or stochastic noise. This paper breaks these assumptions, demonstrating that an $O(\sqrt{T})$ regret bound can be achieved even in the presence of adversarial noise, bandit feedback, and strongly convex loss functions.

STRENGTHS:
This paper presents a direct theoretical improvement, offering significant advancements in the field. It is well-written and theoretically solid, providing a robust analysis and clear insights into the online non-stochastic control problem with adversarial noise, bandit feedback, and strongly convex loss functions.

WEAKNESSES:
1. The citation for Optimal rates for bandit non-stochastic control is incorrect; it was mistakenly written as NeurIPS 2024.

2.This paper could benefit from some additional discussion. While this work presents a significant improvement in a specific scenario of online bandit control, it is equally important to address the challenge of designing a single algorithm that can achieve theoretical guarantees across different scenarios simultaneously. For instance, you might consider the problem proposed by the recent work "Handling Heterogeneous Curvatures in Bandit LQR Control" from ICML 2024. I believe that discussing this issue in the related work and future work sections would add significant value to the paper.

QUESTIONS:
No questions.

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the Linear Quadratic Control (LQC) problem with adversarial perturbations, bandit feedback models, and non-quadratic cost. The authors propose an algorithm that achieves $\\mathcal{O}(\\sqrt{T})$ optimal regret for bandit non-stochastic control with strongly-convex and smooth cost functions in the presence of adversarial perturbations, which improves the known $\\mathcal{O}(T^{2/3})$ of the previous work of Cassel and Koren [2020].\n\nThe dynamic system (partially observable linear time-invariant (LTI)) is defined as in Eq.(1). This work is largely inspired by the previous work of Suggala et al. [2024] which achieves optimal regret guarantee in a more restricted setting.\n\nSTRENGTHS:\n1. Though I have skimmed the proof of several lemmas, the analysis part seems to be rigorous and mathematically correct. \n2. The delayed mechanism to de-correlate the recent m iterates looks interesting, which may be of use in the other delayed feedback setting.\n\nWEAKNESSES:\n1. The specific contribution of this work towards the previous work of Suggala et al. [2024]  is still a little bit unclear. According to Line 382 to 387, it seem that the most important algorithmic contribution is the delay mechanism.  \n2. Not certain what it means by \"preserves an estimation of Hessian $H_t$ for free\" in Line 236. It seems related to Assumption 5 which provides the $H_t$ to the learner directly at the end of each iteration. I wonder if this sort of assumptions is general, and whether it is reasonable in the LTI control problem. \n\nTypo:\n1. line 248, length to lengthy. \n2. Definition 3, $f_t$ should be $f$? \n\nOther than these two issues, I haven't observed any specific weaknesses in this work.\n\nQUESTIONS:\nThe questions are raised in the weakness section. I am willing to re-evaluate the scores if these questions are properly answered.\n\nLIMITATIONS:\nThis paper is pure theoretical and does not have any limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}