Title: Online Control with Adversarial Disturbance for Continuous-time Linear Systems
Venue: NeurIPS 2024
Accepted: yes
Keywords: control and dynamical system

online learning

continuous-time linear system.
Abstract: We study online control for continuous-time linear systems with finite sampling rates, where the objective is to design an online procedure that learns under non-stochastic noise and performs comparably to a fixed optimal linear controller. 
We present a novel two-level online algorithm, by integrating a higher-level learning strategy and a lower-level feedback control strategy. This method offers a practical and robust solution for online control, which achieves sublinear regret. Our work provides the first nonasymptotic results for controlling continuous-time linear systems with finite number of interactions with the system. Moreover, we examine how to train an agent in domain randomization environments from a non-stochastic control perspective. By applying our method to the SAC (Soft Actor-Critic) algorithm, we achieved improved results in multiple reinforcement learning tasks within domain randomization environments. Our work provides new insights into non-asymptotic analyses of controlling continuous-time systems. Furthermore, our work brings practical intuition into controller learning under non-stochastic environments.

Online Control with Adversarial Disturbance for Continuous-time Linear Systems

Abstract

We study online control for continuous-time linear systems with finite sampling rates, where the objective is to design an online procedure that learns under nonstochastic noise and performs comparably to a fixed optimal linear controller. We present a novel two-level online algorithm, by integrating a higher-level learning strategy and a lower-level feedback control strategy. This method offers a practical and robust solution for online control, which achieves sublinear regret. Our work provides the first nonasymptotic results for controlling continuous-time linear systems with finite number of interactions with the system. Moreover, we examine how to train an agent in domain randomization environments from a non-stochastic control perspective. By applying our method to the SAC (Soft Actor-Critic) algorithm, we achieved improved results in multiple reinforcement learning tasks within domain randomization environments. Our work provides new insights into non-asymptotic analyses of controlling continuous-time systems. Furthermore, our work brings practical intuition into controller learning under non-stochastic environments.

1 Introduction

A major challenge in robotics is to deploy simulated controllers into real-world. This process, known as sim-to-real transfer, can be difficult due to misspecified dynamics, unanticipated real-world perturbations, and non-stationary environments. Various strategies have been proposed to address these issues, including domain randomization, meta-learning, and domain adaptation [20, 10, 21]. Although they have shown great effectiveness in experimental results, training agents within these setups poses a significant challenge. To accommodate different environments, the strategies developed by agents tend to be overly conservative [26, 4] or lead to suboptimal outcomes [43, 27].

In this work, we provide an analysis of the sim-to-real transfer problem from an online control perspective. Online control focuses on iteratively updating the controller after deployment (i.e.,

online) based on collected trajectories. Significant progress has been made in this field by applying insights from online learning to linear control problems [2, 1, 12, 19, 11, 8, 6, 16].

Following this line of work, we approach the sim-to-real transfer issue for continuous-time linear systems as a non-stochastic control problem, as explored in previous works [19, 11, 8]. These studies provide regret bounds for an online controller that lacks prior knowledge of system perturbations. However, a gap remains as no previous analysis has specifically investigated continuous-time systems, but real world systems often evolve continuously in time.

Existing literature on online continuous control is limited [42, 22, 13, 32]. Most continuous control research emphasizes the development of model-free algorithms, such as policy iteration, under the assumption of noise absence. Recently, [8] examined online continuous-time linear quadratic control problem and achieves sublinear regret. However, it relies on the assumption of standard Brownian noise instead of non-stochastic noise that may not always hold true in real-world applications. This leads us to the crucial question:

Is it possible to design an online non-stochastic control algorithm in a continuous-time setting that achieves sublinear regret?

Our work addresses this question by proposing a two-level online controller. The higher-level controller symbolizes the policy learning process and updates the policy at a low frequency to minimize regret. Conversely, the lower-level controller delivers high-frequency feedback control input to reduce discretization error. Our proposed algorithm results in regret bounds for continuoustime linear control in the face of non-stochastic disturbances.

Furthermore, we implement the ideas from our theoretical analysis and test them in several experiments. Note that the key difference between our algorithm and traditional online policy optimization is that we utilize information from past states with some skips to enable faster adaptation to environmental changes. Although the aforementioned concepts are often adopted experimentally as frame stacking and frame skipping, there is relatively little known about the appropriate scenarios for applying these techniques. Our analysis and experiments demonstrate that these techniques are particularly effective in developing adaptive policies for uncertain environments. We choose the task of training agents in a domain randomization environment to evaluate our method, and the results confirm that these techniques substantially improve the agents' performance.

2 Related Works

The control theory of linear dynamical systems under disturbances has been thoroughly examined in various contexts, such as linear quadratic stochastic control [7], robust control [37, 23], system identification [17, 24, 9, 25]. However, most of these problems are investigated in non-robust settings, with robust control being the sole exception where adversarial perturbations in the dynamic are permitted. In this scenario, the controller solves for the optimal linear controller in the presence of worst-case noise. Nonetheless, the algorithms designed in this context can be overly conservative as they optimize over the worst-case noise, a scenario that is rare in real-world applications. We will elaborate on the difference between robust control and online non-stochastic control in Section 3.

Online Control There has been a recent surge of interest in online control, as demonstrate by studies such as [2, 1, 12]. In online control, the player interacts with the environment and updates the policy in each round aiming to achieve sublinear regret. In scenarios with stochastic Gaussian noise, [12] provides the first efficient algorithm with an O ( √ T ) regret bound. However, in real-world applications, the assumption of Gaussian distribution is often unfulfilled.

[3] pioneers research on non-stochastic online control, where the noises can be adversarial. Under general convex costs, they introduce the Disturbance-Action Policy Class. Using an online convex optimization (OCO) algorithm with memory, they achieve an O ( √ T ) regret bound. Subsequent studies extend this approach to other scenarios, such as quadratic costs [8], partial observations [36, 35] or unknown dynamical systems [19, 11]. Other works yield varying theoretical guarantees like online competitive ratio [15, 33].

Online Continuous Control Compared to online control, there has been relatively little research on model-based continuous-time control. Most continuous control works focus on developing model-free

algorithms such as policy iteration (e.g. [42, 22, 32]), typically assuming zero-noise. This is because analyzing the system when transition dynamics are represented by differential equations, rather than recurrence formulas, poses a significant challenge.

Recently, [8] studies online continuous-time linear quadratic control with standard Brownian noise and unknown system dynamics. They propose an algorithm based on the least-square method, which estimates the system's coefficients and solves the corresponding Riccati equation. The papers [34, 14] also focus on online control setups with continuous-time stochastic linear systems and unknown dynamics. They achieve O ( √ T log T ) regret by different approaches. [34] uses the Thompson sampling algorithm to learn optimal actions. [14] takes a randomized-estimates policy to balance exploration and exploitation. The main difference between [8, 34, 14] and our paper is that they consider stochastic noise of Brownian motion which can be quite stringent and may fail in real-world applications, while the noise in our setup is non-stochastic. This makes our analysis completely different from theirs.

Domain Randomization Domain randomization, which is proposed by [39], is a commonly used technique for training agents to adapt to different (real) environments by training in randomized simulated environments. From the empirical perspective, many previous works focus on designing efficient algorithms for learning in a randomized simulated environment (by randomizing environmental settings, such as friction coefficient) such that the algorithm can adapt well in a new environment, [29, 44, 26, 28, 30]. Other works study how to effectively randomize the simulated environment so that the trained algorithm would generalize well in other environments [43, 27, 38]. However, prior research has not explored how to apply certain theoretical analysis ideas to train agents in domain-randomized environments. Limited previous works, such as [10] and [21], concentrate on theoretically analyzing the sim-to-real gap within specific domain randomization models but they do not test their algorithms in real domain randomization environments.

3 Problem Setting

In this paper, we consider the online non-stochastic control for continuous-time linear systems. Therefore, we provide a brief overview below and define our notations.

3.1 Continuous-time Linear Systems

The Linear Dynamical System can be considered a specific case of a continuous Markov decision process with linear transition dynamics. The state transitions are governed by the following equation:

where x t is the state at time t , u t is the action taken by the controller at time t , and w t represents the disturbance at time t . Follow the setup of [3], we assume x 0 = 0 . We do not make any strong assumptions about the distribution of w t , and we also assume that the distribution of w t is unknown to the learner beforehand. This implies that the disturbance sequence w t can be selected adversarially.

When the action u t is applied to the state x t , a cost c t ( x t , u t ) is incurred. Here, we assume that the cost function c t is convex. However, this cost is not known in advance and is only revealed after the action u t is implemented at time t . In the system described above, an online policy π is defined as a function that maps known states to actions, i.e., u t = π ( { x ξ | ξ ∈ [0 , t ] } ) . Our goal, then, is to design an algorithm that determines such an online policy to minimize the cumulative cost incurred. Specifically, for any algorithm A , the cost incurred over a time horizon T is:

In scenarios where the policy is linear (i.e., a linear controller), such that u t = -Kx t , we use J ( K ) to denote the cost of a policy K ∈ K from a certain class K .

3.2 Difference between Robust and Online Non-stochastic Control

While both robust and online non-stochastic control models incorporate adversarial noise, it's crucial to understand that their objectives differ significantly.

The objective function for robust control, as seen in [37, 23], is defined as:

Meanwhile, the objective function for online non-stochastic control, as discussed in [3], is:

Note that the robust control approach seeks to directly minimize the cost function, while online non-stochastic control targets the minimization of regret, which is the discrepancy between the actual cost and the cost associated with a baseline policy. Additionally, in robust control, the noise at each step can depend on the preceding policy, whereas in online non-stochastic control, all the noise is predetermined (though unknown to the player).

3.3 Assumptions

We operate under the following assumptions throughout this paper. To be concise, we denote ∥ · ∥ as the L 2 operator norm of the vector and matrix. Firstly, we make assumptions concerning the system dynamics and noise:

Assumption 1. The matrices that govern the dynamics are bounded, meaning ∥ A ∥ ≤ κ A and ∥ B ∥ ≤ κ B , where κ A and κ B are constants. Moreover, the perturbation and its derivative are both continuous and bounded: ∥ w t ∥ , ∥ ˙ w t ∥ ≤ W , with W being a constant.

These assumptions ensure that we can bound the states and actions, as well as their first and secondorder derivatives. Next, we make assumptions regarding the cost function:

Assumption 2. The costs c t ( x, u ) are convex in x and u . Additionally, if there exists a constant D such that ∥ x ∥ , ∥ u ∥ ≤ D , then we have the following inequalities of the costs: | c t ( x, u ) | ≤ βD 2 , ∥∇ x c t ( x, u ) ∥ , ∥∇ u c t ( x, u ) ∥ ≤ GD , | c t 1 ( x, u ) -c t 2 ( x, u ) | ≤ L | t 1 -t 2 | D 2 , where β , G and L are constants corresponding to the cost function. This assumption implies that if the differences between states and actions are small, then the error in their cost will also be relatively small.

3.4 Strongly Stable Policy

We next describe our baseline policy class introduced in [12]. Note that the continuous system and the discrete system are different. If we consider the approximation over a relatively small interval h , we get

Therefore, if we consider the transition of a discrete system x i +1 = ˜ Ax i + ˜ Bu i + ˜ w i , we get the approximation ˜ A ≈ I + hA , ˜ B ≈ hB . Hence, we extend the definition of a strongly stable policy [12, 3] in the discrete system to the continuous system as follows:

Definition 1. A linear policy K is ( κ, γ ) -strongly stable if, for any h > 0 that is sufficiently small, there exist matrices L h , P such that I + h ( A -BK ) = PL h P -1 , with the following two conditions:

The norm of L h is strictly smaller than unity and dependent on h , i.e., ∥ L h ∥ ≤ 1 -hγ .

2. The controller and transforming matrices are bounded, i.e., ∥ K ∥ ≤ κ and ∥ P ∥ , ∥ ∥ P -1 ∥ ∥ ≤ κ . The above definition ensures the system can be stabilized by a linear controller K .

3.5 Regret Formulation

To evaluate the designed algorithm, we follow the setup in [12, 3] and use regret, which is defined as the cumulative difference between the cost incurred by the policy of our algorithm and the cost incurred by the best policy in hindsight. Let K denotes the class of strongly stable linear policies, i.e. K = { K : K is ( κ, γ ) -strongly stable } . Then we try to minimize the regret of algorithm:

4 Algorithm Design

In this section, we outline the design of our algorithm and formally define the concepts involved in deriving our main theorem. We summarize our algorithm design as follows:

First, we discretize the total time period T into smaller intervals of length h . We use the information at each point x h , x 2 h , . . . and u h , u 2 h , . . . to approximate the actual cost of each time interval, leveraging the continuity assumption. This process does introduce some discretization errors.

Next, we employ the Disturbance-Action policy (DAC) [3]. This policy selects the action based on the current time step and the estimations of disturbances from several past steps. This policy can approximate the optimal linear policy in hindsight when we choose suitable parameters. However, the optimal policy K ∗ is unknown, so we cannot directly acquire the optimal choice. To overcome this, we employ the OCO with memory framework [5] to iteratively adjust the DAC policy parameter M t to approximate the optimal solution M ∗ .

After that, we introduce the concept of the ideal state y t and ideal action v t that approximate the actual state x t and action u t . Note that both the state and policy depend on all DAC policy parameters M 1 , M 2 , . . . , M t . Yet, the OCO with memory framework only considers the previous H steps. Therefore, we need to consider ideal state and action. y t and v t represent the state the system would reached if it had followed the DAC policy { M t -H , . . . , M t } at all time steps from t -H to t , under the assumption that the state x t -H was 0 .

From all the analysis above, we can decompose the regret as three parts: the discretization error R 1 , the regret of the OCO with memory R 2 , and the approximation error between the ideal cost and the actual cost R 3 .

Then we will formally introduce out method and define all the concepts. In the subsequent discussion, we use shorthand notation to denote the cost, state, control, and disturbance variables c ih , x ih , u ih , and w ih as c i , x i , u i , and w i , respectively.

First, we need to define the Disturbance-Action Policy Class(DAC) for continuous systems:

Definition 2. The Disturbance-Action Policy Class(DAC) is defined as:

where K is a fixed strongly stable policy, l is a parameter that signifies the dimension of the policy class, M t = { M 1 t , . . . , M l t } is the weighting parameter of the disturbance at step t , and ˆ w t is the estimated disturbance:

We note that this definition differs from the DAC policy in discrete systems [3] as we utilize the estimation of disturbance over an interval [ t, t + h ] instead of only the noise in time t . It counteracts the second-order residue term of the Taylor expansion of x t and is also an online policy as it only requires information from the previous state.

Our higher-level controller adopts the OCO with memory framework. A technical challenge lies in balancing the approximation error and OCO regret. To achieve a low approximation error, we desire the policy update interval H to be inversely proportional to the sampling distance h . However, this relationship may lead to large OCO regret. To mitigate this issue, we introduce a new parameter m = Θ( 1 h ) , representing the lookahead window. We update the parameter M t only once every m iterations, further reducing the OCO regret without negatively impacting the approximation error:

Where g t is a function corresponding to the loss function c t and we will introduce later in Algorithm 1. For notational convenience and to avoid redundancy, we denote ˜ M [ t/m ] = M t . We can then define the ideal state and action. Due to the properties of the OCO with memory structure, we need to consider only the previous Hm states and actions, rather than all states. As a result, we introduce the definition of the ideal state and action. During the interval t ∈ [ im, ( i +1) m -1] , the learning policy remains unchanged, so we could define the ideal state and action follow the definition in [3]:

Definition 3. The ideal state y t and action v t at time t ∈ [ im, ( i +1) m -1] are defined as

where the notation indicates that we assume the state x t -H is 0 and that we apply the DAC policy ( ˜ M i -H , . . . , ˜ M i ) at all time steps from t -Hm to t .

We can also define the ideal cost in this interval follow the definition in [3]:

Definition 4. The ideal cost function during the interval t ∈ [ im, ( i +1) m -1] is defined as follows:

With all the concepts presented above, we are now prepared to introduce our algorithm:

Algorithm 1 Continuous two-level online control algorithm

Input: step size η , sample distance h , policy update parameters H,m , parameters κ, γ, T . Define sample numbers n = ⌈ T/h ⌉ , OCO policy update times p = ⌈ n/m ⌉ .

Denote the discretization time r = km + s . Use the action u t = -Kx r + h ∑ Hm i =1 ˜ M i k ˆ w r -i during the time period t ∈ [ rh, ( r +1) h ] . Observe the new state x r +1 at time ( r +1) h and record ˆ w r according to Equation (1).

end for

Define the function g k ( M ) = f k ( M,...,M ) .

Update OCO policy end for

5 Main Result

In this section, we present the primary theorem of online continuous control regret analysis:

Theorem 1. Under Assumption 1, 2, a step size of η = Θ( √ m Th ) , and a DAC policy update frequency m = Θ( 1 h ) , Algorithm 1 attains a regret bound of

With the sampling distance h = Θ( 1 √ T ) , and the OCO policy update parameter H = Θ(log( T )) , Algorithm 1 achieves a regret bound of

Theorem 1 demonstrates a regret that matches the regret of a discrete system [3]. Despite the analysis of a continuous system differing from that of a discrete system, we can balance discretization error, approximation error, and OCO with memory regret by selecting an appropriate update frequency for the policy. Here, O ( · ) and Θ( · ) are abbreviations for the polynomial factors of universal constants in the assumption.

While we defer the detailed proof to the appendix, we outline the key ideas and highlight them below.

˜

M

= Π

(

˜

M

-

η

∇

k

+1

M

k

g

k

( ˜

M

k

)

)

.

Challenge and Proof Sketch We first explain why we cannot directly apply the methods for discrete nonstochastic control from [3] to our work. To utilize Assumption 2, it is necessary first to establish a union bound over the states. In a discrete-time system, it can be easily proved by applying the dynamics inequality ∥ x t +1 ∥ ≤ a ∥ x t ∥ + b (where a < 1 ) and the induction method presented in [3]. However, for a continuous-time system, a different approach is necessary because we only have the differential equation instead of the state recurrence formula.

To overcome this challenge, we employ Gronwall's inequality to bound the first and second-order derivatives in the neighborhood of the current state. We then use these bounded properties, in conjunction with an estimation of previous noise, to bound the distance to the next state. Through an iterative application of this method, we can argue that all states and actions are bounded.

Another challenge is that we need to discretize the system but we must overcome the curse of dimensionality caused by discretization. In continuous-time systems, the number of states is inversely proportional to the discretization parameter h , which also determines the size of the OCO memory buffer. Our regret is primarily composed of three components: the error caused by discretization R 1 , the regret of OCO with memory R 2 and the difference between the actual cost and the approximate cost R 3 . The discretization error R 1 is O ( hT ) , therefore if we achieve O ( √ T ) regret, we must choose h no more than O ( 1 √ T ) .

If we update the OCO with memory parameter at each timestep follow the method in [3], we will incur the regret of OCO with memory R 2 = O ( H 2 . 5 √ T ) . The difference between the actual cost and the approximate cost R 3 = O ( T (1 -hγ ) H ) . To achieve sublinear regret for the third term, we must choose H = O ( log T hγ ) , but since h is no more than O ( 1 √ T ) , H will be larger than Θ( √ T ) , therefore the second term R 2 will definitely exceed O ( √ T ) .

Therefore, we adjust the frequency of updating the OCO parameters by introducing a new parameter m , using a two-level approach and update the OCO parameters once in every m steps. This will incur the third term R 3 = O ( T (1 -hγ ) Hm ) but keep the OCO with memory regret R 2 = O ( H 2 . 5 √ T ) , so we can choose H = O ( log T γ ) and m = O ( 1 h ) . Then the term of R 2 is O ( √ T log T ) and we achieve the same regret compare with the discrete system.

6 Experiments

In this section, we apply our theoretical analysis to the practical training of agents. First we highlight the key difference between our algorithm and traditional online policy optimization.

Stack: While standard online policy optimization learns the optimal policy from the current state u t = ϕ ( x t ) , an optimal non-stochastic controller employs the DAC policy as outlined in Definition 2. Leveraging information from past states aids the agent in adapting to dynamic environments.

Skip: Different from the analysis in [3], in a continuous-time system we update the state information every few steps, rather than updating it at every step. This solves the curse of dimensionality caused by discretization in continuous-time system.

The above inspires us with an intuitive strategy for training agents by stacking past observations with some observations to skip. We denote this as Stack & skip for convenience. Stack & skip is frequently used as a heuristic in reinforcement learning, yet little was known about when and why such a technique could boost agent performance.

How should we evaluate our algorithm in a non-stochastic environment? We opt for learning an optimal policy within a domain randomization environment. In this context, each model's parameters are randomly sampled from a predetermined task distribution. We train policies to optimize performance across various simulated models [41, 29].

We observe that learning in Domain Randomization (DR) significantly differs from stochastic or robust learning problems. In DR, sampling from environmental variables occurs at the beginning of each episode, rather than at every step, distinguishing it from stochastic learning where randomness is step-wise independent and identically distributed. This episodic sampling approach allows agents in DR to exploit environmental conditions and adapt to episodic changes

ReplayBuffer

RandomizedEnvironment

Observation

Stack&Skip

ObservationAugmentationModule

RLAgent

within an episode. On the other hand, robust learning focuses on worst-case scenarios depending on an agent's policy. DR, in contrast, is concerned with the distribution of conditions aimed at broad applicability rather than worst-case perturbations.

In the context of non-stochastic control, the disturbance, while not disclosed to the learner beforehand, remains fixed throughout the episode and does not adaptively respond to the control policy. This setup in non-stochastic control shows a clear parallel to domain randomization: fixed yet unknown disturbances in non-stochastic control mirror the unknown training environments in DR. As the agent continually interacts with these environments, it progressively adapts, mirroring the adaptive process observed in domain randomization. Therefore, we propose evaluating our algorithm within a domain randomization training task. Subsequently, we introduce the details of our experimental setup:

Environment Setting We conduct experiments on the hopper, halfcheetah, and walker2d benchmarks using the MuJoCo simulator [40]. The randomized parameters include environmental physical parameters such as damping and friction, as well as the agent properties such as torso size. We set the range of our domain randomization to follow a distribution with default parameters as the mean value, shown in Table 1. When training in the domain randomization environment, the parameter is uniformly sampled from this distribution. To an-

alyze the result of generalization, we only change one of the parameters and keep the other parameters as the mean of its distribution in each test environment. We conducted experiments using NVIDIA A40 graphics card.

Algorithm Design and Baseline We design a practical meta-algorithm that converts any standard deep RL algorithm into a domain-adaptive algorithm, shown in Figure 2. In this algorithm, we augment the original state observation o old t at time t with past observations, resulting in o new t = [ o old t , o old t -m , . . . , o old t -( h -1) m ] . Here h is the number of past states we leverage and m is the number of states we skip when we get each of the past states. For clarity in our results, we selected the SAC algorithm for evaluation. We use a variant of Soft Actor-Critic (SAC) [31] and leverage past states with some skip as our algorithm. We compare our algorithm with the standard SAC algorithm training on domain randomization environments as our baseline.

Impact of Frame Stack and Frame Skip To understand the effects of the frame stack number h and frame skip number m , we carried out experiments in the hopper environment with different h and m . For each parameter we train with 3 random seeds and take the average. Figure 3 shows that the performance increases significantly when the frame stack number is increased from 1 to 3 , and

remains roughly unchanged when the frame stack number continues to climb up. Figure 4 shows that the optimal frame skip number is 3 , while both too large or too small frame skip numbers result in sub-optimal results. Therefore, in the following experiments we fix the parameter h = 3 , m = 3 . We train our algorithm with this parameter and standard SAC on hopper and test the performance on more environments. Figure 5 shows that our algorithm outperforms the baseline in all environments.

360

360

360

360

SAC

SAC

SAC with skip interval 1

SAC with skip interval 1

SAC with frame stack 2

SAC with frame stack 2

SAC with skip interval 3

SAC with skip interval 3

350

SAC with frame stack 3

350

SAC with frame stack 3

350

SAC with skip interval 5

350

SAC with skip interval 5

SAC with frame stack 5

SAC with frame stack 5

340

340

340

340

Reward

Reward

Reward

Reward

330

330

330

330

320

320

320

320

310

310

310

310

300

1.0

1.5

2.0

2.5

3.0

300

1.2

1.3

1.4

1.5

1.6

1.7

300

1.0

1.5

2.0

2.5

3.0

300

1.2

1.3

1.4

1.5

1.6

1.7

foot_friction

head_height

foot_friction

head_height

360

360

360

360

SAC

SAC

SAC

SAC

350

SAC with history

350

SAC with history

350

SAC with history

350

SAC with history

340

340

340

340

Reward

Reward

Reward

Reward

330

330

330

330

320

320

320

320

310

310

310

310

300

300

300

300

0.50

0.75

1.00

1.25

1.50

1.0

1.5

2.0

2.5

3.0

1.2

1.4

1.6

0.03

0.04

0.05

0.06

0.07

joint_damping

foot_friction

head_height

torso_size

Results on Other Environments Each algorithm was trained using three distinct random seeds in the half-cheetah and walker2d domain randomization (DR) environments. Consistent with previous experiments, we employed a frame stack number of h = 3 and frame skip number of m = 3 . The comparative performance of our algorithm and the baseline algorithm, across various domain parameters, is presented in Figure 6. The result clearly demonstrates that our algorithm consistently outperforms the baseline in all evaluated test environments.

1050

1050

1050

SAC

SAC

SAC

1025-

SAC with history

1025

SAC with history

1025

SAC with history

1000

1000

1000

975

975

975

Reward

Reward

Reward

950

950

950

925

925

925

006

006

900

875

875

875

850

0.0050 0.00750.0100 0.0125 0.0150

850

3

4

5

6

7

850

0.0400.0450.0500.0550.060

joint_damping

foot_friction

torso_size

400

400

400

SAC

SAC

SAC

380

SACwithhistory

380-

SACwithhistory

380

SACwithhistory

360

360

360

Reward

340

Reward

340

Reward

340-

320

320

320-

300

300

300

280

280

280-

260

260

260-

0.06

0.080.100.12

0.14

600

800100012001400

0.03

0.040.05

0.06

0.07

joint_damping

density

torso_size

7 Conclusion, Limitations and Future Directions

In this paper, we propose a two-level online controller for continuous-time linear systems with adversarial disturbances, aiming to achieve sublinear regret. This approach is grounded in our examination of agent training in domain randomization environments from an online control perspective. At the higher level, our controller employs the Online Convex Optimization (OCO) with memory framework to update policies at a low frequency, thus reducing regret. The lower level uses the DAC policy to align the system's actual state more closely with the idealized setting.

In our empirical evaluation, applying our algorithm's core principles to the SAC (Soft Actor-Critic) algorithm led to significantly improved results in multiple reinforcement learning tasks within domain randomization environments. This highlights the adaptability and effectiveness of our approach in practical scenarios.

It is important to note that our theoretical analysis depends on the known dynamics of the system and the assumption of convex costs. This reliance could represent a limitation to our method, as it may not adequately address scenarios where these conditions do not hold or where system dynamics are incompletely understood. For future research, there are several promising directions in online non-stochastic control of continuous-time systems. These include extending our methods to systems with unknown dynamics, exploring the impact of assuming strong convexity in cost functions, and shifting the focus from regret to the competitive ratio. Further research can also explore how to utilize historical information more effectively to enhance agent training in domain randomization environments. This might involve employing time series analysis instead of simply incorporating parameters into neural network training.

References

Yasin Abbasi-Yadkori, Peter Bartlett, and Varun Kanade. Tracking adversarial targets. In International Conference on Machine Learning , 2014.

Yasin Abbasi-Yadkori and Csaba Szepesvári. Regret bounds for the adaptive control of linear quadratic systems. In Proceedings of the 24th Annual Conference on Learning Theory , 2011.

Naman Agarwal, Brian Bullins, Elad Hazan, Sham Kakade, and Karan Singh. Online control with adversarial disturbances. In International Conference on Machine Learning , 2019.

Artemij Amiranashvili, Max Argus, Lukas Hermann, Wolfram Burgard, and Thomas Brox. Pre-training of deep rl agents for improved learning under domain randomization. arXiv preprint arXiv:2104.14386 , 2021.

Oren Anava, Elad Hazan, and Shie Mannor. Online learning for adversaries with memory: price of past mistakes. Advances in Neural Information Processing Systems , 2015.

Lachlan Andrew, Siddharth Barman, Katrina Ligett, Minghong Lin, Adam Meyerson, Alan Roytman, and Adam Wierman. A tale of two metrics: Simultaneous bounds on competitiveness and regret. In Conference on Learning Theory , pages 741-763. PMLR, 2013.

Michael Athans. The role and use of the stochastic linear-quadratic-gaussian problem in control system design. IEEE transactions on automatic control , 16(6):529-552, 1971.

Matteo Basei, Xin Guo, Anran Hu, and Yufei Zhang. Logarithmic regret for episodic continuoustime linear-quadratic reinforcement learning over a finite-time horizon. Journal of Machine Learning Research , 2022.

Marco C Campi and PR Kumar. Adaptive linear quadratic gaussian control: the cost-biased approach revisited. SIAM Journal on Control and Optimization , 36(6):1890-1907, 1998.

Xiaoyu Chen, Jiachen Hu, Chi Jin, Lihong Li, and Liwei Wang. Understanding domain randomization for sim-to-real transfer. In International Conference on Learning Representations , 2022.

Xinyi Chen and Elad Hazan. Black-box control for linear dynamical systems. In Conference on Learning Theory , 2021.

Alon Cohen, Avinatan Hasidim, Tomer Koren, Nevena Lazic, Yishay Mansour, and Kunal Talwar. Online linear quadratic control. In International Conference on Machine Learning , 2018.

Tyrone E Duncan, Petr Mandl, and Bo˙ zenna Pasik-Duncan. On least squares estimation in continuous time linear stochastic systems. Kybernetika , 28(3):169-180, 1992.

Mohamad Kazem Shirani Faradonbeh and Mohamad Sadegh Shirani Faradonbeh. Online reinforcement learning in stochastic continuous-time systems. In The Thirty Sixth Annual Conference on Learning Theory , pages 612-656. PMLR, 2023.

Gautam Goel, Naman Agarwal, Karan Singh, and Elad Hazan. Best of both worlds in online control: Competitive ratio and policy regret. arXiv preprint arXiv:2211.11219 , 2022.

Gautam Goel and Adam Wierman. An online algorithm for smoothed regression and lqr control. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2504-2513. PMLR, 2019.

Graham C Goodwin, Peter J Ramadge, and Peter E Caines. Discrete time stochastic adaptive control. SIAM Journal on Control and Optimization , 19(6):829-853, 1981.

Elad Hazan. Introduction to online convex optimization. CoRR , abs/1909.05207, 2019.

Elad Hazan, Sham Kakade, and Karan Singh. The nonstochastic control problem. In Algorithmic Learning Theory , 2020.

Sebastian Höfer, Kostas Bekris, Ankur Handa, Juan Camilo Gamboa, Melissa Mozifian, Florian Golemo, Chris Atkeson, Dieter Fox, Ken Goldberg, John Leonard, et al. Sim2real in robotics and automation: Applications and challenges. IEEE transactions on automation science and engineering , 2021.

Jiachen Hu, Han Zhong, Chi Jin, and Liwei Wang. Provable sim-to-real transfer in continuous domain with partial observations. arXiv preprint arXiv:2210.15598 , 2022.

Yu Jiang and Zhong-Ping Jiang. Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica , 2012.

IS Khalil, JC Doyle, and K Glover. Robust and optimal control . Prentice hall, 1996.

PR Kumar. Optimal adaptive control of linear-quadratic-gaussian systems. SIAM Journal on Control and Optimization , 21(2):163-178, 1983.

Lennart Ljung. System identification . Springer, 1998.

Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher J Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning , pages 1162-1176. PMLR, 2020.

Melissa Mozian, Juan Camilo Gamboa Higuera, David Meger, and Gregory Dudek. Learning domain randomization distributions for training robust locomotion policies. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 6112-6117. IEEE, 2020.

Fabio Muratore, Christian Eilers, Michael Gienger, and Jan Peters. Data-efficient domain randomization with bayesian optimization. IEEE Robotics and Automation Letters , 6(2):911918, 2021.

Fabio Muratore, Michael Gienger, and Jan Peters. Assessing transferability from simulation to reality for reinforcement learning. IEEE transactions on pattern analysis and machine intelligence , 43(4):1172-1183, 2019.

Fabio Muratore, Theo Gruner, Florian Wiese, Boris Belousov, Michael Gienger, and Jan Peters. Neural posterior domain randomization. In Conference on Robot Learning , pages 1532-1542. PMLR, 2022.

Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning . PMLR, 2022.

Syed Ali Asad Rizvi and Zongli Lin. Output feedback reinforcement learning control for the continuous-time linear quadratic regulator problem. In 2018 Annual American Control Conference (ACC) , 2018.

Guanya Shi, Yiheng Lin, Soon-Jo Chung, Yisong Yue, and Adam Wierman. Online optimization with memory and competitive control. Advances in Neural Information Processing Systems , 33:20636-20647, 2020.

Mohamad Kazem Shirani Faradonbeh, Mohamad Sadegh Shirani Faradonbeh, and Mohsen Bayati. Thompson sampling efficiently learns to control diffusion processes. Advances in Neural Information Processing Systems , 35:3871-3884, 2022.

Max Simchowitz. Making non-stochastic control (almost) as easy as stochastic. Advances in Neural Information Processing Systems , 33:18318-18329, 2020.

Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non-stochastic control. In Conference on Learning Theory , pages 3320-3436. PMLR, 2020.

Robert F Stengel. Optimal control and estimation . Courier Corporation, 1994.

Gabriele Tiboni, Karol Arndt, and Ville Kyrki. Dropo: Sim-to-real transfer with offline domain randomization. Robotics and Autonomous Systems , 166:104432, 2023.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 23-30. IEEE, 2017.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, 2012.

Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 969-977, 2018.

Draguna Vrabie, O Pastravanu, Murad Abu-Khalaf, and Frank L Lewis. Adaptive optimal control for continuous-time linear systems based on policy iteration. Automatica , 2009.

Quan Vuong, Sharad Vikram, Hao Su, Sicun Gao, and Henrik I Christensen. How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies? arXiv preprint arXiv:1903.11774 , 2019.

Sergey Zakharov, Wadim Kehl, and Slobodan Ilic. Deceptionnet: Network-driven domain randomization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 532-541, 2019.

In the appendix we define n as the smallest integer greater than or equal to T h , and we use the shorthand c ih , x ih , u ih , and w ih as c i , x i , u i , and w i , respectively. First we provide the proof of our main theorem there.

A Proof of Theorem 1

Theorem 1. Under Assumption 1, 2, a step size of η = Θ( √ m Th ) , and a DAC policy update frequency m = Θ( 1 h ) , Algorithm 1 attains a regret bound of

With the sampling distance h = Θ( 1 √ T ) , and the OCO policy update parameter H = Θ(log( T )) , Algorithm 1 achieves a regret bound of

Proof. We denote u ∗ t = K ∗ x ∗ t as the optimal state and action that follows the policy specified by K ∗ , where K ∗ = arg max K ∈K J T ( K ) .

We then discretize and decompose the regret as follows:

where R 0 represents the discretization error.

We define p as the smallest integer greater than or equal to n m , then the first term can be further decomposed as

where the last equality is by the definition of the idealized cost function.

Let us denote

Then we have the regret decomposition as

We then separately upper bound each of the four terms.

The term R 0 represents the error caused by discretization, which decreases as the number of sampling points increases and the sampling distance h decreases. This is because more sampling points make our approximation of the continuous system more accurate. Using Lemma 3, we get the following upper bound: R 0 ≤ O ( hT ) .

The term R 1 represents the difference between the actual cost and the approximate cost. For a fixed h , this error decreases as the number of sample points looked ahead m increases, while it increases as the sampling distance h decreases. This is because the closer adjacent points are, the slower the convergence after approximation. By Lemma 4 we can bound it as R 1 ≤ O ( n (1 -hγ ) Hm ) .

The term R 2 is incurred due to the regret of the OCO with memory algorithm. Note that this term is determined by learning rate η and the policy update frequency m . Choosing suitable parameters and using Lemma 5, we can obtain the following upper bound: R 2 ≤ O ( √ n/h ) .

The term R 3 represents the difference between the ideal optimal cost and the actual optimal cost. Since the accuracy of the DAC policy approximation of the optimal policy depends on its degree of freedom l , a higher degree of freedom leads to a more accurate approximation of the optimal policy. We use Lemma 6 and choose l = Hm to bound this error: R 3 ≤ O ( n (1 -hγ ) Hm ) .

By summing up these four terms and taking m = Θ( 1 h ) , we get:

Finally, we choose h = Θ ( 1 √ T ) , m = Θ ( 1 h ) , H = Θ( log ( T )) , the regret is bounded by

B Key Lemmas

In this section, we will primarily discuss the rationale behind the proof of our key lemmas. First, we need to prove all the states and actions are bounded.

Lemma 2. Under Assumption 1 and 2, choosing arbitrary h in the interval [0 , h 0 ] where h 0 is a constant only depends on the parameters in the assumption, we have for any t and policy M i , ∥ x t ∥ , ∥ y t ∥ , ∥ u t ∥ , ∥ v t ∥ ≤ D , ∥ ˙ x t ∥ ≤ D , ∥ x t -y t ∥ , ∥ u t -v t ∥ ≤ κ 2 (1 + κ )(1 -hγ ) Hm +1 D . In particular, taking all the M t = 0 and K = K ∗ , we can also obtain the inequality of the optimal solution: ∥ x ∗ t ∥ , ∥ u ∗ t ∥ ≤ D .

The proof of this Lemma mainly use the Gronwall inequality and the induction method. Then we analyze the discretization error of the system.

Lemma 3. Under Assumption 2, Algorithm 1 attains the following bound of R 0 :

This lemma indicates that the discretization error is directly proportional to the sample distance h . In other words, increasing the number of sampling points leads to more accurate estimation of system.

Then we analysis the difference between ideal cost and actual cost. The following lemma describes the upper bound of the error by approximating the ideal state and action:

Lemma 4. Under Assumption 1 and 2, Algorithm 1 attains the following bound of R 1 :

From this lemma, it is evident that for a fixed sample distance h , the error diminishes as the number of sample points looked ahead m increases. However, as the sampling distance h decreases, the convergence rate of this term becomes slower. Therefore, it is not possible to select an arbitrarily small value for h in order to minimize the discretization error R 0 .

We need to demonstrate that the discrepancy between x t and y t , as well as u t and v t , is sufficiently small, given assumption 1. This can be proven by analyzing the state evolution under the DAC policy.

By utilizing Assumption 2 and Lemma 2, we can deduce the following inequality:

Summing over all the terms and use Lemma 2, we can derive an upper bound for R 1 .

Next, we analyze the regret of Online Convex Optimization (OCO) with a memory term. To analyze OCO with a memory term, we provide an overview of the framework established by [5] in online convex optimization. The framework considers a scenario where, at each time step t , an online player selects a point x t from a set K ⊂ R d . At each time step, a loss function f t : K H +1 → R is revealed, and the player incurs a loss of f t ( x t -H , . . . , x t ) . The objective is to minimize the policy regret, which is defined as

In this setup, the first term corresponds to the DAC policy we choose, while the second term is used to approximate the optimal strongly stable linear policy.

Lemma 5. Under Assumption 1 and 2, choosing m = C h and η = Θ( m Th ) , Algorithm 1 attains the following bound of R 2 :

To analyze this term, we can transform the problem into an online convex optimization with memory and utilize existing results presented by [5] for it. By applying their results, we can derive the following bound:

Taking into account the bounds on the diameter, Lipschitz constant, and the gradient, we can ultimately derive an upper bound for R 2 .

Lastly, we aim to establish a bound on the approximation error between the optimal DAC policy and the unknown optimal linear policy.

Lemma 6. Under Assumption 1 and 2, Algorithm 1 attains the following bound of R 3 :

The intuition behind this lemma is that the evolution of states leads to an approximation of the optimal linear policy in hindsight, where u ∗ t = -K ∗ x t if we choose M ∗ = { M i } , where M i = ( K -K ∗ )( I + h ( A -BK ∗ )) i . Although the optimal policy K ∗ is unknown, such an upper bound is attainable because the left-hand side represents the minimum of M ∈ M .

C The evolution of the state

In this section we will prove that using the DAC policy, the states and actions are uniformly bounded. The difference between ideal and actual states and the difference between ideal and actual action is very small.

We begin with expressions of the state evolution using DAC policy:

Lemma 7. We have the evolution of the state and action:

where Ψ t,i represent the coefficients of ˆ w t -i :

Proof. Define Q h = I + h ( A -BK ) . Using the Taylor expansion of x t and denoting r t as the second-order residue term, we have

Then we calculate the difference between w i and ˆ w i :

Using the definition of DAC policy and the difference between disturbance, we have

where the last equality is by recursion and Ψ t,i represent the coefficients of ˆ w t -i .

Then we calculate the coefficients of w t -i and get the following result:

By the ideal definition of y t +1 and v t (only consider the effect of the past Hm steps while planning, assume x t -Hm = 0 ), taking l = Hm we have

Then we prove the norm of the transition matrix is bounded.

Lemma 8. We have the following bound of the transition matrix:

Proof. By the definition of strongly stable policy, we know

By the definition of Ψ t,i , we have

where the first inequality is due to equation 2, assumption 1 and the condition of ∥ ∥ M i t ∥ ∥ ≤ a (1 -hγ ) i -1 .

After that, we can uniformly bound the state x t and its first and second-order derivative.

Lemma 9. For any t ∈ [0 , T ] , choosing arbitrary h in the interval [0 , h 0 ] where h 0 is a constant only depends on the parameters in the assumption, we have ∥ x t ∥ ≤ D 1 , ∥ ˙ x t ∥ ≤ D 2 , ∥ ¨ x t ∥ ≤ D 3 and the estimatation of disturbance is bounded by ∥ ˆ w t ∥ ≤ W 0 . Moreover, D 1 , D 2 , D 3 are only depend on the parameters in the assumption.

Proof. We prove this lemma by induction. When t = 0 , it is clear that x 0 satisfies this condition. Suppose x t ≤ D 1 , ˙ x t ≤ D 2 , ¨ x t ≤ D 3 , ˆ w t ≤ W 0 for any t ≤ t 0 , where t 0 = kh is the k -th discretization point. Then for t ∈ [ t 0 , t 0 + h ] , we first prove that ˙ x t ≤ D 2 , ¨ x t ≤ D 3 .

By Assumption 1 and our definition of u t , we know that for any t ∈ [ t 0 , t 0 + h ] . Thus, we have

where the first inequality is by the induction hypothesis ˆ w t ≤ W 0 for any t ≤ t 0 and M i k ≤ (1 -hγ ) i -1 , the second inequality is by the induction hypothesis x t ≤ D 1 for any t ≤ t 0 .

For any t ∈ [ t , t + h ] , because we choose the fixed policy u ≡ u , so we have ˙ u = 0 and

By the Newton-Leibniz formula, we have for any ζ ∈ [0 , h ] ,

By Gronwall inequality, we have

Then we have

Then we have

where the first inequality is by the relation ξ ≤ ζ , the second inequality is by the relation ζ ≤ h and the bounding property of first-order derivative, the third inequality is by the induction hypothesis and the last inequality is due to h ≤ 1 .

By the relation ∥ ¨ x t ∥ ≤ κ A ∥ ˙ x t ∥ + W , we have

So we choose D 3 = κ A D 2 + W . By the equation, we have

where in the second line we use the Newton-Leibniz formula, the inequality is by the conclusion ∥ ¨ x t ∥ ≤ D 3 which we have proved before. By Assumption 1, we have

Choosing D 3 = κ A D 2 + W , W 0 = W + hD 3 = W + h ( κ A D 2 + W ) , we get

Using the notation

When h < γ 2 κ A (1+exp( κ A )) , we have β 1 < 1 2 . Taking D 2 = 2 β 2 we get

So we have proved that for any t ∈ [ t 0 , t 0 + h ] , ∥ ˙ x t ∥ ≤ D 2 , ∥ ¨ x t ∥ ≤ D 3 , ∥ ˆ w t ∥ ≤ W 0 .

Then we choose suitable D 1 and prove that for any t ∈ [ t 0 , t 0 + h ] , ∥ x t ∥ ≤ D 1 .

Using Lemma 7, we have

By the induction hypothesis of bounded state and estimation noise in [0 , t 0 ] together with Lemma 8, we have

Then, by the Taylor expansion and the inequality ˙ x t ≤ D 2 , we have for any ζ ∈ [0 , h ] ,

Therefore we have

In the last inequality we use h ≤ 1 .

By the relation D 2 = β 2 / (1 -β 1 ) and β 1 ≤ 1 2 , we know that

Using the notation

We have ∥ x t + ζ ∥ ≤ γ 1 D 1 + γ 2 .

From the equation of γ 1 we know that when h ≤ 1 4( κ A + κ B κ )(1+exp( κ A )) we have γ 1 ≤ 1 2 . Then we choose D 1 = 2 γ 2 , we finally get

Finally, set

By the relationship D 1 = 2 γ 2 , D 2 = 2 β 2 , D 3 = κ A D 2 + W , W 0 = W + hD 3 , we can verify the induction hypothesis. Moreover, we know that D 1 , D 2 , D 3 are not depend on h . Therefore we have proved the claim.

The last step is then to bound the action and the approximation errors of states and actions.

Lemma 2. Under Assumption 1 and 2, choosing arbitrary h in the interval [0 , h 0 ] where h 0 is a constant only depends on the parameters in the assumption, we have for any t and policy M i , ∥ x t ∥ , ∥ y t ∥ , ∥ u t ∥ , ∥ v t ∥ ≤ D , ∥ ˙ x t ∥ ≤ D , ∥ x t -y t ∥ , ∥ u t -v t ∥ ≤ κ 2 (1 + κ )(1 -hγ ) Hm +1 D . In particular, taking all the M t = 0 and K = K ∗ , we can also obtain the inequality of the optimal solution: ∥ x ∗ t ∥ , ∥ u ∗ t ∥ ≤ D .

Proof. By Lemma 8, we have

By Lemma 9 we know that for any h in [0 , h 0 ] , where

we have ∥ x t ∥ ≤ D 1 .

By Lemma 7, Lemma 8 and Lemma 9, we have

Via the definition of x t , y t , we have

For the actions

we can derive the bound

By Lemma 9, taking D = max { D 1 , D 2 , ˜ D 1 , κD 1 + W 0 γ , κ ˜ D 1 + W 0 γ } , we get the following inequality: ∥ x t ∥ , ∥ y t ∥ , ∥ u t ∥ , ∥ v t ∥ ≤ D , ∥ ˙ x t ∥ ≤ D .

We also have

In particular, the optimal policy can be recognized as taking the DAC policy with all the M t equal to 0 and the fixed strongly stable policy K = K ∗ . So we also have ∥ x ∗ t ∥ , ∥ u ∗ t ∥ ≤ D .

Now we have finished the analysis of evolution of the states. It will be helpful to prove the key lemmas in this paper.

D Proof of Lemma 3

In this section we will prove the following lemma:

Lemma 3. Under Assumption 2, Algorithm 1 attains the following bound of R 0 :

Proof. By Assumption 2 and Lemma 2, since we use the unchanged policy u t in the interval t ∈ [ ih, ( i +1) h ] , we have

Therefore we have

A similar bound can easily be established by lemma 2 about the optimal state and policy:

Taking sum of the two terms we get R 0 ≤ ( G + L ) D 2 hT .

E Proof of Lemma 4

In this section we will prove the following lemma:

Lemma 4. Under Assumption 1 and 2, Algorithm 1 attains the following bound of R 1 :

Proof. Using Lemma 2 and Assumption 2, have the approximation error between ideal cost and actual cost bounded as,

where the first inequality is by triangle inequality, the second inequality is by Assumption 2, Lemma 2, and the third inequality is by Lemma 2.

With this, we have

F Proof of Lemma 5

Before we start the proof of Lemma 5, we first present an overview of the online convex optimization (OCO) with memory framework. Consider the setting where, for every t , an online player chooses some point x t ∈ K ⊂ R d , a loss function f t : K H +1 ↦→ R is revealed, and the learner suffers a loss of f t ( x t -H , . . . , x t ) . We assume a certain coordinate-wise Lipschitz regularity on f t of the form such that, for any j ∈ { 1 , . . . , H } , for any x 1 , . . . , x H , ˜ x j ∈ K

In addition, we define ˜ f t ( x ) = f t ( x, . . . , x ) , and we let

The resulting goal is to minimize the policy regret, which is defined as

Algorithm 2 Online Gradient Descent with Memory (OGD-M)

Input: Step size η , functions { f t } T t = m . Initialize x 0 , . . . , x H -1 ∈ K arbitrarily.

Play x t , suffer loss f t ( x t -H , . . . , x t ) .

end for

To minimize this regret, a commonly used algorithm is the Online Gradient descent. By running the Algorithm 2, we may bound the policy regret by the following lemma:

Lemma 10. Let { f t } T t =1 be Lipschitz continuous loss functions with memory such that ˜ f t are convex. Then by runnning algorithm 2 itgenerates a sequence { x t } T t =1 such that

Furthermore, setting η = D f √ G f ( G f + LH 2 ) T implies that

Proof. By the standard OGD analysis [18], we know that

In addition, we know by the Lipschitz property, for any t ≥ H , we have

and so we have that

It follows that

T

∑

t

=

H

x

∈K

T

∑

t

=

H

D

2

f

η

In this setup, the first term corresponds to the DAC policy we make, and the second term is used to approximate the optimal strongly stable linear policy. It is worth noting that the cost of OCO with memory depends on the update frequency H . Therefore, we propose a two-level online controller. The higher-level controller updates the policy with accumulated feedback at a low frequency to reduce the regret, whereas a lower-level controller provides high-frequency updates of the DAC policy to reduce the discretization error. In the following part, we define the update distance of the DAC policy as l = Hm , where m is the ratio of frequency between the DAC policy update and OCO memory policy update. Formally, we update the value of M t once every m transitions, where g t represents a loss function.

From now on, we denote ˜ M t = M tm for the convenience to remove the duplicate elements. By the definition of ideal cost, we know that it is a well-defined definition.

By Lemma 7 we know that where

So we know that y t and y t are linear combination of M t , therefore

is convex in M t . So we can use the OCO with memory structure to solve this problem.

By Lemma 9 we know that y t and v t are bounded by D . Then we need to calculate the diameter, Lipchitz constant, and gradient bound of this function f i . In the following, we choose the DAC policy parameter l = Hm .

Lemma 11. (Bounding the diameter) We have

.

Proof. By the definition of M , taking l = Hm we know that

f

t

(

x

t

-

H

, . . . , x

t

)

-

min

f

t

(

x, . . . , x

)

+

2

f

TG

η

+

LH

2

ηG

f

T .

Lemma 12. (Bounding the Lipschitz Constant) Consider two policy sequences { ˜ M i -H . . . ˜ M i -k . . . ˜ M i } and { ˜ M i -H . . . ˆ M i -k . . . ˜ M i } which differ in exactly one policy played at a time step t -k for k ∈ { 0 , . . . , H } . Then we have that

where C is a constant.

Proof. By the definition we have

Where the first inequality is by ∥ Q j h ∥ ≤ κ 2 (1 -hγ ) j -1 ≤ κ 2 and lemma 9 of bounded estimation disturbance, the second inequality is by the fact that M i -k have taken m times, the last equality is by m = C h . Furthermore, we have that

Therefore using Assumption 2, Lemma 9 and Lemma 2 we immediately get that

Lemma 13. (Bounding the Gradient) We have the following bound for the gradient:

Proof. Since M is a matrix, the ℓ 2 norm of the gradient ∇ M f t corresponds to the Frobenius norm of the ∇ M f t matrix. So it will be sufficient to derive an absolute value bound on ∇ M [ r ] p,q f t ( M,...,M ) for all r, p, q . To this end, we consider the following calculation. Using lemma 9 we get that y t ( M...M ) , v t ( M...M ) ≤ D . Therefore, using Assumption 2 we have that

We now bound the quantities on the right-hand side:

Similarly,

Combining the above inequalities with

gives the bound that

Finally we prove Lemma 5:

Lemma 5. Under Assumption 1 and 2, choosing m = C h and η = Θ( m Th ) , Algorithm 1 attains the following bound of R 2 :

By Lemma 11, Lemma 12, and Lemma 13 we have

G Proof of Lemma 6

In this section, we will prove the approximation value of DAC policy and optimal policy is sufficiently small. First, we introduce the following:

Lemma 14. For any two ( κ, γ ) -strongly stable matrices K ∗ , K , there exists M = ( M 1 , . . . , M Hm ) where

such that

Proof. Denote Q h ( K ) = I + h ( A -BK ) , Q h ( K ∗ ) = I + h ( A -BK ∗ ) . By Lemma 7 we have

Consider the following calculation for i ≤ Hm and M i = ( K -K ∗ ) ( I + h ( A -BK ∗ )) i -1 :

where the final equality follows as the sum telescopes. Therefore, we have that

Then we obtain that

Using Definition 1 and Lemma 7 we finally get

We also have

where the inequality is by Definition 1 and Lemma 8.

Finally, we have

where the second inequality is by Assumption 2.

Then we can prove our main lemma:

Lemma 6. Under Assumption 1 and 2, Algorithm 1 attains the following bound of R 3 :

Proof. By choosing

We know that

Therefore choose a = 2 κ 3 we have M = { M i } in the DAC policy update class M .

Then we have the analysis of the regret:

where the first inequality is by Lemma 2 and the second inequality is by Lemma 14.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clarify our contributions and basic problem setups in both abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of our paper in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions and a complete proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose the experiment details in Section 6.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Our code is very simple, just use the traditional SAC algorithm with one line implement. Our main contribution is the theoretical analysis.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the training and test details in 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For each experiment we use 3 random seeds and take the average.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specify all the computational resources in 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is about the theory on online control, which does not seem to have evident societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We add citations for all datasets we used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of online non-stochastic control within continuous-time linear dynamical systems. To this end, the authors proposed a two-level online algorithm, where the higher level deals with a discretized reduced problem to minimize the performance measure of regret, while the lower level handles the discretization error. This work achieves an $O(\\sqrt{T} \\log T)$ regret, which is the same as that in the discretized system. Finally, empirical evaluations validate the effectiveness of the proposed method.\n\nSTRENGTHS:\nThis work investigates the meaningful problem of controlling a continuous-time linear dynamical system. This work provides an $O(\\sqrt{T} \\log T)$ regret, which is in the same order compared with the result in discretized systems. Furthermore, the authors have conducted sufficient experiments to study the effects of frame stack and frame skip, and show the effectiveness of the proposed method.\n\nWEAKNESSES:\nAlthough this work has done a nice job in handling continuous-time systems in the online non-stochastic control setup, my major concern is that the technical novelty seems insufficient. \n\nSpecifically, as the authors have said, two technical challenges exist to achieve the main Theorem 1. The first one is to control the magnitude of the states, and the second one is to handle the curse of dimensionality caused by discretization. As for the first technical challenge, the idea of \"employ Gronwall’s inequality to bound the first and second-order derivatives in the neighborhood of the current state\" (Line 207-209) seems to be pretty natural because of the continuous nature of the system. And the second technical challenge seems to be leveraging the *lazy-update* paradigm to facilitate some kind of parameter tuning for the desired regret bound. Both technical challenges seem to be a little minor and thus insufficient from my point of view. Could the authors provide more detailed explanations on the technical challenges of dealing with continuous-time systems compared with the analysis in a discretized one?\n\nQUESTIONS:\nIn this part, I list some other questions or suggestions:\n1. In Line 90, the authors said \"However, it is unclear why domain randomization works in theory\". However, this work does not illuminate the role of domain randomization in theory either because the DR only appears in the experiments part. I suggest that the authors could revise the sentence to avoid unnecessary misunderstanding of this work's contributions.\n2. Section 3.2, in my opinion, could be put in the appendix because the relationship with robust control should be explained as a part of the related work, but not necessarily to be mentioned when introducing the problem setup of *this* work.\n3. Assumption 1, compared with the standard non-stochastic disturbances in discretized systems, uses an extra condition of $\\|\\dot{w}_t\\| \\le W$. I guess that this additional condition is used to solve the first technical challenge (employ Gronwall’s inequality to bound the first and second-order derivatives in the neighborhood of the current state)? Am I right? Besides, the comparison of Assumption 1 with the previous one should be explicitly explained in the next version.\n4. Definition 1 is a little different from the standard strong stability condition in discretized systems. In discretized systems, a linear controller is strongly stable if $A - BK = P L P^{-1}$. However, in this work, the strong stability holds for $I + h(A-BK)$. By setting $h=1$ as a special case, this assumption is still different with an extra identity matrix $I$. Why are they different? And the comparison with the standard discretized strong stability assumption should be explained in the next version.\n5. In Section 4, some citations of the pioneering work [Agarwal et al., ICML 2019] are missing. For example,  Definition 2,3,4 should cite [Agarwal et al., ICML 2019]. I understand that there exist minor differences in the definitions in this work compared with those in [Agarwal et al., ICML 2019] for discretized systems. However, correct citations should still be provided to avoid unnecessary misunderstandings of this work's contributions.\n6. In the equation after Line 172, the notation of '$g_t$' is not defined. I noticed that $g_t$ is first defined in Algorithm 1 on the next page.\n7. In Figure 2, the font sizes of 'Replay' and 'Buffer' seem to be different?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and, if applicable, potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an algorithm for the non-stochastic control problem for continuous-time systems. In particular, the proposed algorithm discretizes the continuous-time system and balances the sampling time for discretization and update frequence of the online learning algorithm, in order to obtain a sublinear regret guarantee. The theoretical results are validated in numerical experiments.\n\nSTRENGTHS:\nThe paper is well written and studies an interesting research question. In order to obtain their results, the authors make use of novel proof techniques and design ideas for the proposed algorithm.\n\nWEAKNESSES:\nIn my opinion, the main weakness of this work is that the presentation of the results is often unclear and/or confusing. In particular, \n- the way the authors use to denote time steps is very confusing and often $t+1$ and $t+h$ seem to be used interchangeably (compare, e.g., line 8 'Use the action...' in Algorithm 1). Similarly $t$ and $i$ seem to be used interchangeably in Lemma 12. As this paper works with both, continuous and discrete-time, I believe it is highly important to use a notation that is less ambiguous.\n- the matrices $M$ of the DAC policy are sometimes used with square brackets (for either the sub- or superscript) and sometimes without apparently without meaning different matrices\n- $n$ and $m$ are most frequently used to denote the state and input dimension of the dynamial system, I suggest using different constants in Algorithm 1\n- There are many typos in the proofs in the appendix, for example there should not be an 'a' in (2), the constant $a$ is generally not defined (at some point you introduce the condition $|M^i_t | \\leq a(1-h\\gamma)^{i-1}$, but this does not match the DAC policy class in Algorithm 1?),  I believe the definition of $\\gamma_1$ after line 555 is wrong, above line 582, you require $n=\\lfloor\\frac{T}{h}\\rfloor$ contrary to your definition in Algorithm 1 (which, however, you require earlier on in the proof; similar arguments hold for $p$ after line 592), and after line 622, the first sum should start at $i=1$ by definition of $u_t$, whereas it should be $Q_h^i$ instead of $Q_h^{i-1}$ when replacing $x_t(M)-x_t^*$. More generally, I believe that the results are qualitatively correct, but the constants (which are omitted in the $\\mathcal O$-notation) are frequently wrong so that I was not able to verify all the presented results\n\nQUESTIONS:\nI am not sure whether I fully understand why updating the DAC policy at each time step (compared to every $m$-th time step as you propose) does not work. Intuitively, I would assume that updating the DAC policy more frequently yields better performance, and therefore also a lower regret bound. After Lemma 10 (page 24) the authors argue 'It is worth noting that the cost of OCO with memory depends on the the update frequency $H$.' However, $H$ is not immediately connected to the update frequency, but rather denotes the size of the memory in the OCO with memory framework described above Lemma 10. Could you please clarify why updating the DAC policy less often leads to improved performance (i.e., lower regret)?\n\nLIMITATIONS:\nFrom the proof (e.g., after line 549), you seem to assume $x_0=0$. While this assumption is very commonly adopted in the related literature, it should be stated somewhere in the main body of the text explicitly. Furthermore, I am not sure whether this assumption is restrictive: It requires the system to be initialized at i) a steady state of the system and ii) the steady state of the system, that the benchmark policy $K^*$ aims to stabilize.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper studies the sample complexity of online control for continuous-time linear systems. A novel two-level online algorithm was proposed. Specifically, a sublinear regret is guaranteed. Finally, the authors applied their method to the SAC algorithm and achieves improved simulation results.

STRENGTHS:
This paper studies the sample complexity of online control with adversarial disturbances for continuous-time linear systems, providing a complementary perspective to discrete-time linear systems and achieving the same sublinear rate.

WEAKNESSES:
1. There is a gap between the theoretical results and the experimental results. It is not clear how the setting studied in this paper is implemented in the experiments. More discussion is needed.


2. There are several minor writing or notation errors that need to be checked carefully. For example:

line 21 they "has"

line 36 it "reliers"

line 104, missing definition of $\mathcal{K}$

QUESTIONS:
How the setting studied in this paper is related to the experiments.

LIMITATIONS:
As mentioned in the paper, the theroical results relies on the access of system model and the loss function is convex. Extentions to model-free methods should be discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work tackles the challenge of applying simulated controllers to real-world scenarios to manage continuous-time linear systems that face unpredictable disturbances. It introduces a double-layer control approach that balances slow policy updates with quick feedback adjustments. This way it effectively minimizing errors over time. The work is tested on various real world tests in simulation environments and it shows that it significantly boosts the performance of agents, even under uncertain situations.\n\nSTRENGTHS:\n-\tIntroduction of a new two-level online control strategy that manages both policy learning & feedback control\n-\tGood theoretical analysis that achieves sublinear regret\n-\tValidation of theory with practical experiments using domain randomization in various robotic simulation environments\n\nWEAKNESSES:\n-\tThe paper lacks a comprehensive comparative analysis with other state-of-the-art methods.\n-\tExperiments are conducted in simulated environments with domain randomization. There is limited discussion on how well these results generalize to physical systems outside of controlled simulations\n\nQUESTIONS:\n-\tIn Section 3.1: the authors assumes access to \\cdot{x}/t at each time step. This is a strong assumption, as measuring the state derivative directly is often not possible in practice. Work should discuss implications of this assumption & show it can be extended to more realistic case of only having access to state measurements.\n-\tRegret defn in section 3.4 compares to the best linear policy in hindsight. Why this is an appropriate comparator class? Why not best non-linear policy? more discussion needed on this choice of benchmark & limitation\n-\tAssumption 1 is quite strict. What if disturbances may occasionally have large spikes leading to violating these bounds (which happens in practical setting)? include discussion on this case\n-\tAssumption 2 on the cost function seems to imply the costs are quadratic (due to the D^2 term). But the paper also says the costs are convex, which is a more general class. Clarify precise assumption on the cost function here\n-\tDefinition 1 of a strongly stable linear policy is central to the paper, but the motivation behind this definition is not entirely clear. \n-\tThe challenge of trading off approximation error & OCO regret in section 4 is subtle. More details needed on why this trade-off arises & how this two-level algorithm resolves it.\n-\t$w_t$ notion is inconsistent. line \\sims 100 w_t is treated as a random variable with an unknown distribution. In line 98, it is a specific realization of the disturbance at time t, rather than a random variable.\n\nLIMITATIONS:\n-\tStrict assumptions restrict the applicability of finding systems that meet these criteria\n-\tMethod is tailored for non-stochastic disturbances, leaving out scenarios where disturbances might have stochastic elements.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}