Online Control with Adversarial Disturbance for Continuous-time Linear Systems

Authors: jingwei  li
jing  dong
can  chang
baoxiang  wang
jingzhao  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 04:03:29

Abstract: We study online control for continuous-time linear systems with finite sampling rates, where the objective is to design an online procedure that learns under non-stochastic noise and performs comparably to a fixed optimal linear controller. 
We present a novel two-level online algorithm, by integrating a higher-level learning strategy and a lower-level feedback control strategy. This method offers a practical and robust solution for online control, which achieves sublinear regret. Our work provides the first nonasymptotic results for controlling continuous-time linear systems with finite number of interactions with the system. Moreover, we examine how to train an agent in domain randomization environments from a non-stochastic control perspective. By applying our method to the SAC (Soft Actor-Critic) algorithm, we achieved improved results in multiple reinforcement learning tasks within domain randomization environments. Our work provides new insights into non-asymptotic analyses of controlling continuous-time systems. Furthermore, our work brings practical intuition into controller learning under non-stochastic environments.

Main paper: 
 introduction: 
 A major challenge in robotics is to deploy simulated controllers into real-world. This process, known as sim-to-real transfer, can be difficult due to misspecified dynamics, unanticipated real-world perturbations, and non-stationary environments. Various strategies have been proposed to address these issues, including domain randomization, meta-learning, and domain adaptation [20, 10, 21] . Although they have shown great effectiveness in experimental results, training agents within these setups poses a significant challenge. To accommodate different environments, the strategies developed by agents tend to be overly conservative [26, 4] or lead to suboptimal outcomes [43, 27] .Online Control There has been a recent surge of interest in online control, as demonstrate by studies such as [2, 1, 12] . In online control, the player interacts with the environment and updates the policy in each round aiming to achieve sublinear regret. In scenarios with stochastic Gaussian noise, [12] provides the first efficient algorithm with an O( √ T ) regret bound. However, in real-world applications, the assumption of Gaussian distribution is often unfulfilled.[3] pioneers research on non-stochastic online control, where the noises can be adversarial. Under general convex costs, they introduce the Disturbance-Action Policy Class. Using an online convex optimization (OCO) algorithm with memory, they achieve an O( √ T ) regret bound. Subsequent studies extend this approach to other scenarios, such as quadratic costs [8] , partial observations [36, 35] or unknown dynamical systems [19, 11] . Other works yield varying theoretical guarantees like online competitive ratio [15, 33] .Online Continuous Control Compared to online control, there has been relatively little research on model-based continuous-time control. Most continuous control works focus on developing model-free algorithms such as policy iteration (e.g. [42, 22, 32] ), typically assuming zero-noise. This is because analyzing the system when transition dynamics are represented by differential equations, rather than recurrence formulas, poses a significant challenge.Recently, [8] studies online continuous-time linear quadratic control with standard Brownian noise and unknown system dynamics. They propose an algorithm based on the least-square method, which estimates the system's coefficients and solves the corresponding Riccati equation. The papers [34, 14] also focus on online control setups with continuous-time stochastic linear systems and unknown dynamics. They achieve O( √ T log T ) regret by different approaches. [34] uses the Thompson sampling algorithm to learn optimal actions. [14] takes a randomized-estimates policy to balance exploration and exploitation. The main difference between [8, 34, 14] and our paper is that they consider stochastic noise of Brownian motion which can be quite stringent and may fail in real-world applications, while the noise in our setup is non-stochastic. This makes our analysis completely different from theirs. Domain Randomization Domain randomization, which is proposed by [39] , is a commonly used technique for training agents to adapt to different (real) environments by training in randomized simulated environments. From the empirical perspective, many previous works focus on designing efficient algorithms for learning in a randomized simulated environment (by randomizing environmental settings, such as friction coefficient) such that the algorithm can adapt well in a new environment, [29, 44, 26, 28, 30] . Other works study how to effectively randomize the simulated environment so that the trained algorithm would generalize well in other environments [43, 27, 38] . However, prior research has not explored how to apply certain theoretical analysis ideas to train agents in domain-randomized environments. Limited previous works, such as [10] and [21] , concentrate on theoretically analyzing the sim-to-real gap within specific domain randomization models but they do not test their algorithms in real domain randomization environments. 
problem setting: 
 In this paper, we consider the online non-stochastic control for continuous-time linear systems. Therefore, we provide a brief overview below and define our notations.The Linear Dynamical System can be considered a specific case of a continuous Markov decision process with linear transition dynamics. The state transitions are governed by the following equation:ẋt = Ax t + Bu t + w t ,where x t is the state at time t, u t is the action taken by the controller at time t, and w t represents the disturbance at time t. Follow the setup of [3] , we assume x 0 = 0. We do not make any strong assumptions about the distribution of w t , and we also assume that the distribution of w t is unknown to the learner beforehand. This implies that the disturbance sequence w t can be selected adversarially.When the action u t is applied to the state x t , a cost c t (x t , u t ) is incurred. Here, we assume that the cost function c t is convex. However, this cost is not known in advance and is only revealed after the action u t is implemented at time t. In the system described above, an online policy π is defined as a function that maps known states to actions, i.e., u t = π({x ξ |ξ ∈ [0, t]}). Our goal, then, is to design an algorithm that determines such an online policy to minimize the cumulative cost incurred. Specifically, for any algorithm A, the cost incurred over a time horizon T is:J T (A) = T 0 c t (x t , u t )dt .In scenarios where the policy is linear (i.e., a linear controller), such that u t = -Kx t , we use J(K) to denote the cost of a policy K ∈ K from a certain class K.While both robust and online non-stochastic control models incorporate adversarial noise, it's crucial to understand that their objectives differ significantly.The objective function for robust control, as seen in [37, 23] , is defined as:min u1 max w 1:T min u2 . . . min ut max w T J T (A) ,Meanwhile, the objective function for online non-stochastic control, as discussed in [3] , is:min A max w 1:T (J T (A) -min K∈K J T (K)) .Note that the robust control approach seeks to directly minimize the cost function, while online non-stochastic control targets the minimization of regret, which is the discrepancy between the actual cost and the cost associated with a baseline policy. Additionally, in robust control, the noise at each step can depend on the preceding policy, whereas in online non-stochastic control, all the noise is predetermined (though unknown to the player).We operate under the following assumptions throughout this paper. To be concise, we denote ∥ • ∥ as the L 2 operator norm of the vector and matrix. Firstly, we make assumptions concerning the system dynamics and noise: Assumption 1. The matrices that govern the dynamics are bounded, meaning ∥A∥ ≤ κ A and ∥B∥ ≤ κ B , where κ A and κ B are constants. Moreover, the perturbation and its derivative are both continuous and bounded: ∥w t ∥ , ∥ ẇt ∥ ≤ W , with W being a constant.These assumptions ensure that we can bound the states and actions, as well as their first and secondorder derivatives. Next, we make assumptions regarding the cost function: Assumption 2. The costs c t (x, u) are convex in x and u. Additionally, if there exists a constant D such that ∥x∥, ∥u∥ ≤ D, then we have the following inequalities of the costs:|c t (x, u)| ≤ βD 2 , ∥∇ x c t (x, u)∥ , ∥∇ u c t (x, u)∥ ≤ GD, |c t1 (x, u) -c t2 (x, u)| ≤ L|t 1 -t 2 |D 2 ,where β,G and L are constants corresponding to the cost function. This assumption implies that if the differences between states and actions are small, then the error in their cost will also be relatively small.We next describe our baseline policy class introduced in [12] . Note that the continuous system and the discrete system are different. If we consider the approximation over a relatively small interval h, we getx t+h = x t + t+h s=t ẋs ds =x t + t+h s=t Ax s + Bu s + w s ds ≈x t + h(Ax t + Bu t + w t ) = (I + hA)x t + hBu t + hw t .Therefore, if we consider the transition of a discrete system x i+1 = Ãx i + Bu i + wi , we get the approximation Ã ≈ I + hA, B ≈ hB. Hence, we extend the definition of a strongly stable policy [12, 3] in the discrete system to the continuous system as follows: Definition 1. A linear policy K is (κ, γ)-strongly stable if, for any h > 0 that is sufficiently small, there exist matrices L h , P such that I + h(A -BK) = P L h P -1 , with the following two conditions:1. The norm of L h is strictly smaller than unity and dependent on h, i.e., ∥L h ∥ ≤ 1 -hγ.2. The controller and transforming matrices are bounded, i.e., ∥K∥ ≤ κ and ∥P ∥, P -1 ≤ κ.The above definition ensures the system can be stabilized by a linear controller K.To evaluate the designed algorithm, we follow the setup in [12, 3] and use regret, which is defined as the cumulative difference between the cost incurred by the policy of our algorithm and the cost incurred by the best policy in hindsight. Let K denotes the class of strongly stable linear policies, i.e. K = {K : K is (κ, γ)-strongly stable}. Then we try to minimize the regret of algorithm:min A max w 1:T Regret(A) = min A max w 1:T (J T (A) -min K∈K J T (K)) . 
algorithm design: 
 In this section, we outline the design of our algorithm and formally define the concepts involved in deriving our main theorem. We summarize our algorithm design as follows:First, we discretize the total time period T into smaller intervals of length h. We use the information at each point x h , x 2h , . . . and u h , u 2h , . . . to approximate the actual cost of each time interval, leveraging the continuity assumption. This process does introduce some discretization errors.Next, we employ the Disturbance-Action policy (DAC) [3] . This policy selects the action based on the current time step and the estimations of disturbances from several past steps. This policy can approximate the optimal linear policy in hindsight when we choose suitable parameters. However, the optimal policy K * is unknown, so we cannot directly acquire the optimal choice. To overcome this, we employ the OCO with memory framework [5] to iteratively adjust the DAC policy parameter M t to approximate the optimal solution M * .After that, we introduce the concept of the ideal state y t and ideal action v t that approximate the actual state x t and action u t . Note that both the state and policy depend on all DAC policy parameters M 1 , M 2 , . . . , M t . Yet, the OCO with memory framework only considers the previous H steps. Therefore, we need to consider ideal state and action. y t and v t represent the state the system would reached if it had followed the DAC policy {M t-H , . . . , M t } at all time steps from t -H to t, under the assumption that the state x t-H was 0.From all the analysis above, we can decompose the regret as three parts: the discretization error R 1 , the regret of the OCO with memory R 2 , and the approximation error between the ideal cost and the actual cost R 3 .Then we will formally introduce out method and define all the concepts. In the subsequent discussion, we use shorthand notation to denote the cost, state, control, and disturbance variables c ih , x ih , u ih , and w ih as c i , x i , u i , and w i , respectively.First, we need to define the Disturbance-Action Policy Class(DAC) for continuous systems: Definition 2. The Disturbance-Action Policy Class(DAC) is defined as:u t = -Kx t + l i=1 M i t ŵt-i ,where K is a fixed strongly stable policy, l is a parameter that signifies the dimension of the policy class, M t = {M 1 t , . . . , M l t } is the weighting parameter of the disturbance at step t, and ŵt is the estimated disturbance:EQUATIONWe note that this definition differs from the DAC policy in discrete systems [3] as we utilize the estimation of disturbance over an interval [t, t + h] instead of only the noise in time t. It counteracts the second-order residue term of the Taylor expansion of x t and is also an online policy as it only requires information from the previous state.Our higher-level controller adopts the OCO with memory framework. A technical challenge lies in balancing the approximation error and OCO regret. To achieve a low approximation error, we desire the policy update interval H to be inversely proportional to the sampling distance h. However, this relationship may lead to large OCO regret. To mitigate this issue, we introduce a new parameter m = Θ( 1 h ), representing the lookahead window. We update the parameter M t only once every m iterations, further reducing the OCO regret without negatively impacting the approximation error:M t+1 = Π M (M t -η∇g t (M )) if t mod m == 0 , M t otherwise .Where g t is a function corresponding to the loss function c t and we will introduce later in Algorithm 1. For notational convenience and to avoid redundancy, we denote M[t/m] = M t . We can then define the ideal state and action. Due to the properties of the OCO with memory structure, we need to consider only the previous Hm states and actions, rather than all states. As a result, we introduce the definition of the ideal state and action. During the interval t ∈ [im, (i + 1)m -1], the learning policy remains unchanged, so we could define the ideal state and action follow the definition in [3] : Definition 3. The ideal state y t and action v t at time t ∈ [im, (i + 1)m -1] are defined asy t = x t ( Mi-H , ..., Mi ), v t = -Ky t + l j=1 M j i w t-i .where the notation indicates that we assume the state x t-H is 0 and that we apply the DAC policy Mi-H , . . . , Mi at all time steps from t -Hm to t.We can also define the ideal cost in this interval follow the definition in [3] :Definition 4.The ideal cost function during the interval t ∈ [im, (i + 1)m -1] is defined as follows: f i Mi-H , . . . , Mi = (i+1)m-(M ) = f k (M, . . . , M ).Update OCO policy Mk+1 = Π M Mk -η∇g k ( Mk ) . end for 
main result: 
 In this section, we present the primary theorem of online continuous control regret analysis: Theorem 1. Under Assumption 1, 2, a step size of η = Θ( m T h ), and a DAC policy update frequency m = Θ( 1 h ), Algorithm 1 attains a regret bound ofJ T (A) -min K∈K J T (K) ≤ O(nh(1 -hγ) H h ) + O( √ nh) + O(T h) .With the sampling distance h = Θ( 1 √ T ), and the OCO policy update parameter H = Θ(log(T )), Algorithm 1 achieves a regret bound ofJ T (A) -min K∈K J T (K) ≤ O √ T log (T ) .Theorem 1 demonstrates a regret that matches the regret of a discrete system [3] . Despite the analysis of a continuous system differing from that of a discrete system, we can balance discretization error, approximation error, and OCO with memory regret by selecting an appropriate update frequency for the policy. Here, O(•) and Θ(•) are abbreviations for the polynomial factors of universal constants in the assumption.While we defer the detailed proof to the appendix, we outline the key ideas and highlight them below. Challenge and Proof Sketch We first explain why we cannot directly apply the methods for discrete nonstochastic control from [3] to our work. To utilize Assumption 2, it is necessary first to establish a union bound over the states. In a discrete-time system, it can be easily proved by applying the dynamics inequality ∥x t+1 ∥ ≤ a∥x t ∥ + b (where a < 1) and the induction method presented in [3] . However, for a continuous-time system, a different approach is necessary because we only have the differential equation instead of the state recurrence formula.To overcome this challenge, we employ Gronwall's inequality to bound the first and second-order derivatives in the neighborhood of the current state.We then use these bounded properties, in conjunction with an estimation of previous noise, to bound the distance to the next state. Through an iterative application of this method, we can argue that all states and actions are bounded.Another challenge is that we need to discretize the system but we must overcome the curse of dimensionality caused by discretization. In continuous-time systems, the number of states is inversely proportional to the discretization parameter h, which also determines the size of the OCO memory buffer. Our regret is primarily composed of three components: the error caused by discretization R 1 , the regret of OCO with memory R 2 and the difference between the actual cost and the approximate costR 3 . The discretization error R 1 is O(hT ), therefore if we achieve O( √ T ) regret, we must choose h no more than O( 1 √ T ).If we update the OCO with memory parameter at each timestep follow the method in [3] , we will incur the regret of OCO with memory R 2 = O(H 2.5 √ T ). The difference between the actual cost and the approximate cost R 3 = O(T (1 -hγ) H ). To achieve sublinear regret for the third term, we must chooseH = O( log T hγ ), but since h is no more than O( 1 √ T ), H will be larger than Θ( √ T ),therefore the second term R 2 will definitely exceed O(Therefore, we adjust the frequency of updating the OCO parameters by introducing a new parameter m, using a two-level approach and update the OCO parameters once in every m steps. This will incur the third termEQUATIONT ) and we achieve the same regret compare with the discrete system. 
experiments: 
 In this section, we apply our theoretical analysis to the practical training of agents. First we highlight the key difference between our algorithm and traditional online policy optimization.1. Stack: While standard online policy optimization learns the optimal policy from the current state u t = ϕ(x t ), an optimal non-stochastic controller employs the DAC policy as outlined in Definition 2. Leveraging information from past states aids the agent in adapting to dynamic environments. 2. Skip: Different from the analysis in [3] , in a continuous-time system we update the state information every few steps, rather than updating it at every step. This solves the curse of dimensionality caused by discretization in continuous-time system.The above inspires us with an intuitive strategy for training agents by stacking past observations with some observations to skip. We denote this as Stack & skip for convenience. Stack & skip is frequently used as a heuristic in reinforcement learning, yet little was known about when and why such a technique could boost agent performance.How should we evaluate our algorithm in a non-stochastic environment? We opt for learning an optimal policy within a domain randomization environment. In this context, each model's parameters are randomly sampled from a predetermined task distribution. We train policies to optimize performance across various simulated models [41, 29] . We observe that learning in Domain Randomization (DR) significantly differs from stochastic or robust learning problems.In DR, sampling from environmental variables occurs at the beginning of each episode, rather than at every step, distinguishing it from stochastic learning where randomness is step-wise independent and identically distributed. This episodic sampling approach allows agents in DR to exploit environmental conditions and adapt to episodic changes within an episode. On the other hand, robust learning focuses on worst-case scenarios depending on an agent's policy. DR, in contrast, is concerned with the distribution of conditions aimed at broad applicability rather than worst-case perturbations.In the context of non-stochastic control, the disturbance, while not disclosed to the learner beforehand, remains fixed throughout the episode and does not adaptively respond to the control policy. This Environment Setting We conduct experiments on the hopper, halfcheetah, and walker2d benchmarks using the MuJoCo simulator [40] . The randomized parameters include environmental physical parameters such as damping and friction, as well as the agent properties such as torso size.We set the range of our domain randomization to follow a distribution with default parameters as the mean value, shown in Table 1 . When training in the domain randomization environment, the parameter is uniformly sampled from this distribution. To analyze the result of generalization, we only change one of the parameters and keep the other parameters as the mean of its distribution in each test environment. We conducted experiments using NVIDIA A40 graphics card.We design a practical meta-algorithm that converts any standard deep RL algorithm into a domain-adaptive algorithm, shown in Figure 2 . In this algorithm, we augment the original state observation o old t at time t with past observations, resulting ino new t = [o old t , o old t-m , . . . , o old t-(h-1)m ].Here h is the number of past states we leverage and m is the number of states we skip when we get each of the past states. For clarity in our results, we selected the SAC algorithm for evaluation. We use a variant of Soft Actor-Critic (SAC) [31] and leverage past states with some skip as our algorithm. We compare our algorithm with the standard SAC algorithm training on domain randomization environments as our baseline.To understand the effects of the frame stack number h and frame skip number m, we carried out experiments in the hopper environment with different h and m. For each parameter we train with 3 random seeds and take the average. Figure 3 shows that the performance increases significantly when the frame stack number is increased from 1 to 3, and remains roughly unchanged when the frame stack number continues to climb up. Figure 4 shows that the optimal frame skip number is 3, while both too large or too small frame skip numbers result in sub-optimal results. Therefore, in the following experiments we fix the parameter h = 3, m = 3. We train our algorithm with this parameter and standard SAC on hopper and test the performance on more environments. Figure 5 shows that our algorithm outperforms the baseline in all environments. Results on Other Environments Each algorithm was trained using three distinct random seeds in the half-cheetah and walker2d domain randomization (DR) environments. Consistent with previous experiments, we employed a frame stack number of h = 3 and frame skip number of m = 3.The comparative performance of our algorithm and the baseline algorithm, across various domain parameters, is presented in Figure 6 . The result clearly demonstrates that our algorithm consistently outperforms the baseline in all evaluated test environments. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We specify all the training and test details in 6.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion, limitations and future directions: 
 In this paper, we propose a two-level online controller for continuous-time linear systems with adversarial disturbances, aiming to achieve sublinear regret. This approach is grounded in our examination of agent training in domain randomization environments from an online control perspective. At the higher level, our controller employs the Online Convex Optimization (OCO) with memory framework to update policies at a low frequency, thus reducing regret. The lower level uses the DAC policy to align the system's actual state more closely with the idealized setting.In our empirical evaluation, applying our algorithm's core principles to the SAC (Soft Actor-Critic) algorithm led to significantly improved results in multiple reinforcement learning tasks within domain randomization environments. This highlights the adaptability and effectiveness of our approach in practical scenarios.It is important to note that our theoretical analysis depends on the known dynamics of the system and the assumption of convex costs. This reliance could represent a limitation to our method, as it may not adequately address scenarios where these conditions do not hold or where system dynamics are incompletely understood. For future research, there are several promising directions in online non-stochastic control of continuous-time systems. These include extending our methods to systems with unknown dynamics, exploring the impact of assuming strong convexity in cost functions, and shifting the focus from regret to the competitive ratio. Further research can also explore how to utilize historical information more effectively to enhance agent training in domain randomization environments. This might involve employing time series analysis instead of simply incorporating parameters into neural network training.In the appendix we define n as the smallest integer greater than or equal to T h , and we use the shorthand c ih , x ih , u ih , and w ih as c i , x i , u i , and w i , respectively. First we provide the proof of our main theorem there.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: For each experiment we use 3 random seeds and take the average.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We specify all the computational resources in 6.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Our work is about the theory on online control, which does not seem to have evident societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We add citations for all datasets we used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Theorem 1. Under Assumption 1, 2, a step size of η = Θ( m T h ), and a DAC policy update frequency m = Θ( 1 h ), Algorithm 1 attains a regret bound ofJ T (A) -min K∈K J T (K) ≤ O(nh(1 -hγ) H h ) + O( √ nh) + O(T h) .With the sampling distance h = Θ( 1 √ T ), and the OCO policy update parameter H = Θ(log(T )), Algorithm 1 achieves a regret bound ofJ T (A) -min K∈K J T (K) ≤ O √ T log (T ) .Proof. We denote u * t = K * x * t as the optimal state and action that follows the policy specified by K * , where K * = arg max K∈K J T (K).We then discretize and decompose the regret as follows:J T (A) -min K∈K J T (K) = T 0 c t (x t , u t )dt - T 0 c t (x * t , u * t )dt = n-1 i=0 (i+1)h ih c t (x t , u t )dt - n-1 i=0 (i+1)h ih c t (x * t , u * t )dt = h n-1 i=0 c i (x i , u i ) - n-1 i=0 c i (x * i , u * i ) + R 0 ,where R 0 represents the discretization error.We define p as the smallest integer greater than or equal to n m , then the first term can be further decomposed asn-1 i=0 c i (x i , u i ) - n-1 i=0 c i (x * i , u * i ) = p-1 i=0 (i+1)m-1 j=im c i (x i , u i ) - p-1 i=0 (i+1)m-1 j=im c i (x * i , u * i ) = p-1 i=0   (i+1)m-1 j=im c i (x i , u i ) - (i+1)m-1 j=im c i (y i , v i )   + p-1 i=0 (i+1)m-1 j=im c i (y i , v i ) - p-1 i=0 (i+1)m-1 j=im c i (x * i , u * i ) = p-1 i=0   (i+1)m-1 j=im c i (x i , u i ) -f i ( Mi-H , . . . , Mi )   + p-1 i=0 f i ( Mi-H , . . . , Mi ) -min M ∈M p-1 i=0 f i (M, . . . , M ) + min M ∈M p-1 i=0 f i (M, . . . , M ) - p-1 i=0 (i+1)m-1 j=im c i (x * i , u * i ) ,where the last equality is by the definition of the idealized cost function.Let us denoteR 1 = p-1 i=0   (i+1)m-1 j=im c i (x i , u i ) -f i ( Mi-H , . . . , Mi )   , R 2 = p-1 i=0 f i ( Mi-H , . . . , Mi ) -min M ∈M p-1 i=0 f i (M, . . . , M ) , R 3 = min M ∈M p-1 i=0 f i (M, . . . , M ) - p-1 i=0 (i+1)m-1 j=im c i (x * i , u * i ) .Then we have the regret decomposition asRegret(T ) = h(R 1 + R 2 + R 3 ) + O(hT ) .We then separately upper bound each of the four terms.The term R 0 represents the error caused by discretization, which decreases as the number of sampling points increases and the sampling distance h decreases. This is because more sampling points make our approximation of the continuous system more accurate. Using Lemma 3, we get the following upper bound: R 0 ≤ O(hT ).The term R 1 represents the difference between the actual cost and the approximate cost. For a fixed h, this error decreases as the number of sample points looked ahead m increases, while it increases as the sampling distance h decreases. This is because the closer adjacent points are, the slower the convergence after approximation. By Lemma 4 we can bound it asR 1 ≤ O(n(1 -hγ) Hm ).The term R 2 is incurred due to the regret of the OCO with memory algorithm. Note that this term is determined by learning rate η and the policy update frequency m. Choosing suitable parameters and using Lemma 5, we can obtain the following upper bound: R 2 ≤ O( n/h).The term R 3 represents the difference between the ideal optimal cost and the actual optimal cost. Since the accuracy of the DAC policy approximation of the optimal policy depends on its degree of freedom l, a higher degree of freedom leads to a more accurate approximation of the optimal policy. We use Lemma 6 and choose l = Hm to bound this error: R 3 ≤ O(n(1 -hγ) Hm ).By summing up these four terms and taking m = Θ( 1 h ), we get:Regret(T ) ≤ O(nh(1 -hγ) H h ) + O( √ nh) + O(hT ) .h = Θ 1 √ T , m = Θ 1 h , H = Θ(log(T )), the regret is bounded by Regret(T ) ≤ O( √ T log(T )) .In this section, we will primarily discuss the rationale behind the proof of our key lemmas. First, we need to prove all the states and actions are bounded.Lemma 2. Under Assumption 1 and 2, choosing arbitrary h in the interval [0, h 0 ] where h 0 is a constant only depends on the parameters in the assumption, we have for any t and policy M i ,∥x t ∥, ∥y t ∥, ∥u t ∥, ∥v t ∥ ≤ D, ∥ ẋt ∥ ≤ D, ∥x t -y t ∥, ∥u t -v t ∥ ≤ κ 2 (1 + κ)(1 -hγ) Hm+1 D.In particular, taking all the M t = 0 and K = K * , we can also obtain the inequality of the optimal solution:∥x * t ∥, ∥u * t ∥ ≤ D.The proof of this Lemma mainly use the Gronwall inequality and the induction method. Then we analyze the discretization error of the system.Lemma 3. Under Assumption 2, Algorithm 1 attains the following bound of R 0 :R 0 = n-1 i=0 (i+1)h ih (c t (x t , u t ) -c t (x * t , u * t ))dt -h n-1 i=0 (c i (x i , u i ) -c i (x * i , u * i )) ≤ (G + L)D 2 hT .This lemma indicates that the discretization error is directly proportional to the sample distance h. In other words, increasing the number of sampling points leads to more accurate estimation of system.Then we analysis the difference between ideal cost and actual cost. The following lemma describes the upper bound of the error by approximating the ideal state and action: Lemma 4. Under Assumption 1 and 2, Algorithm 1 attains the following bound of R 1 :R 1 = p-1 i=0   (i+1)m-1 j=im c i (x i , u i ) -f i Mi-H , . . . , Mi   ≤ nGD 2 κ 2 (1 + κ)(1 -hγ) Hm+1 .From this lemma, it is evident that for a fixed sample distance h, the error diminishes as the number of sample points looked ahead m increases. However, as the sampling distance h decreases, the convergence rate of this term becomes slower. Therefore, it is not possible to select an arbitrarily small value for h in order to minimize the discretization error R 0 .We need to demonstrate that the discrepancy between x t and y t , as well as u t and v t , is sufficiently small, given assumption 1. This can be proven by analyzing the state evolution under the DAC policy.By utilizing Assumption 2 and Lemma 2, we can deduce the following inequality:|c t (x t , u t ) -c t (y t , v t )| ≤ |c t (x t , u t ) -c t (y t , u t )| + |c t (y t , u t ) -c t (y t , v t )| ≤ GD∥x t -y t ∥ + GD∥u t -v t ∥ .Summing over all the terms and use Lemma 2, we can derive an upper bound for R 1 .Next, we analyze the regret of Online Convex Optimization (OCO) with a memory term. To analyze OCO with a memory term, we provide an overview of the framework established by [5] in online convex optimization. The framework considers a scenario where, at each time step t, an online player selects a point x t from a set K ⊂ R d . At each time step, a loss function f t : K H+1 → R is revealed, and the player incurs a loss of f t (x t-H , . . . , x t ). The objective is to minimize the policy regret, which is defined asPolicyRegret = T t=H f t (x t-H , . . . , x t ) -min x∈K T t=H f t (x, . . . , x) .In this setup, the first term corresponds to the DAC policy we choose, while the second term is used to approximate the optimal strongly stable linear policy. Lemma 5. Under Assumption 1 and 2, choosing m = C h and η = Θ( m T h ), Algorithm 1 attains the following bound of R 2 :R 2 = p-1 i=0 f i ( Mi-H , . . . , Mi ) -min M ∈M p-1 i=0 f i (M, . . . , M ) ≤ 4a γ GDC 2 κ 2 (κ + 1)W 0 κ B γ ( GDCκ 2 (κ + 1)W 0 κ B γ + C 2 κ 3 κ B W 0 H 2 ) n h .To analyze this term, we can transform the problem into an online convex optimization with memory and utilize existing results presented by [5] for it. By applying their results, we can derive the following bound:T t=H f t (x t-H , . . . , x t ) -min x∈K T t=H f t (x, . . . , x) ≤ O D G f (G f + LH 2 ) T .Taking into account the bounds on the diameter, Lipschitz constant, and the gradient, we can ultimately derive an upper bound for R 2 .Lastly, we aim to establish a bound on the approximation error between the optimal DAC policy and the unknown optimal linear policy.Lemma 6. Under Assumption 1 and 2, Algorithm 1 attains the following bound of R 3 :R 3 = min M ∈M p-1 i=0 f i (M, ..., M ) - p-1 i=0 (i+1)m-1 j=im c i (x * i , u * i ) ≤ 3n(1 -hγ) Hm GDW 0 κ 3 a(lhκ B + 1) .The intuition behind this lemma is that the evolution of states leads to an approximation of the optimal linear policy in hindsight, where u * t = -K * x t if we choose M * = {M i }, where M i = (K -K * )(I + h(A -BK * )) i . Although the optimal policy K * is unknown, such an upper bound is attainable because the left-hand side represents the minimum of M ∈ M.In this section we will prove that using the DAC policy, the states and actions are uniformly bounded. The difference between ideal and actual states and the difference between ideal and actual action is very small. We begin with expressions of the state evolution using DAC policy: Lemma 7. We have the evolution of the state and action:x t+1 = Q l+1 h x t-l + h 2l i=0 Ψ t,i ŵt-i , y t+1 = h 2Hm i=0 Ψ t,i ŵt-i , v t = -Ky t + h Hm j=1 M j t ŵt-j .where Ψ t,i represent the coefficients of ŵt-i :Ψ t,i = Q i h 1 i≤l + h l j=0 Q j h BM i-j t-j 1 i-j∈[1,l] . Proof. Define Q h = I + h(A -BK).Using the Taylor expansion of x t and denoting r t as the second-order residue term, we havex t+1 = x t + h ẋt + h 2 r t = x t + h(Ax t + Bu t + w t ) + h 2 r t .Then we calculate the difference between w i and ŵi :ŵt -w t = x t+1 -x t -h(Ax t + Bu t + w t ) h = hr t .Using the definition of DAC policy the difference between disturbance, we havex t+1 = x t + h Ax t + B -Kx t + h l i=1 M i t ŵt-i + ŵt -hr t + h 2 r t = (I + h(A -BK))x t + h Bh l i=1 M i t ŵt-i + ŵt = Q h x t + h Bh l i=1 M i t ŵt-i + ŵt = Q 2 h x t-1 + h Q h Bh l i=1 M i t-1 ŵt-1-i + ŵt-1 + h Bh l i=1 M i t ŵt-i + ŵt = Q l+1 h x t-l + h 2l i=0 Ψ t,i ŵt-i ,where the last equality is by recursion and Ψ t,i represent the coefficients of ŵt-i .Then we calculate the coefficients of w t-i and get the following result:Ψ t,i = Q i h 1 i≤l + h l j=0 Q j h BM i-j t-j 1 i-j∈[1,l] .By the ideal definition of y t+1 and v t (only consider the effect of the past Hm steps while planning, assume x t-Hm = 0), taking l = Hm we havey t+1 = h 2Hm i=0 Ψ t,i ŵt-i , v t = -Ky t + h Hm j=1 M j t ŵt-j .Then we prove the norm of the transition matrix is bounded. Lemma 8. We have the following bound of the transition matrix:∥Ψ t,i ∥ ≤ a(lhκ B + 1)κ 2 (1 -hγ) i-1 .Proof. By the definition of strongly stable policy, we knowEQUATIONBy the definition of Ψ t,i , we have∥Ψ t,i ∥ = Q i h 1 i≤l + h l j=0 Q j h BM i-j t-j 1 i-j∈[1,l] ≤ κ 2 (1 -hγ) i + ah l j=1 κ B κ 2 (1 -hγ) j (1 -hγ) i-j-1 ≤ κ 2 (1 -hγ) i + alhκ B κ 2 (1 -hγ) i-1 ≤ a(lhκ B + 1)κ 2 (1 -hγ) i-1 ,where the first inequality is due to equation 2, assumption 1 and the condition ofM i t ≤ a(1 - hγ) i-1 .After that, we can uniformly bound the state x t and its first and second-order derivative. 9. For any t ∈ [0, T ], choosing arbitrary h in the interval [0, h 0 ] where h 0 is a constant only depends on the parameters in the assumption, we have ∥x t ∥ ≤ D 1 , ∥ ẋt ∥ ≤ D 2 , ∥ẍ t ∥ ≤ D 3 and the estimatation of disturbance is bounded by ∥ ŵt ∥ ≤ W 0 . Moreover, D 1 , D 2 , D 3 are only depend on the parameters in the assumption.Proof. We prove this lemma by induction. When t = 0, it is clear that x 0 satisfies this condition. Suppose x t ≤ D 1 , ẋt ≤ D 2 , ẍt ≤ D 3 , ŵt ≤ W 0 for any t ≤ t 0 , where t 0 = kh is the k-th discretization point. Then for t ∈ [t 0 , t 0 + h], we first prove that ẋt ≤ D 2 , ẍt ≤ 3 .By Assumption 1 and our definition of u t , we know that for any t ∈ [t 0 , t 0 + h]. Thus, we have∥ ẋt ∥ = ∥Ax t + Bu t + w t ∥ = ∥Ax t + B(-Kx t0 + h l i=1 M i k ŵk-i ) + w t ∥ ≤ κ A ∥x t ∥ + κ B κ∥x t0 ∥ + h l i=1 (1 -hγ) i-1 W 0 + W ≤ κ A ∥x t ∥ + κ B κD 1 + W 0 γ + W ,where the first inequality is by the induction hypothesis ŵt ≤ W 0 for any t ≤ t 0 and M i k ≤ (1 -hγ) i-1 , the second inequality is by the induction hypothesis x t ≤ D 1 for any t ≤ t 0 .For any t ∈ [t 0 , t 0 + h], because we choose the fixed policy u t ≡ u t0 , so we have ut = 0 and∥ẍ t ∥ = ∥A ẋt + B ut + ẇt ∥ = ∥A ẋt + ẇt ∥ ≤ κ A ∥ ẋt ∥ + W .ζ ∈ [0, h], ẋt0+ζ -ẋt0 = ζ 0 ẍt0+ξ d ξ .Then we have∥ ẋt0+ζ ∥ ≤ ∥ ẋt0 ∥ + ζ 0 ∥ẍ t0+ξ ∥d ξ ≤ ∥ ẋt0 ∥ + ζ 0 (κ A ∥ ẋt0+ξ ∥ + W )d ξ = ∥ ẋt0 ∥ + W ζ + κ A ζ 0 ∥ ẋt0+ξ ∥d ξ .By Gronwall inequality, we have∥ ẋt0+ζ ∥ ≤ ∥ ẋt0 ∥ + W ζ + ζ 0 (∥ ẋt0 ∥ + W ξ) exp(κ A (ζ -ξ))d ξ .Then we have∥ ẋt0+ζ ∥ ≤ ∥ ẋt0 ∥ + W ζ + ζ 0 (∥ ẋt0 ∥ + W ζ) exp(κ A ζ))d ξ = (∥ ẋt0 ∥ + W ζ)(1 + ζ exp(κ A ζ)) ≤ κ A ∥x t0 ∥ + κ B κD 1 + W 0 γ + W + W h (1 + h exp(κ A h)) ≤ (κ A + κ B κ)D 1 + W 0 γ + W + W h (1 + h exp(κ A h)) ≤ (κ A + κ B κ)D 1 + W 0 γ + 2W (1 + exp(κ A )) ,where the first inequality is by relation ξ ≤ ζ, the second inequality is by the relation ζ ≤ h and the bounding property of first-order derivative, the third inequality is by the induction hypothesis and the last inequality is due to h ≤ 1.t ∥ ≤ κ A ∥ ẋt ∥ + W , we have ∥ẍ t0+ζ ∥ ≤ κ A D 2 + W . So we choose D 3 = κ A D 2 + W . By the equation, we have ∥ ŵt -w t ∥ = x t+1 -x t -h(Ax t + Bu t + w t ) h = x t+1 -x t -h ẋt h = h 0 ( ẋt+ξ -ẋt )dξ h = h 0 ξ 0 ẍt+ζ dζdξ h ≤ h 0 ξ 0 ∥ẍ t+ζ ∥dζdξ h ≤ hD 3 ,where in the second line we use the Newton-Leibniz formula, the inequality is by the conclusion ∥ẍ t ∥ ≤ D 3 which we have proved before. By Assumption 1, we have∥ ŵt ∥ ≤ W + hD 3 . Choosing D 3 = κ A D 2 + W , W 0 = W + hD 3 = W + h(κ A D 2 + W ), we get ∥ ẋt0+ζ ∥ ≤ ((κ A + κ B κ)D 1 + W 0 γ + 2W )(1 + exp(κ A )) ≤ ((κ A + κ B κ)D 1 + W + h(κ A D 2 + W ) γ + 2W )(1 + exp(κ A )) ≤ D 2 hκ A γ (1 + exp(κ A )) + (κ A + κ B κ)D 1 + (1 + h + 2γ)W γ (1 + exp(κ A ))) .Using the notationβ 1 = hκ A γ (1 + exp(κ A )) , β 2 = (κ A + κ B κ)D 1 + 2(1 + γ)W γ (1 + exp(κ A )) . When h < γ 2κ A (1+exp(κ A )) , we have β 1 < 1 2 . Taking D 2 = 2β 2 we get ∥ ẋt0+ζ ∥ ≤ β 1 D 2 + β 2 ≤ D 2 .t ∈ [t 0 , t 0 + h], ∥ ẋt ∥ ≤ D 2 , ∥ẍ t ∥ ≤ D 3 , ∥ ŵt ∥ ≤ W 0 .Then we choose suitable D 1 and prove that for any t∈ [t 0 , t 0 + h], ∥x t ∥ ≤ D 1 .Using Lemma 7, we havex t+1 = h t i=0 Ψ t,i ŵt-i .By the induction hypothesis of bounded state and estimation noise in [0, t 0 ] together with Lemma 8, we have∥x t+1 ∥ ≤ h t i=0 (lhκ B + 1)κ 2 (1 -hγ) i (W + hD 3 ) ≤ (lhκ B + 1)κ 2 (W + hD 3 ) γ .by the Taylor expansion and the inequality ẋt ≤ D 2 , we have for any ζ ∈ [0, h],∥x t+1 -x t+ζ ∥ = ∥ h ζ ẋt+ξ dξ∥ ≤ (h -ζ)D 2 ≤ hD 2 .Therefore we have∥x t+ζ ∥ ≤ ∥x t+1 ∥ + hD 2 ≤ (lhκ B + 1)κ 2 (W + hD 3 ) γ + hD 2 = (lhκ B + 1)κ 2 W (1 + h) γ + hD 2 (lhκ B + 1)κ 2 κ A γ + 1 ≤ (lκ B + 1)2κ 2 W γ + hD 2 (lκ B + 1)κ 2 κ A γ + 1 .In the last inequality we use h ≤ 1.D 2 = β 2 /(1 -β 1 ) and β 1 ≤ 1 2 , we know that D 2 ≤ 2 (κ A + κ B κ)D 1 + 2(1 + γ)W γ (1 + exp(κ A )).Using the notationγ 1 = 2h(κ A + κ B κ)(1 + exp(κ A )) , γ 2 = (lκ B + 1)2κ 2 W γ + 4 (1 + γ)W γ (1 + exp(κ A )) (lκ B + 1)κ 2 κ A γ + 1 .We have∥x t+ζ ∥ ≤ γ 1 D 1 + γ 2 .From the equation of γ 1 we know that when h ≤ 1 4(κ A +κ B κ)(1+exp(κ A )) we have γ 1 ≤ 1 2 . Then we choose D 1 = 2γ 2 , we finally get∥x t+ζ ∥ ≤ γ 1 D 1 + γ 2 ≤ D 1 .Finally, seth 0 = min 1, γ κ A (1 + exp(κ A )) , 1 4(κ A + κ B κ)(1 + exp(κ A )),By the relationship D 1 = 2γ 2 , D 2 = 2β 2 , D 3 = κ A D 2 + W , W 0 = W + hD 3 ,we can verify the induction hypothesis. Moreover, we know that D 1 , D 2 , D 3 are not depend on h. Therefore we have proved the claim.The last step is then to bound the action and the approximation errors of states and actions. Lemma 2. Under Assumption 1 and 2, choosing arbitrary h in the interval [0, h 0 ] where h 0 is a constant only depends on the parameters in the assumption, we have for any t and policy M i ,∥x t ∥, ∥y t ∥, ∥u t ∥, ∥v t ∥ ≤ D, ∥ ẋt ∥ ≤ D, ∥x t -y t ∥, ∥u t -v t ∥ ≤ κ 2 (1 + κ)(1 -hγ) Hm+1 D.In particular, taking all the M t = 0 and K = K * , we can also obtain the inequality of the optimal solution: ∥x * t ∥, ∥u * t ∥ ≤ D.Proof. By Lemma 8, we have∥Ψ t,i ∥ ≤ a(lhκ B + 1)κ 2 (1 -hγ) i-1 .By Lemma 9 we know that for any h in [0, h 0 ], whereh 0 = min 1, γ κ A (1 + exp(κ A )) , 1 4(κ A + κ B κ)(1 + exp(κ A )) ,we have ∥x t ∥ ≤ 1 .By Lemma 7, Lemma 8 and Lemma 9, we have∥y t+1 ∥ = ∥h 2Hm i=0 Ψ t,i ŵt-i ∥ ≤ hW 0 2Hm i=0 a(lhκ B + 1)κ 2 (1 -hγ) i-1 ≤ aW 0 (lhκ B + 1)κ 2 γ = D1 .Via the definition of x t , y t , we have∥x t -y t ∥ ≤ κ 2 (1 -hγ) Hm+1 ∥x t-Hm ∥ ≤ κ 2 (1 -hγ) Hm+1 D 1 .For the actionsu t = -Kx t + h Hm i=1 M i t ŵt-i , v t = -Ky t + h Hm i=1 M i t ŵt-i ,we can derive the bound∥u t ∥ ≤ ∥Kx t ∥ + h Hm i=1 M i t ŵt-i ≤ κ ∥x t ∥ + W 0 h Hm i=1 a(1 -hγ) i-1 ≤ κD 1 + aW 0 γ , ∥v t ∥ ≤ ∥Ky t ∥ + h Hm i=1 M i t ŵt-i ≤ κ ∥y t ∥ + W 0 h Hm i=1 a(1 -hγ) i-1 ≤ κ D1 + aW 0 γ , ∥u t -v t ∥ ≤ ∥K∥ ∥x t -y t ∥ ≤ κ 3 (1 -hγ) Hm+1 D 1 .By Lemma 9, taking D = max{D 1 , D 2 , D1 , κD 1 + W0 γ , κ D1 + W0 γ }, we get the following inequality:∥x t ∥, ∥y t ∥, ∥u t ∥, ∥v t ∥ ≤ D, ∥ ẋt ∥ ≤ D.We also have∥x t -y t ∥ + ∥u t -v t ∥ ≤ κ 2 (1 -hγ) Hm+1 D 1 + κ 3 (1 -hγ) Hm+1 D 1 ≤ κ 2 (1 + κ)(1 -hγ) Hm+1 D .In particular, the optimal policy can be recognized as taking the DAC policy with all the M t equal to 0 and the fixed strongly stable policy K = K * . So we also have ∥x * t ∥, ∥u * t ∥ ≤ D.Now we have finished the analysis of evolution of the states. It will be helpful to prove the key lemmas in this paper.In this section we will prove the following lemma: Lemma 3. Under Assumption 2, Algorithm 1 attains the following bound of R 0 :R 0 = n-1 i=0 (i+1)h ih (c t (x t , u t ) -c t (x * t , u * t ))dt -h n-1 i=0 (c i (x i , u i ) -c i (x * i , u * i )) ≤ (G + L)D 2 hT .By Assumption 2 and Lemma 2, since we use the unchanged policy u t in the interval t ∈ [ih, (i + 1)h], we have|c t (x t , u t ) -c ih (x ih , u ih )| ≤ |c t (x t , u t ) -c t (x ih , u ih )| + |c t (x ih , u ih ) -c ih (x ih , u ih )| ≤ max x ∥∇ x c t (x, u)∥ ∥x t -x ih ∥ + L(t -ih)D 2 ≤ GD∥ t ih ẋs ds∥ + L(t -ih)D 2 ≤ (G + L)D 2 (t -ih) .Therefore we have| n-1 i=0 (i+1)h ih c t (x t , u t )dt -h n-1 i=0 c i (x i , u i )| =| n-1 i=0 (i+1)h ih (c t (x t , u t ) -c ih (x ih , u ih ))dt| ≤(G + L)D 2 n-1 i=0 (i+1)h ih (t -ih)dt = 1 2 (G + L)D 2 nh 2 = 1 2 (G + L)D 2 hT .A similar bound can easily be established by lemma 2 about the optimal state and policy:| n-1 i=0 (i+1)h ih c t (x * t , u * t )dt - n-1 i=0 c i (x * i , u * i )| ≤ 1 2 (G + L)D 2 hT .Taking sum of the two terms we get R 0 ≤ (G + L)D 2 hT .In this section we will prove the following lemma: Lemma 4. Under Assumption 1 and 2, Algorithm 1 attains the following bound of R 1 :R 1 = p-1 i=0   (i+1)m-1 j=im c i (x i , u i ) -f i Mi-H , . . . , Mi   ≤ nGD 2 κ 2 (1 + κ)(1 -hγ) Hm+1 .Proof. Using Lemma 2 and Assumption 2, have the approximation error between ideal cost and actual cost bounded as,|c t (x t , u t ) -c t (y t , v t )| ≤ |c t (x t , u t ) -c t (y t , u t )| + |c t (y t , u t ) -c t (y t , v t )| ≤ GD∥x t -y t ∥ + GD∥u t -v t ∥ ≤ GD 2 κ 2 (1 + κ)(1 -hγ) Hm+1, where the first inequality is by triangle inequality, the second inequality is by Assumption 2, Lemma 2, and the third inequality is by Lemma 2.With this, we haveR 1 = p-1 i=0   (i+1)m-1 j=im c i (x i , u i ) -f i ( Mi-H , ..., Mi )   = p-1 i=0   (i+1)m-1 j=im c i (x i , u i ) - (i+1)m-1 j=im c i (y i , v i )   ≤ p-1 i=0 (i+1)m-1 j=im GD 2 κ 2 (1 + κ)(1 -hγ) Hm+1 ≤ nGD 2 κ 2 (1 + κ)(1 -hγ) Hm+1 .Before start the proof of Lemma 5, we first present an overview of the online convex optimization (OCO) with memory framework. Consider the setting where, for every t, an online player chooses some point x t ∈ K ⊂ R d , a loss function f t : K H+1 → R is revealed, and the learner suffers a loss of f t (x t-H , . . . , x t ). We assume a certain coordinate-wise Lipschitz regularity on f t of the form such that, for any j ∈ {1, . . . , H}, for anyx 1 , . . . , x H , xj ∈ K |f t (x 1 , . . . , x j , . . . , x H ) -f t (x 1 , . . . , xj , . . . , x H )| ≤ L ∥x j -xj ∥ .In addition, we define ft (x) = f t (x, . . . , x), and we letG f = sup t∈{1,...,T },x∈K ∇ ft (x) , D f = sup x,y∈K ∥x -y∥ .The resulting goal is to minimize the policy regret, which is defined asRegret = T t=H f t (x t-H , . . . , x t ) -min x∈K T t=H f t (x, . . . , x) .Algorithm 2 Online Gradient Descent with Memory (OGD-M) Input:Step size η, functions {f t } T t=m . Initialize x 0 , . . . , x H-1 ∈ K arbitrarily. for t = H, . . . , T do Play x t , suffer loss f t (x t-H , . . . , x t ).Set x t+1 = Π K x t -η∇ ft (x) . end forTo minimize this regret, a commonly used algorithm is the Online Gradient descent. By running the Algorithm 2, we may bound the policy regret by the following lemma: Lemma 10. Let {f t } T t=1 be Lipschitz continuous loss functions with memory such that ft are convex. Then by runnning algorithm 2 itgenerates a sequence {x t } T t=1 such thatT t=H f t (x t-H , . . . , x t ) -min x∈K T t=H f t (x, . . . , x) ≤ D 2 f η + T G 2 f η + LH 2 ηG f T . Furthermore, setting η = D f √ G f (G f +LH 2 )T implies that PolicyRegret ≤ 2D f G f (G f + LH 2 ) T .Proof. By the standard OGD analysis [18] , we know thatT t=H ft (x t ) -min x∈K T t=H ft (x) ≤ D 2 f η + T G 2 η.In addition, we know by the Lipschitz property, for any t ≥ H, we have|f t (x t-H , . . . , x t ) -f t (x t , . . . , x t )| ≤ L H j=1 ∥x t -x t-j ∥ ≤ L H j=1 j l=1 ∥x t-l+1 -x t-l ∥ ≤ L H j=1 j l=1 η ∇ ft-l (x t-l ) ≤ LH 2 ηG,and so we have thatT t=H f t (x t-H , . . . , x t ) - T t=H f t (x t , . . . , x t ) ≤ T LH 2 ηG. follows that T t=H f t (x t-H , . . . , x t ) -min x∈K T t=H f t (x, . . . , x) ≤ D 2 f η + T G 2 f η + LH 2 ηG f T .In this setup, the first term corresponds to the DAC policy we make, and the second term is used to approximate the optimal strongly stable linear policy. It is worth noting that the cost of OCO with memory depends on the update frequency H. Therefore, we propose a two-level online controller. The higher-level controller updates the policy with accumulated feedback at a low frequency to reduce the regret, whereas a lower-level controller provides high-frequency updates of the DAC policy to reduce the discretization error. In the following part, we define the update distance of the DAC policy as l = Hm, where m is the ratio of frequency between the DAC policy update and OCO memory policy update. Formally, we update the value of M t once every m transitions, where g t represents a loss function.M t+1 = Π M (M t -η∇g t (M )) if t%m == 0 M t otherwise .From now on, we denote Mt = M tm for the convenience to remove the duplicate elements. By the definition of ideal cost, we know that it is a well-defined definition.By Lemma 7 we know thaty t+1 = h 2Hm i=0 Ψ t,i ŵt-i , v t = -Ky t + h Hm j=1 M j t ŵt-j ,whereΨ t,i = Q i h 1 i≤l + h l j=0 Q j h BM i-j t-j 1 i-j∈[1,l] .So we know that y t and y t are linear combination of M t , thereforef i Mi-H , . . . , Mi = (i+1)m-1 t=im c t y t Mi-H , . . . , Mi , v t Mi-H , . . . , Mi .is convex in M t . So we can use the OCO with memory structure to solve this problem.By Lemma 9 we know that y t and v t are bounded by D. Then we need to calculate the diameter, Lipchitz constant, and gradient bound of this function f i . In the following, we choose the DAC policy parameter l = Hm. Lemma 11. (Bounding the diameter) We haveD f = sup Mi,Mj ∈M ∥M i -M j ∥ ≤ 2a hγ .Proof. By the definition of M, taking l = Hm we know that sup Mi,Mj ∈M ∥M i -M j ∥ ≤ Hm k=1 ∥M k i -M k j ∥ ≤ Hm k=1 2a(1 -hγ)f i Mi-H . . . Mi-k . . . Mi -f i Mi-H . . . Mi-k . . . Mi ≤ C 2 κ 3 κ B W 0 Hm j=0 ∥ M j i-k -M j i-k ∥ ,where C is a constant.Proof. By the definition we have∥y t -ỹt ∥ = ∥h 2Hm i=0 h Hm j=0 Q j h B(M i-j t-j -M i-j t-j )1 i-j∈[1,Hm] ŵt-i ∥ ≤ h 2 κ 2 κ B W 0 2Hm i=0 Hm j=0 ∥M i-j t-j -M i-j t-j ∥1 i-j∈[1,Hm] ≤ h 2 κ 2 κ B W 0 m Hm j=0 ∥ M j i-k -M j i-k ∥ = hCκ 2 κ B W 0 Hm j=0 ∥ M j i-k -M j i-k ∥ .Where the first inequality is by ∥Q j h ∥ ≤ κ 2 (1 -hγ) j-1 ≤ κ 2 and lemma 9 of bounded estimation disturbance, the second inequality is by the fact that M i-k have taken m times, the last equality is by m = C h . Furthermore, we have that∥v t -ṽt ∥ = ∥ -K (y t -ỹt ) ∥ ≤ hCκ 3 κ B W 0 Hm j=0 M j i-k -M j i-k .Therefore using Assumption 2, Lemma 9 and Lemma 2 we immediately get thatf i Mi-H . . . Mi-k . . . Mi -f i Mi-H . . . Mi-k . . . Mi ≤ C 2 κ 3 κ B W 0 Hm j=0 ∥ M j i-k -M j i-k ∥ .Lemma 13. (Bounding the Gradient) We have the following bound for the gradient:∥∇ M f t (M . . . M )∥ F ≤ GDCκ 2 (κ + 1)W 0 κ B γProof. Since M is a matrix, the ℓ 2 norm of the gradient ∇ M f t corresponds to the Frobenius norm of the ∇ M f t matrix. So it will be sufficient to derive an absolute value bound on ∇ M [r] p,q f t (M, . . . , M ) for all r, p, q. To this end, we consider the following calculation. Using lemma 9 we get that y t (M . . . M ), v t (M . . . M ) ≤ D. Therefore, using Assumption 2 we have that∇ M [r] p,q c t (M . . . M ) ≤ GD ∂y t (M ) ∂M [r] p,q + ∂v t (M . . . M ) ∂M [r] p,q.now bound the quantities on the right-hand side:δy t (M . . . M ) δM [r] p,q = h 2Hm i=0 h Hm j=1 ∂Q j h BM [i-j] ∂M [r] p,q ŵt-i 1 i-j∈[1,H] ≤ h 2 r+Hm i=r ∂Q i-r h BM [r] ∂M [r] p,q w t-i ≤ h 2 κ 2 W 0 κ B 1 hγ = hκ 2 W 0 κ B γ .Similarly,∂v t (M . . . M ) ∂M [r] p,q ≤ κ δy t (M . . . M ) δM [r] p,q ≤ κ hκ 2 W 0 κ B γ ≤ hκ 3 W 0 κ B γ .Combining the above inequalities withf i Mi-H , . . . , Mi = (i+1)m-1 t=im c t y t Mi-H , . . . , Mi , v t Mi-H , . . . , Mi .gives the bound that∥∇ M f t (M . . . M )∥ F ≤ GDCκ 2 (κ + 1)W 0 κ B γ .Finally we prove Lemma 5:Lemma 5. Under Assumption 1 and 2, choosing m = C h and η = Θ( m T h ), Algorithm 1 attains the following bound of R 2 :R 2 = p-1 i=0 f i ( Mi-H , . . . , Mi ) -min M ∈M p-1 i=0 f i (M, . . . , M ) ≤ 4a γ GDC 2 κ 2 (κ + 1)W 0 κ B γ ( GDCκ 2 (κ + 1)W 0 κ B γ + C 2 κ 3 κ B W 0 H 2 ) n h .Proof. By Lemma 10 we haveR 2 ≤ 2D f G f (G f + LH 2 ) pBy Lemma 11, Lemma 12, and Lemma 13 we haveR 2 ≤ 2D f G f (G f + LH 2 ) p ≤ 2 2a hγ GDCκ 2 (κ + 1)W 0 κ B γ ( GDCκ 2 (κ + 1)W 0 κ B γ + C 2 κ 3 κ B W 0 H 2 ) n m ≤ 4a γ GDC 2 κ 2 (κ + 1)W 0 κ B γ ( GDCκ 2 (κ + 1)W 0 κ B γ + C 2 κ 3 κ B W 0 H 2 ) n h .In section, we will prove the approximation value of DAC policy and optimal policy is sufficiently small. First, we introduce the following: Lemma 14. For any two (κ, γ)-strongly stable matrices K * , K, there exists M = M 1 , . . . , M Hm whereM i = (K -K * ) (I + h(A -BK * )) i-1 , such that c t (x t (M ), u t (M )) -c t (x * t , u * t ) ≤ GDW 0 κ 3 a(lhκ B + 1)(1 -hγ) Hm . Proof. Denote Q h (K) = I + h(A -BK), Q h (K * ) = I + h(A -BK * ). By Lemma 7 we havex * t+1 = h t i=0 Q i h (K * ) ŵt-i .Consider the following calculation for i ≤ Hm andM i = (K -K * ) (I + h(A -BK * )) i-1 : Ψ t,i (M, . . . , M ) = Q i h (K) + h i j=1 Q i-j h (K)BM j = Q i h (K) + h i j=1 Q i-j h (K)B (K -K * ) Q j-1 h (K * ) = Q i h (K) + i j=1 Q i-j h (K)(Q h (K * ) -Q h (K))Q j-1 h (K * ) = Q i h (K * ) ,where the final equality follows as the sum telescopes. Therefore, we have thatx t+1 (M ) = h Hm i=0 Q i h (K * ) ŵt-i + h t i=Hm+1 Ψ t,i ŵt-i .Then we obtain thatx t+1 (M ) -x * t+1 ≤ hW 0 t i=Hm+1 (∥Ψ t,i (M * )∥ + ∥Q i h (K * )∥) .Using Definition 1 and Lemma 7 we finally getx t+1 (M ) -x * t+1 ≤ hW 0 ( t i=Hm+1 ((lhκ B + 1)aκ 2 (1 -hγ) i-1 ) + κ 2 (1 -hγ) i ) ≤ W 0 (lhκ B + 2)aκ 2 (1 -hγ) Hm .also have∥u * t -u t (M )∥ = -K * x * t + Kx t (M ) -h Hm i=0 M i ŵt-i = ∥(K -K * )x * t + K(x t (M ) -x * t ) -h Hm i=0 M i ŵt-i ∥ = ∥(K -K * )h t-1 i=0 Q i h (K * ) ŵt-i + K(x t (M ) -x * t ) -h Hm i=0 M i ŵt-i ∥ = ∥K(x t (M ) -x * t ) -h t-1 i=Hm+1 (K -K * )Q i-1 h (K * ) ŵt-i ∥ = ∥Kh t-1 i=Hm+1 (Ψ t,i -Q i-1 h (K * )) ŵt-i -h t-1 i=Hm+1 (K -K * )Q i-1 h (K * ) ŵt-i ∥ = h t-1 i=Hm+1 K * Q i-1 h (K * ) + Ψ t,i ŵt-i ≤ W 0 κ((1 -hγ) Hm + a(lhκ B + 1)κ 2 (1 -hγ) Hm ) = W 0 κ(a(lhκ B + 1)κ 2 + 1)(1 -hγ) Hm ), where the inequality is by Definition 1 and Lemma 8.Finally, we have|c t (x t (M ), u t (M )) -c t (x * t , u * t )| ≤ |c t (x t (M ), u t (M )) -c t (x * t , u t (M ))| + |c t (x * t , u t (M )) -c t (x * t , u * t )| ≤GD|x t (M ) -x * t | + GD|u t (M ) -u * t | ≤GDW 0 κ 3 a(lhκ B + 1)(1 -hγ) Hm ,where the second inequality is by Assumption 2.Then we can prove our main lemma: Lemma 6. Under Assumption 1 and 2, Algorithm 1 attains the following bound of R 3 :R 3 = min M ∈M p-1 i=0 f i (M, ..., M ) - p-1 i=0 (i+1)m-1 j=im c i (x * i , u * i ) ≤ 3n(1 -hγ) Hm GDW 0 κ 3 a(lhκ B + 1) . Proof. By choosing M i = (K -K * ) (I + h(A -BK * )) i-1 .We know that∥M i ∥ = ∥ (K -K * ) (I + h(A -BK * )) i-1 ∥ ≤ 2κ 3 (1 -γ) i-1 .Therefore choose a = 2κ 3 we have M = {M i } in the DAC policy update class M.Then we have the analysis of the regret:R 3 = min M ∈M p-1 i=0 f i (M, ..., M ) - p-1 i=0 (i+1)m-1 j=im c i (x * i , u * i ) ≤ min M ∈M p-1 i=0 (i+1)m-1 j=im c i (x i (M ), u i (M )) - p-1 i=0 (i+1)m-1 j=im c i (x * i , u * i ) + nκ 2 (1 + κ)(1 -hγ) Hm+1 D ≤ 3n(1 -hγ) Hm GDW 0 κ 3 a(lhκ B + 1), where the first inequality is by Lemma 2 and the second inequality is by Lemma 14.Justification: Our code is simple, just use the traditional SAC algorithm with one line implement. Our main contribution is the theoretical analysis.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: Summary: Most of the existing works on online control have been on discrete-time linear systems, except for a handful of works that consider stochastic noise. This paper studies an online control problem with adversarial disturbances for a continuous time linear system. This paper proposes a two-level online controller to solve this problem. The higher-level controller symbolizes the policy learning process and updates the policy at a low frequency to minimize regret. The lower-level controller delivers high-frequency feedback control input to reduce discretization error. The paper shows that the proposed algorithm achieves sublinear regret in the face of non-stochastic disturbances. The performance of the proposed algorithm is illustrated using simulation experiments in the Mujoco setting. 

We received four expert reviews, with the scores 7, 6, 6, 4, and the average score is 5.75. Reviewers are generally positive about the technical contribution and algorithmic novelty. They are also satisfied with the simulation experiments. One reviewer has expressed concerns about two aspects of technical proofs, in particular about the difficulty of analysis and the novel contribution. I believe that the authors have given a detailed and satisfactory answer to this comment. 

The reviewers have given many critical feedback on improving the quality of the paper. The authors have acknowledged them and agreed to incorporate these changes in their revision. Please make sure that you update your paper based on these comments while preparing the final submission.

Reviews: review: paper_summary: paper_summary: The work tackles the challenge of applying simulated controllers to real-world scenarios to manage continuous-time linear systems that face unpredictable disturbances. It introduces a double-layer control approach that balances slow policy updates with quick feedback adjustments. This way it effectively minimizing errors over time. The work is tested on various real world tests in simulation environments and it shows that it significantly boosts the performance of agents, even under uncertain situations.
strength_weakness: strength_weakness: -	The paper lacks a comprehensive comparative analysis with other state-of-the-art methods.
-	Experiments are conducted in simulated environments with domain randomization. There is limited discussion on how well these results generalize to physical systems outside of controlled simulations
questions: questions: -	In Section 3.1: the authors assumes access to \cdot{x}/t at each time step. This is a strong assumption, as measuring the state derivative directly is often not possible in practice. Work should discuss implications of this assumption & show it can be extended to more realistic case of only having access to state measurements.
-	Regret defn in section 3.4 compares to the best linear policy in hindsight. Why this is an appropriate comparator class? Why not best non-linear policy? more discussion needed on this choice of benchmark & limitation
-	Assumption 1 is quite strict. What if disturbances may occasionally have large spikes leading to violating these bounds (which happens in practical setting)? include discussion on this case
-	Assumption 2 on the cost function seems to imply the costs are quadratic (due to the D^2 term). But the paper also says the costs are convex, which is a more general class. Clarify precise assumption on the cost function here
-	Definition 1 of a strongly stable linear policy is central to the paper, but the motivation behind this definition is not entirely clear. 
-	The challenge of trading off approximation error & OCO regret in section 4 is subtle. More details needed on why this trade-off arises & how this two-level algorithm resolves it.
-	$w_t$ notion is inconsistent. line \sims 100 w_t is treated as a random variable with an unknown distribution. In line 98, it is a specific realization of the disturbance at time t, rather than a random variable.
limitations: limitations: -	Strict assumptions restrict the applicability of finding systems that meet these criteria
-	Method is tailored for non-stochastic disturbances, leaving out scenarios where disturbances might have stochastic elements.
paper_summary: paper_summary: This paper studies the sample complexity of online control for continuous-time linear systems. A novel two-level online algorithm was proposed. Specifically, a sublinear regret is guaranteed. Finally, the authors applied their method to the SAC algorithm and achieves improved simulation results.
strength_weakness: strength_weakness: 1. There is a gap between the theoretical results and the experimental results. It is not clear how the setting studied in this paper is implemented in the experiments. More discussion is needed.


2. There are several minor writing or notation errors that need to be checked carefully. For example:

line 21 they "has"

line 36 it "reliers"

line 104, missing definition of $\mathcal{K}$
questions: questions: How the setting studied in this paper is related to the experiments.
limitations: limitations: As mentioned in the paper, the theroical results relies on the access of system model and the loss function is convex. Extentions to model-free methods should be discussed.
paper_summary: paper_summary: This paper presents an algorithm for the non-stochastic control problem for continuous-time systems. In particular, the proposed algorithm discretizes the continuous-time system and balances the sampling time for discretization and update frequence of the online learning algorithm, in order to obtain a sublinear regret guarantee. The theoretical results are validated in numerical experiments.
strength_weakness: strength_weakness: In my opinion, the main weakness of this work is that the presentation of the results is often unclear and/or confusing. In particular, 
- the way the authors use to denote time steps is very confusing and often $t+1$ and $t+h$ seem to be used interchangeably (compare, e.g., line 8 'Use the action...' in Algorithm 1). Similarly $t$ and $i$ seem to be used interchangeably in Lemma 12. As this paper works with both, continuous and discrete-time, I believe it is highly important to use a notation that is less ambiguous.
- the matrices $M$ of the DAC policy are sometimes used with square brackets (for either the sub- or superscript) and sometimes without apparently without meaning different matrices
- $n$ and $m$ are most frequently used to denote the state and input dimension of the dynamial system, I suggest using different constants in Algorithm 1
- There are many typos in the proofs in the appendix, for example there should not be an 'a' in (2), the constant $a$ is generally not defined (at some point you introduce the condition $|M^i_t | \leq a(1-h\gamma)^{i-1}$, but this does not match the DAC policy class in Algorithm 1?),  I believe the definition of $\gamma_1$ after line 555 is wrong, above line 582, you require $n=\lfloor\frac{T}{h}\rfloor$ contrary to your definition in Algorithm 1 (which, however, you require earlier on in the proof; similar arguments hold for $p$ after line 592), and after line 622, the first sum should start at $i=1$ by definition of $u_t$, whereas it should be $Q_h^i$ instead of $Q_h^{i-1}$ when replacing $x_t(M)-x_t^*$. More generally, I believe that the results are qualitatively correct, but the constants (which are omitted in the $\mathcal O$-notation) are frequently wrong so that I was not able to verify all the presented results
questions: questions: I am not sure whether I fully understand why updating the DAC policy at each time step (compared to every $m$-th time step as you propose) does not work. Intuitively, I would assume that updating the DAC policy more frequently yields better performance, and therefore also a lower regret bound. After Lemma 10 (page 24) the authors argue 'It is worth noting that the cost of OCO with memory depends on the the update frequency $H$.' However, $H$ is not immediately connected to the update frequency, but rather denotes the size of the memory in the OCO with memory framework described above Lemma 10. Could you please clarify why updating the DAC policy less often leads to improved performance (i.e., lower regret)?
limitations: limitations: From the proof (e.g., after line 549), you seem to assume $x_0=0$. While this assumption is very commonly adopted in the related literature, it should be stated somewhere in the main body of the text explicitly. Furthermore, I am not sure whether this assumption is restrictive: It requires the system to be initialized at i) a steady state of the system and ii) the steady state of the system, that the benchmark policy $K^*$ aims to stabilize.
paper_summary: paper_summary: This paper considers the problem of online non-stochastic control within continuous-time linear dynamical systems. To this end, the authors proposed a two-level online algorithm, where the higher level deals with a discretized reduced problem to minimize the performance measure of regret, while the lower level handles the discretization error. This work achieves an $O(\sqrt{T} \log T)$ regret, which is the same as that in the discretized system. Finally, empirical evaluations validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: Although this work has done a nice job in handling continuous-time systems in the online non-stochastic control setup, my major concern is that the technical novelty seems insufficient. 

Specifically, as the authors have said, two technical challenges exist to achieve the main Theorem 1. The first one is to control the magnitude of the states, and the second one is to handle the curse of dimensionality caused by discretization. As for the first technical challenge, the idea of "employ Gronwall’s inequality to bound the first and second-order derivatives in the neighborhood of the current state" (Line 207-209) seems to be pretty natural because of the continuous nature of the system. And the second technical challenge seems to be leveraging the *lazy-update* paradigm to facilitate some kind of parameter tuning for the desired regret bound. Both technical challenges seem to be a little minor and thus insufficient from my point of view. Could the authors provide more detailed explanations on the technical challenges of dealing with continuous-time systems compared with the analysis in a discretized one?
questions: questions: In this part, I list some other questions or suggestions:
1. In Line 90, the authors said "However, it is unclear why domain randomization works in theory". However, this work does not illuminate the role of domain randomization in theory either because the DR only appears in the experiments part. I suggest that the authors could revise the sentence to avoid unnecessary misunderstanding of this work's contributions.
2. Section 3.2, in my opinion, could be put in the appendix because the relationship with robust control should be explained as a part of the related work, but not necessarily to be mentioned when introducing the problem setup of *this* work.
3. Assumption 1, compared with the standard non-stochastic disturbances in discretized systems, uses an extra condition of $\|\dot{w}_t\| \le W$. I guess that this additional condition is used to solve the first technical challenge (employ Gronwall’s inequality to bound the first and second-order derivatives in the neighborhood of the current state)? Am I right? Besides, the comparison of Assumption 1 with the previous one should be explicitly explained in the next version.
4. Definition 1 is a little different from the standard strong stability condition in discretized systems. In discretized systems, a linear controller is strongly stable if $A - BK = P L P^{-1}$. However, in this work, the strong stability holds for $I + h(A-BK)$. By setting $h=1$ as a special case, this assumption is still different with an extra identity matrix $I$. Why are they different? And the comparison with the standard discretized strong stability assumption should be explained in the next version.
5. In Section 4, some citations of the pioneering work [Agarwal et al., ICML 2019] are missing. For example,  Definition 2,3,4 should cite [Agarwal et al., ICML 2019]. I understand that there exist minor differences in the definitions in this work compared with those in [Agarwal et al., ICML 2019] for discretized systems. However, correct citations should still be provided to avoid unnecessary misunderstandings of this work's contributions.
6. In the equation after Line 172, the notation of '$g_t$' is not defined. I noticed that $g_t$ is first defined in Algorithm 1 on the next page.
7. In Figure 2, the font sizes of 'Replay' and 'Buffer' seem to be different?
limitations: limitations: The authors have adequately addressed the limitations and, if applicable, potential negative societal impact of their work.

Comments: title: Thank you
Thank you!
Response to further feedback
Thanks for the feedback
comment: We thank the reviewer for acknowledging our work. We will incorporate the reviewer's valuable suggestions in the next version of our paper.
Thank you for the detailed response. Some of the answers have helped clarify the questions raised in my review. I have no further questions at this point. I hope the authors can leverage the rebuttal answers (specifically A1, A2 and A6) in the subsequent version of this manuscript.
We greatly appreciate the reviewer's feedback and suggestions! We will carefully revise these notations to ensure a clearer distinction between discrete and continuous systems. The square brackets in line 173 is a floor function and it is used to simplify notation in subsequent statements. Since our policy parameter updates only once every $m$ steps, the $M$ remains the same within the interval $[im,(i+1)m]$, and we use the floor function to denote the same $M$. The square brackets in Algorithm 1 follow the notation of [1], but we did not realize it overlaps with the previous floor function notation. We will correct this typo in the next version of the paper.  

We once again thank the reviewer for their careful reading of our paper, which helps improve the quality of our work.

References

[1] Agarwal, Naman, et al. "Online control with adversarial disturbances." International Conference on Machine Learning. PMLR, 2019.
Thank you for your detailed response. Your explanations have clarified most of my concerns, most importantly A1. Below are some clarifications on the points I raised in my original review:
* A2: I appreciate your effort to clarify the time indices. However, I am not sure whether removing $h$ would be helpful. My suggestion would have been to, e.g., denote continuous time by $t$ (but keep the use of $h$), discrete time by $k\in\mathbb{N}$, and connecting them by $k=t/h$ to clearly differentiate between continuous time and discrete time steps.
* A3: I understood the meaning of the sub- and superscripts of $M$, but I am confused regarding the use of square brackets, e.g., in line 161 (where you use none), line 173 (where the subscript has square brackets) and Algorithm 1 (where the superscript has square brackets in the definition of the DAC policy update class only).
* A5: Since the benchmark policy is stable, the corresponding state trajectory will converge to $0$ in the absence of disturbances and to a neighborhood of $0$ for bounded disturbances. Thus, I believe that assuming $x_0=0$ does not just correspond to assuming that the system is initialized at steady state, but at a very specific steady state. However, I also believe that allowing any initialization would only lead to an additional constant cost upper bounding the regret of the transient trajectory, which would be neglected in the $\mathcal{O}$-notation anyway. Therefore, I do not think that this is a major issue.

Once again, I would like to thank the authors for their detailed responses. I would like to keep my original score, as it is already relatively high.
We sincerely thank the reviewer for the rapid and detailed feedback, which will greatly assist us in enhancing the quality of our work.

We appreciate the reviewer's acknowledgment that our responses to A2 and A3 have addressed your concerns and affirmed that our assumptions are not stronger than those in discretized systems. Although the solution to the second challenge is algorithmically simple, identifying the solution is nontrivial. The key is to see that the learning frequency for online control and the feedback frequency for reducing discretization errors are mismatched. This mismatch is unique to the continuous time system, and up to our knowledge, identified the first time. Overall, our main contribution is that we **provide the first non-asymptotic results for controlling continuous-time linear systems with non-stochastic noise**.

Regarding your suggestion to include comparisons with previous assumptions in discretized systems (A2 and A3) to aid reader comprehension, we will include these comparisons in the appendix.

Thank you again for your constructive feedback.
Thanks for the detailed feedback and explanations. I have read the response carefully, where A2 and A3 have solved my concerns raised in the original review. I believe these assumptions are not stronger than those in the discretized system. I also agree with the technical challenges mentioned in A1. 

However, in my opinion, the technical contributions are still a little weak. I agree with the first challenge that bounding the magnitude of states requires novel components in continuous systems. However, the second one that the authors have mentioned still seems to be a pretty natural operation, which uses lazy updates to achieve delicated parameter tuning. I do not see much challenge about this point. Besides, I think that the comparisons with previous assumptions in the discretized systems (A2 and A3) are necessary to help readers better understand whether the assumptions here are strong or not. If there are not enough spaces in the main paper, I suggest that the authors could put them in the appendix. Based on this, I am more inclined to believe another round of polishment would be better. Therefore I keep my current score.

At last, thank the authors again for the detailed responses.