SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that Assumption 2.2 is on the joint properties of both the dynamical system and the policy class when composed together in a closed loop. The motivation is to generalize two key properties of linear systems under typical reasonable controllers: 1) the effect of past decisions on the current state decays exponentially fast, and 2) if the system is initialized near the origin, it remains near the origin. We generalize these properties via  $\varepsilon$ -time-varying contractive perturbation (Definition 2.6) and  $\varepsilon$ -time-varying stability (Definition 2.7) respectively. Although Assumption 2.2 may seem complicated to understand, it is less restrictive than the assumptions in the most closely related work (e.g., [137]) that focus on linear dynamics.
Compared to other settings where contractive perturbation holds globally [1, 8, 54], Assumption 2.2 only requires it to hold locally in a bounded ball  $B(0, R_C)$ , which becomes important in nonlinear settings. This brings a new challenge because we need to guarantee that the starting state stays within  $B(0, R_C)$  whenever we apply this property in the proof. Therefore, in Assumption 2.2 we assume  $R_C > R_S + C \| x_0 \|$ . Similarly, to leverage the Lipschitzness/smoothness property, we require  $\mathcal{X} \supseteq B(0, R_x)$  where  $R_x \geq C(R_S + C \| x_0 \|) + R_S$  and  $\mathcal{U} = \{\pi(x, \theta) \mid x \in \mathcal{X}, \theta \in \Theta, \pi \in \mathcal{G}\}$ . Since the coefficients in Assumption 2.1 depend on  $\mathcal{X}$  and  $\mathcal{U}$ , we will set  $\mathcal{X} = B(0, R_x)$  and  $R_x = C(R_S + C \| x_0 \|) + R_S$  by default when presenting these constants. The goal is to ensure that the controller never leaves the region where contractive perturbation applies, which is critical for our analysis and again generalizes properties found in the literature (e.g., Examples 2.1, 2.2 and H.1).
For some systems, verifying Assumption 2.2 is straightforward (e.g., Example 2.1). In other cases, we can rely on the following lemma, which can convert a time-invariant version of the property to general time-varying one. We defer its proof to Appendix C
Lemma 2.8. Suppose Assumption  $\boxed{2.2}$  holds for  $\varepsilon = 0$  and  $(R_C, C, \rho, R_S)$ , which satisfies  $R_C > (C + 1)R_S$ . Suppose Assumption  $\boxed{2.1}$  also holds and let  $\mathcal{X} := B(0, R_x)$ , where  $R_x = (C + 1)^2 R_S$ . Then, Assumption  $\boxed{2.2}$  also holds for  $\hat{\varepsilon} > 0$ ,  $(\hat{R}_C, \hat{C}, \hat{\rho}, \hat{R}_S)$ , and  $x_0$  that satisfies  $(\hat{R}_C - \hat{R}_S) / C$ . Here,  $\hat{R}_S, \hat{R}_C, \hat{\rho}$  are arbitrary constants that satisfies  $R_S < \hat{R}_S < \hat{R}_C < R_C / (C + 1)$  and  $\rho < \hat{\rho} < 1$ . The positive constants  $\hat{\varepsilon}$  and  $\hat{C}$  are given detailed expressions in Appendix  $\boxed{C}$
Remark 2.9. Lemma 2.8 can also be useful when applied to some parameterized controllers for time-invariant nonlinear systems. For example, the well-known "computed torque control" feedback linearization controllers for robotic manipulators (see, e.g., [55]) renders the closed-loop dynamics exponentially stable about an equilibrium, and the feedback gains can be parameterized. Thus, it satisfies Assumption 2.2 in a neighborhood about the equilibrium, via Lemma 2.8 Even with time-invariant dynamics, the time-varying costs (such as tracking a trajectory determined online) provide a setting where selecting the policy parameters online can be beneficial.
3 Method and Theoretical Results
Our algorithm, Gradient-Based Adaptive Policy Selection (GAPS), is inspired by the classic online gradient descent (OGD) algorithm [32, 56], with a novel approach for approximating the gradient of the surrogate stage cost  $F_{t}$ . In the context of online optimization, OGD works as follows. At each time  $t$ , the current stage cost describes how good the learner's current decision  $\theta_{t}$  is. The learner updates its decision by taking a gradient step with respect to this cost. Mapping this intuition to online policy selection, the ideal OGD update rule would be the following.
Definition 3.1 (Ideal OGDP Update). At time step  $t$ , update  $\theta_{t + 1} = \prod_{\Theta}(\theta_t - \eta \nabla F_t(\theta_t))$
This is because the surrogate cost  $F_{t}$  (Definition 2.4) characterizes how good  $\theta_{t}$  is for time  $t$  if we had applied  $\theta_{t}$  from the start, i.e., without the impact of other historical policy parameters  $\theta_{0:t-1}$ . However, since the complexity of computing  $\nabla F_{t}$  exactly grows proportionally to  $t$ , the ideal OGD becomes intractable when the horizon  $T$  is large.
For  $\{g,\pi \}_{\times T}$  to be in  $\mathcal{G}$ , it must satisfy other assumptions about contractive perturbation and stability that we impose on  $\mathcal{G}$  but does not need to occur in real problem instances. We only use this assumption in the proof of Theorem 3.6 and it can be made without the loss of generality for time-invariant dynamics and policy classes.
As outlined in Algorithm  $\boxed{1}$ , GAPS uses  $G_{t}$  to approximate  $\nabla F_t(\theta_t)$  efficiently. To see this, we compare the decompositions, with key differences highlighted in colored text:
$$
\nabla F _ {t} \left(\theta_ {t}\right) = \sum_ {b = 0} ^ {t} \frac {\partial f _ {t | 0}}{\partial \theta_ {t - b}} \Bigg | _ {x _ {0}, \left(\theta_ {t}\right) _ {\times (t + 1)}} \quad \text {a n d} \quad G _ {t} = \sum_ {b = 0} ^ {\min  \{B - 1, t \}} \frac {\partial f _ {t | 0}}{\partial \theta_ {t - b}} \Bigg | _ {x _ {0}, \theta_ {0: t}}. \tag {4}
$$
GAPS uses two techniques to efficiently approximate  $\nabla F_t(\theta_t)$ . First, we replace the ideal sequence  $(\theta_{t})_{\times (t + 1)}$  by the actual sequence  $\theta_{0:t}$ . This enables computing gradients along the actual trajectory experienced by the online policy without re-simulating the trajectory under  $\theta_{t}$ . Second, we truncate the whole historical dependence to at most  $B$  steps. This bounds the memory used by GAPS.  $\varepsilon$ -time-varying contractive perturbation is the key to bound the bias of  $G_{t}$ : Intuitively, in the first step, although  $\theta_{\tau}$  becomes more different with  $\theta_{t}$  as  $\tau$  decreases, its impact on  $f_{t|0}$  decays more quickly (exponentially); In the second step, the terms that we discard are exponentially small with respect to  $B$ . We provide a rigorous bound of the bias in Theorem 3.2 and a proof outline in Appendix D.
Algorithm 1 Gradient-based Adaptive Policy Selection (GAPS)
Require: Learning rate  $\eta$ , buffer length  $B$ , initial  $\theta_0$ .
1: for  $t = 0, \dots, T - 1$  do
2: Observe the current state  $x_{t}$ .
3: Pick the control action  $u_{t} = \pi_{t}(x_{t},\theta_{t})$
4: Incur the stage cost  $c_{t} = f_{t}(x_{t},u_{t})$
5: Compute the approximated gradient:  $G_{t} = \sum_{b = 0}^{\min \{B - 1,t\}}\frac{\partial f_{t|0}}{\partial\theta_{t - b}}\Bigg{|}_{x_{0},\theta_{0:t}}$
6: Perform the update  $\theta_{t + 1} = \prod_{\Theta}(\theta_t - \eta G_t)$
7: end for
Algorithm  $\boxed{1}$  presents GAPS in its simplest form. Although the expression of the partial derivatives contains  $\theta_{0:t}$ , the time- and space-efficient implementation of GAPS only requires to store  $B$  partial derivatives for  $B$  previous time steps. Details are given in Algorithm  $\boxed{2}$  in Appendix  $\boxed{B}$ .
Compared to many previous online control algorithms that take a reduction approach based on OCO with Memory, our algorithm can be much more computationally efficient (see Appendix [4] for an empirical comparison). Specifically, these works [136] take a different finite-memory reduction approach toward reducing the online control problem to OCO with Memory [33] by completely removing the dependence on policy parameters before time step  $t - B$  for a fixed memory length  $B$ . In the finite-memory reduction, one must "imaginarily" reset the state at time  $t - B$  to be 0 and then use the  $B$ -step truncated multi-step cost function  $f_{t|t - B}(0,\theta_{t - B:t})$  in the OGD with Memory algorithm [1]. When applied to our setting, this is equivalent to replacing  $G_{t}$  in line of Algorithm 1 by  $G_{t}' = \sum_{b=0}^{B-1} \frac{\partial f_{t|t-B}}{\partial \theta_{t-b}} |_{0,(\theta_t)_{\times(B+1)}}$ . However, the estimator  $G_{t}'$  has limitations compared with  $G_{t}$  in GAPS. First, computing  $G_{t}'$  requires oracle access to the partial derivatives of the dynamics and cost functions for arbitrary state and actions. Second, even if those are available,  $G_{t}'$  is less computationally efficient than  $G_{t}$  in GAPS, especially when the policy is expensive to execute. Taking MPC (Example [2.1]) as an example, computing  $G_{t}'$  at every time step requires solving  $B$  MPC optimization problems when re-simulating the system, where  $B = \Omega(\log T)$ . In contrast, computing  $G_{t}$  in GAPS only requires solving one MPC optimization problem and  $O(B)$  matrix multiplications to update the partial derivatives. One may wonder how significant this improvement is, and if it affects the regret. To address this concern, we compare GAPS with Ideal OGD and OGD with Memory in the setting of MPC with confidence coefficients for a 2D double integrator. The simulation results show that GAPS achieve very similar regret with the two benchmarks, while the improvement on computation efficiency is significant (see Appendix [4]).
3.1 Bounds on Truncation Error
We now present the first part of our main result, which states that the actual stage cost  $f_{t}(x_{t},u_{t})$  incurred by GAPS is close to the ideal surrogate cost  $F_{t}(\theta_{t})$ , and the approximated gradient  $G_{t}$  is close to the ideal gradient  $\nabla F_{t}(\theta_{t})$ . In other words, GAPS mimics the ideal OGD update (Definition 3.1).
Theorem 3.2. Suppose Assumptions  $\boxed{2.1}$  and  $\boxed{2.2}$  hold. Let  $\{(x_t, u_t, \theta_t)\}_{t \in \mathcal{T}}$  denote the trajectory of GAPS (Algorithm  $\boxed{1}$ ) with buffer size  $B$  and learning rate  $\eta \leq \Omega((1 - \rho)\varepsilon)$ . Then, we have
$|f_{t}(x_{t},u_{t}) - F_{t}(\theta_{t})| = O\big((1 - \rho)^{-3}\eta \big)$  and  $\| G_t - \nabla F_t(\theta_t)\| = O\big((1 - \rho)^{-5}\eta +(1 - \rho)^{-1}\rho^B\big),$  where  $\Omega (\cdot)$  and  $O(\cdot)$  hide the dependence on the Lipschitz/smoothness constants defined in Assumption 2.1 and C in contractive perturbation - see details in Appendix D.1
We defer the proof of Theorem 3.2 to Appendix D.1. Note that this result does not require any convexity assumptions on the surrogate cost  $F_{t}$ .
3.2 Regret Bounds for GAPS: Convex Surrogate Cost
The second part of our main result studies the case when the surrogate cost  $F_{t}$  is a convex function. This assumption is explicitly required or satisfied by the policy classes and dynamical systems in many prior works on online control and online policy selection [1, 3, 6, 54].
The error bounds in Theorem 3.2 can reduce the problem of GAPS' regret bound in control to the problem of OGD's regret bound in online optimization, where the following result is well known: When the surrogate cost functions  $F_{t}$  are convex, the ideal OGD update (Definition 3.1) achieves the regret bound  $\sum_{t=0}^{T-1} F_{t}(\theta_{t}) - \min_{\theta \in \Theta} \sum_{t=0}^{T-1} F_{t}(\theta) = O(\sqrt{T})$ , when the step size  $\eta$  is of the order  $1/\sqrt{T}$ . By taking the biases on the stage costs and the gradients into consideration, we derive the adaptive regret bound in Theorem 3.3. Besides the adaptive regret, one can use a similar reduction approach to "transfer" other regret guarantees for OGD in online optimization to GAPS in control. We include the derivation of a dynamic regret bound as an example in Appendix E.
Theorem 3.3. Under the same assumptions as Theorem [3.2] if we additionally assume  $F_{t}$  is convex for every time  $t$  and  $\mathrm{diam}(\Theta)$  is bounded by a constant  $D$ , then GAPS achieves adaptive regret
$$
R ^ {A} (T) = O \big (\eta^ {- 1} + (1 - \rho) ^ {- 5} \eta T + (1 - \rho) ^ {- 1} \rho^ {B} T + (1 - \rho) ^ {- 1 0} \eta^ {3} T + (1 - \rho) ^ {- 2} \rho^ {2 B} \eta T \big),
$$
where  $O(\cdot)$  hides the same constants as in Theorem 3.2 and  $D$  — see details in Appendix D.1
We discuss how to choose the learning rate and the regret it achieves in the following corollary.
Corollary 3.4. Under the same assumptions as Theorem 3.3, suppose the horizon length  $T \gg \frac{1}{1 - \rho}$  and the buffer length  $B \geq \frac{1}{2}\log (T) / \log (1 / \rho)$ . If we set  $\eta = (1 - \rho)^{\frac{5}{2}}T^{-\frac{1}{2}}$ , then GAPS achieves adaptive regret  $R^{A}(T) = O((1 - \rho)^{-\frac{5}{2}}T^{\frac{1}{2}})$ .
We defer the proof of Theorem 3.3 to Appendix D.1 Compared to the (static) policy regret bounds of [1, 3], our bound is tighter by a factor of  $\log T$ . The key observation is that the impact of a past policy parameter  $\theta_{t - b}$  on the current stage cost  $c_{t}$  decays exponentially with respect to  $b$  (see Appendix D for details). In comparison, the reduction-based approach first approximates  $c_{t}$  with  $\hat{c}_t$  that depends on  $\theta_{t - B + 1:t}$ , and then applies general OCO with memory results on  $\hat{c}_t$  [1, 3]. General OCO with memory cannot distinguish the different magnitudes of the contributions that  $\theta_{t - B + 1:t}$  make to  $\hat{c}_t$ , which leads to the regret gap of  $B = O(\log T)$ .
In the more restrictive setting of linear time-invariant dynamics with the DAC policy class, the results of a concurrent work [10] can also be used to close the log  $T$  gap on static regret of online policy selection. In comparison, Theorem 3.3 considers more general time-varying dynamics and adopts the stronger metric of adaptive regret. As a practical matter, the follow-the-regularized-leader type of algorithm used by [10] is often (much) less computationally efficient than a gradient-based algorithm like GAPS. Nevertheless, [10] made distinct contributions by allowing the state space to be a general Banach space and providing a lower bound for OCO with unbounded memory.
3.3 Regret Bounds for GAPS: Nonconvex Surrogate Cost
The third part of our main result studies the case when the surrogate cost  $F_{t}$  is nonconvex. Before presenting the result, we formally define the variation intensity that measures how much the system changes over the whole horizon.
Definition 3.5 (Variation Intensity). Let  $\{g_t, \pi_t, f_t\}_{t \in \mathcal{T}}$  be a sequence of dynamics/policy/cost functions that the environment provides. The variation intensity  $V$  of this sequence is defined as
$$
\sum_ {t = 1} ^ {T - 1} \sup  _ {x \in \mathcal {X}, u \in \mathcal {U}} \| g _ {t} (x, u) - g _ {t - 1} (x, u) \| + \sup  _ {x \in \mathcal {X}, \theta \in \Theta} \| \pi_ {t} (x, \theta) - \pi_ {t - 1} (x, \theta) \| + \sup  _ {x \in \mathcal {X}, u \in \mathcal {U}} | f _ {t} (x, u) - f _ {t - 1} (x, u) |.
$$
Variation intensity is used as a measure of hardness for changing environments in the literature of online optimization that often appear in regret upper bounds (see [57] for an overview). Definition 3.5 generalizes one of the standard definitions to online policy selection. Using this definition, we present our main result for GAPS applied to nonconvex surrogate costs using the metric of local regret (3).
Theorem 3.6. Under the same assumptions as Theorem 3.2, if we additionally assume that  $\Theta = \mathbb{R}^d$  for some integer  $d$ , then GAPS satisfies local regret
$$
R ^ {L} (T) = O \left(\frac {1 + V}{(1 - \rho) ^ {3} \eta} + \frac {\eta T}{(1 - \rho) ^ {6}} + \frac {\rho^ {B} T}{(1 - \rho) ^ {2}} + \frac {\eta^ {3} T}{(1 - \rho) ^ {1 3}} + \frac {\rho^ {2 B} \eta T}{(1 - \rho) ^ {5}}\right),
$$
where  $O(\cdot)$  hides the same constants as in Theorem 3.2—see details in Appendix F.
Corollary 3.7. Under the same assumptions as Theorem 3.6, suppose the horizon length  $T \gg \frac{1}{1 - \rho}$  and the buffer length  $B \geq \frac{1}{2}\log (T) / \log (1 / \rho)$ . If we set  $\eta = (1 - \rho)^{\frac{3}{2}}(1 + V)^{\frac{1}{2}}T^{-\frac{1}{2}}$ , GAPS achieves local regret  $R^{L}(T) = O((1 - \rho)^{-\frac{9}{2}}(1 + V)^{\frac{1}{2}}T^{\frac{1}{2}})$ .
We defer the proof of Theorem 3.6 to Appendix F. Note that the local regret will be sublinear in  $T$  if the variation intensity  $V = o(T)$ . To derive the local regret guarantee in Theorem 3.6 we address additional challenges compared to the convex case. First, we derive a local regret guarantee for OGD in online nonconvex optimization. We cannot directly apply results from the literature because they do not use ordinary OGD, and it is difficult to apply algorithms like Follow-the-Perturbed-Leader [e.g. 58] to online policy selection due to constraints on information and step size. Then, to transfer the regret bound from online optimization to online policy selection, we show how to convert the measure of variation defined on  $F_{0:T-1}$  to our variation intensity  $V$  defined on  $\{g_t, \pi_t, f_t\}_{t \in \mathcal{T}}$ .
A limitation of Theorem 3.6 is that we need to assume  $\Theta$  is a whole Euclidean space so that GAPS will not converge to a point at the boundary of  $\Theta$  that is not a stationary point. Example 2.2 and Appendix H.3 show that one can re-parameterize the policy class to satisfy this assumption in some cases. Relaxing this assumption is our future work.
In this section we compare GAPS to strong baseline algorithms in settings based on Examples 2.1 and 2.2. Details are deferred to Appendix II due to space limitations. Appendix II also includes a third experiment comparing GAPS to a bandit-based algorithm for selecting the planning horizon in MPC, and a computation time comparison between GAPS and the alternative gradient approximation of [II].
Figure 2: Comparing GAPS and baseline [11] for online adaptation of a confidence parameter for MPC with disturbance predictions. Left: Confidence parameter. Right: Per-step cost. Shaded bands show  $10\% -90\%$  quantile range over randomized disturbance properties. See body for details.
MPC confidence parameter. We compare GAPS to the follow-the-leader-type method of [11] for tuning a scalar confidence parameter in model-predictive control with noisy disturbance predictions. The setting is close to Example [2.1] but restricted to satisfy the conditions of the theoretical guarantees in [11]. We consider the scalar system  $x_{t + 1} = 2x_t + u_t + w_t$  under non-stochastic disturbances  $w_t$  with the cost  $f_t(x_t, u_t) = x_t^2 + u_t^2$ . For  $t = 0$  to 100, the predictions of  $w_t$  are corrupted by a large amount of noise. After  $t > 100$ , the prediction noise is instantly reduced by a factor of 100. In this setup, an ideal algorithm should learn to decrease confidence level at first to account for the noise, but then increase to  $\lambda \approx 1$  when the predictions become accurate.
Figure 2 shows the values of the confidence parameter  $\lambda$  and the per-timestep cost generated by each algorithm. Both methods are initialized to  $\lambda = 1$ . The method of [11] rapidly adjusts to an appropriate confidence level at first, while GAPS adjusts more slowly but eventually reaches the same value. However, when the accuracy changes, GAPS adapts more quickly and obtains lower costs towards the end of the simulation. In other words, we see that GAPS behaves essentially like an instance of Ideal OGD with constant step size, which is consistent with our theoretical results (Theorem 3.2).
(a) Linear gains  $k_{p}$ ,  $k_{d}$  tuned by GAPS compared to LQR optimal.
Figure 3: Comparing GAPS and LQR baseline in nonlinear inverted pendulum system. Shaded bands show  $\pm 1$  standard deviation over the randomness of the disturbances. See body for details.
(b) Cumulative cost difference.
Linear controller of nonlinear time-varying system. We apply GAPS to tune the gain parameters of a linear feedback controller in a nonlinear inverted pendulum system. Every 100 seconds, the pendulum mass changes. The system reflects the smooth nonlinear dynamics and nonconvex surrogate costs in Example 2.2 although it differs in other details (see Appendices H.3 and I.2). We compare GAPS to a strong and dynamic baseline that deploys the infinite-horizon linear-quadratic regulator (LQR) optimal controller for the linearized dynamics at each mass. We simulate two disturbances: 1) i.i.d. Gaussian, and 2) Ornstein-Uhlenbeck random walk.
Figure 3a shows the controller parameters tuned by GAPS, along with the baseline LQR-optimal gains, for each disturbance type. The derivative gain  $k_{d}$  closely follows LQR for i.i.d. disturbances but diverges for random-walk disturbances, where LQR is no longer optimal. This is reflected in the cumulative cost difference between GAPS and LQR, shown in Figure 3b. GAPS nearly matches LQR under i.i.d. disturbances, but significantly outperforms it when the disturbance is a random walk. The results show that GAPS can both 1) adapt to step changes in dynamics on a single trajectory almost as quickly as the comparator that benefits from knowledge of the near-optimal analytic solution, and 2) outperform the comparator in more general settings where the analytic solution no longer applies.
In this paper, we study the problem of online adaptive policy selection under a general contractive perturbation property. We propose GAPS, which can be implemented more efficiently and with less information than existing algorithms. Under convexity assumptions, we show that GAPS achieves adaptive policy regret of  $O(\sqrt{T})$ , which closes the log  $T$  gap between online control and OCO left open by previous results. When convexity does not hold, we show that GAPS achieves local regret of  $O(\sqrt{(1 + V)T})$ , where  $V$  is the variation intensity of the time-varying system. This is the first local regret bound on online policy selection attained without any convexity assumptions on the surrogate cost functions. Our numerical simulations demonstrate the effectiveness of GAPS, especially for fast adaptation in time-varying settings.
Our work motivates interesting future research directions. For example, a limitation is that GAPS assumes all policy parameters can stabilize the system and satisfy contractive perturbation. A recent work on online policy selection relaxed this assumption by a bandit-based algorithm but requires  $\Theta$  to be a finite set [59]. An interesting future direction to study is what regret guarantees can be achieved when  $\Theta$  is a continuous parameter set and not all of the candidate policies satisfy these assumptions.
[57] Aryan Mokhtari, Shahin Shahrampour, Ali Jadbabaie, and Alejandro Ribeiro. Online optimization in dynamic environments: Improved regret rates for strongly convex problems. In 2016 IEEE 55th Conference on Decision and Control (CDC), pages 7195-7201. IEEE, 2016.
[58] Arun Sai Suggala and Praneeth Netrapalli. Online non-convex learning: Following the perturbed leader is optimal. In Algorithmic Learning Theory, pages 845–861. PMLR, 2020.
[59] Yingying Li, James A Preiss, Na Li, Yiheng Lin, Adam Wierman, and Jeff S Shamma. Online switching control with stability and regret guarantees. In Learning for Dynamics and Control Conference, pages 1138–1151. PMLR, 2023.
[60] Rolf Schneider. Convex Bodies: the Brunn-Minkowski Theory. Cambridge University Press, 2014.
[61] Hongyu Zhou, Zirui Xu, and Vasileios Tzoumas. Efficient online learning with memory via Frank-Wolfe optimization: Algorithms with bounded dynamic regret and applications to control. CoRR, abs/2301.00497, 2023.
[62] Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-2368, 2013.
[63] Ofer Dekel, Jian Ding, Tomer Koren, and Yuval Peres. Bandits with switching costs:  $T^{2/3}$  regret. In ACM Symposium on the Theory of Computing, pages 459-467, 2014.
[64] Yiheng Lin, Yang Hu, Guannan Qu, Tongxin Li, and Adam Wierman. Bounded-regret MPC via perturbation analysis: Prediction error, constraints, and nonlinearity. In Advances in Neural Information Processing Systems, 2022.