SlowGuess/NeurIPS_2023_OCR huggingface dataset:
These conditions are relatively standard for bandit convex optimization algorithms, and are needed for the novel BCO-with-memory algorithm which underpins our control algorithm.
Assumption 2.7 (Adversary).  $\{c_t(\cdot ,\cdot),\mathbf{w}_t^{\mathrm{adv}},\mathbf{e}_t^{\mathrm{adv}}\}_{t = 1}^T$  is chosen by the adversary ahead of time.
2.2 Disturbance Response Controllers
Regret compares the excess cost from executing our proposed control algorithm with respect to the cost of the best algorithm in hindsight from a given policy class. In particular, low regret against a rich policy class is a very strong near-optimality guarantee. We take the comparator policy class  $\Pi$  to be the set of disturbance response controllers (DRC), formally given by the following definition.
Definition 2.8 (Disturbance Response Controllers). The disturbance response controller (DRC) and the DRC policy class are defined as:
- A disturbance response controller (DRC) is a policy  $\pi_{M}$  parametrized by  $M = (M^{[j]})_{j=0}^{\bar{H}}$ , where  $H \in \mathbb{Z}_{++}$ , a sequence of  $H$  matrices in  $\mathbb{R}^{d_{\mathbf{u}} \times d_{\mathbf{y}}}$  s.t. the control at time  $t$  given by  $\pi_{M}$  is  $\mathbf{u}_t^{\pi_M} = \sum_{j=0}^{\bar{H}} M^{[j]} \mathbf{y}_{t-j}^{\mathbf{nat}}$ . We shorthand  $\mathbf{u}_t^M \stackrel{\text{def}}{=} \mathbf{u}_t^{\pi_M}$ .
- The DRC policy class parametrized by  $H \in \mathbb{Z}_{++}$ ,  $R \in \mathbb{R}_+$  is the set of all disturbance response controller with bounded length  $H$  and norm  $R$ :  $\mathcal{M}(H, R) = \{M = (M^{[j]})_{j=0}^{\bar{H}} \mid \| M \|_{\ell_1, \mathrm{op}} = \sum_{j=0}^{\bar{H}} \| M^{[j]} \|_{\mathrm{op}} \leq R\}$ .
Previous works have demonstrated the richness of the DRC policy class. In particular, Theorem 1 from Simchowitz et al. [2020] has established that the DRC policy class generalizes the state-of-art benchmark class of stabilizing linear dynamic controllers (LDC) with error  $e^{-\Theta(H)}$ .
2.3 Approach and Technical Challenges
The classical approach in online nonstochastic control of stable/stabilizable systems is to reduce to a problem of online convex optimization with memory. This insight relies on the exponentially decaying effect of past states and controls on the present, which allows approximating the cost functions as functions of the most recent controls.
A core technical challenge lies in the bandit convex optimization problem obtained from the bandit control problem. In the bandit setting, no gradient information is given to the learner, and thus the learner needs to construct a low-bias gradient estimator. Previous work uses the classical spherical gradient estimator proposed by Flaxman et al. [2004], but the regret guarantee is suboptimal. We would like to leverage the ellipsoidal gradient estimator proposed by Hazan and Levy [2014]. However, when extending to loss functions with memory, there is no clear mechanism for obtaining a low-bias bound for general convex functions. We exploit the quadratic structure of the LQR/LQG cost functions to build EBCO-M (Algorithm 1), which uses ellipsoidal gradient estimators. We note that even outside of the control applications, EBCO-M may be of independent interests in bandit online learning theory.
3 BCO with Memory: Quadratic and Strongly Convex Functions
As with previous works, our control algorithm will depend crucially on a generic algorithm for bandit convex optimization with memory (BCO-M). We present a new online bandit convex optimization with memory algorithm that explores the structure of quadratic costs to achieve near-optimal regret.
3.1 Setting and working assumptions
In the BCO-M setting with memory length  $H$ , we consider an algorithm playing against an adversary. At time  $t$ , the algorithm is asked to play its choice of  $y_{t}$  in the convex constraint set  $\mathcal{K}$ . The adversary chooses a loss function  $F_{t}: \mathcal{K}^{H} \to \mathbb{R}_{+}$  which takes as input the algorithm's current play as well as its previous  $\bar{H}$  plays. The algorithm then observes a cost  $F_{t}(y_{t - \bar{H}}, \dots, y_{t})$  (and no other information about  $F_{t}(\cdot)$ ) before it chooses and plays the next action  $y_{t + 1}$ . The goal is to minimize regret with respect to the expected loss, which is the excessive loss incurred by the algorithm compared to the best fixed decision in  $\mathcal{K}$ :
$$
\operatorname {R e g r e t} _ {T} \stackrel {\text {d e f}} {=} \sum_ {t = H} ^ {T} \mathbb {E} \left[ F _ {t} \left(y _ {t - \bar {H}}, \dots , y _ {t}\right) \right] - \min  _ {x \in \mathcal {K}} \sum_ {t = H} ^ {T} \mathbb {E} \left[ F _ {t} (x, \dots , x) \right].
$$
For notation convenience, we will at times shorthand  $y_{t - \bar{H}:t} \stackrel{\mathrm{def}}{=} (y_{t - \bar{H}}, \ldots, y_t) \in \mathcal{K}^H$ .
3.1.1 BCO-M assumptions
We make the following assumptions on the loss functions  $\{F_t\}_{t = H}^T$  and the constraint set  $\mathcal{K}$ .
Assumption 3.1 (Constraint set).  $\mathcal{K}$  is convex, closed, and bounded with non-empty interior.  $\operatorname{diam}(\mathcal{K}) = \sup_{z,z' \in \mathcal{K}} \| z - z' \|_2 \leq D$ .
Assumption 3.2 (Loss functions). The loss functions chosen by the adversary obeys the following regularity and curvature assumptions:
-  $F_{t}:\mathcal{K}^{H}\to R_{+}$  is quadratic and  $\beta$  -smooth:
- Quadratic:  $\exists W_t \in \mathbb{R}^{nH \times nH}, b_t \in \mathbb{R}^{nH}, c_t \in \mathbb{R}$  such that  $F_t(w) = w^\top W_t w + b_t^\top w + c_t, \forall w \in \mathcal{K}^H$ .
- Smooth:  $W_{t} \preceq \beta I_{nH \times nH}$ .
-  $F_{t}:\mathcal{K}^{H}\to \mathbb{R}_{+}$  is  $\sigma$ -strongly convex in its induced unary form:  $f_{t}:\mathcal{K}\rightarrow \mathbb{R}_{+}$  with  $f_{t}(z) = F_{t}(z,\ldots ,z)$  is  $\sigma$ -strongly convex, i.e.  $f_{t}(z)\geq f_{t}(z^{\prime}) + \nabla f_{t}(z^{\prime})^{\top}(z - z^{\prime}) + \frac{\sigma}{2}\| z - z^{\prime}\|_{2}^{2},\forall z,z^{\prime}\in \mathcal{K}.$
-  $F_{t}$  satisfies the following diameter and gradient bound on  $\mathcal{K}$ :  $\exists B, L > 0$  such that
$$
B = \sup  _ {w, w ^ {\prime} \in \mathcal {K} ^ {H}} | F _ {t} (w) - F _ {t} \left(w ^ {\prime}\right) |, L = \sup  _ {w \in \mathcal {K} ^ {H}} \| \nabla F _ {t} (w) \| _ {2}.
$$
In the online control problems, when formulating the cost function  $c_{t}$  as a function  $F_{t}$  of the most recent  $H$  controls played, the function  $F_{t}$  itself may depend on the entire history of the algorithm through step  $t - H$ . Therefore, it is essential to analyze the regret guarantee of our BCO-M algorithm when playing against an adversary that can be  $(t - H)$ -adaptive, giving rise to the following assumption.
Assumption 3.3 (Adversarial adaptivity). The adversary chooses  $F_{t}$  independently of the noise  $u_{t - \bar{H};t}$  which is drawn by the algorithm in the  $H$  most recent steps, but possibly not independently of earlier noises.
Note that Assumption 3.3 is minimal for BCO: if this fails, then in the subcase of a delayed loss, the adversary may fully control the agent's observations, resulting in no possibility of learning.
Self-concordant barrier. The algorithm makes use of a self-concordant barrier  $R(\cdot)$  of  $\mathcal{K}$  as the regularization function in the updates.
Definition 3.4 (Self-concordant barrier). A three-time continuously differentiable function  $R(\cdot)$  over a closed convex set  $\mathcal{K} \subset \mathbb{R}^n$  with non-empty interior is a  $\nu$ -self-concordant barrier of  $\mathcal{K}$  if it satisfies the following two properties:
1. (Boundary property) For any sequence  $\{x_{n}\}_{n\in \mathbb{N}}\subset \mathrm{int}(\mathcal{K})$  such that  $\lim_{n\to \infty}x_n = x\in \partial \mathcal{K}$ ,  $\lim_{n\to \infty}R(x_n) = \infty$ .
2. (Self-concordant)  $\forall x\in \operatorname {int}(\mathcal{K}),h\in \mathbb{R}^n,$
(a)  $|\nabla^3 R(x)[h,h,h]|\leq 2|\nabla^2 R(x)[h,h]|^{3 / 2}.$
(b)  $|\langle \nabla R(x),h\rangle |\leq \sqrt{\nu} |\nabla^{2}R(x)[h,h]|^{1 / 2}.$
3.2 Algorithm specification and regret guarantee
We present EBCO-M (Algorithm 1) for online bandit convex optimization with memory. The key novelty is the use of an ellipsoidal gradient estimator. It is difficult to establish a low-bias guarantee for ellipsoidal gradient estimator for general convex loss functions. However, thanks to the quadratic structure of the loss functions in LQR/LQG problems, we can show provable low bias for the ellipsoidal gradient estimator, and therefore achieve optimal regret.
Algorithm 1 Ellipsoidal BCO with memory (EBCO-M)
1: Input: Convex, closed set  $\mathcal{K} \subseteq \mathbb{R}^n$  with non-empty interior, time horizon  $T$ , memory length  $H$ , step size  $\eta$ ,  $\nu$ -self-concordant barrier  $R(\cdot)$  over  $\mathcal{K}$ , convexity strength parameter  $\sigma$ .  
2: Initialize  $x_t = \arg \min_{x \in \mathcal{K}} R(x), \forall t = 1, \dots, H$ .  
3: Compute  $A_t = (\nabla^2 R(x_t) + \eta \sigma t I)^{-1/2}, \forall t = 1, \dots, H$ .  
4: Sample  $u_1, \dots, u_H \sim S^{n-1}$  i.i.d. uniformly at random.  
5: Set  $y_t = x_t + A_t u_t, \forall t = 1, \dots, H$ .  
6: Set  $g_t = 0, \forall t = 1, \dots, \bar{H}$ .  
7: Play  $y_1, \dots, y_{\bar{H}}$ .  
8: for  $t = H, \dots, T$  do  
9: Play  $y_t$ , suffer loss  $F_t(y_{t-\bar{H};t})$ .  
10: Store  $g_t = n F_t(y_{t-\bar{H};t}) \sum_{i=0}^{\bar{H}} A_{t-i}^{-1} u_{t-i}$ .  
11: Set  $x_{t+1} = \arg \min_{x \in \mathcal{K}} \sum_{s=H}^{t} (g_{s-\bar{H}}^\top x + \frac{\sigma}{2} \|x - x_{s-\bar{H}}\|^2) + \frac{1}{\eta} R(x)$ .  
12: Compute  $A_{t+1} = (\nabla^2 R(x_{t+1}) + \eta \sigma (t+1) I)^{-1/2}$ .  
13: Sample  $u_{t+1} \sim S^{n-1}$  uniformly at random.  
14: Set  $y_{t+1} = x_{t+1} + A_{t+1} u_{t+1}$ .  
15: end for
Before analyzing the regret, we first make note of two properties of Algorithm 1.
Remark 3.5 (Delayed dependence). In Algorithm 1,  $x_{t}$  is independent of  $u_{t - \bar{H}:t}$ ,  $\forall t$ , and therefore  $A_{t}$  is independent of  $u_{t - \bar{H}:t}$  as  $A_{t}$  is determined by  $x_{t}$ .
Remark 3.6 (Correctness).  $y_{t}$  played by Algorithm 1 lies in  $\mathcal{K}$ :  $\| y_{t} - x_{t}\|_{\nabla^{2}R(x_{t})}^{2} = \| A_{t}u_{t}\|_{\nabla^{2}R(x_{t})}^{2} \leq \| u_{t}\|_{2}^{2} = 1$ , and by Proposition C.1, the Dikin ellipsoid centered at  $x_{t}$  is contained in  $\mathcal{K}$ .
We give the following two theorems on regret guarantees of the EBCO-M algorithm. In particular, Theorem 3.6.B is a slight generalization of Theorem 3.6.A in its relaxation on the curvature assumption on the sequence of loss functions  $F_{H:T}$ .
Theorem 3.6.A (EBCO-M regret with strong convexity). For any sequence of cost functions  $\{F_t\}_{t = H}^T$  satisfying Assumption 3.2, constraint set  $\mathcal{K}$  satisfying Assumption 3.1, adversary satisfying Assumption 3.3, and  $H = \mathrm{poly}(\log T)$ , Algorithm 1 satisfies the regret bound
$$
\operatorname {R e g r e t} _ {T} (E B C O - M) = \sum_ {t = H} ^ {T} \mathbb {E} \left[ F _ {t} \left(y _ {t - \bar {H}: t}\right) \right] - \min  _ {x \in \mathcal {K}} \sum_ {t = H} ^ {T} \mathbb {E} \left[ f _ {t} (x) \right] \leq \tilde {\mathcal {O}} \left(\frac {\beta n}{\sigma} \sqrt {T}\right),
$$
where the expectation is taken over the randomness of the exploration noises  $u_{1:T}$ , with  $\tilde{\mathcal{O}}$  hiding all natural parameters  $(B, D, L)$  and logarithmic dependence on  $T$ .
Theorem 3.6.B (EBCO-M regret with conditional strong convexity). Suppose Algorithm 1 is run on  $\mathcal{K}$  satisfying Assumption 3.1 against an adversary satisfying Assumption 3.3 with a sequence of random cost functions  $\{F_t\}_{t = H}^T$  satisfying that
1.  $F_{t}$  is quadratic, convex,  $\beta$ -smooth, has diameter bound  $B$  and gradient bound  $L$  on  $\mathcal{K}^H$ .
2.  $\exists$  filtration  $\mathcal{G}_{1:T}$  such that  $u_{1:T}$  is independent of  $\mathcal{G}_T$ ,  $F_t \in \sigma(\mathbf{u}_{1:t-H}) \cup \mathcal{G}_t$ , and  $F_t$  is conditionally  $\sigma$ -strongly convex in its induced unary form:  $\bar{f}_t(z) \stackrel{\text{def}}{=} \mathbb{E}[f_t(z) \mid u_{1:t-H}, \mathcal{G}_{t-H}]$  is  $\sigma$ -strongly convex.
Then, Algorithm 1 satisfies the same regret bound attained in Theorem 3.6.A, i.e.
$$
R e g r e t _ {T} (E B C O - M) = \sum_ {t = H} ^ {T} \mathbb {E} [ F _ {t} (y _ {t - \bar {H}: t}) ] - \min  _ {x \in \mathcal {K}} \sum_ {t = H} ^ {T} \mathbb {E} [ f _ {t} (x) ] \leq \tilde {\mathcal {O}} \left(\frac {\beta n}{\sigma} \sqrt {T}\right),
$$
where the expectation is taken over the randomness of the exploration noises  $u_{1:T}$  and the random functions  $F_{H:T}$ , with  $\tilde{\mathcal{O}}$  hiding all natural parameters  $(B, D, L)$  and logarithmic dependence on  $T$ .
4 Bandit Controller: Known and Unknown Systems
We will now use our BCO-with-memory algorithm to find an optimal controller (as in Gradu et al. [2020]), arguing that regret in choice of controller transfers into the setting discussed in the previous section. We first consider the case where the system is known, and then reduce the unknown system case to the known system case.
4.1 Known systems
Applying Algorithm 1 to predict controllers<sup>4</sup> with losses given by control losses, we obtain Algorithm 2.
Algorithm 2 Ellipsoidal Bandit Perturbation Controller (EBPC)
1: Input: Time horizon  $T$ , memory length  $H$ , Markov operator  $G$ . BCO-M parameters  $\sigma, \eta$ . Self-concordant barrier  $R(\cdot)$  over  $\mathcal{M}(\bar{H}, R) \subset \mathbb{R}^{H \times d_{\mathbf{u}} \times d_{\mathbf{y}}}$ .  
2: Initialize  $M_1 = \dots = M_H = \arg \min_{M \in \mathcal{M}(H, R)} R(M)$ .  
3: Compute  $A_i = (\nabla^2 R(M_i) + \eta \sigma t I)^{-1/2}, \forall i = 1, \ldots, H$ .  
4: Sample  $\varepsilon_1, \ldots, \varepsilon_H \sim S^{H \times d_{\mathbf{u}} \times d_{\mathbf{y}} - 1}$  i.i.d. uniformly at random.  
5: Set  $\widetilde{M}_i = M_i + \varepsilon_i, \forall i = 1, \ldots, H$ . Set  $g_i = 0, \forall i = 1, \ldots, \bar{H}$ .  
6: Play control  $\mathbf{u}_i = 0$ , incur cost  $c_i(\mathbf{y}_i, \mathbf{u}_i), \forall i = 1, \ldots, \bar{H}$ .  
7: for  $t = H, \ldots, T$  do  
8: Play control  $\mathbf{u}_t = \mathbf{u}_t(\widetilde{M}_t) = \sum_{i=0}^{\bar{H}} \widetilde{M}_t^{[i]} \mathbf{y}_{t-i}^{\mathbf{nat}}$ , incur cost  $c_t(\mathbf{y}_t, \mathbf{u}_t)$ .  
9: Observe  $\mathbf{y}_{t+1}$  and compute signal  $\mathbf{y}_{t+1}^{\mathbf{nat}} = \mathbf{y}_{t+1} - \sum_{i=1}^{t} G^{[i]} \mathbf{u}_{t-i}$ .  
10: Store  $g_t = d_{\mathbf{u}} d_{\mathbf{y}} H c_t(\mathbf{y}_t, \mathbf{u}_t) \sum_{i=0}^{\bar{H}} A_{t-i}^{-1} \varepsilon_{t-i}$ .  
11: Update  $M_{t+1} = \arg \min_{M \in \mathcal{M}(H, R)} \sum_{s=H}^{t} (\langle g_{s-\bar{H}}, M\rangle + \frac{\sigma}{2} \| M - M_{s-\bar{H}}\|^2) + \frac{1}{\eta} R(M)$ .  
12: Compute  $A_{t+1} = (\nabla^2 R(M_{t+1}) + \eta \sigma (t+1) I)^{-1/2}$ .  
13: Sample  $\varepsilon_{t+1} \sim S^{H \times d_{\mathbf{u}} \times d_{\mathbf{y}} - 1}$  uniformly at random. Set  $\widetilde{M}_{t+1} = M_{t+1} + A_{t+1} \varepsilon_{t+1}$ .  
14: end for
Theorem 4.1 (Known system control regret). Consider a linear dynamical system governed by known dynamics  $(A,B,C)$  and the interaction model with adversarially chosen cost functions
4Notation: while our controller  $M$  is typically a tensor, it should be thought of as the output vector of Algorithm 1. As such, the relevant vector and matrix operations in that algorithm will correspond to tensor operations here, and the notation reflects that correspondence. In particular, the inner product on line 11 is an all-dimension tensor dot product and  $A$  is a square "matrix" which acts on tensors of shape  $(H,d_{\mathbf{u}},d_{\mathbf{y}})$
and perturbations satisfying Assumption 2.3, 2.5, 2.6, 2.7. Then running Algorithm 2 with  $H = \Theta(\mathrm{poly}(\log T))$ ,  $\sigma = \sigma_c\left(\sigma_{\mathbf{e}}^2 + \sigma_{\mathbf{w}}\frac{\sigma_{\min}(C)}{1 + \|A\|_{\mathrm{op}}^2}\right)$ , and  $\eta = \Theta\left(\frac{1}{d_{\mathbf{u}}d_{\mathbf{y}}L_cH^3\sqrt{T}}\right)$  guarantees
$$
\mathbb {E} \left[ R e g r e t _ {T} (E B P C) \right] \leq \tilde {\mathcal {O}} \left(\frac {\beta_ {c} d _ {\mathbf {u}} d _ {\mathbf {y}}}{\sigma_ {c}} \sqrt {T}\right),
$$
where regret is defined as in Eq.(2.2), the expectation is taken over the exploration noises  $\varepsilon_{1:T}$  of the algorithm as well as the stochastic components  $\{\mathbf{w}_t^{\mathrm{stoch}},\mathbf{e}_t^{\mathrm{stoch}}\}_{t = 1}^T$  of the perturbations  $\{\mathbf{w}_t,\mathbf{e}_t\}_{t = 1}^T$ , and  $\tilde{\mathcal{O}} (\cdot)$  hides all universal constants, natural parameters, and logarithmic dependence on  $T$ .
4.2 Unknown systems: control after estimation
Note that EBPC (Algorithm 2) relies on the access to the system's Markov operator  $G$ , which is available if and only if the system dynamics  $(A, B, C)$  are known. When the system dynamics is unknown, we can identify the system using a system estimation algorithm, obtain an estimated Markov operator  $\hat{G}$ , and run EBPC with  $G \gets \hat{G}$ . Algorithm 3 outlines the estimation method of system dynamics via least squares.
Algorithm 3 System estimation via least squares (SysEst-LS)
1: Input: estimation sample size  $N$ , system length  $H$ .
2: Initialize:  $\hat{G}^{[t]} = 0, \forall t \geq H$ .
3: Sample and play  $\mathbf{u}_t\sim N(0,I_{d_{\mathbf{u}}\times d_{\mathbf{u}}}),\forall t = 1,\ldots ,N.$
4: Set  $\hat{G}^{[0:\bar{H}]} = \arg \min \sum_{t = H}^{N}\| \mathbf{y}_{t} - \sum_{i = 0}^{\bar{H}}\hat{G}^{[i]}\mathbf{u}_{t - i}\|_{2}^{2}$ .
5: Return  $\hat{G}$ .
Theorem 4.2 (Unknown system control regret). Consider a linear dynamical system governed by unknown dynamics  $(A,B,C)$  and the interaction model with adversarially chosen cost functions and perturbations satisfying Assumption 2.3, 2.5, 2.6, 2.7. Suppose we obtain an estimated Markov operator  $\hat{G}$  from Algorithm 3 with  $N = \lceil \sqrt{T} \rceil$  and  $H = \Theta(\text{poly log } T)$ . Then Algorithm 2 with  $G \gets \hat{G}$ ,  $H \gets 3H$ ,  $\sigma = \frac{1}{8} \sigma_c \sigma_{\mathbf{e}}^2$ , and  $\eta = \Theta \left( \frac{1}{d_{\mathbf{u}} d_{\mathbf{y}} L_c H^3 \sqrt{T}} \right)$  guarantees
where regret is defined as in Eq.(2.2), the expectation is taken over the exploration noises  $\varepsilon_{1:T}$  in Algorithm 2, the sampled Gaussian controls  $\mathbf{u}_{1:N}$  in Algorithm 3, and the stochastic components  $\{\mathbf{w}_t^{\mathrm{stoch}},\mathbf{e}_t^{\mathrm{stoch}}\}_{t = 1}^T$  of the perturbations  $\{\mathbf{w}_t,\mathbf{e}_t\}_{t = 1}^T$ , and  $\tilde{\mathcal{O}} (\cdot)$  hides all universal constants, natural parameters, and logarithmic dependence on  $T$ .
We solve the open problem put forth by Gradu et al. [2020] on the optimal rate for online bandit control for the case of LQR/LQG control, improving to regret  $\tilde{O} (\sqrt{T})$  from  $\tilde{O} (T^{\frac{3}{4}})$  in the semiadversarial noise model and for strongly convex LQR/LQG cost functions. Our method builds upon recent advancements in bandit convex optimization for quadratic functions, providing the first near-optimal regret algorithm for bandit convex optimization with memory in a nonstochastic setting.
It would be interesting to investigate (1) whether the results can be extended to fully adversarial noise, (2) whether a similar stable controller recovery as seen in [Chen and Hazan, 2021] for fully observable systems can be established for partially observable systems, and whether that can be incorporated to extend our result to stabilizable systems even without access to a stabilizing controller.
Elad Hazan acknowledges funding from the Office of Naval Research grant N000142312156, the NSF award 2134040, and Open Philanthropy.
Elad Hazan. Introduction to online convex optimization. Foundations and Trends® in Optimization, 2(3-4):157-325, 2016.
Elad Hazan and Kfir Levy. Bandit convex optimization: Towards tight bounds. Advances in Neural Information Processing Systems, 27, 2014.
Elad Hazan and Karan Singh. Introduction to online nonstochastic control. arXiv preprint arXiv:2211.09619, 2022.
Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning, 69(2):169-192, 2007.
Morteza Ibrahimi, Adel Javanmard, and Benjamin Roy. Efficient reinforcement learning for high dimensional linear quadratic systems. Advances in Neural Information Processing Systems, 25, 2012.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960.
Sahin Lale, Kamyar Azizzadenesheli, Babak Hassibi, and Anima Anandkumar. Adaptive control and regret minimization in linear quadratic gaussian (lqg) setting. In 2021 American Control Conference (ACC), pages 2517-2522. IEEE, 2021.
Kent Quanrud and Daniel Khashabi. Online learning with adversarial delays. Advances in neural information processing systems, 28, 2015.
Ohad Shamir. On the complexity of bandit and derivative-free stochastic convex optimization. In Conference on Learning Theory, pages 3-24. PMLR, 2013.
Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non-stochastic control. In Conference on Learning Theory, pages 3320-3436. PMLR, 2020.
Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 928-936, 2003.