SlowGuess/NeurIPS_2023_OCR huggingface dataset:
PD-3 In practice the expectation in Equation 4 requires estimation. We use an average over 4 copies of the environment for this estimation. For Noisy Ant, we find that a default  $\sigma = 0.15$  performs well for MF-GPC so we do not tune  $\sigma$  further.
B.4 Experiments with adversarial noise
We run MF-GPC on top of a DDPG baseline for the inverted pendulum environment with 1) fast gradient sign method noise [Goodfellow et al., 2014] and 2) noise from a discretized sinusoid. We plot our results in Figure 7
Figure 7: Results from our method on the inverted pendulum environment for fast gradient sign method noise and sinusoid noise.
C Discrete State and Action Spaces
In this section, we consider differentiable parameterized random policies. For finite action space  $\mathcal{U}$ , let  $\Delta^{\mathcal{U}} = \{p:\mathcal{U}\to [0,1]\mid \sum_{a\in \mathcal{U}}p(a) = 1\}$  be the probability simplex over  $\mathcal{U}$ . Our policies, will be parameterized by  $M$  and depend on a window of past Pseudo-disturbances, providing the following distribution over actions.
$$
\mathbf {u} \sim \pi (\cdot | \hat {\mathbf {w}} _ {t - 1: t - h}, M) \in \Delta^ {\mathcal {U}} \tag {7}
$$
The baseline policy, would be built into  $\pi$  in this setting. For example, we could have a softmax neural net policy, and our algorithm adds a residual correction to the logits.
Implementation of PD signals in discrete spaces For discrete spaces, PD-2 defined via (3) is well defined, as we can still create auxiliary  $Q$  functions in the discrete space to produce our signal. Because, we no longer use Gaussian noise, PD-1 (2) can be modified as follows:
$$
\hat {\mathbf {w}} _ {t} = \left(c \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) + \gamma V _ {\pi} \left(\mathbf {x} _ {t + 1}\right) - Q _ {\pi} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right)\right) \nabla_ {M} \log \left(\pi \left(\mathbf {u} _ {t} \mid \hat {\mathbf {w}} _ {t - h: t - 1}, M\right)\right) | _ {M = M _ {t}} \tag {8}
$$
For our zeroth order gradient, we use the REINFORCE:
Algorithm 2 DMF-GPC (Discrete Model-Free Gradient Perturbation Controller)
1: Input: Memory parameter  $h$ , learning rate  $\eta$ , initialization  $M_{1:h}^{1}$ , initial value and  $Q$  functions, base RL algorithm  $\mathcal{A}$ .
2: for  $t = 1\ldots T$  do
3:
$$
\text {S a m p l e} \mathbf {u} _ {t} \sim \pi (\cdot | \hat {\mathbf {w}} _ {t - 1: t - h}, M ^ {t})
$$
4: Observe state  $\mathbf{x}_{t + 1}$ , and cost  $c_{t} = c_{t}(\mathbf{x}_{t},\mathbf{u}_{t})$
5: Compute Pseudo-Disturbance [see (3), (8)]
$$
\hat {\mathbf {w}} _ {t} = \operatorname {P D - e s t i m a t e} \left(\mathbf {x} _ {t + 1}, \mathbf {x} _ {t}, \mathbf {u} _ {t}, c _ {t}\right).
$$
6: Update policy parameters using the stochastic gradient estimate
$$
M ^ {t + 1} \leftarrow M ^ {t} - \eta c _ {t} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) \sum_ {j = 0} ^ {h - 1} \nabla_ {M} \log \left(\pi \left(\mathbf {u} _ {t - j} \mid \hat {\mathbf {w}} _ {t - j - 1: t - j - h}, M\right)\right) | _ {M = M _ {t}}.
$$
8: Optionally, update the baseline policy parameters and its  $Q$ ,  $V$  functions using  $\mathcal{A}$  so that they are Bellman consistent, i.e. they satisfy the policy version of Bellman equation.
D Pseudo-Disturbance Proofs
In this appendix, we have the deferred proofs from Section 3. For convenience, the lemmas have also been restated.
D.1 Proof of Lemma 1
Lemma 5. For time-invariant linear dynamical systems with system matrices  $A$ ,  $B$  and quadratic costs, in expectation, the pseudo disturbances (1) and (2) is a linear transformation of the actual perturbation
$$
\mathbb {E} [ \hat {\mathbf {w}} _ {t} | \mathbf {x} _ {t} ] = T \mathbf {w} _ {t},
$$
Proof. Recall from the theory of the linear quadratic regulator that the value function of an infinite horizon LDS is quadratic Bertsekas [2012],
$$
V (\mathbf {x}) = \mathbf {x} ^ {\top} P \mathbf {x}.
$$
$$
\begin{array}{l} \mathbb {E} [ \nabla_ {\mathbf {u}} (Q (\mathbf {x} _ {t}, \mathbf {u}) - c (\mathbf {x} _ {t}, \mathbf {u})) | _ {\mathbf {u} = \mathbf {u} _ {t}} ] = \mathbb {E} [ \gamma B ^ {\top} P (A \mathbf {x} _ {t} + B \mathbf {u} _ {t}) ] \\ = \mathbb {E} \left[ \gamma B ^ {\top} P \left(A \mathbf {x} _ {t} + B \pi \left(\mathbf {x} _ {t}\right) + B \mathbf {n} _ {t}\right) \right] \\ = \gamma B ^ {\top} P (A \mathbf {x} _ {t} + B \pi (\mathbf {x} _ {t})). \\ \end{array}
$$
By the definition of the signal, we have that
$$
\begin{array}{l} \mathbb {E} [ \hat {\mathbf {w}} _ {t} | \mathbf {x} _ {t} ] = \gamma \mathbb {E} [ V (\mathbf {x} _ {t + 1}) \Sigma^ {- 1} \mathbf {n} _ {t} - \nabla_ {\mathbf {u}} (Q (\mathbf {x} _ {t}, \mathbf {u}) - c (\mathbf {x} _ {t}, \mathbf {u})) | _ {\mathbf {u} = \mathbf {u} _ {t}} ] \\ = \gamma \mathbb {E} \left[ V \left(\mathbf {x} _ {t + 1}\right) \Sigma^ {- 1} \mathbf {n} _ {t} \mid \mathbf {x} _ {t} \right] - \gamma B ^ {\top} P \left(A \mathbf {x} _ {t} + B \pi (\mathbf {x} _ {t})\right). \\ \end{array}
$$
Writing the quadratic value function also to the first term, and denoting  $\| \mathbf{x}\| _P^2 = \mathbf{x}^\top P\mathbf{x}$ , we have that
$$
\begin{array}{l} \mathbb {E} [ V (\mathbf {x} _ {t + 1}) \Sigma^ {- 1} \mathbf {n} _ {t} | \mathbf {x} _ {t} ] = \mathbb {E} [ \| A \mathbf {x} _ {t} + B \pi (\mathbf {x} _ {t}) + \mathbf {w} _ {t} + B \mathbf {n} _ {t} \| _ {P} ^ {2} \Sigma^ {- 1} \mathbf {n} _ {t} | \mathbf {x} _ {t} ] \\ = \Sigma^ {- 1} \mathbb {E} \left[ \mathbf {n} _ {t} \mathbf {n} _ {t} ^ {\top} \right] B ^ {\top} P \left(A \mathbf {x} _ {t} + B \pi (\mathbf {x} _ {t}) + \mathbf {w} _ {t}\right) \\ = B ^ {\top} P \left(A \mathbf {x} _ {t} + B \pi (\mathbf {x} _ {t}) + \mathbf {w} _ {t}\right) \\ \end{array}
$$
We thus conclude,
$$
\begin{array}{l} \mathbb {E} \left[ \hat {\mathbf {w}} _ {t} \mid \mathbf {x} _ {t} \right] = \gamma \mathbb {E} \left[ V \left(\mathbf {x} _ {t + 1}\right) \Sigma^ {- 1} \mathbf {n} _ {t} \mid \mathbf {x} _ {t} \right] - \gamma B ^ {\top} P \left(A \mathbf {x} _ {t} + B \pi (\mathbf {x} _ {t})\right) \\ = \gamma B ^ {\top} P \left(A \mathbf {x} _ {t} + B \pi (\mathbf {x} _ {t}) + \mathbf {w} _ {t}\right) - \gamma B ^ {\top} P \left(A \mathbf {x} _ {t} + B \pi (\mathbf {x} _ {t})\right) \\ = \gamma B ^ {\top} P \mathbf {w} _ {t} = T \mathbf {w} _ {t} \\ \end{array}
$$
Lemma 6. Consider a time-invariant linear dynamical systems with system matrices  $A$ ,  $B$ , along with a linear baseline policy  $\pi$  defined by control law  $\mathbf{u}_t = -K_\pi \mathbf{x}_t$ . Let  $\mathbf{V}_{\pi}^{\mathbf{c}}$  and  $\mathbf{Q}_{\pi}^{\mathbf{c}}$  be value functions for  $\pi$  for i.i.d. zero mean noise with linear costs  $\mathbf{c}(x) := Lx$ , then the PD-signal (3) is a linear transformation
Proof. We first note that for linear rewards, value functions for i.i.d. zero mean noise are equivalent to the value functions without noise. As such, we have the identity
$$
\mathbf {Q} _ {\pi} ^ {\mathbf {c}} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) = \mathbf {c} \left(\mathbf {x} _ {t}\right) + \gamma \mathbf {V} _ {\pi} ^ {\mathbf {c}} \left(A \mathbf {x} _ {t} + B \mathbf {u} _ {t}\right),
$$
and so, we can rewrite or PD-signal as
$$
\begin{array}{l} \hat {\mathbf {w}} _ {t} = \mathbf {V} _ {\pi} ^ {\mathbf {c}} \left(\mathbf {x} _ {t + 1}\right) - \mathbf {V} _ {\pi} ^ {\mathbf {c}} \left(A \mathbf {x} _ {t} + B \mathbf {u} _ {t}\right) \\ = \mathbf {V} _ {\pi} ^ {\mathbf {c}} \left(A \mathbf {x} _ {t} + B \mathbf {u} _ {t} + \mathbf {w} _ {t}\right) - \mathbf {V} _ {\pi} ^ {\mathbf {c}} \left(A \mathbf {x} _ {t} + B \mathbf {u} _ {t}\right). \\ \end{array}
$$
Now, it remains to show that  $\mathbf{V}_{\pi}^{\mathbf{c}}$  is a fixed linear transformation. Indeed, we have
$$
\mathbf {V} _ {\pi} ^ {\mathbf {c}} (x) = \sum_ {t = 0} ^ {\infty} \gamma^ {i} \mathbf {c} (A _ {\pi} ^ {t} x) = \sum_ {t = 0} ^ {\infty} \gamma^ {i} L A _ {\pi} ^ {t} x = L (I - \gamma A _ {\pi}) ^ {- 1} x,
$$
where  $A_{\pi} = A - BK_{\pi}$  is the closed loop dynamics matrix. We now have
$$
\begin{array}{l} \hat {\mathbf {w}} _ {t} = \mathbf {V} _ {\pi} ^ {\mathbf {c}} \left(A \mathbf {x} _ {t} + B \mathbf {u} _ {t} + \mathbf {w} _ {t}\right) - \mathbf {V} _ {\pi} ^ {\mathbf {c}} \left(A \mathbf {x} _ {t} + B \mathbf {u} _ {t}\right) \\ = L \left(I - \gamma A _ {\pi}\right) ^ {- 1} \left[ \left(A \mathbf {x} _ {t} + B \mathbf {u} _ {t} + \mathbf {w} _ {t}\right) - \left(A \mathbf {x} _ {t} + B \mathbf {u} _ {t}\right) \right] \\ = L \left(I - \gamma A _ {\pi}\right) ^ {- 1} \mathbf {w} _ {t}. \\ \end{array}
$$
Now, stability of  $\pi$  dictates  $(I - \gamma A_{\pi})$  is full rank (even for  $\gamma = 1$ ), so if  $L$  is full rank,  $L(I - \gamma A_{\pi})^{-1}$  is full rank.
D.3 Proof of Lemma 3
Lemma 7. Suppose we have a simulator  $f_{sim}$  such that  $\forall \mathbf{x}, \mathbf{u}, \| f_{sim}(\mathbf{x}, \mathbf{u}) - f(\mathbf{x}, \mathbf{u}) \| \leq \delta$ , then Pseudo-Disturbance (4) is approximately equal to the actual perturbation  $\| \widehat{\mathbf{w}}_t - \mathbf{w}_t \| \leq \delta$ .
Proof. This Lemma is immediate from the definition of the dynamics as  $\mathbf{x}_{t + 1} = f(\mathbf{x}_t,\mathbf{u}_t) + \mathbf{w}_t$ .
E Main Result and Dimension-efficient Bandit GPC
Below, we formally state and prove the main result. Subsequent sections attest to the fact that this regret bound is an unconditional improvement over the best known Gradu et al. [2020], Cassel and Koren [2020] in terms of its dependence on dimension and applicability to high-dimensional systems, even for the well-studied setting of linear control.
E.1 Main Result
Using Theorem 9 which we state and prove in subsequent sections, we can prove the main result.
Theorem 8. Consider a modification of Algorithm 3 implemented using  $\hat{\mathbf{w}}_t$  in place of  $\mathbf{w}_t$  and choose the step size as  $\eta = \sqrt{\frac{d_{min}}{d_u}} T^{-3/4}$ , and the exploration radius as  $\delta = \sqrt{d_u d_{min}} T^{-1/4}$ .
If the underlying dynamics are linear and satisfy the assumptions in Section E.2 then then as long as the Pseudo-Disturbance signal  $\hat{\mathbf{w}}_t$  satisfies  $\hat{\mathbf{w}}_t = \mathcal{T}\mathbf{w}_t$ , for some (possibly unknown) invertible map  $\mathcal{T}$ , with  $\mathbf{u}_t$  such that for any sequence of bounded (even adversarial)  $\mathbf{w}_t$  such that the following holds
$$
\sum_ {t} c _ {t} (\mathbf {x} _ {t}, \mathbf {u} _ {t}) - \inf _ {M \in \mathcal {M}} \sum_ {t} c _ {t} (\mathbf {x} _ {t} ^ {M}, \mathbf {u} _ {t} ^ {M}) \leq \widetilde {\mathcal {O}} (\sqrt {d _ {u} d _ {m i n}} \mathrm {p o l y} (\| \mathcal {T} \| \| \mathcal {T} ^ {- 1} \|) T ^ {3 / 4}),
$$
for any any sequence of convex costs  $c_t$ . Further, if the costs  $c_t$  are  $L$ -smooth, the regret can be improved upper bound of
$$
\sum_ {t} c _ {t} (\mathbf {x} _ {t}, \mathbf {u} _ {t}) - \inf _ {M \in \mathcal {M}} \sum_ {t} c _ {t} (\mathbf {x} _ {t} ^ {M}, \mathbf {u} _ {t} ^ {M}) \leq \widetilde {\mathcal {O}} (\mathrm {p o l y} (\| \mathcal {T} \| \| \mathcal {T} ^ {- 1} \|) (d _ {u} d _ {m i n} T) ^ {2 / 3}).
$$
Proof. This follows from that fact that an invertible linear transformation  $\mathcal{T}$  of  $\mathbf{w}_t$  does not change the expressiveness of a DAC policy class  $\mathcal{M}$  except through constants related to the norm of  $\mathcal{T}$ . More specifically, given a DAC policy  $M_{1:h}$  that acts on true disturbances  $\mathbf{w}_s$ , the same exact controls are produced by a DAC policy  $M'$  with  $M_i' = M_i\mathcal{T}^{-1}$  acting on  $\hat{\mathbf{w}}_s = \mathcal{T}\mathbf{w}_s$ . The disturbances are also scaled by  $\mathcal{T}$ . By As such, we can attain equivalent regret bounds with a new policy class  $\mathcal{M}'$  with diameter scaled by  $\| \mathcal{T}^{-1}\|$  and new bound on disturbances  $W' = \| \mathcal{T}\| W$ . In Theorem 9, the hidden dependence on the DAC diameter and disturbance size are polynomial, yielding at most  $\mathrm{poly}(\| \mathcal{T}\| \| \mathcal{T}^{-1}\|)$  scaling in the regret.
E.2 Dimension-Efficient Bandit GPC
Under bandit feedback, the learner can only observe the cost it incurs, and does not have access to function value oracles or gradients of the cost functions. This setting has been studied in detail in linear control subject to adversarial disturbances using both dynamic and static regret settings; we restrict our attention to the latter.
A key characteristic of our proposed algorithm is that it performs exploration in the action space, rather than the policy space. This enables us to obtain a favorable trade-off between the quality of the proxy of the gradient and the amount of modification the objective (via randomized smoothing) is subject to. Leveraging this property, we show that our approach obtains a better regret bound than the best known [Cassel and Koren, 2020, Gradu et al., 2020] in the literature. In particular, the best known regret bounds for this setting scale as  $O(\sqrt{d_x d_u d_{min}} T^{3/4})$ . In contrast, we offer a regret bound of  $O(\sqrt{d_u d_{min}} T^{3/4})$ . This is both a quantitative and a qualitative improvement, and carries over to the case of smooth costs too. In particular, since our bound has no dependence on  $d_x$  whatsoever, it is equally applicable to the high-dimensional setting ( $d_x \gg 1$ ), which existing
methodologies fail to scale to in the bandit setting. We stress that this improvement in the upper bound stems from the right algorithm design, and not just a tighter analysis.
In this section, we analyze Algorithm 3, a minimally modified version of Algorithm 1, which uses delayed gradient updates. As a convention, we hold  $\mathbf{w}_t = 0$  for  $t < 0$  when defining DAC policies in early rounds.
Algorithm 3 Bandit GPC
1: Input: Memory parameter  $h$ , learning rate  $\eta$ , exploration size  $\delta$ , initialization  $M_{1:h}^{1} \in \mathbb{R}^{d_u \times d_x \times h}$  and convex policy set  $\mathcal{M}$ .
2: for  $t = 1 \dots T$  do
3: Use action  $\mathbf{u}_t = \sum_{i=1}^h M_i^t \mathbf{w}_{t-i} + \delta \mathbf{n}_t$ , where  $\mathbf{n}_t$  is drawn iid from a sphere uniformly, i.e.
$$
\mathbf {n} _ {t} \sim \operatorname {U n i f} \left(\mathbb {S} _ {d _ {u}}\right).
$$
4: Observe state  $\mathbf{x}_{t + 1}$ , and cost  $c_{t} = c_{t}(\mathbf{x}_{t},\mathbf{u}_{t})$
5: Store the stochastic gradient estimate
$$
\widehat {\nabla} _ {t} = \frac {d _ {u} c _ {t} (\mathbf {x} _ {t} , \mathbf {u} _ {t})}{\delta} \sum_ {i = 0} ^ {h - 1} \mathbf {n} _ {t - i} \otimes \mathbf {w} _ {t - i - 1: t - i - h}
$$
6: Update using delayed gradient with euclidean projection onto  $\mathcal{M}$
$$
M ^ {t + 1} = \Pi_ {\mathcal {M}} \left[ M ^ {t} - \eta \widehat {\nabla} _ {t - h} \right]
$$
We make the following assumptions pertaining to costs and linear dynamics:
1. The underlying dynamics are assumed to be time-invariant and linear, i.e.
where  $\mathbf{x}_t, \mathbf{w}_t \in \mathbb{R}^{d_x}, \mathbf{u}_t \in \mathbb{R}^{d_u}, \| w_t \| \leq W$  and  $\| B \| \leq \kappa$ .
2. The linear system is  $(\kappa, \alpha)$ -strongly stable:  $\exists Q, L$  such that
$$
A = Q L Q ^ {- 1},
$$
where  $\| Q^{-1}\| ,\| Q\| \leq \kappa ,\| L\| \leq 1 - \alpha .$
3. The time-varying online cost functions  $c_{t}(\mathbf{x}_{t},\mathbf{u}_{t})$  are convex and satisfy for all  $\| \mathbf{x}\| ,\| \mathbf{u}\| \leq D$  that
$$
c _ {t} (\mathbf {x}, \mathbf {u}) \leq C \max  \{D ^ {2}, 1 \}, \quad \text {a n d} \quad \| \nabla_ {\mathbf {u}} c (\mathbf {x}, \mathbf {u}) \|, \| \nabla_ {\mathbf {x}} c (\mathbf {x}, \mathbf {u}) \| \leq G \max  \{D, 1 \}.
$$
4. We define our comparator set  $\mathcal{M} = \mathcal{M}_1\times \dots \times \mathcal{M}_h$  where
$$
\mathcal {M} _ {i} = \left\{M \in \mathbb {R} ^ {d _ {u} \times d _ {x}}: \| M \| \leq 2 \kappa^ {4} (1 - \alpha) ^ {i} \right\}
$$
as in [Cassel and Koren, 2020]. Let  $d_{min} = \min \{d_x,d_u\}$
The second assumption may be relaxed to that of stabilizability, the case when the linear system by itself might be unstable, however the learner is provided with a suboptimal linear controller  $K_{0}$  such that  $A + BK_{0}$  is  $(\kappa, \alpha)$ -strongly stable, via a blackbox reduction outlined in Proposition 6 (Appendix A) in Cassel et al. [2022].
Theorem 9. Choosing the step size as  $\eta = \sqrt{\frac{d_{min}}{d_u}} T^{-3/4}$ , and the exploration radius as  $\delta = \sqrt{d_u d_{min}} T^{-1/4}$ , the regret of Algorithm 3 is upper bounded as
$$
\operatorname {R e g r e t} (\mathcal {A}) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} c _ {t} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) \right] - \inf  _ {M \in \mathcal {M}} \sum_ {t = 1} ^ {T} c _ {t} \left(\mathbf {x} _ {t} ^ {M}, \mathbf {u} _ {t} ^ {M}\right) \leq \widetilde {\mathcal {O}} \left(\sqrt {d _ {u} d _ {\min }} T ^ {3 / 4}\right).
$$
Furthermore, if the costs  $c_t$  is  $L$ -smooth, then choosing  $\delta = (d_u d_{min})^{1/3} T^{-1/6}, \eta = d_{min}^{1/3} / (d_u^{2/3} T^{2/3})$ , the regret incurred by the algorithm admits a tighter upper bound of
$$
\operatorname {R e g r e t} (\mathcal {A}) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} c _ {t} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) \right] - \inf  _ {M \in \mathcal {M}} \sum_ {t = 1} ^ {T} c _ {t} \left(\mathbf {x} _ {t} ^ {M}, \mathbf {u} _ {t} ^ {M}\right) \leq \widetilde {\mathcal {O}} \left(\left(d _ {u} d _ {\min } T\right) ^ {2 / 3}\right).
$$
E.3 Idealized Cost and Proof of Theorem 9
We will prove our result by creating a proxy loss with memory which accurately estimates the cost, showing that our update provides a low bias gradient estimate with suitably small variance. This will allow us to prove a regret bound on our proxy-losses, which we then translate to a regret bound on the policy itself.
Following [Agarwal et al., 2019, Cassel and Koren, 2020], we introduce a transfer matrix that describes the effect of recent disturbances on the state.
Definition 10. For any  $i < 2h$ , define the disturbance-state transfer matrix Let
$$
\Psi_ {i} (M ^ {1: h}) = A ^ {i} \mathbf {1} _ {i} \leq h + \sum_ {j = 1} ^ {h} A ^ {j} B M _ {i - j - 1} ^ {h - j + 1} \mathbf {1} _ {i - j - 1 \in [ 1, h ]}
$$
We can also create a transfer matrix for the effect of injected noise in the control on the state:
Definition 11. The noise transfer matrix is defined as  $\Phi_{i} = A^{i}B$
We have the following representation of the state
$$
\mathbf {x} _ {t + 1} = A ^ {h + 1} \mathbf {x} _ {t - h} + \sum_ {i = 0} ^ {2 h} \Psi_ {i} \left(M ^ {1: h}\right) \mathbf {w} _ {t - i} + \delta \sum_ {i = 0} ^ {h - 1} \Phi_ {i} \mathbf {n} _ {t - i} \tag {9}
$$
We are also interested in counterfactual state trajectories using non-stationary DAC policies. In particular, we have
Definition 12. The idealized state using policies  $M^{1:h}$  is defined as
$$
y _ {t + 1} \left(M ^ {1: h}\right) = \sum_ {i = 0} ^ {2 h} \Psi_ {i} \left(M ^ {1: h}\right) \mathbf {w} _ {t - i}.
$$
Similarly, the idealized cost is defined as
$$
C _ {t} (M ^ {1: h}) = c _ {t} \left(y _ {t} (M ^ {1: h}), \sum_ {i = 1} ^ {h} M _ {i} ^ {h} \mathbf {w} _ {t - i}\right).
$$
The univariate generalization of the idealized state and cost are
$$
\tilde {y} _ {t} (M) = y _ {t} (\underbrace {M , M , \ldots M} _ {h \text {t i m e s}}), \quad \widetilde {C} _ {t} (M) = C _ {t} (\underbrace {M , M , \ldots M} _ {h \text {t i m e s}}).
$$
We also define  $F_{t}(\mathbf{u}_{1:h}) = c_{t}\left(\sum_{i=0}^{h-1} A^{i}(B\mathbf{u}_{i} + \mathbf{w}_{t-1-i}), \mathbf{u}_{h}\right)$  representing the instantaneous cost as a function of the last  $h$  controls.
$$
\widetilde {C} _ {t} (M) = F _ {t} \left(\sum_ {i = 0} ^ {h - 1} M _ {i} w _ {t - i - h}, \sum_ {i = 0} ^ {h - 1} M _ {i} w _ {t - i - h + 1}, \dots \sum_ {i = 0} ^ {h - 1} M _ {i} w _ {t - i - 1}\right)
$$
We now define a smoothed version of  $F_{t}$ ,  $F_{t,\delta}$  and a smoothed version of  $\tilde{C}_t$ ,  $\tilde{C}_{t,\delta}$  that uses  $F_{t,\delta}$ .
$$
F _ {t, \delta} (\mathbf {u} _ {1: h}) = \mathbb {E} _ {\mathbf {n} _ {1: h} \sim \mathbb {S} _ {d u}} [ F _ {t} (\mathbf {u} _ {1: h} + \delta \mathbf {n} _ {1: h}) ]
$$
$$
\widetilde {C} _ {t, \delta} (M) = F _ {t, \delta} \left(\sum_ {i = 0} ^ {h - 1} M _ {i} w _ {t - i - h}, \sum_ {i = 0} ^ {h - 1} M _ {i} w _ {t - i - h + 1} \dots \sum_ {i = 0} ^ {h - 1} M _ {i} w _ {t - i - 1}\right)
$$
We also use the following notation for idealized costs fixing a realization of the exploration noise:
$$
C _ {t} (M | \mathbf {n} _ {1: h}) = F _ {t, \delta} \left(\sum_ {i = 0} ^ {h - 1} M _ {i} w _ {t - i - h} + \mathbf {n} _ {1}, \sum_ {i = 0} ^ {h - 1} M _ {i} w _ {t - i - h + 1} + \mathbf {n} _ {2}, \dots , \sum_ {i = 0} ^ {h - 1} M _ {i} w _ {t - i - 1} + \mathbf {n} _ {h}\right).
$$
Since  $\delta = o(1)$ , the contribution to the state space is negligible, we can use bounds from Definition 5 of [Cassel and Koren, 2020]. In particular, we will use
$$
h = \alpha^ {- 1} \log \left(2 \kappa^ {3} T\right), \quad D _ {x, u} = \max  \left(1 0 \alpha^ {- 1} \kappa^ {4} W (h \kappa + 1), 1\right) \tag {10}
$$
Proof of Theorem 9. First, we state a bound on how large the states can get when modifying a DAC policy online.
Lemma 13. Suppose controls are played according to  $\mathbf{u}_t = \sum_{i=1}^h M_i^t \mathbf{w}_{t-i} + \delta \mathbf{n}_t$  where  $\mathbf{n}_t \sim \mathbb{S}_{d_u}$  and  $\delta = o(1)$ , then
1.  $\| \mathbf{x}_t\| ,\| \mathbf{u}_t\| \leq D_{x,u}$  and  $|c_{t}(\mathbf{x}_{t},\mathbf{u}_{t})|\leq CD_{x,u}^{2}$
2. Let  $x_{t}^{M}, u_{t}^{M}$  correspond to the counterfactual trajectory, playing DAC with parameter  $M$  for all time, then  $|c_{t}(\mathbf{x}_{t}^{M}, \mathbf{u}_{t}^{M}) - \tilde{C}_{t}(M)| \leq \frac{GD_{x,u}^{2}}{T}$
3.  $|c_{t}(\mathbf{x}_{t},\mathbf{u}_{t}) - C_{t}(M^{t - h + 1:t}|\mathbf{n}_{t - h + 1:t})| \leq \frac{GD_{x,u}^{2}}{T}$
The next lemma quantifies both the degree to which an idealized notion of cost tracks the true cost incurred for a DAC policy, and the resultant quality of gradient estimates thus obtained.
Lemma 14. For all  $t$ ,  $C_{t,\delta}$  is convex and
$$
\| \nabla \widetilde {C} _ {t, \delta} (M ^ {t}) - \mathbb {E} [ \widehat {\nabla} _ {t} ] \| \leq \frac {2 \eta d _ {u} h ^ {4} W ^ {2} \kappa^ {3} \widehat {G} G D _ {x , u}}{\delta},
$$
and for all  $M\in \mathcal{M}$
$$
\left| \widetilde {C} _ {t, \delta} (M) - \widetilde {C} _ {t} (M) \right| \leq \delta h G D _ {x, u} \kappa^ {3}.
$$
We begin by observing that for any  $t$  using Lemma 13.3 and the second part of Lemma 14, we have
$$
| c _ {t} (\mathbf {x} _ {t} ^ {M}, \mathbf {u} _ {t} ^ {M}) - \widetilde {C} _ {t, \delta} (M) | \leq \frac {G D _ {x , u} ^ {2}}{T} + \delta h G D _ {x, u} \kappa^ {3}.
$$
A analogous result on the difference between true and idealized costs is stated below, but this time for the online algorithm itself which employs a changing sequence of DAC policies.
Lemma 15.
$$
\left| c _ {t} \left(\mathbf {x} _ {t}, \mathbf {u} _ {t}\right) - \widetilde {C} _ {t} \left(M ^ {t} \mid \mathbf {n} _ {t - h + 1: t}\right) \right| \leq \frac {G D _ {x , u} ^ {2}}{T} + \eta G D _ {x, u} W \kappa^ {3} h ^ {2} \widehat {G}
$$
Similarly, we have using Lemma 15 for any  $t$  that
$$
| c _ {t} (\mathbf {x} _ {t}, \mathbf {u} _ {t}) - \widetilde {C} _ {t, \delta} (M ^ {t}) | \leq \frac {G D _ {x , u} ^ {2}}{T} + \eta G D _ {x, u} W \kappa^ {3} h ^ {2} \widehat {G} + \delta h G D _ {x, u} \kappa^ {3}.
$$
Using this display, we decompose the regret of the algorithm as stated below.
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} c _ {t} (\mathbf {x} _ {t}, \mathbf {u} _ {t}) \right] - \inf  _ {M \in \mathcal {M}} \sum_ {t = 1} ^ {T} c _ {t} (\mathbf {x} _ {t} ^ {M}, \mathbf {u} _ {t} ^ {M}) \\ \leq \sum_ {t = 1} ^ {T} \widetilde {C} _ {t, \delta} (M ^ {t}) - \inf _ {M \in \mathcal {M}} \sum_ {t = 1} ^ {T} \widetilde {C} _ {t, \delta} (M) + 2 G D _ {x, u} ^ {2} + \eta G D _ {x, u} W \kappa^ {3} h ^ {2} \widehat {G} T + 2 \delta h G D _ {x, u} \kappa^ {3} T \\ \end{array}
$$
Next, we use the following regret bound on an abstract implementation of online gradient descent with delayed updates, which we specialize subsequently to our setting.
Lemma 16. Consider a delayed gradient update in Online Gradient Descent, executed as