SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
C _ {\mathrm {l o t}} := \operatorname {p o l y} \left(L _ {\pi_ {\star}}, B _ {A}, B _ {\phi}, L _ {\phi}, L _ {\theta}, L _ {\mathrm {c o s t}}, d _ {\boldsymbol {x}}, d _ {\phi}, \frac {1}{\lambda_ {\mathrm {m i n}} ^ {\star}}, \sigma_ {\boldsymbol {w}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, \| \mathcal {H} (A _ {\star}) \| _ {\mathrm {o p}}, \log \frac {T}{\delta}\right)
$$
denote some lower-order constant, whose precise polynomial dependence may change from line to line. On  $\mathcal{E}$ , by Lemma B.3 we can bound
$$
\left\| \widehat {A} - A _ {\star} \right\| _ {\mathrm {o p}}, \left\| \widehat {A} ^ {-} - A _ {\star} \right\| _ {\mathrm {o p}} \leq C _ {\mathrm {l o t}} \cdot \frac {1}{\sqrt {T}}, \tag {B.2}
$$
and so assuming the burn-in (B.1) is met, we can bound  $\| \widehat{A}^{-} - A_{\star}\|_{\mathrm{op}}\leq \frac{1}{2} r_{\mathrm{est}}(A_{\star})$  and  $\| \widehat{A} -$ $A_{\star}\|_{\mathrm{op}}\leq \frac{1}{2} r_{\mathrm{est}}(A_{\star})$  . This then implies that
$$
\left\| \mathcal {H} \left(A _ {\star}\right) - \mathcal {H} \left(\widehat {A} ^ {-}\right) \right\| _ {\mathrm {o p}} \leq C _ {\mathrm {l o t}} \cdot \frac {1}{\sqrt {T}},
$$
so in particular we can bound
$$
\begin{array}{l} \mathrm {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\mathrm {m i n}} ^ {\star}}, B _ {A}, B _ {\phi}, H, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, \| \mathcal {H} (\widehat {A} ^ {-}) \| _ {\mathrm {o p}}, \log \frac {T}{\delta}\right) \\ \leq \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min} ^ {\star}}, B _ {A}, B _ {\phi}, H, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, \| \mathcal {H} (A _ {\star}) \| _ {\mathrm {o p}}, \log \frac {T}{\delta}\right). \\ \end{array}
$$
We have therefore shown that
$$
\begin{array}{l} \mathcal {J} \left(\widehat {\pi} _ {T}; A _ {\star}\right) - \mathcal {J} \left(\pi_ {\star} \left(A _ {\star}\right); A _ {\star}\right) \leq \frac {C}{T} \cdot \min  _ {\boldsymbol {\Lambda} \in \Omega} \operatorname {t r} \left(\mathcal {H} \left(A _ {\star}\right) \check {\boldsymbol {\Lambda}} ^ {- 1}\right) \cdot \sigma_ {\boldsymbol {w}} ^ {2} \log \frac {6 d _ {\boldsymbol {x}} d _ {\phi}}{\delta} \\ + C _ {\mathrm {l o t}} \cdot \left(\| \widehat {A} - A _ {\star} \| _ {\mathrm {o p}} ^ {3} + \| \widehat {A} ^ {-} - A _ {\star} \| _ {\mathrm {o p}} ^ {3} + \frac {1}{T ^ {2}} + \frac {1}{T} \cdot \| \widehat {A} ^ {-} - A _ {\star} \| _ {\mathrm {o p}}\right) \\ \leq \frac {C}{T} \cdot \min  _ {\boldsymbol {\Lambda} \in \boldsymbol {\Omega}} \operatorname {t r} \left(\mathcal {H} (A _ {\star}) \breve {\boldsymbol {\Lambda}} ^ {- 1}\right) \cdot \sigma_ {\boldsymbol {w}} ^ {2} \log \frac {6 d _ {\boldsymbol {x}} d _ {\phi}}{\delta} + \frac {C _ {\mathrm {l o t}}}{T ^ {3 / 2}}, \\ \end{array}
$$
The final result follows from using Lemma D.4 to bound
$$
\left\| \mathcal {H} \left(A _ {\star}\right) \right\| _ {\mathrm {o p}} \leq \operatorname {p o l y} \left(\left\| A _ {\star} \right\| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\theta}, L _ {\text {c o s t}}, L _ {\pi_ {\star}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, d _ {\boldsymbol {x}}\right).
$$
Proof of Theorem 1. The proof of Theorem 1 is identical to that of Theorem 4, the only difference being that we replace Assumption 13 with Assumption 6. However, by Proposition 6, the conditions of Assumption 13 are met when Assumption 6 holds.  $\square$
B.1 Supporting Lemmas
Lemma B.1. Under Assumptions 1 and 3, the system (1.1) satisfies Assumptions 11 and 12 with
$$
\boldsymbol {\psi} (\boldsymbol {\tau}) \leftarrow I _ {d _ {\boldsymbol {x}}} \otimes \sum_ {h = 1} ^ {H} \phi \left(\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}}\right) \phi \left(\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}}\right) ^ {\top}, \quad D = d _ {\boldsymbol {x}} H B _ {\phi} ^ {2},
$$
$d_{\psi} \gets d_{\phi} d_{\mathbf{x}}$ , and where  $\mathbf{x}_h^{\tau}$  (resp.  $\mathbf{u}_h^{\tau}$ ) denotes the state (resp. input) at step  $h$  of trajectory  $\mathbf{\tau}$ . Furthermore, it satisfies Assumption 10 with  $\mathbb{A}_{\mathcal{R}}$  instantiated with the LC<sup>3</sup> algorithm of [17] and
$$
C _ {\mathcal {R}} = C \cdot H \sqrt {d _ {\phi} (d _ {\phi} + d _ {\boldsymbol {x}} + B _ {A})} \cdot \log (1 + B _ {\phi} H / \sigma_ {\boldsymbol {w}}), \quad p _ {\mathcal {R}} = 3 / 2, \quad \alpha = 1 / 2
$$
for a universal constant  $C$ .
Proof. That Assumption 12 is satisfied is immediate under Assumption 3. It is clear that  $\psi(\tau) \in S_{+}^{d_{\phi}d_{\mathfrak{x}}}$ . To obtain a bound on  $D$ , we only need to bound the trace of  $\psi(\tau)$ :
$$
\begin{array}{l} \operatorname {t r} (\boldsymbol {\psi} (\boldsymbol {\tau})) = \sum_ {h = 1} ^ {H} \operatorname {t r} (I _ {d _ {\boldsymbol {\alpha}}}) \cdot \operatorname {t r} (\phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}})) \phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}})) ^ {\top}) \\ = d _ {\boldsymbol {x}} \sum_ {h = 1} ^ {H} \| \phi \left(\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}}\right)) \| _ {2} ^ {2} \\ \leq d _ {x} H B _ {\phi} ^ {2} \\ \end{array}
$$
where the inequality holds under Assumption 1.
To show that Assumption 10 is satisfied in this setting, we have by Theorem 6 that with probability at least  $1 - \delta$ ,  $\mathrm{LC}^3$  has regret bounded as (using that  $c_{\max} \leq 1$  in the setting of Assumption 10):
$$
\mathcal {R} _ {T} \leq C \cdot H \sqrt {d _ {\phi} \cdot (d _ {\phi} + d _ {\boldsymbol {x}} + B _ {A} + \log \frac {1}{\delta}) \cdot T} \cdot \log (1 + B _ {\phi} H T / \sigma_ {\boldsymbol {w}})
$$
for  $C$  a universal constant. We can therefore take  $\alpha = 1 / 2$ ,  $p_{\mathcal{R}} = 3 / 2$ , and
$$
C _ {\mathcal {R}} = C ^ {\prime} \cdot H \sqrt {d _ {\phi} (d _ {\phi} + d _ {\boldsymbol {x}} + B _ {A})} \cdot \log (1 + B _ {\phi} H / \sigma_ {\boldsymbol {w}}).
$$
Lemma B.2. Let  $\bar{T}_{\ell}$  denote the total number of episodes collected by Algorithm 3 at round  $\ell$ . For
$$
T _ {\ell} \geq \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min } ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {T _ {\ell}}{\delta}\right), \tag {B.3}
$$
on the success event of Lemma B.3, we have  $2T_{\ell} \geq \bar{T}_{\ell}$  and
$$
2 T _ {\ell} \geq \sum_ {i = 1} ^ {\ell - 1} \bar {T} _ {i}.
$$
Proof. Recall that  $T_{\ell} = 2^{\ell}$  and  $N_{\ell} = \lceil T_{\ell} / T_{\ell}^{\mathrm{oed}} \rceil$  for  $T_{\ell}^{\mathrm{oed}} = |\Pi_{\ell}|$ . By Lemma B.3,  $\bar{T}_{\ell}$  can be bounded as
$$
\bar {T} _ {\ell} \leq N _ {\ell} T _ {\ell} ^ {\mathrm {o e d}} + (1 6 + 2 \log T _ {\ell} ^ {\mathrm {o e d}}) T _ {\ell} ^ {\mathrm {o e d}}
$$
and  $T_{\ell}^{\mathrm{oed}}$  can be bounded as
$$
T _ {\ell} ^ {\text {o e d}} \leq \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min } ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {T _ {\ell}}{\delta}\right). \tag {B.4}
$$
Note that we can bound
$$
\begin{array}{l} \bar {T} _ {i} \leq N _ {i} T _ {i} ^ {\mathrm {o e d}} + (1 6 + 2 \log T _ {i} ^ {\mathrm {o e d}}) T _ {i} ^ {\mathrm {o e d}} \\ \leq T _ {i} + (1 7 + 2 \log T _ {i} ^ {\mathrm {o e d}}) T _ {i} ^ {\mathrm {o e d}} \\ \leq T _ {i} + \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\operatorname* {m i n}} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {T _ {\ell}}{\delta}\right). \\ \end{array}
$$
From this it is immediately obvious that  $2T_{\ell} \geq \bar{T}_{\ell}$  as long as (B.3) is satisfied.
To show the second conclusion note that, by our choice of  $T_{i} = 2^{i}$ , we have that  $T_{\ell} \geq \sum_{i=1}^{\ell-1} T_{i}$ , so it therefore remains to show that
$$
T _ {\ell} \geq \sum_ {i = 1} ^ {\ell - 1} \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {T _ {i}}{\delta}\right).
$$
However, we can bound
$$
\sum_ {i = 1} ^ {\ell - 1} \mathrm {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\mathrm {m i n}} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {T _ {i}}{\delta}\right) \leq \mathrm {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\mathrm {m i n}} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {T _ {\ell}}{\delta}\right) \cdot \log T _ {\ell},
$$
so a sufficient condition is
$$
T _ {\ell} \geq \mathrm {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\mathrm {m i n}} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {T _ {\ell}}{\delta}\right) \cdot \log T _ {\ell}
$$
which we see is met when (B.3) holds.
Lemma B.3. Consider running Algorithm 7 with weight matrix  $\mathcal{H}$ , parameter  $\widetilde{N}$ , and confidence  $\delta$ , and rerunning each policy in  $\Pi_{\mathrm{out}} N \leq \widetilde{N}$  times. Then, under Assumptions 1 and 3, with probability at least  $1 - 6\delta$ :
$$
\begin{array}{l} \| \operatorname {v e c} (\widehat {A} - A _ {\star}) \| _ {\mathcal {H}} ^ {2} \leq \frac {6 0}{N T _ {\text {o u t}}} \cdot \min  _ {\boldsymbol {\Lambda} \in \Omega} \operatorname {t r} \left(\mathcal {H} \check {\boldsymbol {\Lambda}} ^ {- 1}\right) \cdot \sigma_ {\boldsymbol {w}} ^ {2} \log \frac {6 d _ {\boldsymbol {x}} d _ {\phi}}{\delta} \\ + \mathrm {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\mathrm {m i n}} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \| \mathcal {H} \| _ {\mathrm {o p}}, \log \frac {\widetilde {N}}{\delta}\right) \cdot \frac {1}{N ^ {2}} \\ \end{array}
$$
where  $\widehat{A}$  denotes the least-squares estimate of  $A_{\star}$  obtained on the data generated by rerunning  $\Pi_{\mathrm{out}}$ . In addition, we have
$$
\left\| \widehat {A} - A _ {\star} \right\| _ {\mathrm {F}} \leq \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min } ^ {\star}}, B _ {A}, B _ {\phi}, \sigma_ {\boldsymbol {w}}, H, \log \frac {\widetilde {N}}{\delta}\right) \cdot \frac {1}{\sqrt {N}}.
$$
$$
T _ {\text {o u t}} \leq \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {\widetilde {N}}{\delta}\right)
$$
and the total number of episodes collected by this procedure is bounded by  $NT_{\mathrm{out}} + (16 + 2\log (T_{\mathrm{out}}))\cdot T_{\mathrm{out}}$ , for  $T_{\mathrm{out}} = |\Pi_{\mathrm{out}}|$ .
Proof. By Lemma B.1, the assumptions of Lemma C.6 and Lemma C.7 are met, so we can therefore apply these results in our setting. By Lemma C.6, the event  $\mathcal{E}_{\mathrm{exp}}$  occurs with probability at least  $1 - \delta$ . Throughout the remainder of the proof we union bound over the success event of Lemma C.7 and  $\mathcal{E}_{\mathrm{exp}}$ , which together occur with probability at least  $1 - 4\delta$ .
$$
\widetilde {\boldsymbol {\Lambda}} := \sum_ {t = 1} ^ {N T _ {\text {o u t}}} \boldsymbol {\psi} (\boldsymbol {\tau} ^ {t}) = I _ {d _ {\boldsymbol {x}}} \otimes \sum_ {t = 1} ^ {N T _ {\text {o u t}}} \sum_ {h = 1} ^ {H} \boldsymbol {\phi} (\boldsymbol {x} _ {h} ^ {t}, \boldsymbol {u} _ {h} ^ {t}) \boldsymbol {\phi} (\boldsymbol {x} _ {h} ^ {t}, \boldsymbol {u} _ {h} ^ {t}) ^ {\top}
$$
denote the features returned by rerunning every policy in  $\Pi_{\mathrm{out}}N$  times. By Lemma C.7, we then have that:
$$
\left\| \widetilde {\boldsymbol {\Lambda}} - N \cdot \sum_ {\pi \in \Pi_ {\mathrm {o u t}}} \check {\boldsymbol {\Lambda}} _ {\pi} \right\| _ {\mathrm {o p}} \leq \underbrace {\frac {\sqrt {T _ {\mathrm {o u t}}} \cdot \sqrt {8 d _ {\phi} d _ {\boldsymbol {x}} \log (1 + 8 \sqrt {N T _ {\mathrm {o u t}}}) + 8 \log 1 / \delta}}{\sqrt {N} \cdot 6 2 7 2 d _ {\phi} d _ {\boldsymbol {x}} \log \frac {6 8 \tilde {N}}{\delta}}} _ {=: \beta} \cdot \lambda_ {\min } \left(N \cdot \sum_ {\pi \in \Pi_ {\mathrm {o u t}}} \check {\boldsymbol {\Lambda}} _ {\pi}\right).
$$
Applying Proposition 3 with  $\mathcal{E}$  the event that the above conclusion holds and  $\Gamma \coloneqq N\cdot \sum_{\pi \in \Pi_{\mathrm{out}}}\check{\Lambda}_{\pi}$ , we obtain that, with probability at least  $1 - \delta$  (using the mapping to the martingale regression setting described in Appendix A.1.1):
$$
\| \operatorname {v e c} (\widehat {A} - A _ {\star}) \| _ {\mathcal {H}} ^ {2} \leq 5 (1 + \zeta) \cdot \sigma_ {\boldsymbol {w}} ^ {2} \log \frac {6 d _ {\boldsymbol {x}} d _ {\phi}}{\delta} \cdot \operatorname {t r} \left(\mathcal {H} \boldsymbol {\Gamma} ^ {- 1}\right)
$$
for  $\zeta = 26\beta^2\lambda_{\max}(\Gamma)\mathrm{tr}(\Gamma^{-1})$  and  $\beta$  as defined above. Since  $\| \phi (\pmb {x},\pmb {u})\| _2\leq B_\phi$  under Assumption 1, we have  $\| \tilde{\Lambda}_{\pi}\|_{2}\leq B_{\phi}^{2}$ , so we can upper bound  $\lambda_{\max}(\Gamma)\leq NT_{\mathrm{out}}B_{\phi}^{2}$ . By Lemma C.7, we can also bound (using that  $D = d_{\pmb{x}}HB_{\phi}^{2}$  by Lemma B.1):
$$
\mathrm {t r} (\Gamma^ {- 1}) \leq \frac {1}{N \cdot 6 2 7 2 d _ {\boldsymbol {x}} H B _ {\phi} ^ {2} \log \frac {6 8 \widetilde {N}}{\delta}}.
$$
Combining these and using that
$$
T _ {\text {o u t}} \leq \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min } ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {\widetilde {N}}{\delta}\right) \tag {B.5}
$$
as shown in Lemma C.8 (and using our bounds on  $C_{\mathcal{R}}$  and  $p_{\mathcal{R}}$  in Lemma B.1), we can therefore bound
$$
\zeta \leq \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {\widetilde {N}}{\delta}\right) \cdot \frac {1}{N}.
$$
Using that  $\mathrm{tr}(\mathcal{H}\Gamma^{-1})\leq \| \mathcal{H}\|_{\mathrm{op}}\cdot \mathrm{tr}(\Gamma^{-1})$  , and the bound on  $\mathrm{tr}(\Gamma^{-1})$  given above, it follows that
$$
\| \mathrm {v e c} (\widehat {A} - A _ {\star}) \| _ {\mathcal {H}} ^ {2} \leq 5 \sigma_ {\pmb {w}} ^ {2} \log \frac {6 d _ {\pmb {x}} d _ {\phi}}{\delta} \cdot \mathrm {t r} (\mathcal {H} \pmb {\Gamma} ^ {- 1}) + \mathrm {p o l y} \left(d _ {\phi}, d _ {\pmb {x}}, \frac {1}{\lambda_ {\mathrm {m i n}} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\pmb {w}}}, H, \| \mathcal {H} \| _ {\mathrm {o p}}, \log \frac {\widetilde {N}}{\delta}\right) \cdot \frac {1}{N ^ {2}}.
$$
Finally, by Lemma C.7, we can bound
$$
\operatorname {t r} \left(\mathcal {H} \boldsymbol {\Gamma} ^ {- 1}\right) \leq \frac {1 2}{N T _ {\mathrm {o u t}}} \cdot \min  _ {\boldsymbol {\Lambda} \in \Omega} \operatorname {t r} \left(\mathcal {H} \check {\boldsymbol {\Lambda}} ^ {- 1}\right).
$$
By Lemma C.8, we can bound the total number of episodes collected by Algorithm 7 by  $(16 + 2\log (T_{\mathrm{out}}))\cdot T_{\mathrm{out}}$
Bound on Frobenius Norm Error. By Lemma C.7, we can lower bound
$$
\lambda_ {\min } (\widetilde {\boldsymbol {\Lambda}}) \geq N \cdot 6 2 7 2 d _ {\boldsymbol {x}} d _ {\phi} H B _ {\phi} ^ {2} \log \frac {6 8 \widetilde {N}}{\delta}.
$$
Furthermore, since  $\| \phi (\pmb {x},\pmb {u})\| _2\leq B_\phi$  , we always have  $\| \widetilde{\Lambda}\|_{\mathrm{op}}\leq NT_{\mathrm{out}}B_{\phi}^{2}$  , which implies  $\widetilde{\Lambda}\preceq NT_{\mathrm{out}}B_{\phi}^{2}\cdot I.$  By Proposition 4, we then have that with probability at least  $1 - \delta$  (again using the mapping to the martingale regression setting described in Appendix A.1.1):
$$
\begin{array}{l} \| \widehat {A} - A _ {\star} \| _ {\mathrm {F}} \leq C \cdot \sigma_ {\boldsymbol {w}} \sqrt {\frac {\log 1 / \delta + d _ {\boldsymbol {x}} d _ {\phi} + \log \det  \left(\frac {T _ {\text {o u t}}}{6 2 7 2 d _ {\boldsymbol {x}} d _ {\phi} H \log \frac {6 8 \widetilde {N}}{\delta}} \cdot I + I\right)}{N \cdot 6 2 7 2 d _ {\boldsymbol {x}} d _ {\phi} H B _ {\phi} ^ {2} \log \frac {6 8 \widetilde {N}}{\delta}}} \\ \leq \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min} ^ {\star}}, B _ {A}, B _ {\phi}, \sigma_ {\boldsymbol {w}}, H, \log \frac {\widetilde {N}}{\delta}\right) \cdot \frac {1}{\sqrt {N}}. \\ \end{array}
$$
Lemma B.4. Under Assumption 3, for any  $\mathcal{H},\mathcal{H}^{\prime}$  we can bound
$$
\min  _ {\tilde {\Lambda} \in \Omega} \operatorname {t r} \left(\mathcal {H} \check {\Lambda} ^ {- 1}\right) \leq \min  _ {\tilde {\Lambda} \in \Omega} 2 \operatorname {t r} \left(\mathcal {H} ^ {\prime} \check {\Lambda} ^ {- 1}\right) + \frac {2 d _ {\boldsymbol {x}} d _ {\phi}}{\lambda_ {\min } ^ {\star}} \cdot \| \mathcal {H} - \mathcal {H} ^ {\prime} \| _ {\mathrm {o p}}.
$$
$$
\begin{array}{l} \min  _ {\check {\Lambda} \in \Omega} \operatorname {t r} \left(\mathcal {H} \check {\Lambda} ^ {- 1}\right) = \min  _ {\check {\Lambda} \in \Omega} \operatorname {t r} \left(\mathcal {H} ^ {\prime} \check {\Lambda} ^ {- 1}\right) + \operatorname {t r} \left(\left(\mathcal {H} - \mathcal {H} ^ {\prime}\right) \check {\Lambda} ^ {- 1}\right) \\ \leq \min _ {\boldsymbol {\Lambda} \in \Omega} \operatorname {t r} (\mathcal {H} ^ {\prime} \check {\boldsymbol {\Lambda}} ^ {- 1}) + \| \mathcal {H} - \mathcal {H} ^ {\prime} \| _ {\mathrm {o p}} \cdot \operatorname {t r} (\check {\boldsymbol {\Lambda}} ^ {- 1}) \\ \end{array}
$$
Under Assumption 3, we know that there exists some  $\tilde{\Lambda}^{\prime}\in \Omega$  such that  $\lambda_{\mathrm{min}}(\tilde{\Lambda}^{\prime})\geq \lambda_{\mathrm{min}}^{\star}$ . We can then bound
$$
\begin{array}{l} \min  _ {\check {\boldsymbol {\Lambda}} \in \Omega} \operatorname {t r} \left(\mathcal {H} ^ {\prime} \check {\boldsymbol {\Lambda}} ^ {- 1}\right) + \| \mathcal {H} - \mathcal {H} ^ {\prime} \| _ {\mathrm {o p}} \cdot \operatorname {t r} \left(\check {\boldsymbol {\Lambda}} ^ {- 1}\right) \\ \leq \min  _ {\tilde {\boldsymbol {\Lambda}} \in \boldsymbol {\Omega}} \operatorname {t r} \left(\mathcal {H} ^ {\prime} \left(\frac {1}{2} \tilde {\boldsymbol {\Lambda}} + \frac {1}{2} \tilde {\boldsymbol {\Lambda}} ^ {\prime}\right) ^ {- 1}\right) + \| \mathcal {H} - \mathcal {H} ^ {\prime} \| _ {\mathrm {o p}} \cdot \operatorname {t r} \left(\left(\frac {1}{2} \tilde {\boldsymbol {\Lambda}} + \frac {1}{2} \tilde {\boldsymbol {\Lambda}} ^ {\prime}\right) ^ {- 1}\right) \\ \leq \min  _ {\boldsymbol {\Lambda} \in \Omega} 2 \operatorname {t r} \left(\mathcal {H} ^ {\prime} \check {\boldsymbol {\Lambda}} ^ {- 1}\right) + 2 \| \mathcal {H} - \mathcal {H} ^ {\prime} \| _ {\mathrm {o p}} \cdot \frac {d _ {\boldsymbol {x}} d _ {\boldsymbol {\phi}}}{\lambda_ {\min} ^ {\star}} \\ \end{array}
$$
C Experiment Design in Arbitrary Dynamical Systems
In this section we generalize somewhat the setting of Section 5. In particular, our goal will now be to collect some set of trajectories  $\mathfrak{D} = \{\pmb{\tau}_t\}_{t=1}^T$ , which minimize
$$
\Phi \left(\frac {1}{T} \sum_ {t = 1} ^ {T} \psi (\boldsymbol {\tau} _ {t})\right)
$$
for some general feature mapping  $\psi : \mathcal{T} \to \mathbb{R}^d$ ,  $\Phi : \mathbb{R}^d \to \mathbb{R}$ , and  $\mathcal{T} = (\mathcal{X} \times \mathcal{U})^H \times \mathcal{X}$  the space of possible state-input trajectories,  $\pmb{\tau} = (x_0, u_0, x_1, \dots, x_{H-1}, u_{H-1}, x_H) \in \mathcal{T}$ . We will assume that  $\psi$  can be decomposed additively as
$$
\boldsymbol {\psi} (\boldsymbol {\tau}) = \sum_ {h = 1} ^ {H} \boldsymbol {\psi} _ {h} \left(\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}\right).
$$
In Section 5 we considered the special case where  $\psi_h(\pmb{x},\pmb{u}) = \phi (\pmb{x},\pmb{u})\phi (\pmb{x},\pmb{u})^\top$ ; in this section  $\psi$  could instead be any arbitrary mapping.
As before, we will be interested in defining optimal exploration with respect to some set of exploration policies,  $\Pi_{\mathrm{exp}}$ . Let
$$
\boldsymbol {\Omega} _ {\boldsymbol {\psi}} := \left\{\mathbb {E} _ {\pi \sim \omega} \left[ \mathbb {E} _ {\pi} [ \boldsymbol {\psi} (\boldsymbol {\tau}) ] \right]: \omega \in \triangle_ {\Pi} \right\}
$$
denote the space of expected value of  $\psi(\tau)$  for mixtures of policies in  $\Pi_{\mathrm{exp}}$ . To distinguish elements  $\Lambda \in \Omega$  from elements in  $\Omega_{\psi}$ , we will let  $\Gamma = \mathbb{E}_{\pi \sim \omega}[\mathbb{E}_{\pi}[\psi(\tau)]]$  refer to elements of  $\Omega_{\psi}$ , and in particular define  $\Gamma_{\pi} \coloneqq \mathbb{E}_{\pi}[\psi(\tau)]$  (where it is assumed that the expectation is collected over trajectories on (5.1)). We will usually denote unnormalized sums of features, e.g.  $\sum_{t=1}^{T} \psi(\tau_t)$ , with  $\Sigma$ . We also define  $\widehat{\Omega}_{\psi}$  to be the space of all possible combinations of  $\psi(\tau)$ :
$$
\widehat {\Omega} _ {\psi} := \left\{\mathbb {E} _ {\tau \sim \omega} [ \psi (\boldsymbol {\tau}) ]: \omega \in \triangle_ {\mathcal {T}} \right\}.
$$
We generalize Assumption 7 and Assumption 8 as follows.
Assumption 9 (Regularity of  $\Phi$ ). We make the following assumptions:
1.  $\Phi$  is convex, differentiable, and  $\beta$ -smooth in the norm  $\|\cdot\|$ :
$$
\left\| \nabla_ {\mathbf {r}} \Phi (\mathbf {r}) - \nabla_ {\mathbf {r} ^ {\prime}} \Phi (\mathbf {r} ^ {\prime}) \right\| _ {*} \leq \beta \cdot \| \mathbf {r} - \mathbf {r} ^ {\prime} \|, \quad \forall \mathbf {r}, \mathbf {r} ^ {\prime} \in \widehat {\Omega} _ {\psi}
$$
for  $\| \cdot \|_{*}$  the dual norm of  $\| \cdot \|$ .
2. There exists some  $M < \infty$  satisfying
$$
\sup_{\boldsymbol {\Gamma}\in \widehat{\Omega}_{\boldsymbol{\psi}}}\sup_{\boldsymbol {\tau}\in \mathcal{T}}|\langle \nabla_{\boldsymbol{\Gamma}}\Phi (\boldsymbol {\Gamma}),\boldsymbol {\psi}(\boldsymbol {\tau})\rangle | \leq M.
$$
Assumption 10 (Regret Minimization Oracle). Let  $\mathrm{cost}_h(\pmb{\tau}) = \langle Q_h, \psi_h(\pmb{\tau}) \rangle$  for some  $Q_h \in \mathbb{R}^d$ , and  $\mathrm{cost}(\pmb{\tau}) = \sum_{h=1}^{H} \mathrm{cost}_h(\pmb{\tau})$  the total cost of trajectory  $\pmb{\tau}$ . We assume we have access to some learner  $\mathbb{A}_{\mathcal{R}}$  which, in the setting when  $|\mathrm{cost}(\pmb{\tau})| \leq 1$  for all  $\pmb{\tau} \in \mathcal{T}$ , is able to achieve low regret on  $\{\mathrm{cost}_h(\cdot, \cdot)\}_{h=1}^H$  with respect to policy class  $\Pi_{\mathrm{exp}}$ . That is, with probability at least  $1 - \delta$ :
$$
\sum_ {t = 1} ^ {T} \mathbb {E} _ {f, \pi_ {t}} [ \operatorname {c o s t} (\boldsymbol {\tau} _ {t}) ] - T \cdot \inf  _ {\pi \in \Pi_ {\exp}} \mathbb {E} _ {f, \pi} [ \operatorname {c o s t} (\boldsymbol {\tau}) ] \leq C _ {\mathcal {R}} \cdot \log^ {p \mathcal {R}} \frac {T}{\delta} \cdot T ^ {\alpha}
$$
for some  $C_{\mathcal{R}} > 0$ ,  $p_{\mathcal{R}} > 0$ , and  $\alpha \in (0,1)$ , and where  $\pi_t$  is the policy  $\mathbb{A}_{\mathcal{R}}$  plays at episode  $t$ .
Algorithm 4 Dynamic Optimal Experiment Design (DYNAMICOED)
1: input: objective  $\Phi$ , number of episodes  $T$  (OR number of iterates  $N$ , episodes per iterate  $K$ ), confidence  $\delta$ , regret minimization algorithm  $\mathbb{A}_{\mathcal{R}}$ , exploration policies  $\Pi_{\mathrm{exp}}$
2: Play any policy  $\pi_{\mathrm{exp}} \in \Pi_{\mathrm{exp}}$  for  $K$  episodes, collect trajectories  $\mathfrak{D}_0 = \{\pmb{\tau}_k^0\}_{k=1}^K$ , set  $\Gamma_0 \gets K^{-1} \sum_{k=1}^{K} \psi(\pmb{\tau}_k^0)$
3: for  $n = 1,2,\ldots ,N$  do
4: Set  $\gamma_{n}\gets \frac{1}{n + 1}$
5: Run  $\mathbb{A}_{\mathcal{R}}$  on cost