SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| (A _ {\star} - \widehat {A} ^ {t}) \phi (\pmb {x} _ {h}, \pmb {u} _ {h}) \| _ {2} \leq \| (A _ {\star} - \widehat {A} ^ {t}) \pmb {\Sigma} _ {t} ^ {1 / 2} \| _ {2} \| \pmb {\Sigma} _ {t} ^ {- 1 / 2} \phi (\pmb {x} _ {h}, \pmb {u} _ {h}) \| _ {2} \\ \leq \left(\| (A _ {\star} - \bar {A} ^ {t}) \boldsymbol {\Sigma} _ {t} ^ {1 / 2} \| _ {2} + \| (\bar {A} ^ {t} - \widehat {A} ^ {t}) \boldsymbol {\Sigma} _ {t} ^ {1 / 2} \| _ {2}\right) \cdot \| \boldsymbol {\Sigma} _ {t} ^ {- 1 / 2} \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) \| _ {2} \\ \leq 2 \beta^ {t} \| \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) \| _ {\boldsymbol {\Sigma} _ {t} ^ {- 1}} \\ \end{array}
$$
where the last inequality follows from the definition of  $\mathrm{BALL}^t$  since  $\widehat{A}^t\in \mathrm{BALL}^t$  by construction, and by the definition of  $\mathcal{E}_1$ . This gives
$$
\left. \left(\mathrm {E}. 1\right) \leq \sum_ {t = 1} ^ {T} c _ {\max } \cdot \mathbb {E} _ {A _ {\star}, \pi^ {t}} \left[ \sum_ {h = 1} ^ {H} \min  \left\{\frac {2 \beta^ {t}}{\sigma_ {\boldsymbol {w}}} \| \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) \| _ {\boldsymbol {\Sigma} _ {i} ^ {- 1}}, 1 \right\} \right]. \right.
$$
By Lemma E.3, with probability  $1 - \delta$  we can bound this as
$$
\leq \underbrace {\frac {2 c _ {\operatorname* {m a x}} \beta^ {T}}{\sigma_ {\boldsymbol {w}}} \cdot \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \min  \left\{\| \phi (\boldsymbol {x} _ {h} ^ {t} , \boldsymbol {u} _ {h} ^ {t}) \| _ {\boldsymbol {\Sigma} _ {t} ^ {- 1}} , 1 \right\}} _ {(a)} + 4 c _ {\operatorname * {m a x}} H \sqrt {T \log 1 / \delta}.
$$
By Cauchy-Schwarz, we can bound  $(a)$  as
$$
(a) \leq \frac {2 c _ {\max } \beta^ {T}}{\sigma_ {\boldsymbol {w}}} \cdot \sqrt {T} \sqrt {\sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \min  \left\{\| \phi (\boldsymbol {x} _ {h} ^ {t} , \boldsymbol {u} _ {h} ^ {t}) \| _ {\boldsymbol {\Sigma} _ {t} ^ {- 1}} ^ {2} , 1 \right\}}.
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \min  \left\{\| \phi \left(\boldsymbol {x} _ {h} ^ {t}, \boldsymbol {u} _ {h} ^ {t}\right) \| _ {\boldsymbol {\Sigma} _ {t} ^ {- 1}} ^ {2}, 1 \right\} H \leq \sum_ {t = 1} ^ {T} \min  \left\{\sum_ {h = 1} ^ {H} \| \phi \left(\boldsymbol {x} _ {h} ^ {t}, \boldsymbol {u} _ {h} ^ {t}\right) \| _ {\boldsymbol {\Sigma} _ {t} ^ {- 1}} ^ {2}, 1 \right\} \\ \leq 2 H \log (\det  (\boldsymbol {\Sigma} _ {T}) \det  (\boldsymbol {\Sigma} _ {0}) ^ {- 1}) \\ \end{array}
$$
where the last inequality uses Lemma B.6 of [17]. Putting all of this together, we have shown that with probability at least  $1 - 2\delta$ , we have
$$
\mathcal {R} _ {T} \leq \frac {2 c _ {\max} \beta^ {T}}{\sigma_ {\boldsymbol {w}}} \cdot \sqrt {T} \cdot \sqrt {2 H \log (\det (\boldsymbol {\Sigma} _ {T}) \det (\boldsymbol {\Sigma} _ {0}) ^ {- 1})} + 4 c _ {\max} H \sqrt {T \log 1 / \delta}.
$$
It remains to bound  $\beta^T$  and  $\log (\operatorname*{det}(\pmb{\Sigma}_T)\operatorname*{det}(\pmb{\Sigma}_0)^{-1})$ . We have  $\pmb{\Sigma}_0 = \lambda I$ , so  $\operatorname*{det}(\pmb{\Sigma}_0) = \lambda^{d_{\phi}}$ . Furthermore, if  $\| \phi (\pmb {x},\pmb {u})\| _2\leq B_\phi$ , then we can bound  $\operatorname*{det}(\pmb {\Sigma}_T)\leq (\lambda +B_\phi^2 TH)^{d_\phi}$ . Putting this together we have
$$
\log (\det (\pmb {\Sigma} _ {T}) \det (\pmb {\Sigma} _ {0}) ^ {- 1}) \leq d _ {\phi} \cdot \log (1 + B _ {\phi} ^ {2} T H / \lambda).
$$
$$
\beta^ {T} = \sqrt {\lambda} B _ {A} + \sigma_ {\pmb {w}} \sqrt {8 d _ {\pmb {x}} \log 5 + 8 \log (T \det (\pmb {\Sigma} _ {T}) \det (\pmb {\Sigma} _ {0}) ^ {- 1} / \delta)}
$$
we can similarly bound
$$
\begin{array}{l} \beta^ {T} / \sigma_ {\boldsymbol {w}} \leq \sqrt {\lambda} B _ {A} / \sigma_ {\boldsymbol {w}} + \sqrt {8 d _ {\boldsymbol {x}} \log 5 + 8 d _ {\phi} \cdot \log (1 + B _ {\phi} ^ {2} T H / \lambda) + 8 \log (T / \delta)} \\ \leq \sqrt {\lambda} B _ {A} / \sigma_ {\pmb {w}} + c \sqrt {d _ {\pmb {x}} + d _ {\phi} \log (1 + B _ {\phi} T H / \lambda) + \log 1 / \delta}. \\ \end{array}
$$
Choosing  $\lambda = \sigma_w^2$  completes the proof.
E.1 Supporting Lemmas
Lemma E.1. Under Assumption 14, we can bound
$$
\mathcal {J} (\pi ; A _ {\star}) - \mathcal {J} (\pi ; A) \leq c _ {\max } \cdot \mathbb {E} _ {A _ {\star}, \pi} \left[ \sum_ {h = 1} ^ {H} \min  \left\{\frac {1}{\sigma_ {\boldsymbol {w}}} \| (A _ {\star} - A) \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) \| _ {2}, 1 \right\} \right].
$$
Proof. Following the proof of Lemma B.3 of [17], and adopting the same notation, we have
$$
\mathcal {J} (\pi ; A _ {\star}) - \mathcal {J} (\pi ; A) \leq \sum_ {h = 1} ^ {H} \mathbb {E} _ {A _ {\star}, \pi} \left[ \sqrt {A _ {h}} \min  \left\{\frac {1}{\sigma_ {\boldsymbol {w}}} \| (A _ {\star} - A) \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) \| _ {2}, 1 \right\} \right].
$$
Under Assumption 14 we have  $A_{h} \leq c_{\max}^{2}$ . Plugging this in gives the result.
Lemma E.2. Let  $\beta^t \coloneqq \sqrt{\lambda} B_A + \sigma_{\mathbf{w}} \sqrt{8d_x \log 5 + 8 \log (T \det(\Sigma_t) \det(\Sigma_0)^{-1} / \delta)}$  and let  $\mathcal{E}_1$  denote the event
$$
\mathcal {E} _ {1} := \left\{\forall t \leq T: \left\| \left(\bar {A} ^ {t} - A _ {\star}\right) \boldsymbol {\Sigma} _ {t} ^ {1 / 2} \right\| _ {\mathrm {o p}} \leq \beta^ {t} \right\}.
$$
Then running  $\mathrm{LC}^3$  we have  $\mathbb{P}_{A_*}[\mathcal{E}_1] \geq 1 - \delta$ .
Proof. The proof of Lemma B.5 of [17] shows that with probability at least  $1 - \delta$ ,
$$
\left\| \left(\bar {A} ^ {t} - A _ {\star}\right) \boldsymbol {\Sigma} _ {t} ^ {1 / 2} \right\| _ {\mathrm {o p}} \leq \sqrt {\lambda} \| A _ {\star} \| _ {\mathrm {o p}} + \sigma_ {\boldsymbol {w}} \sqrt {8 d _ {\boldsymbol {x}} \log 5 + 8 \log (\det (\boldsymbol {\Sigma} _ {t}) \det (\boldsymbol {\Sigma} _ {0}) ^ {- 1} / \delta)}.
$$
The result then follows from this, since  $\| A_{\star} \|_{\mathrm{op}} \leq B_A$ , and a union bound.
Lemma E.3. With probability  $1 - \delta$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \mathbb {E} _ {A _ {\star}, \pi^ {t}} \left[ \sum_ {h = 1} ^ {H} \min  \left\{\frac {2 \beta^ {t}}{\sigma_ {\boldsymbol {w}}} \| \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) \| _ {\boldsymbol {\Sigma} _ {t} ^ {- 1}}, 1 \right\} \right] \leq \frac {2 \beta^ {T}}{\sigma_ {\boldsymbol {w}}} \sum_ {t = 1} ^ {T} \sum_ {h = 1} ^ {H} \min  \left\{\| \phi (\boldsymbol {x} _ {h} ^ {t}, \boldsymbol {u} _ {h} ^ {t}) \| _ {\boldsymbol {\Sigma} _ {t} ^ {- 1}}, 1 \right\} \\ + 4 H \sqrt {T \log {1 / \delta}}. \\ \end{array}
$$
Proof. This is an immediate consequence of Azuma-Hoeffding, since  $\sum_{h=1}^{H} \min \left\{ \frac{2\beta^t}{\sigma_w} \| \phi(\boldsymbol{x}_h, \boldsymbol{u}_h) \|_{\Sigma_t^{-1}}, 1 \right\} \leq H$  almost surely, and from upper bounding
$$
\min \left\{\frac {2 \beta^ {t}}{\sigma_ {\boldsymbol {w}}} \| \phi (\boldsymbol {x} _ {h} ^ {t}, \boldsymbol {u} _ {h} ^ {t}) \| _ {\boldsymbol {\Sigma} _ {t} ^ {- 1}}, 1 \right\} \leq \frac {2 \beta^ {T}}{\sigma_ {\boldsymbol {w}}} \min \left\{\| \phi (\boldsymbol {x} _ {h} ^ {t}, \boldsymbol {u} _ {h} ^ {t}) \| _ {\boldsymbol {\Sigma} _ {t} ^ {- 1}}, 1 \right\}.
$$
F Lower Bounds on Learning in Nonlinear Systems
In this section, we assume that  $\theta_{\star}$  and  $\pi_{\star}$  correspond to the global minimizer:
$$
\boldsymbol {\theta} _ {\star} (A) := \underset {\boldsymbol {\theta} \in \mathbb {R} ^ {d} \boldsymbol {\theta}} {\arg \min } \mathcal {J} (\boldsymbol {\theta}; A), \quad \pi_ {\star} (A) := \underset {\pi \in \Pi^ {\star}} {\arg \min } \mathcal {J} (\pi ; A). \tag {F.1}
$$
Here we formally state the additional assumptions needed in Section 4.2, and provide a formal version of Theorem 2.
Assumption 15. There exists some  $r_{\mu}(A_{\star}) > 0$  such that, for all  $A \in \mathcal{B}_{\mathrm{F}}(A_{\star}, r_{\mu}(A_{\star}))$ ,  $\pi_{\star}(A)$  is unique and, furthermore, there exists some  $\mu > 0$  such that
$$
\mathcal {J} (\boldsymbol {\theta}; A) \geq \mathcal {J} (\boldsymbol {\theta} _ {\star} (A); A) + \frac {\mu}{2} \| \boldsymbol {\theta} - \boldsymbol {\theta} ^ {\star} (A) \| _ {2} ^ {2}.
$$
Assumption 15 will be satisfied in cases where  $\mathcal{J}(\pi^{\theta};A)$  is strongly convex in  $\pmb{\theta}$ , but may hold even when this is not the case. Intuitively, it requires that our controller class is not overparameterized—moving  $\pmb{\theta}$  away from its optimal value will cause the loss to increase. We will additionally make the following regularity assumptions on policies in  $\Pi_{\mathrm{exp}}$  and their induced covariates set,  $\Omega$ .
Assumption 16. There exists some  $\underline{\lambda} >0$  such that, for each  $\Lambda \in \Omega$ , we have  $\lambda_{\min}(\Lambda)\geq \underline{\lambda}$ .
Assumption 16 requires that every exploration policy we consider excites all directions in  $\phi$  space (in contrast, Assumption 3 only assumes there exists some distribution over policies in  $\Pi_{\mathrm{exp}}$  which excite all directions). We remark that this assumption is relatively mild if Assumption 3 holds. As we show in Appendix C.2, under Assumption 3, a mixture over policies,  $\omega$ , satisfying  $\lambda_{\min}(\mathbb{E}_{\pi \sim \omega}[\Lambda_{\pi}] > 0$  can be learned using only a number of samples scaling polynomially in problem parameters. Given  $\omega$ , a policy class  $\Pi_{\mathrm{exp}}$  satisfying Assumption 16 can be obtained by simply mixing  $\omega$  with every other exploration policy. We are now ready to state our main lower bound.
Theorem 7 (Formal Version of Theorem 2). Under Assumptions 1, 2, 4, 5, 13, 15 and 16 and if  $\pi_{\star}$  is defined as in (F.1), as long as  $T \geq C_{\mathrm{lb}}$ , for any  $\omega_{\mathrm{exp}} \in \triangle_{\Pi_{\mathrm{exp}}}$ , we have
for  $\mathcal{B}_T\coloneqq \{A: \| A - A_\star \| _\mathrm{F}^2\leq 5d_xd_\phi /(\underline{\lambda} d_xTH)^{5 / 6}\}$ ,  $\mathbb{E}_{\mathfrak{D}_T\sim A,\omega_{\mathrm{exp}}}[\cdot ] = \mathbb{E}_{\pi \sim \omega_{\mathrm{exp}}}\bigl [\mathbb{E}_{\mathfrak{D}_T\sim A,\pi}[\cdot ]\bigr ]$  denotes the expectation over trajectories generated by running policies  $\pi$  drawn according to  $\omega_{\mathrm{exp}}$  on system  $A$  for  $T$  episodes,  $\widehat{\pi}$  any mapping from observations to policies in  $\Pi^{\star}$ , and
$$
C _ {\mathrm {l b}} := \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, H, \| A _ {\star} \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, L _ {\mathrm {c o s t}}, L _ {\pi_ {\star}}, \sigma_ {\boldsymbol {w}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, \frac {1}{\lambda}, \frac {1}{\mu}, \frac {1}{r _ {\mathrm {c o s t}} (A _ {\star})}, \frac {1}{r _ {\boldsymbol {\theta}} (A _ {\star})}, \frac {1}{r _ {\mu} (A _ {\star})}\right).
$$
Proof of Theorem 7. This proof follows immediately from Lemma F.1, by lower bounding the right-hand side of (F.2) by the min over all policies in  $\Pi$ .
Lemma F.1. Under Assumptions 1, 2, 4, 5, 13, 15 and 16 and if  $\theta_{\star}$  is defined as in (F.1), as long as
$$
T \geq \mathrm {p o l y} (\| A _ {\star} \| _ {\mathrm {o p}}, L _ {\phi}, L _ {\pmb {\theta}}, L _ {\pi_ {\star}}, L _ {\mathrm {c o s t}}, \sigma_ {\pmb {w}}, \sigma_ {\pmb {w}} ^ {- 1}, B _ {\phi}, H, d _ {\pmb {x}}, d _ {\phi}, \underline {{\lambda}} ^ {- 1}, \mu^ {- 1}, r _ {\mathrm {c o s t}} (A _ {\star}) ^ {- 1}, r _ {\pmb {\theta}} (A _ {\star}) ^ {- 1}, r _ {\mu} (A _ {\star}) ^ {- 1}),
$$
for any  $\omega_{\mathrm{exp}} \in \triangle_{\Pi}$ , we have
$$
\min  _ {\hat {\boldsymbol {\theta}}} \max  _ {A \in \mathcal {B} _ {T}} \mathbb {E} _ {\mathfrak {D} _ {T} \sim A, \omega_ {\exp}} [ \mathcal {J} (\widehat {\boldsymbol {\theta}} (\mathfrak {D} _ {T}); A) - \mathcal {J} (\boldsymbol {\theta} _ {\star} (A); A) ] \geq \frac {\sigma_ {\boldsymbol {w}} ^ {2}}{3 T} \cdot \operatorname {t r} (\mathcal {H} (A _ {\star}) \mathbb {E} _ {\pi \sim \omega_ {\exp}} [ \check {\Lambda} _ {\pi} ] ^ {- 1}) - \frac {C _ {\mathrm {l b}}}{T ^ {5 / 4}} \tag {F.2}
$$
for  $\mathcal{B}_T\coloneqq \{A:\| A - A_\star \| _\mathrm{F}^2\leq 5d_\pmb {x}d_\phi /(\underline{\lambda} TH)^{5 / 6}\}$ , where  $\mathbb{E}_{\mathfrak{D}_T\sim A,\omega_{\mathrm{exp}}}[\cdot ] = \mathbb{E}_{\pi \sim \omega_{\mathrm{exp}}}[\mathbb{E}_{\mathfrak{D}_T\sim A,\pi}[\cdot ]]$  denotes the expectation over trajectories generated by running policies  $\pi$  drawn according to  $\omega_{\mathrm{exp}}$  on system  $A$  for  $T$  episodes, and
$$
C _ {\mathrm {l b}} := \mathrm {p o l y} (\| A _ {\star} \| _ {\mathrm {o p}}, L _ {\phi}, L _ {\theta}, L _ {\pi_ {\star}}, L _ {\mathrm {c o s t}}, \sigma_ {\boldsymbol {w}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, B _ {\phi}, H, d _ {\boldsymbol {x}}, d _ {\phi}, \underline {{\lambda}} ^ {- 1}, \mu^ {- 1}, r _ {\mathrm {c o s t}} (A _ {\star}) ^ {- 1}, r _ {\boldsymbol {\theta}} (A _ {\star}) ^ {- 1}, r _ {\mu} (A s t) ^ {- 1}).
$$
Proof. This result is a direct consequence of Theorem 6.1 of [2]—to obtain the result we must only verify that the assumptions of this result are met. We verify each assumption below.
Verifying Assumption 3 of [2]. Part 1 of Assumption 3 of [2] is met by Assumption 15 within diameter  $r_{\mu}(A_{\star})$ . Furthermore, under Assumptions 1, 2, 4, 5 and 13 and by Lemma D.1, the additional parts of Assumption 3 of [2] are also met with diameter  $\min \{r_{\mathrm{cost}}(A_{\star}), r_{\theta}(A_{\star})\}$  and smoothness constant  $\mathrm{poly}(\| A_{\star}\|_{\mathrm{op}}, L_{\phi}, L_{\theta}, L_{\pi_{\star}}, L_{\mathrm{cost}}, \sigma_{\boldsymbol{w}}^{-1}, H, d_{\boldsymbol{x}})$ .
Verifying Assumption 4 and Assumption 5 of [2]. Assumption 4 of [2] is immediately met by Assumption 16. Furthermore, Assumption 5 is met by Lemma F.2 with  $c_{\mathrm{cov}} = 1$ ,  $L_{\mathrm{cov}}(\theta_{\star}, \gamma^2) = \frac{H^2B_\phi^3}{\sigma_w^2} \cdot \mathrm{poly}(d_x)$ , and  $C_{\mathrm{cov}} = 0$ .
Given that these assumptions are met, the result follows noting that, if we run for  $T$  episodes, then the effective horizon is  $d_{\pmb{x}}TH$  (using the mapping from the setting of (1.1) to the martingale regression setting described in Appendix A.1.1). Note that the final bound scales with  $\frac{1}{T}$  instead of  $\frac{1}{d_xTH}$  as we are able to bring the  $d_{\pmb{x}}H$  factor into the  $\check{\Lambda}_{\pi_{\mathrm{exp}}}$  term, since  $\Lambda_{\pi_{\mathrm{exp}}}$  is not normalized by  $d_{\pmb{x}}H$ .
Lemma F.2. Under Assumption 1, for any policy distribution  $\omega \in \triangle_{\Pi_{\mathrm{exp}}}^{}$  and  $A,A^{\prime}$  , we have
$$
\mathbb {E} _ {\pi \sim \omega} [ \boldsymbol {\Lambda} _ {A, \pi} ] \preceq \mathbb {E} _ {\pi \sim \omega} [ \boldsymbol {\Lambda} _ {A ^ {\prime}, \pi} ] + \frac {H ^ {2} B _ {\phi} ^ {3}}{\sigma_ {\boldsymbol {w}} ^ {2}} \cdot \mathrm {p o l y} (d _ {\boldsymbol {x}}) \cdot \| A - A ^ {\prime} \| _ {\mathrm {F}} \cdot I.
$$
Proof. We will prove that the desired bound follows for a particular  $\pi \in \Pi_{\mathrm{exp}}$ , which immediately implies that it holds for  $\omega \in \triangle_{\Pi_{\mathrm{exp}}}$ . By definition we have
$$
\boldsymbol {\Lambda} _ {A, \pi} = \int \left(\sum_ {h = 1} ^ {H} \phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}}) \phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}}) ^ {\top}\right) \cdot f _ {A, \pi} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau}
$$
$$
f _ {A, \pi} (\boldsymbol {\tau}) = \prod_ {h = 1} ^ {H} f _ {A} \left(\boldsymbol {x} _ {h + 1} ^ {\boldsymbol {\tau}} \mid \boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}}\right) \pi_ {h} \left(\boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}} \mid \boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}\right).
$$
Fix some  $\pmb{v} \in S^{d_{\phi} - 1}$ , and note that, given the expression above, we have
$$
\boldsymbol {v} ^ {\top} \boldsymbol {\Lambda} _ {A, \pi} \boldsymbol {v} = \int \sum_ {h = 1} ^ {H} (\boldsymbol {v} ^ {\top} \phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}})) ^ {2} \cdot f _ {A, \pi} (\boldsymbol {\tau}) d \boldsymbol {\tau}.
$$
$$
\nabla_ {A} \boldsymbol {v} ^ {\top} \boldsymbol {\Lambda} _ {A, \pi} \boldsymbol {v} = \int \sum_ {h = 1} ^ {H} (\boldsymbol {v} ^ {\top} \phi (\boldsymbol {x} _ {h} ^ {\tau}, \boldsymbol {u} _ {h} ^ {\tau})) ^ {2} \cdot \nabla_ {A} f _ {A, \pi} (\tau) \mathrm {d} \tau
$$
$$
= \int \sum_ {h = 1} ^ {H} \left(\boldsymbol {v} ^ {\top} \phi \left(\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}}\right)\right) ^ {2} \cdot f _ {A, \pi} (\boldsymbol {\tau}) \nabla_ {A} \log f _ {A, \pi} (\boldsymbol {\tau}) d \boldsymbol {\tau}.
$$
As in the proof of Lemma D.1, we have, for any  $\Delta$
$$
\nabla_ {A} \log f _ {A, \pi} (\boldsymbol {\tau}) [ \Delta ] = \sum_ {h = 1} ^ {H} \frac {1}{\sigma_ {\boldsymbol {w}} ^ {2}} (\boldsymbol {x} _ {h + 1} ^ {\boldsymbol {\tau}} - A \phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}})) \cdot \Delta \phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}}),
$$
so we can bound
$$
\| \nabla_ {A} \log f _ {A, \pi} (\boldsymbol {\tau}) \| _ {\mathrm {o p}} \leq \frac {B _ {\phi}}{\sigma_ {\boldsymbol {w}} ^ {2}} \sum_ {h = 1} ^ {H} \| \boldsymbol {x} _ {h + 1} ^ {\boldsymbol {\tau}} - A \phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \boldsymbol {u} _ {h} ^ {\boldsymbol {\tau}}) \| _ {2}.
$$
Furthermore, we can also bound  $(\pmb{v}^{\top}\phi(\pmb{x}_h^\tau,\pmb{u}_h^\tau))^2 \leq B_\phi^2$ . We therefore have
$$
\begin{array}{l} \| \nabla_ {A} \boldsymbol {v} ^ {\top} \boldsymbol {\Lambda} _ {A, \pi} \boldsymbol {v} \| _ {\mathrm {o p}} \leq H B _ {\phi} ^ {3} \int \sum_ {h = 1} ^ {H} \| \boldsymbol {x} _ {h + 1} ^ {\tau} - A \phi (\boldsymbol {x} _ {h} ^ {\tau}, \boldsymbol {u} _ {h} ^ {\tau}) \| _ {2} \cdot f _ {A, \pi} (\tau) \mathrm {d} \tau \\ \leq \frac {H ^ {2} B _ {\phi} ^ {3}}{\sigma_ {\boldsymbol {w}} ^ {2}} \cdot \operatorname {p o l y} (d _ {\boldsymbol {x}}) \\ \end{array}
$$
where the last inequality follows from Lemma A.1. It follows from the Mean Value Theorem that
$$
\left| \boldsymbol {v} ^ {\top} \boldsymbol {\Lambda} _ {A, \pi} \boldsymbol {v} - \boldsymbol {v} ^ {\top} \boldsymbol {\Lambda} _ {A ^ {\prime}, \pi} \boldsymbol {v} \right| \leq \frac {H ^ {2} B _ {\phi} ^ {3}}{\sigma_ {\boldsymbol {w}} ^ {2}} \cdot \operatorname {p o l y} \left(d _ {\boldsymbol {x}}\right) \cdot \| A - A ^ {\prime} \| _ {\mathrm {F}}.
$$
As this holds for all  $\pmb{v} \in S^{d-1}$ , it follows that
$$
\left\| \boldsymbol {\Lambda} _ {A, \pi} - \boldsymbol {\Lambda} _ {A ^ {\prime}, \pi} \right\| _ {\mathrm {o p}} \leq \frac {H ^ {2} B _ {\phi} ^ {3}}{\sigma_ {\boldsymbol {w}} ^ {2}} \cdot \operatorname {p o l y} \left(d _ {\boldsymbol {x}}\right) \cdot \left\| A - A ^ {\prime} \right\| _ {\mathrm {F}}.
$$
In this section, we provide additional details on our experimental results presented in Section 6. All experiments were run on a machine with 56 Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz CPUs, and 64GB RAM. All code was implemented in PyTorch.
G.1 Details on Problem Settings and Controller Parameterizations
We first expand on the precise definitions of the systems considered. As noted in Section 6, for the drone and car examples we set  $H = 50$ , and for the system of Section 1.1 we set  $H = 10$ . In addition, for all examples the noise is distributed as  $\pmb{w}_h \sim \mathcal{N}(0, 0.1 \cdot I)$ . In all cases we set  $\gamma^2 = 10H$  (where  $\gamma^2$  is a bound on  $\mathbb{E}_{\pi_{\mathrm{exp}}}[\sum_{h=1}^{H} \pmb{u}_h^\top \pmb{u}_h]$ ), and we therefore let  $\Pi_{\mathrm{exp}}$  denote the set of all policies satisfying  $\mathbb{E}_{\pi_{\mathrm{exp}}}[\sum_{h=1}^{H} \pmb{u}_h^\top \pmb{u}_h] \leq \gamma^2$ .
G.1.1 System of Section 1.1 (Figure 1)
The dynamics for this system are given by
$$
\boldsymbol {x} _ {h + 1} = 0. 8 \boldsymbol {x} _ {h} + \boldsymbol {u} _ {h} - \sum_ {i = 1} ^ {1 0} 3 \phi_ {i} (\boldsymbol {x} _ {h}) + \boldsymbol {w} _ {h}
$$
for  $\phi_i(\pmb{x}) = \max \{1 - 100(\pmb{x} - c_i)^2, 0\}$ , and  $\mathrm{cost}(\pmb{x}, \pmb{u}) = (\pmb{x} - c_1)^2 + 100^{-1} \cdot \pmb{u}^2$ . We set
$$
c _ {1} = 1 0, c _ {2} = - 1 4, c _ {3} = - 1 1, c _ {4} = - 8, c _ {5} = - 5, c _ {6} = - 2, c _ {7} = 1, c _ {8} = 4, c _ {9} = 7.
$$
This then corresponds to a system in the form (1.1) with
$$
A _ {\star} = [ 0. 8, 1, - 3, \dots , - 3 ], \quad \phi (\boldsymbol {x}, \boldsymbol {u}) = [ \boldsymbol {x}, \boldsymbol {u}, \phi_ {1} (\boldsymbol {x}), \dots , \phi_ {1 0} (\boldsymbol {x}) ].
$$
For this system, we parameterize our controller class  $\Pi^{\star}$  as, for any  $\pi^{\theta} \in \Pi^{\star}$  with parameter  $\theta$ ,
$$
\pi^ {\boldsymbol {\theta}} (\boldsymbol {x}) = \boldsymbol {\theta} _ {1} \boldsymbol {x} + \sum_ {i = 1} ^ {1 0} \boldsymbol {\theta} _ {i + 1} \phi_ {i} (\boldsymbol {x}) + \boldsymbol {\theta} _ {1 2}.
$$
Note that the form of this controller lets us simply "match" the parameters of the system, and cancel undesirable parameters. Given this, for this system we let  $\pi_{\star}(A)$  be the controller which sets  $\theta_{1:11}$  to cancel the dynamics of the system  $A$ , and set  $\theta_{12} = c_1 = 10$ .
See Appendix G.1.3 for details on the computation of  $\mathcal{H}(A)$  on this system.
G.1.2 Drone System (Figure 2)
The dynamics of this system are given by
$$
\boldsymbol {x} _ {h + 1} = \left[ \begin{array}{c c c c c c} 1 & 0 & 0 & 0. 1 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0. 1 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0. 1 \\ 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 & 1 \end{array} \right] \boldsymbol {x} _ {h} + \left[ \begin{array}{c c c} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0. 1 & 0 & 0 \\ 0 & 0. 1 & 0 \\ 0 & 0 & 0. 1 \end{array} \right] \boldsymbol {u} _ {h} + \left[ \begin{array}{c} 0 \\ 0 \\ 0 \\ 0 \\ - 0. 9 8 \end{array} \right] + \boldsymbol {w} _ {h}. \tag {G.1}
$$
Here we interpret  $[x]_{1:3}$  as the  $x, y$ , and  $z$  positions, respectively, and  $[x]_{4:6}$  as the  $x, y, z$  velocities. This system is therefore equivalent to three double integrator systems, with an affine term (which we interpret as "gravity") affecting only the  $z$  coordinate. We set the cost to
$$
\operatorname {c o s t} (\boldsymbol {x}, \boldsymbol {u}) = \frac {0 . 1}{5} \cdot \sum_ {i = 1} ^ {d _ {\boldsymbol {x}}} [ \boldsymbol {x} ] _ {i} ^ {2} + \frac {1}{5} \cdot [ \boldsymbol {u} ] _ {1} ^ {2} + [ \boldsymbol {u} ] _ {2} ^ {2} + [ \boldsymbol {u} ] _ {3} ^ {2}
$$
$$
A _ {\star} = \left[ \begin{array}{c c c c c c c c c c} 1 & 0 & 0 & 0. 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0. 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0. 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 & 0. 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0. 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0. 1 & - 0. 9 8 \end{array} \right], \quad \phi (\boldsymbol {x}, \boldsymbol {u}) = [ \boldsymbol {x}, \boldsymbol {u}, 1 ].
$$
$$
\pi_ {h} ^ {\boldsymbol {\theta}} (\boldsymbol {x}) = \boldsymbol {\theta} _ {h} ^ {\mathrm {f b}} \boldsymbol {x} + \boldsymbol {\theta} _ {h} ^ {\mathrm {o f f s e t}}
$$
where  $\theta_h^{\mathrm{fb}}\in \mathbb{R}^{3\times 6}$  is the state-feedback portion of the controller, and  $\theta_h^{\mathrm{offset}}\in \mathbb{R}^3$  is an offset term. It can be shown that the optimal controller for a system of the form (G.1) can be parameterized in this way [23]. Furthermore, the optimal parameters can be computed in closed-form. As such, for this system we set  $\pi_{\star}(A)$  to be with the optimal parameters, computed using this closed-form solution.
In addition to computing the optimal controller in closed-form, we can also compute the cost of a controller,  $\mathcal{J}(\pi; A)$ , in closed-form. To compute  $\mathcal{H}(A)$  in this example, we then simply apply the torch.autogradfunctional.hessian function to  $\mathcal{J}(\pi_{\star}(A); A)$ .
G.1.3 Car System (Figure 3)