SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 5. Assume that Assumptions 1, 2, 4 and 5 hold and that  $\lambda_{\mathrm{min}}(\nabla_{\pmb{\theta}}^{2}\mathcal{J}(\pmb {\theta};A_{\star})|_{\pmb {\theta} = \pmb{\theta}_{\star}(A_{\star})}) > 0$  Let  $r_{\pmb{\theta}}(A_{\star}) > 0$  be some value satisfying
$$
r _ {\boldsymbol {\theta}} (A _ {\star}) = \min  \left\{r _ {\mathrm {c o s t}} (A _ {\star}), \operatorname {p o l y} \left(\frac {1}{\lambda_ {\operatorname* {m i n}} (\nabla_ {\boldsymbol {\theta}} ^ {2} \mathcal {J} (\boldsymbol {\theta} ; A _ {\star}) | _ {\boldsymbol {\theta} = \boldsymbol {\theta} _ {\star} (A _ {\star})})}, \| A _ {\star} \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, L _ {\mathrm {c o s t}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, d _ {\boldsymbol {x}}\right) ^ {- 1} \right\}.
$$
Then there exists some function  $\pmb{\theta}_{\star}(A)$  such that, for all  $A\in \mathcal{B}_{\mathrm{F}}(A_{\star};r_{\pmb{\theta}}(A_{\star}))$
-  $\nabla_{\pmb{\theta}}\mathcal{J}(\pmb {\theta};A)|_{\pmb {\theta} = \pmb{\theta}_{*}(A)} = 0,$
-  $\theta_{\star}(A)$  is three-times differentiable in  $A$
and it suffices that we take
$$
L _ {\pi_ {\star}} = \operatorname {p o l y} \left(\frac {1}{\lambda_ {\min } (\nabla_ {\boldsymbol {\theta}} ^ {2} \mathcal {J} (\boldsymbol {\theta} ; A _ {\star}) | _ {\boldsymbol {\theta} = \boldsymbol {\theta} _ {\star} (A _ {\star}))}}, \| A _ {\star} \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, L _ {\mathrm {c o s t}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, d _ {\boldsymbol {x}}\right).
$$
While Proposition 5 shows that there exists some  $\theta_{\star}(A)$  satisfying Assumption 13, it does not directly give a recipe for constructing such a map. The following result shows that under a mild additional assumption, the minimizer of the loss satisfies Assumption 13.
Proposition 6. Let
$$
\boldsymbol{\theta}_{\star}(A):= \operatorname *{arg  min}_{\boldsymbol {\theta}\in \mathbb{R}^{d_{\boldsymbol{\theta}}}}\mathcal{J}(\boldsymbol {\theta};A).
$$
Then under Assumptions 1, 2 and 4 to 6, there exists some  $r_{\pmb{\theta}}(A_{\star}) > 0$  and  $L_{\pi_{\star}} < \infty$  such that  $\pmb{\theta}_{\star}(A)$  satisfies Assumption 13.
The scaling of  $L_{\pi_\star}$  in Proposition 6 can be shown to match that of Proposition 5, but in general  $r_\theta(A_\star)$  could be smaller than the value of  $r_\theta(A_\star)$  given in Proposition 5. In particular, in the setting of Proposition 6, we can only show that  $r_\theta(A_\star)$  scales with  $\min_{\theta \notin \mathcal{B}_2(\theta_\star(A_\star); r)} \mathcal{J}(\theta; A_\star) - \mathcal{J}(\theta_\star(A_\star); A_\star)$  for some  $r > 0$  which scales inverse polynomially in problem parameters. While we can show that  $\mathcal{J}(\theta; A_\star) - \mathcal{J}(\theta_\star(A_\star); A_\star)$  scales inverse polynomially in problem parameters, including in  $\lambda_{\min}(\nabla_\theta^2 \mathcal{J}(\theta; A_\star)|_{\theta = \theta_\star(A_\star)})$ , for  $\theta$  approximately a distance of  $r$  from  $\theta_\star(A_\star)$ , it is possible  $\mathcal{J}(\theta; A_\star)$  has some local minimizer  $\theta'$  arbitrarily far away from  $\theta_\star(A_\star)$ , such that  $\mathcal{J}(\theta'; A_\star)$  and  $\mathcal{J}(\theta_\star(A_\star); A_\star)$  are arbitrarily close, in which case  $\Delta^\star$ , and therefore  $r_\theta(A_\star)$ , could be arbitrarily small. The failure mode here is that, while  $\theta_\star(A_\star)$  may be the global minimum of  $\mathcal{J}(\theta; A_\star)$ , for  $A$  arbitrarily close to  $A_\star$ , the global minimum of  $\mathcal{J}(\theta; A)$  could instead be near  $\theta'$ , which would render the map  $\theta_\star(A)$  discontinuous.
By making further assumptions on  $\mathcal{J}(\pmb{\theta}; A_{\star})$  which exclude this case, we can obtain a value of  $r_{\pmb{\theta}}(A_{\star})$  scaling similarly to in Proposition 5. For example, in the following, we show that under the assumption that  $\mathcal{J}(\pmb{\theta}; A)$  is convex, this holds.
Proposition 7. Assume that there exists some  $r_{\mathrm{conv}}(A_\star) > 0$  such that, for all  $A \in \mathcal{B}_{\mathrm{F}}(A_\star; r_{\mathrm{conv}}(A_\star))$ ,  $\mathcal{J}(\theta; A)$  is convex in  $\theta$ , and set
$$
\boldsymbol{\theta}_{\star}(A) = \operatorname *{arg  min}_{\boldsymbol {\theta}\in \mathbb{R}^{d_{\boldsymbol{\theta}}}}\mathcal{J}(\boldsymbol {\theta};A).
$$
Then we have that  $\theta_{\star}$  satisfies Assumption 13 with
$$
\begin{array}{l} r _ {\boldsymbol {\theta}} \left(A _ {\star}\right) = \min  \left\{r _ {\operatorname {c o n v}} \left(A _ {\star}\right), r _ {\operatorname {c o s t}} \left(A _ {\star}\right), \right. \\ \left. \mathrm {p o l y} \left(\frac {1}{\lambda_ {\operatorname* {m i n}} (\nabla_ {\boldsymbol {\theta}} ^ {2} \mathcal {J} (\boldsymbol {\theta} ; A _ {\star}) | _ {\boldsymbol {\theta} = \boldsymbol {\theta} _ {\star} (A _ {\star})})}, \| A _ {\star} \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, L _ {\mathrm {c o s t}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, d _ {\boldsymbol {x}}\right) ^ {- 1} \right\} \\ \end{array}
$$
$$
L _ {\pi_ {\star}} = \mathrm {p o l y} \left(\frac {1}{\lambda_ {\mathrm {m i n}} (\nabla_ {\pmb {\theta}} ^ {2} \mathcal {I} (\pmb {\theta} ; A _ {\star}) | _ {\pmb {\theta} = \pmb {\theta} _ {\star} (A _ {\star})})}, \| A _ {\star} \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\pmb {\theta}}, L _ {\mathrm {c o s t}}, \sigma_ {\pmb {w}} ^ {- 1}, H, d _ {\pmb {x}}\right).
$$
Note that, if  $\mathcal{J}(\pmb{\theta}; A)$  is  $\mu$ -strongly convex in  $\pmb{\theta}$  for all  $A$  near  $A_{\star}$ , we can lower bound  $\lambda_{\min}(\nabla_{\pmb{\theta}}^{2}\mathcal{J}(\pmb{\theta}; A_{\star})|_{\pmb{\theta} = \pmb{\theta}_{\star}(A_{\star})}) \geq \mu$ .
Approximating the Controller Loss. In order to efficiently direct our exploration, it is convenient to derive a quadratic approximation to the controller loss. The following result shows that, under our assumptions, this is indeed possible.
Lemma D.2 (Formal Version of Proposition 1). Under Assumptions 1, 2, 4 to 5 and 13, for  $\widehat{A} \in \mathcal{B}_{\mathrm{F}}(A_{\star}; \min \{r_{\mathrm{cost}}(A_{\star}), r_{\theta}(A_{\star})\})$ , we have
$$
\mathcal {J} (\boldsymbol {\theta} _ {\star} (\widehat {A}); A _ {\star}) - \mathcal {J} (\boldsymbol {\theta} _ {\star} (A _ {\star}); A _ {\star}) \leq \operatorname {v e c} (\widehat {A} - A _ {\star}) ^ {\top} \mathcal {H} (A _ {\star}) \operatorname {v e c} (\widehat {A} - A _ {\star}) + M [ \widehat {A} - A _ {\star}, \widehat {A} - A _ {\star}, \widehat {A} - A _ {\star} ].
$$
for some tensor  $M$  such that
$$
\| M [ \widehat {A} - A _ {\star}, \widehat {A} - A _ {\star}, \widehat {A} - A _ {\star} ] \| _ {\mathrm {o p}} \leq \mathrm {p o l y} (L _ {\pi_ {\star}}, \| A _ {\star} \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, L _ {\mathrm {c o s t}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, d _ {\boldsymbol {x}}) \cdot \| \widehat {A} - A _ {\star} \| _ {\mathrm {o p}} ^ {3}.
$$
In practice we do not know  $\mathcal{H}(A_{\star})$  and must estimate it. The following result shows that the distance between  $\mathcal{H}(A_{\star})$  and  $\mathcal{H}(\widehat{A})$  can be bounded.
Lemma D.3. Under Assumptions 1, 2, 4, 5 and 13, and if  $\widehat{A} \in \mathcal{B}_{\mathrm{F}}(A_{\star}; \min \{r_{\mathrm{cost}}(A_{\star}), r_{\theta}(A_{\star})\})$ , we can bound
$$
\| \mathcal {H} (A _ {\star}) - \mathcal {H} (\widehat {A}) \| _ {\mathrm {o p}} \leq \operatorname {p o l y} \left(L _ {\pi_ {\star}}, \| A _ {\star} \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\theta}, L _ {\text {c o s t}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, d _ {\boldsymbol {x}}\right) \cdot \| \widehat {A} - A _ {\star} \| _ {\mathrm {o p}}.
$$
D.1 Proof of Smoothness of Nonlinear System
We let  $f_{\pmb{w}}(\cdot)$  denote the density of the noise (which, by assumption, is simply an isotropic Gaussian density). We let  $f_{A,\theta}(\cdot)$  denote the density over trajectories induced by playing controller  $\pmb{\theta}$  on system  $A$ . We will overload notation somewhat and let  $f_{A,\theta}(\pmb{x}_{h+1} \mid \pmb{\tau}_{1:h})$  denote the density over  $\pmb{x}_{h+1}$  induced by playing controller  $\pmb{\theta}$  given trajectory  $\pmb{\tau}_{1:h}$ . Note that  $f_{A,\theta}(\pmb{x}_{h+1} \mid \pmb{\tau}_{1:h}) = f_{\pmb{w}}(\pmb{x}_{h+1} - A\phi(\pmb{x}_h^\tau, \pi_h^\theta(\pmb{\tau}_{1:h})))$  and
$$
f _ {A, \boldsymbol {\theta}} (\boldsymbol {\tau}) = \prod_ {h = 1} ^ {H} f _ {A, \boldsymbol {\theta}} \left(\boldsymbol {x} _ {h + 1} \mid \boldsymbol {\tau} _ {1: h}\right).
$$
Throughout this section we let  $\boldsymbol{x}_h^\tau$  (resp.  $\boldsymbol{u}_h^\tau$ ) denote the state (resp. input) at step  $h$  of trajectory  $\boldsymbol{\tau}$ . Under our regularity assumptions (Assumptions 1, 2, 4 and 5) and since the noise is Gaussian, we can swap derivatives and integrals, which we make use of throughout the following proofs.
Proof of Lemma D.1. Let  $\mathrm{cost}(\tau)$  denote the cost of trajectory  $\pmb{\tau}$ . Then we have
$$
\mathcal {J} (\boldsymbol {\theta}; A) = \int \operatorname {c o s t} (\boldsymbol {\tau}) f _ {A, \boldsymbol {\theta}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau}.
$$
Let  $A_{t} \coloneqq A + t_{1}\Delta_{1}^{A} + t_{2}\Delta_{2}^{A} + t_{3}\Delta_{3}^{A}$  and  $\pmb{\theta}_{s} \coloneqq \pmb{\theta} + s_{1}\Delta_{1}^{\pmb{\theta}} + s_{2}\Delta_{2}^{\pmb{\theta}} + s_{3}\Delta_{3}^{\pmb{\theta}}$ , for some  $\Delta_i^A$  and  $\Delta_j^\theta$  which we assume satisfy  $\| \Delta_i^A\|_{\mathrm{op}},\| \Delta_j^\theta \|_{\mathrm{op}}\leq 1$ . Rather than differentiating  $\mathcal{J}(\pmb {\theta};A)$  with respect to  $\pmb{\theta}$  or  $A$ , we will differentiate  $\mathcal{J}(\pmb {\theta}_s;A_t)$  with respect to some  $x_{1},x_{2},x_{3},x_{4}\in \{t_{1},t_{2},t_{3},s_{1},s_{2},s_{3}\}$ . Note that, for example,
$$
\frac {\mathrm {d}}{\mathrm {d} t _ {1}} \mathcal {J} (\boldsymbol {\theta} _ {s}; A _ {t}) | _ {t = s = 0} = \nabla_ {A} \mathcal {J} (\boldsymbol {\theta}, A) [ \Delta_ {1} ^ {A} ],
$$
i.e. the directional gradient of  $\mathcal{J}(\pmb{\theta}, A)$  with respect to  $A$  in direction  $\Delta_1^A$ , and that this similarly holds for gradients with respect to other  $t_i, s_j$ , or higher-order derivatives. Thus, if we can show that  $\mathcal{J}(\pmb{\theta}_s; A_t)$  is differentiable with respect to any  $x_1, x_2, x_3, x_4 \in \{t_1, t_2, t_3, s_1, s_2, s_3\}$ , and this holds for any choice of  $\Delta_i^A, \Delta_j^A$ , then we have that  $\mathcal{J}(\pmb{\theta}, A)$  is four-times differentiable with respect to  $\pmb{\theta}$  and  $A$ . Furthermore, we can bound the operator norm of  $\nabla_A \mathcal{J}(\pmb{\theta}, A)$ , by bounding the value of  $\frac{\mathrm{d}}{\mathrm{d}t_1} \mathcal{J}(\pmb{\theta}_s; A_t)|_{t=s=0}$  for all  $\Delta_1^A$  satisfying  $\| \Delta_1^A \|_{\mathrm{op}} \leq 1$  (and we can similarly bound the operator norm of the higher order derivatives of  $\mathcal{J}(\pmb{\theta}, A)$ ).
$\mathcal{J}(\theta; A)$  is Differentiable. Let  $x_1, x_2, x_3, x_4 \in \{t_1, t_2, t_3, s_1, s_2, s_3\}$ . We have
$$
\frac {\mathrm {d}}{\mathrm {d} x _ {1}} \mathcal {J} (\boldsymbol {\theta} _ {s}; A _ {t}) = \frac {\mathrm {d}}{\mathrm {d} x _ {1}} \int \operatorname {c o s t} (\boldsymbol {\tau}) f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau}
$$
$$
\begin{array}{l} = \int \frac {f _ {A _ {t} , \theta_ {s}} (\tau)}{f _ {A _ {t} , \theta_ {s}} (\tau)} \frac {\mathrm {d}}{\mathrm {d} x _ {1}} f _ {A _ {t}, \theta_ {s}} (\tau) \operatorname {c o s t} (\tau) \mathrm {d} \tau \\ = \int \frac {\mathrm {d}}{\mathrm {d} x _ {1}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \cdot \operatorname {c o s t} (\boldsymbol {\tau}) f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau}. \\ \end{array}
$$
Differentiating this gives
$$
\begin{array}{l} \frac {\mathrm {d}}{\mathrm {d} x _ {2}} \frac {\mathrm {d}}{\mathrm {d} x _ {1}} \mathcal {J} (\boldsymbol {\theta} _ {s}; A _ {t}) = \frac {\mathrm {d}}{\mathrm {d} x _ {2}} \int \frac {\mathrm {d}}{\mathrm {d} x _ {1}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \cdot \operatorname {c o s t} (\boldsymbol {\tau}) f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau} \\ = \int \left(\frac {\mathrm {d}}{\mathrm {d} x _ {1}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau})\right) \left(\frac {\mathrm {d}}{\mathrm {d} x _ {2}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau})\right) \cdot \operatorname {c o s t} (\boldsymbol {\tau}) f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau} \\ + \int \frac {\mathrm {d}}{\mathrm {d} x _ {2}} \frac {\mathrm {d}}{\mathrm {d} x _ {1}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \cdot \operatorname {c o s t} (\boldsymbol {\tau}) f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau}, \\ \end{array}
$$
$$
\begin{array}{l} \frac{\mathrm{d}}{\mathrm{d}x_{3}}\frac{\mathrm{d}}{\mathrm{d}x_{2}}\frac{\mathrm{d}}{\mathrm{d}x_{1}}\mathcal{J}(\boldsymbol{\theta}_{s};A_{t}) = \int \left(\frac{\mathrm{d}}{\mathrm{d}x_{1}}\log f_{A_{t},\boldsymbol{\theta}_{s}}(\boldsymbol {\tau})\right)\left(\frac{\mathrm{d}}{\mathrm{d}x_{2}}\log f_{A_{t},\boldsymbol{\theta}_{s}}(\boldsymbol {\tau})\right)\left(\frac{\mathrm{d}}{\mathrm{d}x_{3}}\log f_{A_{t},\boldsymbol{\theta}_{s}}(\boldsymbol {\tau})\right) \cdot \cos (\boldsymbol {\tau})f_{A_{t},\boldsymbol{\theta}_{s}}(\boldsymbol {\tau})\mathrm{d}\boldsymbol{\tau} \\ + \int \left(\frac {\mathrm {d}}{\mathrm {d} x _ {3}} \frac {\mathrm {d}}{\mathrm {d} x _ {1}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau})\right) \left(\frac {\mathrm {d}}{\mathrm {d} x _ {2}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau})\right) \cdot \operatorname {c o s t} (\boldsymbol {\tau}) f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau} \\ + \int \left(\frac {\mathrm {d}}{\mathrm {d} x _ {1}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau})\right) \left(\frac {\mathrm {d}}{\mathrm {d} x _ {3}} \frac {\mathrm {d}}{\mathrm {d} x _ {2}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau})\right) \cdot \operatorname {c o s t} (\boldsymbol {\tau}) f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau} \\ + \int \frac {\mathrm {d}}{\mathrm {d} x _ {3}} \frac {\mathrm {d}}{\mathrm {d} x _ {2}} \frac {\mathrm {d}}{\mathrm {d} x _ {1}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \cdot \operatorname {c o s t} (\boldsymbol {\tau}) f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau} \\ + \int \left(\frac {d}{d x _ {2}} \frac {d}{d x _ {1}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau})\right) \left(\frac {d}{d x _ {3}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau})\right) \cdot \operatorname {c o s t} (\boldsymbol {\tau}) f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) d \boldsymbol {\tau}. \\ \end{array}
$$
The fourth derivative of  $\mathcal{J}(\pmb {\theta};A)$  can be similarly calculated by differentiating  $\frac{\mathrm{d}}{\mathrm{d}x_3}\frac{\mathrm{d}}{\mathrm{d}x_2}\frac{\mathrm{d}}{\mathrm{d}x_1}\mathcal{J}(\pmb {\theta}_s;A_t)$ ; we omit it for brevity. We have
$$
\begin{array}{l} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) = \log \prod_ {h = 1} ^ {H} f _ {A _ {t}, \boldsymbol {\theta} _ {s}} \left(\boldsymbol {x} _ {h + 1} ^ {\boldsymbol {\tau}} \mid \boldsymbol {\tau} _ {1: h}\right) \\ = \sum_ {h = 1} ^ {H} \log f _ {\boldsymbol {w}} \left(\boldsymbol {x} _ {h + 1} ^ {\boldsymbol {\tau}} - A _ {\boldsymbol {t}} \phi \left(\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \pi^ {\boldsymbol {\theta} _ {s}} \left(\boldsymbol {\tau} _ {1: h}\right)\right)\right) \\ = \sum_ {h = 1} ^ {H} - \frac {1}{2 \sigma_ {\boldsymbol {w}} ^ {2}} \| \boldsymbol {x} _ {h + 1} ^ {\boldsymbol {\tau}} - A _ {\boldsymbol {t}} \phi \left(\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \pi^ {\boldsymbol {\theta} _ {s}} \left(\boldsymbol {\tau} _ {1: h}\right)\right) \| _ {2} ^ {2} + C \\ \end{array}
$$
for some  $C$  which does not depend on  $\pmb{t}$  or  $\pmb{s}$ . Given that  $\phi (\pmb {x},\pmb {u})$  is four-times differentiable in  $\pmb{u}$  and  $\pi_h^{\theta_s}(\tau_{1:h})$  is four-times differentiable in  $\pmb{x}$  (which hold by Assumption 4 and Assumption 5), it is clear that  $\log f_{A_t,\theta_s}(\tau)$  is four-times differentiable in  $t_i$  or  $s_i$ , regardless of the choice of  $\Delta_i^A$  or  $\Delta_i^\theta$ . This proves the first result.
Norm Bounds on Gradient. Note that
$$
\frac {\mathrm {d}}{\mathrm {d} t _ {i}} \log f _ {A _ {t}, \boldsymbol {\theta} _ {s}} (\boldsymbol {\tau}) | _ {\boldsymbol {t} = \boldsymbol {s} = 0} = \sum_ {h = 1} ^ {H} \frac {1}{\sigma_ {\boldsymbol {w}} ^ {2}} \left(\boldsymbol {x} _ {h + 1} ^ {\boldsymbol {\tau}} - A \phi \left(\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \pi_ {h} ^ {\boldsymbol {\theta}} \left(\boldsymbol {\tau} _ {1: h}\right)\right)\right) ^ {\top} \cdot \Delta_ {i} ^ {A} \phi \left(\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \pi_ {h} ^ {\boldsymbol {\theta}} \left(\boldsymbol {\tau} _ {1: h}\right)\right),
$$
$$
\frac{\mathrm{d}}{\mathrm{d}s_{i}}\log f_{A_{t},\boldsymbol{\theta}_{s}}(\boldsymbol {\tau})|_{\boldsymbol {t} = \boldsymbol {s} = 0} = \sum_{h = 1}^{H}\frac{1}{\sigma_{\boldsymbol{w}}^{2}} (x_{h + 1}^{\boldsymbol{\tau}} - A\phi (x_{h}^{\boldsymbol{\tau}},\pi_{h}^{\boldsymbol{\theta}}(\boldsymbol{\tau}_{1:h})))^{\top}\cdot A\nabla_{\boldsymbol{u}}\phi (x_{h}^{\boldsymbol{\tau}},\pi_{h}^{\boldsymbol{\theta}}(\boldsymbol{\tau}_{1:h}))\cdot \nabla_{\boldsymbol{\theta}}\pi_{h}^{\boldsymbol{\theta}}(\boldsymbol{\tau}_{1:h})\cdot \Delta_{i}^{\boldsymbol{\theta}}.
$$
Furthermore, differentiating these expressions further with respect to  $t_j$  or  $s_j$  will simply yield higher-order derivates of  $\phi(\boldsymbol{x}, \boldsymbol{u})$  and  $\pi_h^\theta(\tau_{1:h})$ . Using the norm bounds on the gradient of  $\phi(\boldsymbol{x}, \boldsymbol{u})$  and  $\pi_h^\theta(\tau_{1:h})$  given in Assumption 4 and Assumption 5, and the norm bound of  $\phi(\boldsymbol{x}, \boldsymbol{u})$  given in Assumption 1, we can then bound
$$
\| \nabla_ {A} ^ {(i)} \nabla_ {\boldsymbol {\theta}} ^ {(j)} \log f _ {A, \boldsymbol {\theta}} (\boldsymbol {\tau}) \| _ {\mathrm {o p}} \leq \operatorname {p o l y} (\| A \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, \sigma_ {\boldsymbol {w}} ^ {- 1}) \cdot \sum_ {h = 1} ^ {H} (1 + \| \boldsymbol {x} _ {h + 1} ^ {\boldsymbol {\tau}} - A \phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \pi_ {h} ^ {\boldsymbol {\theta}} (\boldsymbol {\tau} _ {1: h})) \| _ {2})
$$
for  $i,j\in \{0,1,2,3,4\}$  satisfying  $1\leq i + j\leq 4$  (where we have used the fact noted above that, to bound the operator norm of  $\nabla_A^{(i)}\nabla_\theta^{(j)}\log f_{A,\pmb{\theta}}(\pmb{\tau})$ , it suffices to bound the directional gradient in every direction). It follows that we can bound
$$
\begin{array}{l} \left\| \nabla_ {A} ^ {(i)} \nabla_ {\boldsymbol {\theta}} ^ {(j)} \mathcal {J} (\boldsymbol {\theta}; A) \right\| _ {\mathrm {o p}} \\ \leq \operatorname {p o l y} (\| A \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, \sigma_ {\boldsymbol {w}} ^ {- 1}) \cdot \int \left(\sum_ {h = 1} ^ {H} (1 + \| \boldsymbol {x} _ {h + 1} ^ {\boldsymbol {\tau}} - A \phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}}, \pi_ {h} ^ {\boldsymbol {\theta}} (\boldsymbol {\tau} _ {1: h})) \| _ {2})\right) ^ {4} \cdot \cos (\boldsymbol {\tau}) f _ {A, \boldsymbol {\theta}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau} \\ \stackrel {(a)} {\leq} \operatorname {p o l y} (\| A \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, \sigma_ {\boldsymbol {w}} ^ {- 1}) \cdot \sqrt {\int \operatorname {c o s t} (\boldsymbol {\tau}) ^ {2} f _ {A , \boldsymbol {\theta}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau}} \\ \cdot \sqrt {\int \left(\sum_ {h = 1} ^ {H} \left(1 + \| \boldsymbol {x} _ {h + 1} ^ {\boldsymbol {\tau}} - A \phi (\boldsymbol {x} _ {h} ^ {\boldsymbol {\tau}} , \pi_ {h} ^ {\boldsymbol {\theta}} (\boldsymbol {\tau} _ {1 : h})) \| _ {2}\right)\right) ^ {8} f _ {A , \boldsymbol {\theta}} (\boldsymbol {\tau}) \mathrm {d} \boldsymbol {\tau}} \\ \stackrel {(b)} {\leq} \operatorname {p o l y} (\| A \| _ {\mathrm {o p}}, B _ {\phi}, L _ {\phi}, L _ {\theta}, L _ {\mathrm {c o s t}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, d _ {\boldsymbol {x}}) \\ \end{array}
$$
where  $(a)$  follows from Cauchy-Schwarz, and  $(b)$  follows from Lemma A.1 and Assumption 2, since we have assumed  $A\in \mathcal{B}_{\mathrm{F}}(A_{\star};r_{\mathrm{cost}}(A_{\star}))$
Proof of Proposition 5. Existence and Differentiability of  $\theta_{\star}$ . By Lemma D.1 we have that  $\mathcal{J}(\pmb{\theta}; A)$  is four-times differentiable in its arguments. By the Implicit Function Theorem, since  $\lambda_{\min}(\nabla_{\pmb{\theta}}^{2}\mathcal{J}(\pmb{\theta}; A_{\star})|_{\pmb{\theta} = \pmb{\theta}_{\star}(A_{\star})}) > 0$  by assumption, we have that there exists some  $r_{\pmb{\theta}}^{\prime}(A_{\star}) > 0$  and unique function  $\pmb{\theta}_{\star}(A)$  defined on  $\mathcal{B}_{\mathrm{F}}(A_{\star}; r_{\pmb{\theta}}^{\prime}(A_{\star}))$  such that  $\nabla_{\pmb{\theta}}\mathcal{J}(\pmb{\theta}; A)|_{\pmb{\theta} = \pmb{\theta}_{\star}(A)} = 0$ , and  $\pmb{\theta}_{\star}(A)$  is three-times differentiable (note that, while the Implicit Function Theorem is typically stated to give that the resulting function is only one-time differentiable, it can be extended to  $k$ -times differentiable, assuming the implicit equation is  $k$ -times differentiable [60]).
By Lemma D.1 and the continuity of eigenvalues, it follows that for  $A$  close enough to  $A_{\star}$ , we have  $\lambda_{\min}(\nabla_{\pmb{\theta}}^{2}\mathcal{J}(\pmb {\theta};A)|_{\pmb {\theta} = \pmb{\theta}_{\star}(A)})\geq \frac{1}{2}\lambda_{\min}(\nabla_{\pmb{\theta}}^{2}\mathcal{J}(\pmb {\theta};A_{\star})|_{\pmb {\theta} = \pmb{\theta}_{\star}(A_{\star})) > 0$ . We can therefore apply the Implicit Function Theorem as above to any  $A$  satisfying this, to get that there exists some unique  $\widetilde{\pmb{\theta}}_{\star}(A^{\prime})$  defined for all  $A^{\prime}$  near  $A$  such that  $\nabla_{\pmb{\theta}}\mathcal{J}(\pmb {\theta};A^{\prime})|_{\pmb {\theta} = \widetilde{\pmb{\theta}}_{\star}(A^{\prime})} = 0$  and  $\widetilde{\pmb{\theta}}_{\star}(A^{\prime})$  is differentiable.
By the uniqueness of  $\theta_{\star}(A)$  on  $\mathcal{B}_{\mathrm{F}}(A_{\star};r_{\theta}^{\prime}(A_{\star}))$  , it follows that any  $\widetilde{\theta}_{\star}(A^{\prime})$  defined in this way must be identical to  $\theta_{\star}(A)$  on  $\mathcal{B}_{\mathrm{F}}(A_{\star};r_{\theta}^{\prime}(A_{\star}))$  (assuming the regions on which they are defined overlaps). We can therefore define  $\theta_{\star}(A)$  to simply be the extension of  $\theta_{\star}(A)$  to all such  $\widetilde{\theta}_{\star}(A)$  , defined for all  $A$  near  $A_{\star}$  such that  $\lambda_{\min}(\nabla_{\pmb{\theta}}^{2}\mathcal{J}(\pmb {\theta};A)|_{\pmb {\theta} = \pmb{\theta}_{\star}(A)})\geq \frac{1}{2}\lambda_{\min}(\nabla_{\pmb{\theta}}^{2}\mathcal{J}(\pmb {\theta};A_{\star})|_{\pmb {\theta} = \pmb{\theta}_{\star}(A_{\star}))$  , and will have that  $\theta_{\star}(A)$  is three-times differentiable and satisfies  $\nabla_{\pmb{\theta}}\mathcal{J}(\pmb {\theta};A)|_{\pmb {\theta} = \pmb{\theta}_{\star}(A)} = 0$  for all such  $A$