SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To benchmark the performance of these approaches, we consider an affine system with dynamics corresponding to that of a simplified 3-D drone (i.e., 3-D double integrator with a gravity term), and a nonlinear system with dynamics corresponding to that of a 2-D car. For both systems, we choose  $H = 50$ , and plot the value of  $\mathcal{J}(\widehat{\pi}_t; A_\star) - \mathcal{J}(\pi_\star(A_\star); A_\star)$  for  $\widehat{\pi}_t$  the certainty-equivalence controller computed on the estimate of the system obtained at time  $t$ . For the drone, we let  $\Pi^\star$  be the class of linear-affine feedback controllers, and for the car,  $\Pi^\star$  is a set of nonlinear controllers with dimension 4. While the optimal controller for the drone can be computed in closed-form, for the car we rely on a sampling-based routine to find an approximately optimal controller. The model-task hessian  $\mathcal{H}(\widehat{A}^\ell)$  is computed via automatic differentiation. For the exploration policies of Task-Driven Exploration and Uniform Exploration,  $\Pi_{\mathrm{exp}}$ , we rely on MPC-style sampling based methods. For all approaches, we require that  $\mathbb{E}_{A,\pi_{\mathrm{exp}}}[\sum_{h=1}^{H} \|u_h\|_2^2] \leq \gamma^2$  for some  $\gamma^2 > 0$  and all  $\pi_{\mathrm{exp}} \in \Pi_{\mathrm{exp}}$ . On all examples, we implement DYNAMICOED with  $\mathbb{A}_{\mathcal{R}}$  a posteriori sampling-inspired version of the LC<sup>3</sup> of [17]. Figures 1 and 3 show performance averaged over 100 trials, and Figure 2 over 200 trials. Additional experimental details can be found in Appendix G.
As illustrated in Figures 1 to 3, our approach yields a non-trivial gain over existing approaches on all systems. In particular, in Figures 2 and 3 it improves on the sample complexity of existing approaches by roughly a factor of 2—for example, in the drone system, reaching excess controller cost of 10 after less than 20 episodes, as compared to over 40 episodes for existing approaches.
Our implementation is very modular, and any piece (for example, the parameterization of  $\Pi_{\mathrm{exp}}$  and  $\Pi^{\star}$ , the policy optimizer, or the exploration routine) can be easily replaced with other procedures. Our results highlight that, even when using, for example, a possibly suboptimal policy optimizer, exploring so as to minimize uncertainty in the model-task hessian yields a non-trivial gain. We expect that this would hold true regardless of the policy optimizer used—the model-task hessian will adapt to the structure of the policy optimizer. Integration of our approach with deep model-based RL approaches is an interesting direction for future work, but we believe the approach will scale to these settings as well.
In this work, we have characterized the instance-optimal rate of learning controllers in nonlinear dynamical systems. To the best of our knowledge, this is the first work to obtain an optimal sample complexity for learning in nonlinear dynamical systems. Furthermore, our experimental results demonstrate the effectiveness of our proposed algorithm in realistic nonlinear systems. This work opens the door for several interesting directions for future work. First, it is not clear that the assumption on uniform feature excitation, Assumption 3, is necessary. Can this be removed with a more refined analysis (or shown to be necessary)? Second, while in many settings  $\phi$  is known or can be effectively represented by random features, in some settings it is helpful to learn it [12]. Can we obtain end-to-end guarantees on learning both  $\phi$  and  $A_{\star}$ ? Finally, it is of much interest to extend our experimental results to larger-scale systems for real-world deployment.
AW would like to thank Kevin Tully for helpful discussions. The work of AW is supported by NSF HDR 62-0221. The work of KJ is supported in part by NSF TRIPODS 2023166 and CIF 2007036.
[1] Horia Mania, Michael I Jordan, and Benjamin Recht. Active learning for nonlinear system identification with guarantees. J. Mach. Learn. Res., 23:32-1, 2022.
[2] Andrew J. Wagenmaker, Max Simchowitz, and Kevin Jamieson. Task-optimal exploration in linear dynamical systems. In International Conference on Machine Learning, pages 10641-10652. PMLR, 2021.
[3] Jean-Jacques E Slotine, Weiping Li, et al. Applied nonlinear control, volume 199. Prentice hall Englewood Cliffs, NJ, 1991.
[4] Karl J Åström and Björn Wittenmark. Adaptive control. Courier Corporation, 2013.
[5] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[6] Guanya Shi, Xichen Shi, Michael O'Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural lander: Stable drone landing control using learned dynamics. In 2019 International Conference on Robotics and Automation (ICRA), pages 9784-9790. IEEE, 2019.
[7] Lennart Ljung. System identification. Springer, 1998.
[8] Duy Nguyen-Tuong and Jan Peters. Model learning for robot control: a survey. Cognitive processing, 12:319-340, 2011.
[9] Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411-444, 2022.
[10] Max Simchowitz, Horia Mania, Stephen Tu, Michael I Jordan, and Benjamin Recht. Learning without mixing: Towards a sharp analysis of linear system identification. arXiv preprint arXiv:1802.08334, 2018.
[11] Guanya Shi, Kamyar Azizzadenesheli, Michael O'Connell, Soon-Jo Chung, and Yisong Yue. Meta-adaptive nonlinear control: Theory and algorithms. Advances in Neural Information Processing Systems, 34:10013-10025, 2021.
[12] Michael O'Connell, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics, 7(66):eabm6597, 2022.
[13] Nicholas M Boffi, Stephen Tu, and Jean-Jacques E Slotine. Regret bounds for adaptive nonlinear control. In Learning for Dynamics and Control, pages 471-483. PMLR, 2021.
[14] Yuda Song and Wen Sun. Pc-mlxp: Model-based reinforcement learning with policy cover guided exploration. In International Conference on Machine Learning, pages 9801–9811. PMLR, 2021.
[15] Spencer M Richards, Navid Azizan, Jean-Jacques Slotine, and Marco Pavone. Adaptive-control-oriented meta-learning for nonlinear systems. arXiv preprint arXiv:2103.04490, 2021.
[16] Ali Rahimi and Benjamin Recht. Uniform approximation of functions with random bases. In 2008 46th annual allerton conference on communication, control, and computing, pages 555-561. IEEE, 2008.
[17] Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Information theoretic regret bounds for online nonlinear control. Advances in Neural Information Processing Systems, 33:15312-15325, 2020.
[55] Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning, pages 263-272. PMLR, 2017.
[56] Andrew Wagenmaker and Kevin Jamieson. Instance-dependent near-optimal policy identification in linear mdps via online experiment design. arXiv preprint arXiv:2207.02575, 2022.
[57] Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pages 2681-2691. PMLR, 2019.
[58] Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. Advances in Neural Information Processing Systems, 34:25746-25759, 2021.
[59] Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning. arXiv preprint arXiv:2211.04974, 2022.
[60] Jean Dieudonné. Foundations of modern analysis. Read Books Ltd, 2011.
[61] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
A Technical Tools
Additional Notation. We let  $S^{d-1}$  refer to the unit ball in  $d$  dimensions and  $\mathbb{S}_{+}^{d}$  (resp.  $\mathbb{S}_{++}^{d}$ ) the set of positive semi-definite matrices (resp. positive definite matrices) in  $\mathbb{R}^{d \times d}$ . Throughout,  $\mathcal{O}(\cdot)$  denotes standard big-O notation,  $\widetilde{\mathcal{O}}(\cdot)$  hides additional logarithmic factors.  $\mathrm{poly}(\cdot)$  denotes some term that is polynomial in its arguments, with exponents absolute constants.  $\triangle_{\mathcal{X}}$  denotes the set of distributions over set  $\mathcal{X}$ .
Lemma A.1. Let  $\pmb{w}_i \sim \mathcal{N}(0, I_{d_\infty})$  for all  $i \in \{1, 2, \dots, n\}$ . Then,
$$
\mathbb {E} \left[ \left(\sum_ {i = 1} ^ {n} \| \boldsymbol {w} _ {i} \| _ {2}\right) ^ {c} \right] \leq n ^ {2} \cdot \operatorname {p o l y} (d _ {\boldsymbol {x}}).
$$
for  $c$  an absolute constant.
Proof. We first bound
$$
\left(\sum_ {i = 1} ^ {n} \| \boldsymbol {w} _ {i} \| _ {2}\right) ^ {c} \leq n \cdot \max  _ {i} \| \boldsymbol {w} _ {i} \| _ {2} ^ {c} \leq n \cdot \sum_ {i} \| \boldsymbol {w} _ {i} \| _ {2} ^ {c}.
$$
The result then follows since we can bound the  $\mathbb{E}[\| \pmb{w}_i\|_2^c] \leq \mathrm{poly}(d)$  for  $\pmb{w}_i \sim \mathcal{N}(0, I)$  and  $c$  an absolute constant.
Lemma A.2 (Lemma I.4 of [2]). Assume  $A, B \in \mathbb{S}_{++}^{d}$ ,  $\|A - B\|_{\mathrm{op}} \leq \epsilon$ , and  $\epsilon < \lambda_{\min}(B)$ . Then
$$
\left\| A ^ {- 1} - B ^ {- 1} \right\| _ {\mathrm {o p}} \leq \frac {\epsilon}{\lambda_ {\operatorname* {m i n}} (B) (\lambda_ {\operatorname* {m i n}} (B) - \epsilon)}.
$$
A.1 Martingale Regression in General Norms
For the following two results, we consider the martingale regression setting of [2] (referred to as the MDM setting). In particular, we consider observations of the form
$$
y _ {t} = \left\langle \boldsymbol {\mu} ^ {\star}, \boldsymbol {z} _ {t} \right\rangle + w _ {t}, \tag {A.1}
$$
for  $y_{\mathfrak{t}} \in \mathbb{R}$ , unknown parameter  $\pmb{\mu}^{\star} \in \mathbb{R}^{d_{\mu}}$ ,  $w_{\mathfrak{t}} \mid \mathcal{F}_{\mathfrak{t}-1} \sim \mathcal{N}(0, \sigma_{\pmb{w}}^{2})$ , and  $\pmb{z}_{\mathfrak{t}} \in \mathcal{F}_{\mathfrak{t}-1}$ -measurable, for a filtration  $(\mathcal{F}_{\mathfrak{t}})_{\mathfrak{t} \geq 1}$ . This setting therefore encompasses general stochastic processes where the observations are linear—the evolution of  $\pmb{z}_{\mathfrak{t}}$  could be arbitrary.
We consider the setting where we interact with (A.1) for T steps, collecting observations  $\{(z_{\mathrm{t}},y_{\mathrm{t}})\}_{\mathrm{t} = 1}^{\mathrm{T}}$  and then form the least-squares estimate of  $\pmb{\mu}^{\star}$ :
$$
\widehat {\boldsymbol {\mu}} = \left(\sum_ {t = 1} ^ {T} z _ {t} z _ {t} ^ {\top}\right) ^ {- 1} \sum_ {t = 1} ^ {T} z _ {t} y _ {t}.
$$
We also denote  $\Sigma_{\mathrm{T}} := \sum_{t=1}^{\mathrm{T}} z_{t} z_{t}^{\top}$ . The following results characterize the estimation error of  $\widehat{\mu}$  in the  $M$ -norm and 2-norm.
Proposition 3 (Theorem 7.2 of [2]). Fix any matrices  $\Gamma \in \mathbb{S}_{++}^{d_{\mu}}$ ,  $M \in \mathbb{S}_{+}^{d_{\mu}}$ , with  $M \neq 0$ . Given a parameter  $\beta \in (0,1/4)$ , define the event
$$
\mathcal {E} := \left\{\left\| \boldsymbol {\Sigma} _ {\mathbf {T}} - \boldsymbol {\Gamma} \right\| _ {\mathrm {o p}} \leq \beta \lambda_ {\min } (\boldsymbol {\Gamma}) \right\}.
$$
Then, if  $\mathcal{E}$  holds, the following holds with probability at least  $1 - \delta$ :
$$
\| \widehat {\boldsymbol {\mu}} - \boldsymbol {\mu} ^ {\star} \| _ {M} ^ {2} \leq 5 (1 + \zeta) \cdot \sigma_ {\boldsymbol {w}} ^ {2} \log \frac {6 d _ {\boldsymbol {\mu}}}{\delta} \cdot \operatorname {t r} (M \boldsymbol {\Gamma} ^ {- 1})
$$
where  $\zeta = 26\beta^{2}\lambda_{\max}(\mathbf{T})\mathrm{tr}(\mathbf{T}^{-1})$
Proposition 4 (Lemma E.1 of [2]). On the event
$$
\mathcal {E} _ {\mathrm {o p}} := \left\{\lambda_ {\min } \left(\boldsymbol {\Sigma} _ {\mathrm {T}}\right) \geq \underline {{\Delta}} \mathrm {T}, \boldsymbol {\Sigma} _ {\mathrm {T}} \preceq \mathrm {T} \bar {\boldsymbol {\Gamma}} _ {\mathrm {T}} \right\},
$$
then we have that with probability at least  $1 - \delta$ :
$$
\| \widehat {\boldsymbol {\mu}} - \boldsymbol {\mu} ^ {\star} \| _ {2} \leq C \cdot \sigma_ {\boldsymbol {w}} \sqrt {\frac {\log 1 / \delta + d _ {\boldsymbol {\mu}} + \log \det  (\bar {\Gamma} _ {\mathrm {T}} / \underline {{\lambda}} + I)}{\underline {{\lambda}} \mathrm {T}}}.
$$
A.1.1 Connection Between (1.1) and (A.1)
We will apply the results Proposition 3 and Proposition 4 in the setting of (1.1) in order to obtain estimation bounds on  $A_{\star}$ . As the setting of (1.1) has vector observations, we briefly describe here how it can be mapped into the setting described above.
Recall that (1.1) evolves as
$$
\boldsymbol {x} _ {h + 1} = A _ {\star} \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) + \boldsymbol {w} _ {h}, \quad h = 1, \dots , H,
$$
for  $\pmb{x}_h \in \mathbb{R}^{d_x}$ ,  $\phi(\pmb{x}, \pmb{u}) \in \mathbb{R}^{d_\phi}$ , and  $A_\star \in \mathbb{R}^{d_x \times d_\phi}$ . We assume that  $\pmb{x}_1$  is some fixed starting state. Assume that we have run for  $T$  episodes, and collected observations  $\{(x_1^t, u_1^t, x_2^t, \ldots, x_h^t, u_h^t, x_{h+1}^t)\}_{t=1}^T$ . Now let  $\pmb{\mu}^\star \coloneqq \mathrm{vec}(A_\star)$ . Furthermore, for any  $t, h$ , and  $i \in [d_x]$ , let  $\mathfrak{t} = (t, h, i)$  and  $\pmb{z}_{\mathfrak{t}} = [\pmb{0}_{d_\phi(i-1)}, \phi(\pmb{x}_h^t, \pmb{u}_h^t), \pmb{0}_{d_\phi(d_x-i)}] \in \mathbb{R}^{d_x d_\phi}$  where  $\pmb{0}_d$  denotes the zero vector of length  $d$ . Then we see that
$$
[ \boldsymbol {x} _ {h + 1} ^ {t} ] _ {i} = \langle \boldsymbol {\mu} ^ {\star}, \boldsymbol {z} _ {t} \rangle + [ \boldsymbol {w} _ {h} ^ {t} ] _ {i}.
$$
Setting  $y_{\mathfrak{t}} = [\pmb{x}_{h + 1}^{t}]_{i}$  and  $w_{\mathfrak{t}} = [\pmb{w}_{h}^{t}]_{i}$ , it is clear that this follows the observation model of (A.1) with  $d_{\mu} = d_{\phi}d_{x}$  and  $\mathrm{T} = d_{\pmb{x}}TH$ . It is also straightforward to see that the measurability assumptions of the setting of (A.1) are satisfied by this.
B Proof of Main Result
Algorithm 3 Optimal Exploration in Nonlinear Systems (Full Version of Algorithm 1)
1: inputs: number of episodes to run  $T$ , cost function  $(\mathrm{cost}_h)^H_{h=1}$ , confidence  $\delta$ , control policies  $\Pi^{\star}$ , exploration policies  $\Pi_{\mathrm{exp}}$
2:  $\widehat{A}^1\gets$  anything,  $\ell_T\gets \lceil \log_2T / 8\rceil$
3: for  $\ell = 1,2,3,\ldots ,\ell_{T}$  do
4:  $T_{\ell}\gets 2^{\ell},\delta_{\ell}\gets \delta /12\ell^{2}$
5: Compute estimate of cost matrix:  $\mathcal{H}_{\ell}\gets \mathcal{H}(\widehat{A}^{\ell})$
6:  $\Pi_{\ell}\gets \mathsf{LEARNEXP}\Pi (\mathcal{H}_{\ell},T_{\ell},\delta_{\ell},\mathbb{A}_{\mathcal{R}},\Pi_{\mathrm{exp}})$  (Algorithm 7), with  $\mathbb{A}_{\mathcal{R}}$  the LC3 algorithm [17]
7: Rerun each policy in  $\Pi_{\ell}$ $N_{\ell} = \lceil T_{\ell} / |\Pi_{\ell}| \rceil$  times, denote collected data  $\mathfrak{D}_{\ell}$
8: Estimate system parameters
9:
$$
\widehat {A} ^ {\ell + 1} = \arg \min  _ {A} \sum_ {h = 1} ^ {H} \sum_ {\left(\boldsymbol {x} _ {h + 1}, \boldsymbol {u} _ {h}, \boldsymbol {x} _ {h}\right) \in \mathfrak {D} _ {\ell}} \| \boldsymbol {x} _ {h + 1} - A \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) \| _ {2} ^ {2}
$$
10: return  $\widehat{\pi}_T\gets \pi_\star (\widehat{A}^{\ell_T + 1})$
Theorem 4 (Full Version of Theorem 1). Assume Assumptions 1 to 5 and 13 hold. Then if
$$
T \geq \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, H, B _ {A}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, L _ {\mathrm {c o s t}}, L _ {\pi_ {\star}}, \sigma_ {\boldsymbol {w}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, \frac {1}{\lambda_ {\min} ^ {\star}}, \log \frac {T}{\delta}\right) \cdot \max  \left\{1, \frac {1}{r _ {\mathrm {c o s t}} (A _ {\star}) ^ {2}}, \frac {1}{r _ {\boldsymbol {\theta}} (A _ {\star}) ^ {2}} \right\}, \tag {B.1}
$$
with probability at least  $1 - \delta$ , Algorithm 3 plays exploration policies  $\pi_{\mathrm{exp}} \in \Pi_{\mathrm{exp}}$  at every episode, runs for at most  $T$  episodes, and the controller  $\widehat{\pi}_T$  returned Algorithm 3 satisfies, with probability at least  $1 - \delta$ :
$$
\mathcal {J} (\widehat {\pi} _ {T}; A _ {\star}) - \mathcal {J} (\pi_ {\star} (A _ {\star}); A _ {\star}) \leq \frac {\sigma_ {\boldsymbol {w}} ^ {2}}{T} \cdot \min  _ {\boldsymbol {\Lambda} \in \Omega} \operatorname {t r} \left(\mathcal {H} (A _ {\star}) \breve {\boldsymbol {\Lambda}} ^ {- 1}\right) \cdot C \log \frac {6 d _ {\boldsymbol {x}} d _ {\phi}}{\delta} + \frac {C _ {\mathrm {l o t}}}{T ^ {3 / 2}}
$$
for  $C$  a universal constant and
$$
C _ {\text {p o l y}} := \text {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, H, B _ {A}, B _ {\phi}, L _ {\phi}, L _ {\boldsymbol {\theta}}, L _ {\text {c o s t}}, L _ {\pi_ {\star}}, \sigma_ {\boldsymbol {w}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, \frac {1}{\lambda_ {\min} ^ {\star}}, \log \frac {T}{\delta}\right).
$$
Proof. Let  $\mathcal{E}_{\ell}$  denote that the good event of Lemma B.3 holds at round  $\ell$  which, by Lemma B.3, occurs with probability at least  $1 - 6\delta_{\ell}$ . By our setting of  $\delta_{\ell} = \delta /12\ell^{2}$ , we have that the total failure
probability of  $\mathcal{E}_{\ell}$  for all  $\ell$  is bounded as
$$
\sum_ {\ell = 1} ^ {\infty} 6 \cdot \frac {\delta}{1 2 \ell^ {2}} \leq \delta .
$$
Henceforth we assume that  $\mathcal{E} \coloneqq \cap_{\ell} \mathcal{E}_{\ell}$  holds. Let  $\widehat{A} \coloneqq \widehat{A}^{\ell_T + 1}$ , and  $\widehat{A}^{-} \coloneqq \widehat{A}^{\ell_T}$ .
Before proceeding to the main proof, we note that the conclusion that Algorithm 3 only explores with policies in  $\Pi_{\mathrm{exp}}$  follows from the definition of LEARNEXPII and LC<sup>3</sup>. Note that LEARNEXPII only interacts with (1.1) through calls to DYNAMICOED, which itself only interacts with (1.1) through calls to  $\mathbb{A}_{\mathcal{R}}$ , instantiated in Algorithm 3 by LC<sup>3</sup>. Inspection of the LC<sup>3</sup> algorithm in [17] reveals that LC<sup>3</sup> only interacts with (1.1) by playing policies in  $\Pi_{\mathrm{exp}}$ , from which the conclusion follows.
Bounding the Number of Episodes. Denote  $T_{\ell}^{\mathrm{oed}} = |\Pi_{\ell}|$ . Note that by construction we always have  $N_{\ell}T_{\ell}^{\mathrm{oed}} \geq T_{\ell}$ . By Lemma B.2, as long as (B.1) is met, we can bound the total number of episodes collected up to and including round  $\ell$  by  $4T_{\ell}$  for  $\ell \in \{\ell_T, \ell_T - 1\}$ . We therefore, in the following, will make use of the fact that
$$
c \cdot T \leq N _ {\ell} K _ {\ell} \leq c ^ {\prime} \cdot T
$$
for  $\ell \in \{\ell_T, \ell_T - 1\}$  and absolute constants  $c, c'$ . Furthermore, it also follows from this that the total number of episodes run by Algorithm 3 is bounded by
$$
4 T _ {\ell_ {T}} = 4 \cdot 2 ^ {\lceil \log T / 8 \rceil} \leq 8 \cdot \frac {T}{8} = T,
$$
so Algorithm 3 runs for at most  $T$  episodes.
Approximating the Controller Loss. Let  $r_{\mathrm{est}}(A_\star) \coloneqq \min \{1, r_{\mathrm{cost}}(A_\star), r_\theta(A_\star)\}$ , for  $r_{\mathrm{cost}}(A_\star)$  as in Assumption 2 and  $r_\theta(A_\star)$  as in Assumption 13. By Lemma D.2, under Assumptions 1, 2, 4, 5 and 13, as long as  $\widehat{A} \in \mathcal{B}_{\mathrm{F}}(A_\star; r_{\mathrm{est}}(A_\star))$ , we have
$$
\begin{array}{l} \mathcal {J} \left(\widehat {\pi} _ {T}; A _ {\star}\right) - \mathcal {J} \left(\pi_ {\star} \left(A _ {\star}\right); A _ {\star}\right) \leq \| \operatorname {v e c} (\widehat {A} - A _ {\star}) \| _ {\mathcal {H} \left(A _ {\star}\right)} ^ {2} \\ + \operatorname {p o l y} \left(L _ {\pi_ {\star}}, \| A _ {\star} \| _ {\mathrm {o p}}, L _ {\phi}, L _ {\boldsymbol {\theta}}, L _ {\text {c o s t}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, d _ {\boldsymbol {x}}\right) \cdot \| \widehat {A} - A _ {\star} \| _ {\mathrm {o p}} ^ {3}. \\ \end{array}
$$
Furthermore, we can bound
$$
\begin{array}{l} \| \operatorname {v e c} (\widehat {A} - A _ {\star}) \| _ {\mathcal {H} (A _ {\star})} ^ {2} = \| \operatorname {v e c} (\widehat {A} - A _ {\star}) \| _ {\mathcal {H} (\widehat {A} ^ {-})} ^ {2} + \operatorname {v e c} (\widehat {A} - A _ {\star}) ^ {\top} (\mathcal {H} (A _ {\star}) - \mathcal {H} (\widehat {A} ^ {-})) \operatorname {v e c} (\widehat {A} - A _ {\star}) \\ \leq \left(\widehat {A} - A _ {\star}\right) ^ {\top} \mathcal {H} \left(\widehat {A} ^ {-}\right) \left(\widehat {A} - A _ {\star}\right) + \| \widehat {A} ^ {-} - A _ {\star} \| _ {\mathrm {o p}} ^ {2} \| \mathcal {H} \left(A _ {\star}\right) - \mathcal {H} \left(\widehat {A} ^ {-}\right) \| _ {\mathrm {o p}}. \\ \end{array}
$$
Bounding the Hessian Estimation Error. On  $\mathcal{E}$ , by Lemma B.3 and as long as (B.1) is met, we have (note that at the final epoch, the plug-in estimator  $\mathcal{H}(\widehat{A}^{-})$  is given as input to DYNAMICOED):
$$
\begin{array}{l} \left\| \operatorname {v e c} \left(\widehat {A} - A _ {\star}\right) \right\| _ {\mathcal {H} \left(\widehat {A} ^ {-}\right)} ^ {2} \leq \frac {6 0}{N _ {\ell_ {T}} T _ {\ell_ {T}} ^ {\mathrm {o e d}}} \cdot \min  _ {\boldsymbol {\Lambda} \in \Omega} \operatorname {t r} \left(\mathcal {H} \left(\widehat {A} ^ {-}\right) \breve {\boldsymbol {\Lambda}} ^ {- 1}\right) \cdot \sigma_ {\boldsymbol {w}} ^ {2} \log \frac {6 d _ {\boldsymbol {x}} d _ {\phi}}{\delta} \\ + \mathrm {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\mathrm {m i n}} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \| \mathcal {H} (\widehat {A} ^ {-}) \| _ {\mathrm {o p}}, \log \frac {T _ {\ell_ {T}}}{\delta}\right) \cdot \frac {1}{N _ {\ell_ {T}} ^ {2}} \\ \leq \frac {C}{T} \cdot \min  _ {\boldsymbol {\Lambda} \in \Omega} \operatorname {t r} \left(\mathcal {H} (\widehat {A} ^ {-}) \breve {\boldsymbol {\Lambda}} ^ {- 1}\right) \cdot \sigma_ {\boldsymbol {w}} ^ {2} \log \frac {6 d _ {\boldsymbol {x}} d _ {\phi}}{\delta} \\ + \operatorname {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \| \mathcal {H} (\widehat {A} ^ {-}) \| _ {\mathrm {o p}}, \log \frac {T}{\delta}\right) \cdot \frac {1}{T ^ {2}} \\ \end{array}
$$
where the last line uses that  $N_{\ell_T}T_{\ell_T}^{\mathrm{oed}}$  is within a constant of  $T$ , and that  $T_{\ell_T}^{\mathrm{oed}}$  can be bounded by
$$
\text {p o l y} \left(d _ {\phi}, d _ {\boldsymbol {x}}, \frac {1}{\lambda_ {\min} ^ {\star}}, B _ {A}, B _ {\phi}, \log \frac {1}{\sigma_ {\boldsymbol {w}}}, H, \log \frac {T}{\delta}\right)
$$
by Lemma B.3. By Lemma B.4 we can bound
$$
\min  _ {\boldsymbol {\Lambda} \in \Omega} \operatorname {t r} \left(\mathcal {H} (\widehat {A} ^ {-}) \breve {\boldsymbol {\Lambda}} ^ {- 1}\right) \leq \min  _ {\boldsymbol {\Lambda} \in \Omega} 2 \operatorname {t r} \left(\mathcal {H} (A _ {\star}) \breve {\boldsymbol {\Lambda}} ^ {- 1}\right) + \frac {2 d _ {\boldsymbol {x}} d _ {\phi}}{\lambda_ {\min } ^ {\star}} \cdot \| \mathcal {H} (A _ {\star}) - \mathcal {H} (\widehat {A} ^ {-}) \| _ {\mathrm {o p}}
$$
and by Lemma D.3, under Assumptions 1, 2, 4, 5 and 13, and as long as  $\widehat{A}^{-} \in \mathcal{B}_{\mathrm{F}}(A_{\star}; r_{\mathrm{est}}(A_{\star}))$ , we can bound
$$
\| \mathcal {H} (A _ {\star}) - \mathcal {H} (\widehat {A} ^ {-}) \| _ {\mathrm {o p}} \leq \mathrm {p o l y} (L _ {\pi_ {\star}}, \| A _ {\star} \| _ {\mathrm {o p}}, L _ {\phi}, L _ {\theta}, L _ {\mathrm {c o s t}}, \sigma_ {\boldsymbol {w}} ^ {- 1}, H, d _ {\boldsymbol {x}}) \cdot \| \widehat {A} ^ {-} - A _ {\star} \| _ {\mathrm {o p}}.
$$