SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {x} _ {h + 1} = \left[ \begin{array}{c c c c c c} 1 & 0 & 0. 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0. 1 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0. 1 \\ 0 & 0 & 0 & 0 & 0 & 1 \end{array} \right] \boldsymbol {x} _ {h} + \left[ \begin{array}{c c} 0 & 0 \\ 0 & 0 \\ 0. 1 \cdot \cos \left(\left[ \boldsymbol {x} _ {h} \right] _ {5}\right) & 0 \\ 0. 1 \cdot \sin \left(\left[ \boldsymbol {x} _ {h} \right] _ {5}\right) & 0 \\ 0 & 0 \\ 0 & 0. 1 \end{array} \right] \boldsymbol {u} _ {h} + \boldsymbol {w} _ {h} \tag {G.2}
$$
where  $[\pmb{x}_h]_5$  denotes the 5th element of  $\pmb{x}_h$ . Here we interpret  $[\pmb{x}_h]_1$  as the  $x$  position,  $[\pmb{x}_h]_2$  as the  $y$  position,  $[\pmb{x}_h]_3$  as the  $x$  velocity,  $[\pmb{x}_h]_4$  as the  $y$  velocity,  $[\pmb{x}_h]_5$  as the angle of orientation (that is, the direction the car is facing), and  $[\pmb{x}_h]_6$  as the angular velocity. The first control dimension, then, corresponds to the "gas", the power given to the car to move forward or backward, and the second control dimension corresponds to altering the direction of the steering wheel. Similar to the drone system, we set the cost to
$$
\operatorname {c o s t} (\boldsymbol {x}, \boldsymbol {u}) = \left[ \begin{array}{c} \boldsymbol {x} \\ \boldsymbol {u} \end{array} \right] ^ {\top} Q \left[ \begin{array}{c} \boldsymbol {x} \\ \boldsymbol {u} \end{array} \right] \quad \text {w i t h} \quad Q = \frac {0 . 1 \cdot I + \boldsymbol {v} _ {1} \boldsymbol {v} _ {1} ^ {\top} + \boldsymbol {v} _ {2} \boldsymbol {v} _ {2} ^ {\top}}{\| 0 . 1 \cdot I + \boldsymbol {v} _ {1} \boldsymbol {v} _ {1} ^ {\top} + \boldsymbol {v} _ {2} \boldsymbol {v} _ {2} ^ {\top} \| _ {\mathrm {o p}}}
$$
for some  $\pmb{v}_1, \pmb{v}_2$ . To write this in the form of (1.1), in order to make the problem more challenging we choose an overparameterized  $\phi(\pmb{x}, \pmb{u})$ :
$$
\boldsymbol {\phi} (\boldsymbol {x}, \boldsymbol {u}) = [ \boldsymbol {x}, \boldsymbol {u}, \cos ([ \boldsymbol {x} ] _ {5}), \sin ([ \boldsymbol {x} ] _ {5}), [ \boldsymbol {u} ] _ {1} \cdot \cos ([ \boldsymbol {x} ] _ {5}), [ \boldsymbol {u} ] _ {1} \cdot \sin ([ \boldsymbol {x} ] _ {5}), [ \boldsymbol {u} ] _ {2} \cdot \cos ([ \boldsymbol {x} ] _ {5}), [ \boldsymbol {u} ] _ {2} \cdot \sin ([ \boldsymbol {x} ] _ {5}) ]
$$
$$
A _ {\star} = \left[ \begin{array}{l l l l l l l l l l l l l l} 1 & 0 & 0. 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0. 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0. 1 & 0 & 0 & 0 \\ 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0. 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 0. 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0. 1 & 0 & 0 & 0 & 0 & 0 & 0 \end{array} \right].
$$
For the car system, the controller class  $\Pi^{\star}$  is a hierarchical controller parameterized by some  $\pmb {\theta}\in \mathbb{R}^{4}$ . This controller first uses PD control to compute a "goal input", the direction we would like to modify the state in, as:
$$
\boldsymbol {u} _ {\text {g o a l}} (\boldsymbol {x}) = - \boldsymbol {\theta} _ {1} [ \boldsymbol {x} ] _ {1: 2} - \boldsymbol {\theta} _ {2} [ \boldsymbol {x} ] _ {3: 4}.
$$
Given the underactuated structure of the system in (G.2), we cannot directly push the state in the direction of  $\mathbf{u}_{\mathrm{goal}}(\mathbf{x})$ . Instead, we set  $\mathbf{u}$  to the following:
$$
\left[ \begin{array}{c} \boldsymbol {u} _ {\text {g o a l}} (\boldsymbol {x}) ^ {\top} \left[ \begin{array}{c} \cos ([ \boldsymbol {x} ] _ {5}) \\ \sin ([ \boldsymbol {x} ] _ {5}) \end{array} \right] \\ - \boldsymbol {\theta} _ {3} ([ \boldsymbol {x} ] _ {5} - \beta_ {\text {g o a l}} (\boldsymbol {x})) - \boldsymbol {\theta} _ {4} [ \boldsymbol {x} ] _ {6} \end{array} \right] \quad \text {f o r} \quad \beta_ {\text {g o a l}} (\boldsymbol {x}) = \tan^ {- 1} ([ \boldsymbol {u} _ {\text {g o a l}} (\boldsymbol {x}) ] _ {2} / [ \boldsymbol {u} _ {\text {g o a l}} (\boldsymbol {x}) ] _ {1}).
$$
Given the complex form of this controller and the dynamics, there does not exist a closed-form way to set  $\theta$  optimally. Instead, for this system, we rely on a simple random search procedure to compute  $\pi_{\star}(A)$ . To find an optimal controller for system  $A$ , we randomly sample parameters  $\theta$ , compute the cost they incur on system  $A$ , and then set  $\pi_{\star}(A)$  to the randomly generated controller with lowest cost. Note that this procedure is not differentiable, but we require  $\pi_{\star}(A)$  is differentiable. To remedy this, in situations where a differentiable  $\pi_{\star}(A)$  is needed (in particular, in the computation of  $\mathcal{H}(A)$ ), rather than returning a single controller, we return the softmax distribution over all controllers sampled, weighting each controller by its estimated cost. As the softmax distribution can be differentiated, this parameterization of  $\pi_{\star}(A)$  is differentiable.
For this system, there does not exist a closed-form expression for  $\mathcal{J}(\pi; A)$  and, as such, to compute  $\mathcal{J}(\pi; A)$ , we simply perform many roll-outs of policy  $\pi$  on system  $A$  and average the cost. Given this and the search-based implementation of  $\pi_{\star}(A)$  outlined above, we found that computing the hessian  $\mathcal{H}(A)$  using the torch.autogradfunctional.hessian as in Appendix G.1.2 was very memory-intensive. Instead, we computed the Jacobian  $G(A) := \nabla_{A'} \mathcal{J}(\pi_{\star}(A'); A)|_{A' = A}$ , and then, in place of  $\mathcal{H}(A)$ , we use  $G(A)G(A)^{\top}$ . To compute  $G(A)$ , we use the torch.autogradfunctional.jacobian function. While using  $G(A)G(A)^{\top}$  in place of  $\mathcal{H}(A)$  is not justified by our theoretical analysis, if we are in settings where  $\pi_{\star}(A)$  is not precisely the minimum of  $\mathcal{J}(\pi; A)$  (which will likely be the case here since we are relying on a sampling-based implementation of  $\pi_{\star}(A)$ , which will incur some small error), then we argue that this is a reasonable metric to use. In particular, in this setting, the approximation of  $\mathcal{J}(\pi_{\star}(\widehat{A}); A_{\star})$  given in Proposition 1 should have an additional first-order term of the form  $G(A)^{\top} \mathrm{vec}(A_{\star} - \widehat{A})$ . As we can upper bound
$$
G (A) ^ {\top} \mathrm {v e c} (A _ {\star} - \widehat {A}) \leq \sqrt {\| \mathrm {v e c} (A _ {\star} - \widehat {A}) \| _ {G (A _ {\star}) G (A _ {\star}) ^ {\top}} ^ {2}},
$$
optimizing for the metric  $G(A)G(A)^{\top}$  instead of  $\mathcal{H}(A)$  can be seen as minimizing the first-order Taylor-approximation of the excess loss. Intuitively, this metric quantifies the sensitivity of the loss to particular parameters in  $A_{\star}$ , and in practice we found that optimizing this metric produced significant improvements over existing methods. The implementation of the example from Section 1.1 relied on this same approximation.
For all methods considered, our implementation follows the basic structure of Algorithm 1: at every epoch, we explore so as to minimize some exploration objective, form an estimate of  $A_{\star}$  on the collected data, and then compute  $\pi_{\star}(\widehat{A}_t)$  on our estimate. Our main experimental results (Figures 1 to 3) show the loss of  $\pi_{\star}(\widehat{A}_t)$  as the time horizon  $t$  increases. For each method, to collect an initial set of data, we begin each trial by exploring randomly for some fixed number of episodes (10 for the drone example, 100 for the car example). The first point in each plot then corresponds to the performance after this initial random exploration. Each aspect of our implementation is modular, and any given component can be easily replaced. Below we highlight our implementation of the exploration routine, and choice of exploration objective, for the various approaches we consider.
G.2.1 Implementation of DYNAMICOED
Implementing the exploration procedure, DYNAMICOED, requires access to a regret minimization oracle. While in principle the LC $^3$  algorithm of [17] could be applied to this problem to give such an oracle, the LC $^3$  algorithm requires access to a computation oracle which is not clear how to implement in practice. To remedy this, we implement a Thompson Sampling-inspired modification to the LC $^3$  algorithm of [17].
The primary computational challenge of implementing the  $\mathrm{LC^3}$  algorithm is the computation of the optimistic policy:
$$
\operatorname *{arg  min}_{\pi \in \Pi_{\exp}}\min_{A\in \mathrm{B}_{\text{ALL}^{t}}}\mathcal{I}^{\exp}(\pi ;A)
$$
where  $\mathcal{J}^{\mathrm{exp}}(\pi; A)$  denotes the exploration cost that is minimized in  $\mathrm{LC}^3$  (i.e. the expected cost on the cost function  $\mathrm{cost}_{h}^{n}(\pmb{x}, \pmb{u}) \gets \frac{1}{M} \cdot \phi(\pmb{x}, \pmb{u})^{\top} (\Xi_{n}) \phi(\pmb{x}, \pmb{u})$  set in DYNAMICOED), and  $\mathrm{BALL}^t$  the confidence set for  $A_{\star}$  at iteration  $t$ .
To avoid solving this optimization, we adopt a Thompson Sampling-inspired variation of this procedure. In particular, at iteration  $t$ , we sample  $\widetilde{A}_t \sim \mathcal{N}(\widehat{A}_t, \Lambda_t^{-1})$ . Standard Thompson Sampling would then compute  $\arg \min_{\pi \in \Pi_{\mathrm{exp}}} \mathcal{J}^{\mathrm{exp}}(\pi; \widetilde{A}_t)$ , but even this can be challenging, so we instead rely on a sampling MPC-inspired approach. Given that we are at state  $x_h$  and have played inputs  $u_1, \ldots, u_{h-1}$ , we aim to approximately solve the following optimization:
$$
\min  _ {\boldsymbol {u} _ {h}, \boldsymbol {u} _ {h + 1}, \dots , \boldsymbol {u} _ {H} \in \mathbb {R} ^ {d} \boldsymbol {u}} \sum_ {h ^ {\prime} = h} ^ {H} \operatorname {c o s t} _ {h} ^ {n} \left(\boldsymbol {x} _ {h ^ {\prime}}, \boldsymbol {u} _ {h ^ {\prime}}\right) \tag {G.3}
$$
$$
\begin{array}{l} \text {s . t .} \quad \boldsymbol {x} _ {h + 1} = \widetilde {A} _ {t} \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}), \sum_ {h = 1} ^ {H} \boldsymbol {u} _ {h} ^ {\top} \boldsymbol {u} _ {h} \leq \gamma^ {2}. \end{array}
$$
To solve this approximately, we sample many possible  $\mathbf{u}$  randomly, compute the value of the objective of (G.3) on the trajectories induced by these  $\mathbf{u}$ , and finally choose the input that minimizes this objective. Rather than playing the entire sequence of chosen inputs, however, we simply play the first input in the sequence, observe the new state on the actual system, and re-solve (G.3) on this new state. Note that the implementation of  $\mathrm{LC}^3$  used for the experiments given in [17] relies on a similar Thompson Sampling-based approximation to the  $\mathrm{LC}^3$  algorithm.
G.2.2 Implementation of Uniform Exploration
The goal of the procedure we have referred to as Uniform Exploration is to collect data which will result in the estimation error,  $\| A_{\star} - \widehat{A}\|_{\mathrm{op}}$ , being minimized, the goal of the method given in [1]. It can be shown that this is equivalent to maximizing  $\lambda_{\min}(\Lambda_T)$ , so this method reduces to choosing
inputs that maximize  $\lambda_{\mathrm{min}}(\Lambda_T)$ . To implement this procedure, we rely on the same sampling-based MPC approach as we outlined above, with the primary difference being that instead of minimizing the objective of (G.3), we choose the inputs that maximize
$$
\lambda_ {\min } \left(\boldsymbol {\Lambda} _ {t} + \sum_ {h = 1} ^ {H} \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) ^ {\top}\right),
$$
where  $\Lambda_{t}$  denote the covariates we have obtained so far at iteration  $t$ . While very similar in spirit to the algorithm of [1], the implementation details are somewhat different than the algorithm proposed in that work. We found that in practice our implementation performed better than directly implementing (a sampling-based variant of) the algorithm from [1], and all reported results for Uniform Exploration are therefore on this version.
G.2.3 Exploring via Cost Minimization
A natural point of comparison to our methods would be to forsake the system identification phase entirely, and simply run standard policy optimization algorithms such as TRPO or PPO [61, 62], to obtain a controller  $\widehat{\pi}_T$ . The primary difficulty with these approaches in the settings we consider is that these algorithms are on-policy, meaning that they primarily roll out trajectories using their current estimate of the optimal policy,  $\widehat{\pi}_t$ , and using the collected data to do policy improvement on  $\widehat{\pi}_t$ . In contrast, our setting is off-policy, in the sense that the learner must explore by playing policies in  $\Pi_{\mathrm{exp}}$ , but return some policy in  $\Pi^{\star}$ . Since in the settings we consider  $\Pi_{\mathrm{exp}} \neq \Pi^{\star}$ , on-policy approaches are simply exploring very differently, and therefore cannot be compared with directly.
This is particularly an issue in our setting where stability may come into play. Indeed, it may be the case that some controller in  $\Pi^{\star}$  will destabilize the system, and cause the norm of the state to increase exponentially in  $h$ . Inducing such trajectories significantly improves one's ability to perform system identification as the signal-to-noise ratio also then increases exponentially. However, to induce this trajectory with a state-feedback controller, the power of the input played by this controller will also increase exponentially in  $h$ . Since we choose  $\Pi_{\mathrm{exp}}$  to include only policies with bounded power, this is not a fair comparison (and, furthermore, is likely not an algorithm one would want to run in practice).
While direct comparison with such approaches is therefore not possible, it is possible to compare against algorithms that, instead of collecting data that minimizes or maximizes objectives such as  $\mathrm{tr}(\mathcal{H}\bar{\Lambda}^{-1})$  or  $\lambda_{\mathrm{min}}(\Lambda)$ , instead simply aims to play policies minimizing  $\mathcal{J}(\pi;A)$ . In principle, such algorithms are similar to approaches such as TRPO in how they perform their exploration—both collect data by aiming to minimize the actual cost we are attempting to find a controller to minimize.
To implement this approach, we rely on a sampling-based MPC algorithm similar to that described in Appendix G.2.1, but where the goal is now to solve
$$
\min  _ {\pi \in \Pi_ {\exp}} \mathcal {J} (\pi ; A).
$$
Note that the key difference between this approach and approaches such as TRPO is that we still only play  $\pi \in \Pi_{\mathrm{exp}}$ . Using this objective to induce exploration, we then simply estimate  $A_{\star}$  on this collected data, and return  $\pi_{\star}(\hat{A})$ . The results of this approach on the drone system are given in Figure 4 (with this cost minimization approach denoted as Cost Minimization Exploration). As this illustrates, this approach is significantly worse than Algorithm 1, and is also outperformed by Uniform Exploration or Random Exploration when the number of episodes is large enough.
G.3 Additional Results
Finally, in this section we present versions of Figures 1 to 3 with error bars in Figures 5 to 7. In all figures, errors bars denote one standard error.
Figure 4: Performance on drone with LC<sup>3</sup> Exploration
Figure 5: Mean excess controller loss on instance of Section 1.1 with error bars
Figure 6: Mean excess controller loss on drone with error bars
Figure 7: Mean excess controller loss on car with error bars