SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andrew Wagenmaker*
Guanya Shi†
Kevin Jamieson†
Learning to control unknown nonlinear dynamical systems is a fundamental problem in reinforcement learning and control theory. A commonly applied approach is to first explore the environment (exploration), learn an accurate model of it (system identification), and then compute an optimal controller with the minimum cost on this estimated system (policy optimization). While existing work has shown that it is possible to learn a uniformly good model of the system [1], in practice, if we aim to learn a good controller with a low cost on the actual system, certain system parameters may be significantly more critical than others, and we therefore ought to focus our exploration on learning such parameters.
In this work, we consider the setting of nonlinear dynamical systems and seek to formally quantify, in such settings, (a) which parameters are most relevant to learning a good controller, and (b) how we can best explore so as to minimize uncertainty in such parameters. Inspired by recent work in linear systems [2], we show that minimizing the controller loss in nonlinear systems translates to estimating the system parameters in a particular, task-dependent metric. Motivated by this, we develop an algorithm able to efficiently explore the system to reduce uncertainty in this metric, and prove a lower bound showing that our approach learns a controller at a near-instance-optimal rate. Our algorithm relies on a general reduction from policy optimization to optimal experiment design in arbitrary systems, and may be of independent interest. We conclude with experiments demonstrating the effectiveness of our method in realistic nonlinear robotic systems<sup>1</sup>.
Controlling nonlinear dynamical systems is a core problem in robotics, cyber-physical systems, and beyond, and a significant body of work in both the control theory and reinforcement learning communities has sought to address this challenge [3-5]. In many real-world scenarios [6-9], the dynamics of the system of interest is unknown, or only a coarse model of them is available, which
$^{*}$ ajwagen@cs.washington.edu
guanyas@andrew.cmu.edu
$\ddagger$  jamieson@cs.washington.edu
$^{1}$ Code: https://github.com/ajwagen/nonlinear_sysid_for_control
significantly increases the challenge of control—not only must we control such systems, we must learn to control them. While a variety of methods exist to address this challenge, a commonly applied approach is to first perform system identification, learning an accurate model of the system's dynamics, and then use this model to obtain a controller. Despite its promising potential, there are still several fundamental questions that must be answered to make this approach practically effective.
Which parameters are most relevant to learning a good controller? Beyond some special cases, little work has been done characterizing how the estimation error from system identification translates to end-to-end suboptimality in the resulting controller of our nonlinear systems. In particular, certain parameters of the system or regions of the state space may be irrelevant to learning a good controller, and coarse estimates of these parameters would suffice, while other parameters may be critical to learning a good controller, and we must therefore estimate these parameters very accurately in order to effectively control the system. In the context of this work, where nonlinearities are considered, the heterogeneity of the parameters is further accentuated. For instance, around a point of equilibrium, some system parameters might be completely inactive, having no impact on the dynamics (see the example in Section 1.1 for an illustration of this).
How can we best explore so as to minimize uncertainty in relevant parameters? Even if we are able to determine which parameters are most important for obtaining a good controller on the true system, it is not obvious how to use this information. How can we direct our system identification phase in order to focus on learning these parameters as quickly as possible, without spending time estimating the parameters of the system less critical for control? This is fundamentally a question of exploration. While it is known in linear systems that random excitation will efficiently explore [10], exploration in nonlinear systems is significantly more challenging since, in order to excite all parameters of interest, non-trivial planning may be required to ensure all relevant states are reached (as is the case in the example considered in Section 1.1).
We address both these questions in a particular class of nonlinear systems parameterized as:
$$
\boldsymbol {x} _ {h + 1} = A _ {\star} \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) + \boldsymbol {w} _ {h}. \tag {1.1}
$$
Here  $\boldsymbol{x}_h \in \mathbb{R}^{d_x}$  denotes the state of the system,  $\boldsymbol{u}_h \in \mathcal{U} \subseteq \mathbb{R}^{d_u}$  the input,  $\boldsymbol{w}_h \sim \mathcal{N}(0, \sigma_w^2 \cdot I)$  random noise,  $\phi(\cdot, \cdot) \in \mathbb{R}^{d_\phi}$  a (possibly nonlinear, known) feature map, and  $A_\star \in \mathbb{R}^{d_x \times d_\phi}$  the (unknown) system parameter. Systems of this form are able to model a variety of real-world settings [11-15]^2, and have been the subject of recent attention in the reinforcement learning community [1, 17, 14], yet the aforementioned questions have remained unanswered. Towards addressing this, in this work we make the following contributions:
1. For systems of the form (1.1), given some cost of interest which we wish to find a controller to minimize, we (a) formally characterize how estimation error translates into suboptimality in the learned controller, under the certainty equivalent control rule and (b) provide a lower bound on the loss of any (sufficiently regular) control rule learned from  $T$  rounds of interaction with (1.1).
2. Motivated by this characterization, we present an algorithm which achieves the instance-optimal rate, with controller loss matching our lower bound. To the best of our knowledge, this is the first statistically optimal algorithm in the setting of nonlinear dynamical systems. Our algorithm relies on a generic reduction from policy optimization to optimal exploration in arbitrary dynamical systems (not necessarily of the form (1.1)), which may be of independent interest.
3. We present numerical experiments on several realistic nonlinear systems which illustrate that our approach—efficiently exploring to reduce uncertainty in parameters most relevant to learning a controller—yields significant gains in practice.
To further motivate our approach, we consider the following example.
1.1 Motivating Example
To motivate the need for effective exploration, we consider a simple 1-D system with nonlinear dynamics given by:
$$
\boldsymbol {x} _ {h + 1} = a _ {1} \boldsymbol {x} _ {h} + a _ {2} \boldsymbol {u} _ {h} + \sum_ {i = 1} ^ {1 0} a _ {i + 2} \phi_ {i} (\boldsymbol {x} _ {h}) + \boldsymbol {w} _ {h}
$$
2In real-world settings,  $\phi$  is typically (1) from physics (i.e., the system structure is known but some parameters such as drag coefficient are unknown [3]), (2) learned using representation learning or meta-learning [12, 15], and/or (3) from random features (e.g., any sufficiently regular, smooth nonlinear system  $\pmb{f}(\pmb{x},\pmb{u})$  can be modeled by (1.1) using  $N$  random features up to a  $1 / \sqrt{N}$  error [16]).
where  $\phi_i(\pmb{x}) = \max \{1 - 100(\pmb{x} - c_i)^2, 0\}$  for some  $c_i$ . We choose  $a_1 = 0.8, a_2 = 1$ , and  $a_3 = \ldots = a_{12} = -3$ . We assume  $a_{1:12}$  are unknown,  $(\phi_i)_{i=1}^{10}$  is known, and set
$$
\operatorname {c o s t} (\boldsymbol {x}, \boldsymbol {u}) = \left(\boldsymbol {x} - c _ {1}\right) ^ {2} + 1 0 0 ^ {- 1} \cdot \boldsymbol {u} ^ {2}.
$$
With this choice of cost, the optimal controller will attempt to direct the state  $\mathbf{x}$  to the equilibrium point  $c_{1}$  and maintain this position. Note that, with our choice of  $\phi_{i}$ ,  $\phi_{i}(\mathbf{x}) \neq 0$  only when  $\mathbf{x}$  is very close to  $c_{i}$ . This renders the parameters  $a_{4:12}$  irrelevant to learning the optimal controller, since  $\phi_{2}, \ldots, \phi_{10}$  will be inactive if we are playing optimally, but learning  $a_{1:3}$  is critical to performing optimally. In particular, the coefficient of the first nonlinearity,  $a_{3}$ , must be learned, as its value significantly changes the dynamics at the goal state.
We illustrate the result of running on this system in Figure 1, comparing our proposed approach (Task-Driven Exploration, Algorithm 1) to the approach which chooses  $\boldsymbol{u}_h \sim \mathcal{N}(0, \sigma_{\boldsymbol{u}}^2)$  (Random Exploration), and the approach proposed in [1] (Uniform Exploration) which seeks to explore so as to estimate  $a_{1:12}$  uniformly well. As can be seen, neither of these latter two approaches are able to learn a good controller, while our approach easily finds a near-optimal controller. The failure modes of each of these approaches is somewhat different. Here Random Exploration fails since the chance of reaching the point  $x_h \sim c_1$  is extremely small if the input is random noise—reaching  $c_1$  requires playing a particular sequence of actions which are very unlikely to be played if  $u_h$  is
Figure 1: Performance on Motivating Example
chosen randomly. The Uniform Exploration approach does, in contrast, plan and, given enough time, is guaranteed to estimate all parameters accurately. However, as it aims to estimate all parameters uniformly well, it will attempt to estimate  $a_{4:12}$  accurately despite their irrelevance to control, which will slow down the rate at which it is able to estimate  $a_3$ . Only our approach, which both plans and takes into account the cost while exploring, is able to reach  $a_3$  enough times to efficiently estimate it, and learn a good controller.
This example illustrates that it is critical both to explore efficiently, and also to let the objective—learning a good controller—guide this exploration. We emphasize that the behavior in this example is only exhibited in nonlinear systems—though taking into account the task while exploring in linear systems is known to yield provable improvements [2], even playing random noise allows every direction to be learned in such systems. In nonlinear systems, however, this is not the case—one may fail to learn completely unless careful planning is performed.
Learning for control. Recently, there has been increased interest in studying control problems from a learning-theoretic perspective, largely for linear system settings such as online LQR or LQG with unknown dynamics [18, 10, 19-26]. In the nonlinear setting, [27-29] provide formal guarantees on system identification in several different classes of nonlinear systems, yet they only consider noiseless systems, or systems that are significantly easier to excite than (1.1). [17] study systems of the form (1.1), but consider only the regret minimization problem. While their bounds would yield a polynomial complexity via an online-to-batch conversion, the resulting guarantee would scale at a  $\mathcal{O}(1 / \sqrt{T})$  rate, significantly slower than our  $\mathcal{O}(1 / T)$  rate. Several additional works in reinforcement with general function approximation encompass nonlinear systems of the form (1.1) [30-32], yet these results also achieve a slow  $\mathcal{O}(1 / \sqrt{T})$  rate. The most relevant work [1] proposes an active learning approach to identify unknown parameters in (1.1), with the goal of minimizing the Euclidean distance in the parameter space. However, they do not provide end-to-end guarantees on learning controllers and, as shown in Section 1.1, this approach could be significantly worse than learning a model with the goal task in mind. Also very related to our work is [2], which seeks to answer a similar set of questions: performing system identification in order to learn a good controller. This work is restricted to the setting of linear dynamics, however, and does not address the additional complexities of nonlinear systems.
System identification, dual control, and iterative learning control. There is a large body of classical work in system identification [7], and our work can be seen as an instance of active system
identification. While a variety of approaches have been proposed for active system identification [33-40], these tend to only consider linear systems, or lack rigorous theoretical guarantees. Recently deep learning approaches have also been applied in system identification [6, 8, 9, 41, 42]. In these works, the system identification phase is separate from the downstream controller design. Instead, in the control community, estimating parameters while simultaneously optimizing for performance has been formulated as a dual or iterative learning control problem [43-45], yet these settings focus on stability or asymptotic convergence whereas our work quantifies the end-to-end suboptimality gap.
Model-based reinforcement learning. This paper falls into the broad category of model-based reinforcement learning (MBRL), where an agent explores the environment to learn a model and then computes an optimal policy using the learned model. On the empirical side, deep MBRL has made exciting progress in many domains [46-48], and several task-aware methods have been designed to improve MBRL's performance [47-49], yet these works lack formal guarantees. On the theoretical side, a variety of different model-based approaches exist [50-55, 14]; however, the majority of these consider restricted settings such as tabular or linear MDPs. Of particular interest is the work of [14] which presents a result in systems of the form (1.1). While they show that polynomial sample complexity is possible, their guarantee scales at a  $\mathcal{O}(1 / T^{1 / 6})$  rate compared to our much faster  $\mathcal{O}(1 / T)$  rate.
Adaptive nonlinear control. Adaptive nonlinear control also seeks to control an unknown nonlinear system with parametric uncertainties [3, 4]. In particular, the key idea of model-reference adaptive control (MRAC) bears affinity to this paper, in that the adaptation law in MRAC adapts unknown parameters in a task-aware manner. There are two main differences between MRAC and our work. First, adaptive control does not explicitly optimize a cost function—the objective is typically tracking error convergence and Lyapunov stability, whereas our framework allows general cost functions—and the focus is typically on asymptotic convergence, while we give non-asymptotic optimality guarantees. Second, adaptive control has by and large been limited to specific system classes (e.g., fully-actuated systems [4, 15]) and policy classes (e.g., policy to directly cancel out the matched uncertainty [12, 13]), whereas our framework allows more general systems and policy classes.
Notation.  $\| \cdot \|_{\mathrm{op}}$  denotes the operator norm (matrix 2-norm),  $\| \cdot \|_{\mathrm{F}}$  the Frobenius norm, and  $\| \cdot \|_M$  the Mahalanobis norm, defined as  $\| \pmb{x} \|_M \coloneqq \sqrt{\pmb{x}^\top M \pmb{x}}$  for  $M \succeq 0$ .  $\operatorname{vec}(A)$  denotes the vectorization of matrix  $A$ .  $\mathcal{B}_p(A; r) \coloneqq \{A' : \| A - A' \|_p \leq r\}$ .  $[H] = \{1, 2, \dots, H\}$ . We let  $\mathbb{E}_A[\cdot]$  denote the expectation over trajectories induced on system with parameter  $A$ , and  $\mathbb{E}_{A,\pi}[\cdot]$  the expectation induced when policy  $\pi$  is played.  $\mathrm{poly}(\cdot)$  denotes some term that is polynomial in its arguments, with exponents absolute constants. We use  $\lesssim$  informally to highlight key parameters in an inequality.
Setting. In this work, we are interested in systems of the form (1.1). We consider the episodic setting, where episodes are of length  $H$ , and assume that each episode starts from a given state  $x_{1}$ . We also assume  $\| A_{\star}\|_{\mathrm{op}}\leq B_A$  for some known  $B_{A} > 0$ . We note that the setting considered here encompasses many real-world systems of interest in robotics and control (e.g., [12, 14, 11, 15] and Section 6).
The goal of the learner is to find a policy (controller)  $\pi = (\pi_h)_{h=1}^H$  which achieves minimal cost on (1.1), for the cost defined by some (known) function  $(\mathrm{cost}_h(\cdot, \cdot))_{h=1}^H$ , with  $\mathrm{cost}_h: \mathbb{R}^{d_x} \times \mathcal{U} \to \mathbb{R}_+$ . For a given policy  $\pi$ , we define the expected cost on system  $A$  as
$$
\mathcal {J} (\pi ; A) := \mathbb {E} _ {A, \pi} \left[ \sum_ {h = 1} ^ {H} \operatorname {c o s t} _ {h} \left(\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}\right) \right].
$$
We consider the following interaction protocol:
1. Learner interacts with (1.1) for  $T$  episodes, at each episode playing a policy  $\pi_{\mathrm{exp}} \in \Pi_{\mathrm{exp}}$ .
2. After  $T$  episodes, the learner proposes a policy  $\widehat{\pi}_T\in \Pi^*$
3. The learner suffers cost  $\mathcal{J}(\widehat{\pi}_T;A_\star)$
The goal of the learner is therefore first to explore and, after  $T$  episodes of exploration, to propose its best guess at the optimal controller for (1.1),  $\widehat{\pi}_T$ . Here we take  $\Pi_{\mathrm{exp}}$  to be a (known) set of admissible exploration policies (for example, policies with bounded input power), and  $\Pi^{\star}$  a (known) set of admissible control policies. We assume that policies in  $\Pi^{\star}$  are deterministic, but allow for randomized policies in  $\Pi_{\mathrm{exp}}$ . Policies may be either open- or closed-loop.
System Notation. We let  $\mathcal{T}$  denote the space of all possible state trajectories,  $\mathcal{T} \subseteq \mathbb{R}^{d_x \times (H + 1)}$ , and, for any  $\pmb{\tau} \in \mathcal{T}$ , let  $\pmb{\tau}_{1:h}$  denote the first  $h$  states and inputs in  $\pmb{\tau}$ . For any policy  $\pi$ , we denote
$$
\boldsymbol {\Lambda} _ {A, \pi} := \mathbb {E} _ {A, \pi} \left[ \sum_ {h = 1} ^ {H} \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) \phi (\boldsymbol {x} _ {h}, \boldsymbol {u} _ {h}) ^ {\top} \right]
$$
the expected covariance induced by playing  $\pi$  on system  $A$ ,  $\Lambda_{\pi} \coloneqq \Lambda_{A_{\star},\pi}$ , and  $\check{\Lambda} \coloneqq I_{d_{\mathbf{x}}} \otimes \Lambda$  the Kronecker product of  $I_{d_{\mathbf{x}}}$  and  $\Lambda$ . Finally, we let  $\Omega$  denote the convex hull of covariance matrices induced by  $\Pi_{\mathrm{exp}}$ ,  $\Omega \coloneqq \{\mathbb{E}_{\pi \sim \omega}[\Lambda_{\pi}] : \omega \in \triangle_{\Pi_{\mathrm{exp}}}\}$ , for  $\triangle_{\Pi_{\mathrm{exp}}}$  the set of distributions over  $\Pi_{\mathrm{exp}}$ .
3.1 Regularity Assumptions
In order to make learning in (1.1) tractable, we need several regularity assumptions.
Assumption 1 (Bounded Features). For all  $\pmb {x}\in \mathbb{R}^{d_x}$  and  $\pmb {u}\in \mathcal{U}$ , we have  $\| \phi (\pmb {x},\pmb {u})\| _2\leq B_\phi$
Assumption 2 (Bounded Cost). There exists some  $r_{\mathrm{cost}}(A_\star) > 0$  such that, for all  $A \in \mathcal{B}_{\mathrm{F}}(A_\star; r_{\mathrm{cost}}(A_\star))$  and all  $\pi \in \Pi^\star$ , we have  $\mathbb{E}_{A,\pi}[(\sum_{h=1}^H \mathrm{cost}_h(\boldsymbol{x}_h, \boldsymbol{u}_h))^2] \leq L_{\mathrm{cost}}$ .
Assumption 3 (Uniform Feature Excitation). There exists  $\omega \in \triangle_{\Pi_{\mathrm{exp}}}$  such that  $\lambda_{\min}(\mathbb{E}_{\pi_{\mathrm{exp}}\sim \omega}[\pmb{\Lambda}_{\pi_{\mathrm{exp}}}]) \geq \lambda_{\min}^{\star}$  for some  $\lambda_{\min}^{\star} > 0$
We remark that these assumptions have appeared before in work on systems of the form (1.1) [1, 17]. In particular, Assumption 3 implies that every direction of  $A_{\star}$  can, in principle, be excited, allowing it to be learned. In order to precisely quantify the optimal rates of learning, we require that our system satisfy certain smoothness assumptions. First, we require that  $\phi(\cdot, \cdot)$  is differentiable in its second argument.
Assumption 4 (Smooth Nonlinearity). For all  $\pmb{x} \in \mathbb{R}^{d_{\pmb{x}}}$  and  $\pmb{u} \in \mathcal{U}$ ,  $\phi(\pmb{x}, \pmb{u})$  is four-times differentiable in  $\pmb{u}$ . Furthermore,  $\| \nabla_{\pmb{u}}^{(i)} \phi(\pmb{x}, \pmb{u}) \|_{\mathrm{op}} \leq L_{\phi}$ ,  $\forall i \in \{1, 2, 3, 4\}$ ,  $\pmb{x} \in \mathbb{R}^{d_{\pmb{x}}}$ , and  $\pmb{u} \in \mathcal{U}$ .
We also require that the class of admissible control policies,  $\Pi^{\star}$ , has a parametric form,  $\Pi^{\star} = \{\pi^{\theta} : \theta \in \mathbb{R}^{d_{\theta}}\}$ , and that the parameterization is smooth in the following sense.
Assumption 5 (Smooth Controller Class).  $\pi_h^\theta (\pmb{\tau}_{1:h})$  is four-times differentiable in  $\pmb{\theta}$  for all  $\pmb{\tau} \in \mathcal{T}$  and  $h \in [H]$ . Furthermore,  $\| \nabla_{\pmb{\theta}}^{(i)} \pi_h^\theta (\pmb{\tau}_{1:h}) \|_{\mathrm{op}} \leq L_{\pmb{\theta}}$  for  $\forall i \in \{1,2,3,4\}$ ,  $\pmb{\theta} \in \mathbb{R}^{d_{\pmb{\theta}}}$ , and  $\pmb{\tau} \in \mathcal{T}$ .
Assumption 5 is satisfied for commonly considered classes of controllers, such as linear controllers, but is also satisfied by more complex classes such as neural network controllers. While the learner may propose any  $\widehat{\pi}_T \in \Pi^\star$ , we are particularly interested in the certainty equivalence decision rule (i.e., the learner decides  $\widehat{\pi}_T$  as if the estimated system is the actual one), defined as:
$$
\pi_ {\star} (A) := \pi^ {\theta_ {\star} (A)} \quad \text {f o r} \quad \theta_ {\star} (A) := \arg \min  _ {\boldsymbol {\theta} \in \mathbb {R} ^ {d _ {\boldsymbol {\theta}}}} \mathcal {J} (\pi^ {\boldsymbol {\theta}}; A). \tag {3.1}
$$
To ensure that  $\pi_{\star}(A)$  is well-defined and sufficiently regular, we make the following assumption.
Assumption 6 (Unique Optimal Controller). We assume that the global minimum of  $\mathcal{J}(\pi^{\theta};A_{\star})$ $\pmb {\theta}_{\star}(A_{\star})$  , is unique, and that  $\nabla_{\pmb{\theta}}^{2}\mathcal{J}(\pi^{\pmb{\theta}};A_{\pmb{\star}})|_{\pmb{\theta} = \pmb{\theta}_{\star}(A_{\star})}\succ 0.$
In general, the policy optimization problem in (3.1) may not be computationally tractable. As we show in Appendix D, the globally optimal decision rule of (3.1) can be replaced with a locally optimal decision rule (i.e.  $\pi_{\star}(A)$  a local minimum of  $\mathcal{J}(\pi; A)$ ). Furthermore, Assumption 6 can be replaced by assuming the differentiability of  $\theta_{\star}(A)$  with respect to  $A$  for  $A$  near  $A_{\star}$ . For ease of exposition, in the main text we assume that Assumption 6 holds and that  $\pi_{\star}(A)$  is defined as in (3.1). With these definitions and under Assumptions 1, 2, 4 and 5, we can show that  $\mathcal{J}(\pi^{\theta}; A_{\star})$  is differentiable in  $\theta$  and, combined with Assumption 6, that  $\theta_{\star}(A)$  is differentiable in  $A$ , for  $A \in \mathcal{B}_{\mathrm{F}}(A_{\star}; r_{\theta}(A_{\star}))$  and some  $r_{\theta}(A_{\star}) > 0$ . We let  $L_{\pi_{\star}}$  denote an upper bound on the norm of the derivatives of  $\theta_{\star}(A)$ .
4 Optimal Exploration in Nonlinear Systems
In this work, we are interested in characterizing the instance-optimal rates of learning a controller  $\pi \in \Pi^{\star}$  which minimizes the loss  $\mathcal{J}(\pi; A_{\star})$ . The following result, a generalization of Proposition 8.2 of [2] to nonlinear systems, is the starting point of our analysis.
Proposition 1 (Informal). Under Assumptions 1, 2 and 4 to 6 and on the system (1.1), we have
$$
\mathcal {J} \left(\pi_ {\star} (\widehat {A}); A _ {\star}\right) - \mathcal {J} \left(\pi_ {\star} \left(A _ {\star}\right); A _ {\star}\right) = \| \operatorname {v e c} \left(A _ {\star} - \widehat {A}\right) \| _ {\mathcal {H} \left(A _ {\star}\right)} ^ {2} + \mathcal {O} ^ {\star} \left(\| A _ {\star} - \widehat {A} \| _ {\mathrm {F}} ^ {3}\right)
$$
$$
\mathcal {H} (A _ {\star}) := \nabla_ {A} ^ {2} \mathcal {J} (\pi_ {\star} (A); A _ {\star}) | _ {A = A _ {\star}}
$$
and where  $\mathcal{O}^{\star}(\cdot)$  hides factors polynomial in the regularity parameters of Assumptions 1 to 6.