Title: Optimal Exploration for Model-Based RL in Nonlinear Systems
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning

control theory

system identification

experiment design

active learning
Abstract: Learning to control unknown nonlinear dynamical systems is a fundamental problem in reinforcement learning and control theory. A commonly applied approach is to first explore the environment (exploration), learn an accurate model of it (system identification), and then compute an optimal controller with the minimum cost on this estimated system (policy optimization). While existing work has shown that it is possible to learn a uniformly good model of the system (Mania et al., 2020), in practice, if we aim to learn a good controller with a low cost on the actual system, certain system parameters may be significantly more critical than others, and we therefore ought to focus our exploration on learning such parameters.

In this work, we consider the setting of nonlinear dynamical systems and seek to formally quantify, in such settings, (a) which parameters are most relevant to learning a good controller, and (b) how we can best explore so as to minimize uncertainty in such parameters. Inspired by recent work in linear systems (Wagenmaker et al., 2021), we show that minimizing the controller loss in nonlinear systems translates to estimating the system parameters in a particular, task-dependent metric. Motivated by this, we develop an algorithm able to efficiently explore the system to reduce uncertainty in this metric, and prove a lower bound showing that our approach learns a controller at a near-instance-optimal rate. Our algorithm relies on a general reduction from policy optimization to optimal experiment design in arbitrary systems, and may be of independent interest. We conclude with experiments demonstrating the effectiveness of our method in realistic nonlinear robotic systems.

Optimal Exploration for Model-Based RL in Nonlinear Systems

Abstract

Learning to control unknown nonlinear dynamical systems is a fundamental problem in reinforcement learning and control theory. A commonly applied approach is to first explore the environment (exploration), learn an accurate model of it (system identification), and then compute an optimal controller with the minimum cost on this estimated system (policy optimization). While existing work has shown that it is possible to learn a uniformly good model of the system [1], in practice, if we aim to learn a good controller with a low cost on the actual system, certain system parameters may be significantly more critical than others, and we therefore ought to focus our exploration on learning such parameters.

In this work, we consider the setting of nonlinear dynamical systems and seek to formally quantify, in such settings, (a) which parameters are most relevant to learning a good controller, and (b) how we can best explore so as to minimize uncertainty in such parameters. Inspired by recent work in linear systems [2], we show that minimizing the controller loss in nonlinear systems translates to estimating the system parameters in a particular, task-dependent metric. Motivated by this, we develop an algorithm able to efficiently explore the system to reduce uncertainty in this metric, and prove a lower bound showing that our approach learns a controller at a near-instance-optimal rate. Our algorithm relies on a general reduction from policy optimization to optimal experiment design in arbitrary systems, and may be of independent interest. We conclude with experiments demonstrating the effectiveness of our method in realistic nonlinear robotic systems 1 .

1 Introduction

Controlling nonlinear dynamical systems is a core problem in robotics, cyber-physical systems, and beyond, and a significant body of work in both the control theory and reinforcement learning communities has sought to address this challenge [3-5]. In many real-world scenarios [6-9], the dynamics of the system of interest is unknown, or only a coarse model of them is available, which

Guanya Shi †

significantly increases the challenge of control-not only must we control such systems, we must learn to control them. While a variety of methods exist to address this challenge, a commonly applied approach is to first perform system identification , learning an accurate model of the system's dynamics, and then use this model to obtain a controller. Despite its promising potential, there are still several fundamental questions that must be answered to make this approach practically effective.

Which parameters are most relevant to learning a good controller? Beyond some special cases, little work has been done characterizing how the estimation error from system identification translates to end-to-end suboptimality in the resulting controller of our nonlinear systems. In particular, certain parameters of the system or regions of the state space may be irrelevant to learning a good controller, and coarse estimates of these parameters would suffice, while other parameters may be critical to learning a good controller, and we must therefore estimate these parameters very accurately in order to effectively control the system. In the context of this work, where nonlinearities are considered, the heterogeneity of the parameters is further accentuated. For instance, around a point of equilibrium, some system parameters might be completely inactive, having no impact on the dynamics (see the example in Section 1.1 for an illustration of this).

How can we best explore so as to minimize uncertainty in relevant parameters? Even if we are able to determine which parameters are most important for obtaining a good controller on the true system, it is not obvious how to use this information. How can we direct our system identification phase in order to focus on learning these parameters as quickly as possible, without spending time estimating the parameters of the system less critical for control? This is fundamentally a question of exploration . While it is known in linear systems that random excitation will efficiently explore [10], exploration in nonlinear systems is significantly more challenging since, in order to excite all parameters of interest, non-trivial planning may be required to ensure all relevant states are reached (as is the case in the example considered in Section 1.1).

We address both these questions in a particular class of nonlinear systems parameterized as:

Here x h ∈ R d x denotes the state of the system, u h ∈ U ⊆ R d u the input, w h ∼ N (0 , σ 2 w · I ) random noise, ϕ ( · , · ) ∈ R d ϕ a (possibly nonlinear, known) feature map, and A ⋆ ∈ R d x × d ϕ the (unknown) system parameter. Systems of this form are able to model a variety of real-world settings [11-15] 2 , and have been the subject of recent attention in the reinforcement learning community [1, 17, 14], yet the aforementioned questions have remained unanswered. Towards addressing this, in this work we make the following contributions:

For systems of the form (1.1), given some cost of interest which we wish to find a controller to minimize, we (a) formally characterize how estimation error translates into suboptimality in the learned controller, under the certainty equivalent control rule and (b) provide a lower bound on the loss of any (sufficiently regular) control rule learned from T rounds of interaction with (1.1).

Motivated by this characterization, we present an algorithm which achieves the instance-optimal rate, with controller loss matching our lower bound. To the best of our knowledge, this is the first statistically optimal algorithm in the setting of nonlinear dynamical systems. Our algorithm relies on a generic reduction from policy optimization to optimal exploration in arbitrary dynamical systems (not necessarily of the form (1.1)), which may be of independent interest.

We present numerical experiments on several realistic nonlinear systems which illustrate that our approach-efficiently exploring to reduce uncertainty in parameters most relevant to learning a controller-yields significant gains in practice.

To further motivate our approach, we consider the following example.

1.1 Motivating Example

To motivate the need for effective exploration, we consider a simple 1-D system with nonlinear dynamics given by:

where ϕ i ( x ) = max { 1 -100( x -c i ) 2 , 0 } for some c i . We choose a 1 = 0 . 8 , a 2 = 1 , and a 3 = . . . = a 12 = -3 . We assume a 1:12 are unknown, ( ϕ i ) 10 i =1 is known, and set

̸

With this choice of cost, the optimal controller will attempt to direct the state x to the equilibrium point c 1 and maintain this position. Note that, with our choice of ϕ i , ϕ i ( x ) = 0 only when x is very close to c i . This renders the parameters a 4:12 irrelevant to learning the optimal controller, since ϕ 2 , . . . , ϕ 10 will be inactive if we are playing optimally, but learning a 1:3 is critical to performing optimally. In particular, the coefficient of the first nonlinearity, a 3 , must be learned, as its value significantly changes the dynamics at the goal state.

We illustrate the result of running on this system in Figure 1, comparing our proposed approach ( Task-Driven Exploration , Algorithm 1) to the approach which chooses u h ∼ N (0 , σ 2 u ) ( Random Exploration ), and the approach proposed in [1] ( Uniform Exploration ) which seeks to explore so as to estimate a 1:12 uniformly well. As can be seen, neither of these latter two approaches are able to learn a good controller, while our approach easily finds a near-optimal controller. The failure modes of each of these approaches is somewhat different. Here Random Exploration fails since the chance of reaching the point x h ∼ c 1 is extremely small if the input is random noise-reaching c 1 requires playing a particular sequence of actions which are very unlikely to be played if u h is

chosen randomly. The Uniform Exploration approach does, in contrast, plan and, given enough time, is guaranteed to estimate all parameters accurately. However, as it aims to estimate all parameters uniformly well, it will attempt to estimate a 4:12 accurately despite their irrelevance to control, which will slow down the rate at which it is able to estimate a 3 . Only our approach, which both plans and takes into account the cost while exploring, is able to reach a 3 enough times to efficiently estimate it, and learn a good controller.

This example illustrates that it is critical both to explore efficiently, and also to let the objectivelearning a good controller-guide this exploration. We emphasize that the behavior in this example is only exhibited in nonlinear systems-though taking into account the task while exploring in linear systems is known to yield provable improvements [2], even playing random noise allows every direction to be learned in such systems. In nonlinear systems, however, this is not the case-one may fail to learn completely unless careful planning is performed.

2 Related Work

Learning for control. Recently, there has been increased interest in studying control problems from a learning-theoretic perspective, largely for linear system settings such as online LQR or LQG with unknown dynamics [18, 10, 19-26]. In the nonlinear setting, [27-29] provide formal guarantees on system identification in several different classes of nonlinear systems, yet they only consider noiseless systems, or systems that are significantly easier to excite than (1.1). [17] study systems of the form (1.1), but consider only the regret minimization problem. While their bounds would yield a polynomial complexity via an online-to-batch conversion, the resulting guarantee would scale at a O (1 / √ T ) rate, significantly slower than our O (1 /T ) rate. Several additional works in reinforcement with general function approximation encompass nonlinear systems of the form (1.1) [30-32], yet these results also achieve a slow O (1 / √ T ) rate. The most relevant work [1] proposes an active learning approach to identify unknown parameters in (1.1), with the goal of minimizing the Euclidean distance in the parameter space. However, they do not provide end-to-end guarantees on learning controllers and, as shown in Section 1.1, this approach could be significantly worse than learning a model with the goal task in mind. Also very related to our work is [2], which seeks to answer a similar set of questions: performing system identification in order to learn a good controller. This work is restricted to the setting of linear dynamics, however, and does not address the additional complexities of nonlinear systems.

System identification, dual control, and iterative learning control. There is a large body of classical work in system identification [7], and our work can be seen as an instance of active system

identification. While a variety of approaches have been proposed for active system identification [33-40], these tend to only consider linear systems, or lack rigorous theoretical guarantees. Recently deep learning approaches have also been applied in system identification [6, 8, 9, 41, 42]. In these works, the system identification phase is separate from the downstream controller design. Instead, in the control community, estimating parameters while simultaneously optimizing for performance has been formulated as a dual or iterative learning control problem [43-45], yet these settings focus on stability or asymptotic convergence whereas our work quantifies the end-to-end suboptimality gap.

Model-based reinforcement learning. This paper falls into the broad category of model-based reinforcement learning (MBRL), where an agent explores the environment to learn a model and then computes an optimal policy using the learned model. On the empirical side, deep MBRL has made exciting progress in many domains [46-48], and several task-aware methods have been designed to improve MBRL's performance [47-49], yet these works lack formal guarantees. On the theoretical side, a variety of different model-based approaches exist [50-55, 14]; however, the majority of these consider restricted settings such as tabular or linear MDPs. Of particular interest is the work of [14] which presents a result in systems of the form (1.1). While they show that polynomial sample complexity is possible, their guarantee scales at a O (1 /T 1 / 6 ) rate compared to our much faster O (1 /T ) rate.

Adaptive nonlinear control. Adaptive nonlinear control also seeks to control an unknown nonlinear system with parametric uncertainties [3, 4]. In particular, the key idea of model-reference adaptive control (MRAC) bears affinity to this paper, in that the adaptation law in MRAC adapts unknown parameters in a task-aware manner. There are two main differences between MRAC and our work. First, adaptive control does not explicitly optimize a cost function-the objective is typically tracking error convergence and Lyapunov stability, whereas our framework allows general cost functionsand the focus is typically on asymptotic convergence, while we give non-asymptotic optimality guarantees. Second, adaptive control has by and large been limited to specific system classes (e.g., fully-actuated systems [4, 15]) and policy classes (e.g., policy to directly cancel out the matched uncertainty [12, 13]), whereas our framework allows more general systems and policy classes.

3 Preliminaries

Notation. ∥ · ∥ op denotes the operator norm (matrix 2-norm), ∥ · ∥ F the Frobenius norm, and ∥ · ∥ M the Mahalanobis norm, defined as ∥ x ∥ M := √ x ⊤ M x for M ⪰ 0 . vec( A ) denotes the vectorization of matrix A . B p ( A ; r ) := { A ′ : ∥ A -A ′ ∥ p ≤ r } . [ H ] = { 1 , 2 , . . . , H } . We let E A [ · ] denote the expectation over trajectories induced on system with parameter A , and E A,π [ · ] the expectation induced when policy π is played. poly( · ) denotes some term that is polynomial in its arguments, with exponents absolute constants. We use ≲ informally to highlight key parameters in an inequality.

Setting. In this work, we are interested in systems of the form (1.1). We consider the episodic setting, where episodes are of length H , and assume that each episodes starts from a given state x 1 . Wealso assume ∥ A ⋆ ∥ op ≤ B A for some known B A > 0 . We note that the setting considered here encompasses many real-world systems of interest in robotics and control (e.g., [12, 14, 11, 15] and Section 6).

The goal of the learner is to find a policy (controller) π = ( π h ) H h =1 which achieves minimal cost on (1.1), for the cost defined by some (known) function (cost h ( · , · )) H h =1 , with cost h : R d x ×U → R + . For a given policy π , we define the expected cost on system A as

We consider the following interaction protocol:

Learner interacts with (1.1) for T episodes, at each episode playing a policy π exp ∈ Π exp .

After T episodes, the learner proposes a policy ̂ π T ∈ Π ⋆ .

The learner suffers cost J ( ̂ π T ; A ⋆ ) .

The goal of the learner is therefore first to explore and, after T episodes of exploration, to propose its best guess at the optimal controller for (1.1), ̂ π T . Here we take Π exp to be a (known) set of admissible exploration policies (for example, policies with bounded input power), and Π ⋆ a (known) set of admissible control policies. We assume that policies in Π ⋆ are deterministic, but allow for randomized policies in Π exp . Policies may be either open- or closed-loop.

System Notation. We let T denote the space of all possible state trajectories, T ⊆ R d x × ( H +1) , and, for any τ ∈ T , let τ 1: h denote the first h states and inputs in τ . For any policy π , we denote

the expected covariance induced by playing π on system A , Λ π := Λ A ⋆ ,π , and ˇ Λ := I d x ⊗ Λ the Kronecker product of I d x and Λ . Finally, we let Ω denote the convex hull of covariance matrices induced by Π exp , Ω := { E π ∼ ω [ Λ π ] : ω ∈ △ Π exp } , for △ Π exp the set of distributions over Π exp .

3.1 Regularity Assumptions

In order to make learning in (1.1) tractable, we need several regularity assumptions.

Assumption 1 (Bounded Features) . For all x ∈ R d x and u ∈ U , we have ∥ ϕ ( x , u ) ∥ 2 ≤ B ϕ .

Assumption 2 (Bounded Cost) . There exists some r cost ( A ⋆ ) > 0 such that, for all A ∈ B F ( A ⋆ ; r cost ( A ⋆ )) and all π ∈ Π ⋆ , we have E A,π [( ∑ H h =1 cost h ( x h , u h )) 2 ] ≤ L cost .

Assumption 3 (Uniform Feature Excitation) . There exists ω ∈ △ Π exp such that λ min ( E π exp ∼ ω [ Λ π exp ]) ≥ λ ⋆ min for some λ ⋆ min > 0 .

We remark that these assumptions have appeared before in work on systems of the form (1.1) [1, 17]. In particular, Assumption 3 implies that every direction of A ⋆ can, in principle, be excited, allowing it to be learned. In order to precisely quantify the optimal rates of learning, we require that our system satisfy certain smoothness assumptions. First, we require that ϕ ( · , · ) is differentiable in its second argument.

Assumption 4 (Smooth Nonlinearity) . For all x ∈ R d x and u ∈ U , ϕ ( x , u ) is four-times differentiable in u . Furthermore, ∥∇ ( i ) u ϕ ( x , u ) ∥ op ≤ L ϕ , ∀ i ∈ { 1 , 2 , 3 , 4 } , x ∈ R d x , and u ∈ U .

We also require that the class of admissible control policies, Π ⋆ , has a parametric form, Π ⋆ = { π θ : θ ∈ R d θ } , and that the parameterization is smooth in the following sense.

Assumption 5 (Smooth Controller Class) . π θ h ( τ 1: h ) is four-times differentiable in θ for all τ ∈ T and h ∈ [ H ] . Furthermore, ∥∇ ( i ) θ π θ h ( τ 1: h ) ∥ op ≤ L θ for ∀ i ∈ { 1 , 2 , 3 , 4 } , θ ∈ R d θ , and τ ∈ T .

Assumption 5 is satisfied for commonly considered classes of controllers, such as linear controllers, but is also satisfied by more complex classes such as neural network controllers. While the learner may propose any ̂ π T ∈ Π ⋆ , we are particularly interested in the certainty equivalence decision rule (i.e., the learner decides ̂ π T as if the estimated system is the actual one), defined as:

To ensure that π ⋆ ( A ) is well-defined and sufficiently regular, we make the following assumption.

Assumption 6 (Unique Optimal Controller) . We assume that the global minimum of J ( π θ ; A ⋆ ) , θ ⋆ ( A ⋆ ) , is unique, and that ∇ 2 θ J ( π θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) ≻ 0 .

In general, the policy optimization problem in (3.1) may not be computationally tractable. As we show in Appendix D, the globally optimal decision rule of (3.1) can be replaced with a locally optimal decision rule (i.e. π ⋆ ( A ) a local minimum of J ( π ; A ) ). Furthermore, Assumption 6 can be replaced by assuming the differentiability of θ ⋆ ( A ) with respect to A for A near A ⋆ . For ease of exposition, in the main text we assume that Assumption 6 holds and that π ⋆ ( A ) is defined as in (3.1). With these definitions and under Assumptions 1, 2, 4 and 5, we can show that J ( π θ ; A ⋆ ) is differentiable in θ and, combined with Assumption 6, that θ ⋆ ( A ) is differentiable in A , for A ∈ B F ( A ⋆ ; r θ ( A ⋆ )) and some r θ ( A ⋆ ) > 0 . We let L π ⋆ denote an upper bound on the norm of the derivatives of θ ⋆ ( A ) .

4 Optimal Exploration in Nonlinear Systems

In this work, we are interested in characterizing the instance-optimal rates of learning a controller π ∈ Π ⋆ which minimizes the loss J ( π ; A ⋆ ) . The following result, a generalization of Proposition 8.2 of [2] to nonlinear systems, is the starting point of our analysis.

Proposition 1 (Informal) . Under Assumptions 1, 2 and 4 to 6 and on the system (1.1) , we have

for

and where O ⋆ ( · ) hides factors polynomial in the regularity parameters of Assumptions 1 to 6.

The quantity H ( A ⋆ ) := ∇ 2 A J ( π ⋆ ( A ); A ⋆ ) | A = A ⋆ , referred to as the model-task Hessian in [2], corresponds to the curvature of the loss of the certainty-equivalence controller π ⋆ ( A ) around A ← A ⋆ . It precisely quantifies how estimation error in each coordinate of A ⋆ translates into suboptimality of the controller-providing an answer to our question of which parameters are most relevant to learning a good controller-and reduces the problem of minimizing the controller loss to estimating A ⋆ in a particular norm. The following result gives a bound on this estimation error, ∥ vec( A ⋆ -̂ A ) ∥ 2 H ( A ⋆ ) .

Proposition 2 (Informal) . Consider interacting with (1.1) for T episodes, and let Λ T = ∑ T t =1 ∑ H h =1 ϕ ( x t h , u t h ) ϕ ( x t h , u t h ) ⊤ denote the observed covariates and

the least-squares estimate of A ⋆ . Recalling that ˇ Λ T = I d x ⊗ Λ T , we have, with high probability:

4.1 Algorithm and Upper Bound

Proposition 2 motivates our algorithmic approach: explore to collect covariates Λ T minimizing tr( H ( A ⋆ ) ˇ Λ -1 T ) . There are two primary challenges to achieving this: we do not know H ( A ⋆ ) , as it depends on the (unknown) parameter A ⋆ and, even if we did know H ( A ⋆ ) , it is not clear how to explore so as to collect data minimizing tr( H ( A ⋆ ) ˇ Λ -1 T ) . We address both of these challenges in Algorithm 1.

Algorithm 1 proceeds in epochs of exponentially increasing length. At each epoch it first approximates H ( A ⋆ ) by computing the model-task Hessian of the estimated system, ̂ A ℓ . Using this approximatiom of H ( A ⋆ ) , it seeks to explore to minimize tr( H ( ̂ A ℓ ) ˇ Λ -1 T ) . This exploration routine is encapsulated in the DYNAMICOED (dynamic optimal experiment design) function, an adaptive experiment-design routine inspired by recent work in reinforcement learning [56] and described in more detail in Section 5. DYNAMICOED returns a set of exploration policies, Π ℓ , which we run to collect data D ℓ . As we will show, the collected covariates, Λ ℓ := ∑ H h =1 ∑ ( u h , x h ) ∈ D ℓ ϕ ( x h , u h ) ϕ ( x h , u h ) ⊤ , satisfy

which implies that DYNAMICOED collects data minimizing tr( H ( ̂ A ℓ ) ˇ Λ -1 ℓ ) at a near-optimal rate. Given the data D ℓ , we form the least-squares estimate of A ⋆ , ̂ A ℓ +1 , and the process repeats. After running for T episodes, the certainty-equivalence controller on the last estimate obtained, ̂ π T = π ⋆ ( ̂ A ℓ T +1 ) , is returned. The following result bounds the suboptimality of ̂ π T as compared to π ⋆ ( A ⋆ ) .

Theorem 1. Under Assumptions 1 to 6, if T ≥ C poly · max { 1 , r cost ( A ⋆ ) -2 , r θ ( A ⋆ ) -2 } , then with probability at least 1 -δ , Algorithm 1 explores with policies in Π exp at every episode, runs for at most T episodes, and returns ̂ π T ∈ Π ⋆ satisfying:

where we recall Ω is the set of possible expected covariates on (1.1) , C is a universal constant, and

Theorem 1 shows that Algorithm 1 is able to explore so as to optimally minimize the exploration loss tr( H ( A ⋆ ) ˇ Λ -1 T ) , up to a lower-order term scaling as T -3 / 2 and polynomially in system parameters. While Propositions 1 and 2 show that collecting data which minimizes tr( H ( A ⋆ ) ˇ Λ -1 T ) is in some sense fundamental, it is not clear it is necessary. We next show that it is indeed necessary.

4.2 Lower Bounds on Learning Controllers

Our goal is to show that, up to constants and lower-order terms, the bound given in Theorem 1 is not improvable, regardless of which controller estimate we use. To obtain such lower bounds, we need several additional assumptions. In particular, we require that the loss J ( π θ ; A ) grows quadratically in the distance θ is from θ ⋆ ( A ) , and strengthen Assumption 3 to ensure (1.1) is sufficiently easy to excite. Formal statements of these conditions are given in Appendix F. Our lower bound is as follows.

Theorem 2 (Informal) . Under Assumptions 1 to 6 and the additional regularity assumptions mentioned above, as long as T ≥ C lb , for any ω exp ∈ △ Π exp , we have

for B T := B F ( A ⋆ ; O ( T -5 / 6 )) , E D T ∼ A,ω exp [ · ] = E π exp ∼ ω exp [ E D T ∼ A,π exp [ · ]] the expectation over trajectories generated by running policies π ∼ ω exp on system A for T episodes, ̂ π any mapping from observations to policies in Π ⋆ , and C lb some value scaling polynomially in problem parameters.

Note that this lower bound holds for any A ⋆ and mapping ϕ , as long as our assumptions are met. Up to constants and lower-order terms, the scaling of Theorem 2 matches that of Theorem 1-both scale with min Λ ∈ Ω tr( H ( A ⋆ ) ˇ Λ -1 ) -which implies that Algorithm 1 is indeed optimal (under certain additional regularity conditions). To the best of our knowledge, this is the first result characterizing the optimal statistical rates for learning in nonlinear dynamical systems. We emphasize that Theorem 2 holds for any decision rule ̂ π -it does not require that we use the certainty equivalence decision rule. As Algorithm 1 does rely on certainty equivalence, this result also implies that the certainty equivalence decision rule is optimal for (certain classes of) nonlinear dynamical systems.

The proof of Theorem 2 builds on the work [2], which shows a similar result for linear dynamical systems. It critically relies on our quadratic decomposition of the controller loss in Proposition 1, which reduces the problem of obtaining a lower bound on controller loss to a lower bound on estimating A ⋆ in the H ( A ⋆ ) norm. Given this, the result can be obtained by applying lower bounds on regression in general norms.

5 Optimal Experiment Design in Arbitrary Dynamical Systems

We turn now to the DYNAMICOED routine, which is the key algorithmic tool we use to prove Theorem 1. DYNAMICOED is a general reduction from policy optimization to optimal experiment design in arbitrary dynamical systems, and is an extension of a recently proposed approach for experiment design in linear MDPs [56]. This section may be of independent interest.

To illustrate the generality of this reduction, in this section we consider the following system:

where x h ∈ X ⊆ R d x denotes the state, u h ∈ U ⊆ R d u the input, and w h ∈ R d w the noise. We take the dynamics ( f h ) H h =1 to be unknown and arbitrary. We assume there is some known featurization of

our system that is of interest, ϕ ( x , u ) → R d ϕ , and an experiment design object on this featurization, Φ : R d ϕ × d ϕ → R . Our goal is to collect some set of trajectories { τ t } T t =1 which minimizes Φ :

As an example, if Φ( Λ ) = log det( Λ ) , this reduces to D -optimal design, and if Φ( Λ ) = tr( H· Λ -1 ) , the setting considered in Section 4, this reduces to weighted A -optimal design. As before, we assume we have access to some set of exploration policies Π exp , and define Λ π and Ω as in Section 3, but with respect to this new feature map ϕ and system (5.1). We also define ̂ Ω to be the space of all possible covariance matrices:

To facilitate efficient experiment design in this setting, we will make the following assumption on Φ . Assumption 7 (Regularity of Φ ) . Φ is regular in the following sense:

Φ is convex, differentiable, and β -smooth in the norm ∥ · ∥ (with dual-norm ∥ · ∥ ∗ ):

There exists some M < ∞ satisfying sup Λ ∈ ̂ Ω sup x ∈X , u ∈U | ϕ ( x , u ) ⊤ ∇ Λ Φ( Λ ) ϕ ( x , u ) | ≤ M.

The key algorithmic assumption we make is access to a regret minimization oracle on (5.1).

Assumption 8 (Regret Minimization Oracle) . Let cost h ( x , u ) = ϕ ( x , u ) ⊤ Q h ϕ ( x , u ) for some Q h ∈ R d ϕ × d ϕ such that | ∑ h cost h ( x h , u h ) | ≤ 1 for all x h ∈ X , u h ∈ U . We assume we have access to some learner A R which is able to achieve low regret on costs { cost h ( · , · ) } H h =1 with respect to policy class Π exp . That is, with probability at least 1 -δ :

for some C R > 0 , p R > 0 , and α ∈ (0 , 1) , and where π t is the policy A R plays at episode t .

Note that the regret minimization algorithm satisfying Assumption 8 may be arbitrary. For example, for linear systems, we could apply provably efficient algorithms for the Linear Quadratic Regulator [25, 20]; for nonlinear systems of the form (1.1) we could apply the LC 3 algorithm of [17]; for more general settings of reinforcement learning with function approximation, algorithms such as BILIN-UCB [30] or E2D [31] could be applied. In practice, though they may not formally satisfy the guarantee of Assumption 8, deep RL approaches could be used. We have the following result.

Theorem 3. Fix T > 0 and denote R := sup Λ , Λ ′ ∈ ̂ Ω ∥ Λ -Λ ′ ∥ . Under Assumption 7, and assuming we have access to a learner A R satisfying Assumption 8 with α = 1 / 2 , DYNAMICOED runs for T episodes on (5.1) , and with probability at least 1 -δ collects data { ( x t h , u t h ) } h ∈ [ H ] ,t ∈ [ T ] satisfying

where ϕ t h := ϕ ( x t h , u t h ) , and we recall Ω is the set of possible expected covariates on (5.1) .

Theorem 3 shows that, given access only to a regret minimization oracle, it is possible to solve experiment design problems on arbitrary dynamical systems. The requirement that α = 1 / 2 is for expositional purposes only-we generalize this result to arbitrary α (and more general feature maps) in Appendix C. Under certain conditions, it can be shown that, if the exploration policies DYNAMICOED runs to collect D are rerun , the newly collected data satisfies a similar guarantee as Theorem 3. This lets us run DYNAMICOED to learn an approximate solution of min Λ ∈ Ω Φ( Λ ) , and then rerun the learned policies as many times as desired to collect additional data approximately minimizing Φ .

5.1 Overview of DYNAMICOED Algorithm

DYNAMICOED is inspired by recent work on experiment design in reinforcement learning [57, 58, 56, 59], and can be seen as an extension of the FWREGRET algorithm of [56] to arbitrary systems. We

refer the reader to [56] for a more in-depth discussion of the FWREGRET algorithm, and briefly sketch its extension to arbitrary systems here (see Appendix C and Algorithm 4 for precise definitions).

Conceptually, DYNAMICOED runs a variant of conditional gradient descent on the objective Φ( Λ ) . At each iteration, n , it computes the gradient of the loss at the current iterate, Ξ n ← ∇ Λ Φ( Λ ) | Λ = Λ n -1 . To run a standard gradient descent algorithm on this objective, we would simply update Λ n -1 by taking a step in the direction Ξ n . However, our objective is to minimize Φ over the constraint set, Ω . Thus, rather than taking a step in the direction Ξ n , we wish to take a step in the direction of steepest descent within the constraint set .

The challenge is that the constraint set in our setting, Ω , is unknown , as it depends on the expectation over trajectories induced on the unknown dynamics ( f h ) H h =1 , and therefore we cannot directly compute this steepest descent direction. The key observation is that the computation of this steepest descent direction is equivalent to solving:

This is simply a policy optimization problem, however, and can be solved approximately by A R under Assumption 8. Thus, in the call to A R on Line 6, we approximate the steepest descent direction, and on Line 7 update Λ n -1 in this direction. Convergence of this procedure to the optimal value, min Λ ∈ Ω Φ( Λ ) , can then be shown by the standard analysis of conditional gradient descent. We remark that, under Assumption 7 and Assumption 8, this argument is completely generic and does not require that our system, (5.1), exhibit any additional properties.

5.2 From Theorem 3 to Theorem 1

In Algorithm 1, our goal is to collect covariates, ˇ Λ -1 T ℓ , on (1.1) such that tr( H ( ̂ A ℓ ) ˇ Λ -1 T ℓ ) is as small as possible. To achieve this, we apply DYNAMICOED to the dynamics (1.1) and objective Φ ℓ ( Λ ) = tr( H ( ̂ A ℓ ) ˇ Λ -1 ) , with Assumption 8 instantiated by the LC 3 algorithm of [17]. By the guarantee given in Theorem 3, after running for a number of episodes N which scales polynomially in problem parameters, DYNAMICOED will collect covariates Λ N such that Φ ℓ ( 1 N Λ N ) ≤ 2 · min Λ ∈ Ω Φ ℓ ( Λ ) , which implies tr( H ( ̂ A ℓ ) ˇ Λ -1 N ) ≤ 2 N · min Λ ∈ Ω tr( H ( ̂ A ℓ ) ˇ Λ -1 ) . By rerunning the policies DYNAMICOED used to collect this Λ N for T ℓ /N additional times, we can ensure tr( H ( ̂ A ℓ ) ˇ Λ -1 T ℓ ) ≲ 1 T ℓ · min Λ ∈ Ω tr( H ( ̂ A ℓ ) ˇ Λ -1 ) as desired.

We remark that the LC 3 algorithm requires access to a computation oracle. As the focus of this work is primarily statistical, we leave addressing this computational challenge for future work. Furthermore, as we show in the following section, computationally efficient, sampling-based implementations of our approach are very effective in practice. We remark as well that the objective we ultimately care about minimizing is tr( H ( A ⋆ ) ˇ Λ -1 ) . As we show, by including a small amount of uniform exploration, we can ensure that the suboptimality incurred optimizing tr( H ( ̂ A ℓ ) ˇ Λ -1 ) instead of tr( H ( A ⋆ ) ˇ Λ -1 ) only contributes to the lower-order terms of the final guarantee in Theorem 1.

6 Experimental Results

Finally, we demonstrate the effectiveness of our proposed approach (Algorithm 1, the Task-Driven Exploration method in Figures 1 to 3) on several systems motivated by robotic applications. We

70

Task-DrivenExploration(Ours)

Mean Excess Controller Loss

RandomExploration

UniformExploration

40

20

10

10

15

20

25

30

35

40

NumberofEpisodes

6000

Task-Driven Exploration(Ours)

ControllerLoss

RandomExploration

5000

UniformExploration

4000

MeanExcess

3000

2000

1000

100

200

300

400

500

600

NumberofEpisodes

compare Algorithm 1 with an approach that plays u h ∼ N (0 , σ 2 u · I ) ( Gaussian Exploration ), and an approach inspired by [1] ( Uniform Exploration ), which seeks to estimate A ⋆ uniformly well, playing inputs that reduce ∥ ̂ A -A ⋆ ∥ op .

To benchmark the performance of these approaches, we consider an affine system with dynamics corresponding to that of a simplified 3-D drone (i.e., 3-D double integrator with a gravity term), and a nonlinear system with dynamics corresponding to that of a 2-D car. For both systems, we choose H = 50 , and plot the value of J ( ̂ π t ; A ⋆ ) -J ( π ⋆ ( A ⋆ ); A ⋆ ) for ̂ π t the certainty-equivalence controller computed on the estimate of the system obtained at time t . For the drone, we let Π ⋆ be the class of linear-affine feedback controllers, and for the car, Π ⋆ is a set of nonlinear controllers with dimension 4. While the optimal controller for the drone can be computed in closed-form, for the car we rely on a sampling-based routine to find an approximately optimal controller. The model-task hessian H ( ̂ A ℓ ) is computed via automatic differentiation. For the exploration policies of Task-Driven Exploration and Uniform Exploration , Π exp , we rely on MPC-style sampling based methods. For all approaches, we require that E A,π exp [ ∑ H h =1 ∥ u h ∥ 2 2 ] ≤ γ 2 for some γ 2 > 0 and all π exp ∈ Π exp . On all examples, we implement DYNAMICOED with A R a posterior samplinginspired version of the LC 3 of [17]. Figures 1 and 3 shows performance averaged over 100 trials, and Figure 2 over 200 trials. Additional experimental details can be found in Appendix G.

As illustrated in Figures 1 to 3, our approach yields a non-trivial gain over existing approaches on all systems. In particular, in Figures 2 and 3 it improves on the sample complexity of existing approaches by roughly a factor of 2-for example, in the drone system, reaching excess controller cost of 10 after less than 20 episodes, as compared to over 40 episodes for existing approaches.

Our implementation is very modular, and any piece (for example, the parameterization of Π exp and Π ⋆ , the policy optimizer, or the exploration routine) can be easily replaced with other procedures. Our results highlight that, even when using, for example, a possibly suboptimal policy optimizer, exploring so as to minimize uncertainty in the model-task hessian yields a non-trivial gain. We expect that this would hold true regardless of the policy optimizer used-the model-task hessian will adapt to the structure of the policy optimizer. Integration of our approach with deep model-based RL approaches is an interesting direction for future work, but we believe the approach will scale to these settings as well.

7 Conclusion

In this work, we have characterized the instance-optimal rate of learning controllers in nonlinear dynamical systems. To the best of our knowledge, this is the first work to obtain an optimal sample complexity for learning in nonlinear dynamical systems. Furthermore, our experimental results demonstrate the effectiveness of our proposed algorithm in realistic nonlinear systems. This work opens the door for several interesting directions for future work. First, it is not clear that the assumption on uniform feature excitation, Assumption 3, is necessary. Can this be removed with a more refined analysis (or shown to be necessary)? Second, while in many settings ϕ is known or can be effectively represented by random features, in some settings it is helpful to learn it [12]. Can we obtain end-to-end guarantees on learning both ϕ and A ⋆ ? Finally, it is of much interest to extend our experimental results to larger-scale systems for real-world deployment.

References

Horia Mania, Michael I Jordan, and Benjamin Recht. Active learning for nonlinear system identification with guarantees. J. Mach. Learn. Res. , 23:32-1, 2022.

Andrew J Wagenmaker, Max Simchowitz, and Kevin Jamieson. Task-optimal exploration in linear dynamical systems. In International Conference on Machine Learning , pages 1064110652. PMLR, 2021.

Jean-Jacques E Slotine, Weiping Li, et al. Applied nonlinear control , volume 199. Prentice hall Englewood Cliffs, NJ, 1991.

Karl J ˚ Astr¨ om and Bj¨ orn Wittenmark. Adaptive control . Courier Corporation, 2013.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Guanya Shi, Xichen Shi, Michael O'Connell, Rose Yu, Kamyar Azizzadenesheli, Animashree Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural lander: Stable drone landing control using learned dynamics. In 2019 International Conference on Robotics and Automation (ICRA) , pages 9784-9790. IEEE, 2019.

Lennart Ljung. System identification . Springer, 1998.

Duy Nguyen-Tuong and Jan Peters. Model learning for robot control: a survey. Cognitive processing , 12:319-340, 2011.

Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems , 5:411-444, 2022.

Max Simchowitz, Horia Mania, Stephen Tu, Michael I Jordan, and Benjamin Recht. Learning without mixing: Towards a sharp analysis of linear system identification. arXiv preprint arXiv:1802.08334 , 2018.

Guanya Shi, Kamyar Azizzadenesheli, Michael O'Connell, Soon-Jo Chung, and Yisong Yue. Meta-adaptive nonlinear control: Theory and algorithms. Advances in Neural Information Processing Systems , 34:10013-10025, 2021.

Michael O'Connell, Guanya Shi, Xichen Shi, Kamyar Azizzadenesheli, Anima Anandkumar, Yisong Yue, and Soon-Jo Chung. Neural-fly enables rapid learning for agile flight in strong winds. Science Robotics , 7(66):eabm6597, 2022.

Nicholas M Boffi, Stephen Tu, and Jean-Jacques E Slotine. Regret bounds for adaptive nonlinear control. In Learning for Dynamics and Control , pages 471-483. PMLR, 2021.

Yuda Song and Wen Sun. Pc-mlp: Model-based reinforcement learning with policy cover guided exploration. In International Conference on Machine Learning , pages 9801-9811. PMLR, 2021.

Spencer M Richards, Navid Azizan, Jean-Jacques Slotine, and Marco Pavone. Adaptive-controloriented meta-learning for nonlinear systems. arXiv preprint arXiv:2103.04490 , 2021.

Ali Rahimi and Benjamin Recht. Uniform approximation of functions with random bases. In 2008 46th annual allerton conference on communication, control, and computing , pages 555-561. IEEE, 2008.

Sham Kakade, Akshay Krishnamurthy, Kendall Lowrey, Motoya Ohnishi, and Wen Sun. Information theoretic regret bounds for online nonlinear control. Advances in Neural Information Processing Systems , 33:15312-15325, 2020.

Yasin Abbasi-Yadkori, D´ avid P´ al, and Csaba Szepesv´ ari. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24:2312-2320, 2011.

Max Simchowitz, Ross Boczar, and Benjamin Recht. Learning linear dynamical systems with semi-parametric least squares. arXiv preprint arXiv:1902.00768 , 2019.

Horia Mania, Stephen Tu, and Benjamin Recht. Certainty equivalence is efficient for linear quadratic control. Advances in Neural Information Processing Systems , 32, 2019.

Alon Cohen, Tomer Koren, and Yishay Mansour. Learning linear-quadratic regulators efficiently with only √ T regret. arXiv preprint arXiv:1902.06223 , 2019.

Sarah Dean, Horia Mania, Nikolai Matni, Benjamin Recht, and Stephen Tu. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics , 20(4): 633-679, 2020.

Chenkai Yu, Guanya Shi, Soon-Jo Chung, Yisong Yue, and Adam Wierman. The power of predictions in online control. Advances in Neural Information Processing Systems , 33: 1994-2004, 2020.

Andrew Wagenmaker and Kevin Jamieson. Active learning for identification of linear dynamical systems. In Conference on Learning Theory , pages 3487-3582. PMLR, 2020.

Max Simchowitz and Dylan Foster. Naive exploration is optimal for online lqr. In International Conference on Machine Learning , pages 8937-8948. PMLR, 2020.

Max Simchowitz, Karan Singh, and Elad Hazan. Improper learning for non-stochastic control. In Conference on Learning Theory , pages 3320-3436. PMLR, 2020.

Dylan Foster, Tuhin Sarkar, and Alexander Rakhlin. Learning nonlinear dynamical systems from a single trajectory. In Learning for Dynamics and Control , pages 851-861. PMLR, 2020.

Samet Oymak. Stochastic gradient descent learns state equations with nonlinear activations. In conference on Learning Theory , pages 2551-2579. PMLR, 2019.

Yahya Sattar and Samet Oymak. Non-asymptotic and accurate learning of nonlinear dynamical systems. The Journal of Machine Learning Research , 23(1):6248-6296, 2022.

Simon Du, Sham Kakade, Jason Lee, Shachar Lovett, Gaurav Mahajan, Wen Sun, and Ruosong Wang. Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning , pages 2826-2836. PMLR, 2021.

Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

Zixiang Chen, Chris Junchi Li, Angela Yuan, Quanquan Gu, and Michael I Jordan. A general framework for sample-efficient function approximation in reinforcement learning. arXiv preprint arXiv:2209.15634 , 2022.

Raman Mehra. Optimal input signals for parameter estimation in dynamic systems-survey and new results. IEEE Transactions on Automatic Control , 19(6):753-768, 1974.

L´ aszl´ o Gerencs´ er and H˚ akan Hjalmarsson. Adaptive input design in system identification. In Proceedings of the 44th IEEE Conference on Decision and Control , pages 4988-4993. IEEE, 2005.

Dimitrios Katselis, Cristian R Rojas, H˚ akan Hjalmarsson, and Mats Bengtsson. Applicationoriented finite sample experiment design: A semidefinite relaxation approach. IFAC Proceedings Volumes , 45(16):1635-1640, 2012.

Ian R Manchester. Input design for system identification via convex relaxation. In 49th IEEE Conference on Decision and Control (CDC) , pages 2041-2046. IEEE, 2010.

Cristian R Rojas, James S Welsh, Graham C Goodwin, and Arie Feuer. Robust optimal experiment design for system identification. Automatica , 43(6):993-1008, 2007.

Graham Clifford Goodwin and Robert L Payne. Dynamic system identification: experiment design and data analysis . Academic press, 1977.

Kristian Lindqvist and H˚ akan Hjalmarsson. Identification for control: Adaptive input design using convex optimization. In Proceedings of the 40th IEEE Conference on Decision and Control (Cat. No. 01CH37228) , volume 5, pages 4326-4331. IEEE, 2001.

L´ aszl´ o Gerencs´ er, Jonas M˚ artensson, and H˚ akan Hjalmarsson. Adaptive input design for arx systems. In 2007 European Control Conference (ECC) , pages 5707-5714. IEEE, 2007.

Grady Williams, Nolan Wagener, Brian Goldfain, Paul Drews, James M Rehg, Byron Boots, and Evangelos A Theodorou. Information theoretic mpc for model-based reinforcement learning. In 2017 IEEE International Conference on Robotics and Automation (ICRA) , pages 1714-1721. IEEE, 2017.

Guanya Shi, Wolfgang H¨ onig, Xichen Shi, Yisong Yue, and Soon-Jo Chung. Neural-swarm2: Planning and control of heterogeneous multirotor swarms using learned interactions. IEEE Transactions on Robotics , 38(2):1063-1079, 2021.

Aleksandr Aronovich Feldbaum. Dual control theory. i. Avtomatika i Telemekhanika , 21(9): 1240-1249, 1960.

Ali Mesbah. Stochastic model predictive control with active uncertainty learning: A survey on dual control. Annual Reviews in Control , 45:107-117, 2018.

Douglas A Bristow, Marina Tharayil, and Andrew G Alleyne. A survey of iterative learning control. IEEE control systems magazine , 26(3):96-114, 2006.

Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Modelbased reinforcement learning for atari. arXiv preprint arXiv:1903.00374 , 2019.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129-14142, 2020.

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems , 31, 2018.

Yashwanth Kumar Nakka, Anqi Liu, Guanya Shi, Anima Anandkumar, Yisong Yue, and SoonJo Chung. Chance-constrained trajectory optimization for safe exploration and learning of nonlinear systems. IEEE Robotics and Automation Letters , 6(2):389-396, 2020.

Ian Osband and Benjamin Van Roy. Model-based reinforcement learning and the eluder dimension. Advances in Neural Information Processing Systems , 27, 2014.

Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-based rl in contextual decision processes: Pac bounds and exponential improvements over model-free approaches. In Conference on learning theory , pages 2898-2933. PMLR, 2019.

Alekh Agarwal, Sham Kakade, and Lin F Yang. Model-based reinforcement learning with a generative model is minimax optimal. In Conference on Learning Theory , pages 67-83. PMLR, 2020.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning , pages 7304-7312. PMLR, 2019.

Mohammad Gheshlaghi Azar, Ian Osband, and R´ emi Munos. Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning , pages 263-272. PMLR, 2017.

Andrew Wagenmaker and Kevin Jamieson. Instance-dependent near-optimal policy identification in linear mdps via online experiment design. arXiv preprint arXiv:2207.02575 , 2022.

Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning , pages 2681-2691. PMLR, 2019.

Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. Advances in Neural Information Processing Systems , 34:25746-25759, 2021.

Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning. arXiv preprint arXiv:2211.04974 , 2022.

Jean Dieudonn´ e. Foundations of modern analysis . Read Books Ltd, 2011.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

A Technical Tools

Additional Notation. We let S d -1 refer to the unit ball in d dimensions and S d + (resp. S d ++ ) the set of positive semi-definite matrices (resp. positive definite matrices) in R d × d . Throughout, O ( · ) denotes standard big-O notation, ˜ O ( · ) hides additional logarithmic factors. poly( · ) denotes some term that is polynomial in its arguments, with exponents absolute constants. △ X denotes the set of distributions over set X .

Lemma A.1. Let w i ∼ N (0 , I d x ) for all i ∈ { 1 , 2 , . . . , n } . Then,

for c an absolute constant.

Proof. We first bound

The result then follows since we can bound the E [ ∥ w i ∥ c 2 ] ≤ poly( d ) for w i ∼ N (0 , I ) and c an absolute constant.

Lemma A.2 (Lemma I.4 of [2]) . Assume A,B ∈ S d ++ , ∥ A -B ∥ op ≤ ϵ , and ϵ < λ min ( B ) . Then

A.1 Martingale Regression in General Norms

For the following two results, we consider the martingale regression setting of [2] (referred to as the MDM setting). In particular, we consider observations of the form

for y t ∈ R , unknown parameter µ ⋆ ∈ R d µ , w t | F t -1 ∼ N (0 , σ 2 w ) , and z t F t -1 -measurable, for a filtration ( F t ) t ≥ 1 . This setting therefore encompasses general stochastic processes where the observations are linear-the evolution of z t could be arbitrary.

Weconsider the setting where we interact with (A.1) for T steps, collecting observations { ( z t , y t ) } T t =1 , and then form the least-squares estimate of µ ⋆ :

We also denote Σ T := ∑ T t =1 z t z ⊤ t . The following results characterize the estimation error of ̂ µ in the M -norm and 2-norm.

̸

Proposition 3 (Theorem 7.2 of [2]) . Fix any matrices Γ ∈ S d µ ++ , M ∈ S d µ + , with M = 0 . Given a parameter β ∈ (0 , 1 / 4) , define the event

Then, if E holds, the following holds with probability at least 1 -δ :

where ζ = 26 β 2 λ max ( Γ )tr( Γ -1 ) .

Proposition 4 (Lemma E.1 of [2]) . On the event

then we have that with probability at least 1 -δ :

A.1.1 Connection Between (1.1) and (A.1)

We will apply the results Proposition 3 and Proposition 4 in the setting of (1.1) in order to obtain estimation bounds on A ⋆ . As the setting of (1.1) has vector observations, we briefly describe here how it can be mapped into the setting described above.

Recall that (1.1) evolves as

for x h ∈ R d x , ϕ ( x , u ) ∈ R d ϕ , and A ⋆ ∈ R d x × d ϕ . We assume that x 1 is some fixed starting state. Assume that we have run for T episodes, and collected observations { ( x t 1 , u t 1 , x t 2 , . . . , x t h , u t h , x t h +1 ) } T t =1 . Now let µ ⋆ := vec( A ⋆ ) . Furthermore, for any t, h , and i ∈ [ d x ] , let t = ( t, h, i ) and z t = [ 0 d ϕ ( i -1) , ϕ ( x t h , u t h ) , 0 d ϕ ( d x -i ) ] ∈ R d x d ϕ where 0 d denotes the zero vector of length d . Then we see that

Setting y t = [ x t h +1 ] i and w t = [ w t h ] i , it is clear that this follows the observation model of (A.1) with d µ = d ϕ d x and T = d x TH . It is also straightforward to see that the measurability assumptions of the setting of (A.1) are satisfied by this.

B Proof of Main Result

Algorithm 3 Optimal Exploration in Nonlinear Systems (Full Version of Algorithm 1)

1: inputs: number of episodes to run T , cost function (cost h ) H h =1 , confidence δ , control policies Π ⋆ , exploration policies Π exp

2: ̂ A 1 ← anything, ℓ T ←⌈ log 2 T/ 8 ⌉

3: for ℓ = 1 , 2 , 3 , . . . , ℓ T do

4: T ℓ ← 2 ℓ , δ ℓ ← δ/ 12 ℓ 2

5: Compute estimate of cost matrix: H ℓ ←H ( ̂ A ℓ )

6: Π ℓ ← LEARNEXP Π( H ℓ , T ℓ , δ ℓ , A R , Π exp ) (Algorithm 7), with A R the LC 3 algorithm [17]

7: Rerun each policy in Π ℓ N ℓ = ⌈ T ℓ / | Π ℓ |⌉ times, denote collected data D ℓ

8: Estimate system parameters

9:

Theorem 4 (Full Version of Theorem 1) . Assume Assumptions 1 to 5 and 13 hold. Then if

with probability at least 1 -δ , Algorithm 3 plays exploration policies π exp ∈ Π exp at every episode, runs for at most T episodes, and the controller ̂ π T returned Algorithm 3 satisfies, with probability at least 1 -δ :

for C a universal constant and

Proof. Let E ℓ denote that the good event of Lemma B.3 holds at round ℓ which, by Lemma B.3, occurs with probability at least 1 -6 δ ℓ . By our setting of δ ℓ = δ/ 12 ℓ 2 , we have that the total failure

probability of E ℓ for all ℓ is bounded as

Henceforth we assume that E := ∩ ℓ E ℓ holds. Let ̂ A := ̂ A ℓ T +1 , and ̂ A -:= ̂ A ℓ T .

Before proceeding to the main proof, we note that the conclusion that Algorithm 3 only explores with policies in Π exp follows from the definition of LEARNEXP Π and LC 3 . Note that LEARNEXP Π only interacts with (1.1) through calls to DYNAMICOED, which itself only interacts with (1.1) through calls to A R , instantiated in Algorithm 3 by LC 3 . Inspection of the LC 3 algorithm in [17] reveals that LC 3 only interacts with (1.1) by playing policies in Π exp , from which the conclusion follows.

Bounding the Number of Episodes. Denote T oed ℓ = | Π ℓ | . Note that by construction we always have N ℓ T oed ℓ ≥ T ℓ . By Lemma B.2, as long as (B.1) is met, we can bound the total number of episodes collected up to and including round ℓ by 4 T ℓ for ℓ ∈ { ℓ T , ℓ T -1 } . We therefore, in the following, will make use of the fact that

for ℓ ∈ { ℓ T , ℓ T -1 } and absolute constants c, c ′ . Furthermore, it also follows from this that the total number of episodes run by Algorithm 3 is bounded by

so Algorithm 3 runs for at most T episodes.

Approximating the Controller Loss. Let r est ( A ⋆ ) := min { 1 , r cost ( A ⋆ ) , r θ ( A ⋆ ) } , for r cost ( A ⋆ ) as in Assumption 2 and r θ ( A ⋆ ) as in Assumption 13. By Lemma D.2, under Assumptions 1, 2, 4, 5 and 13, as long as ̂ A ∈ B F ( A ⋆ ; r est ( A ⋆ )) , we have

Furthermore, we can bound

Bounding the Hessian Estimation Error. On E , by Lemma B.3 and as long as (B.1) is met, we have (note that at the final epoch, the plug-in estimator H ( ̂ A -) is given as input to DYNAMICOED):

where the last line uses that N ℓ T T oed ℓ T is within a constant of T , and that T oed ℓ T can be bounded by

by Lemma B.3. By Lemma B.4 we can bound

and by Lemma D.3, under Assumptions 1, 2, 4, 5 and 13, and as long as ̂ A -∈ B F ( A ⋆ ; r est ( A ⋆ )) , we can bound

Let

denote some lower-order constant, whose precise polynomial dependence may change from line to line. On E , by Lemma B.3 we can bound

and so assuming the burn-in (B.1) is met, we can bound ∥ ̂ A --A ⋆ ∥ op ≤ 1 2 r est ( A ⋆ ) and ∥ ̂ A -A ⋆ ∥ op ≤ 1 2 r est ( A ⋆ ) . This then implies that

so in particular we can bound

We have therefore shown that

The final result follows from using Lemma D.4 to bound

Proof of Theorem 1. The proof of Theorem 1 is identical to that of Theorem 4, the only difference being that we replace Assumption 13 with Assumption 6. However, by Proposition 6, the conditions of Assumption 13 are met when Assumption 6 holds.

B.1 Supporting Lemmas

Lemma B.1. Under Assumptions 1 and 3, the system (1.1) satisfies Assumptions 11 and 12 with

d ψ ← d ϕ d x , and where x τ h (resp. u τ h ) denotes the state (resp. input) at step h of trajectory τ . Furthermore, it satisfies Assumption 10 with A R instantiated with the LC 3 algorithm of [17] and

for a universal constant C .

Proof. That Assumption 12 is satisfied is immediate under Assumption 3. It is clear that ψ ( τ ) ∈ S d ϕ d x + . To obtain a bound on D , we only need to bound the trace of ψ ( τ ) :

where the inequality holds under Assumption 1.

To show that Assumption 10 is satisfied in this setting, we have by Theorem 6 that with probability at least 1 -δ , LC 3 has regret bounded as (using that c max ≤ 1 in the setting of Assumption 10):

for C a universal constant. We can therefore take α = 1 / 2 , p R = 3 / 2 , and

Lemma B.2. Let ¯ T ℓ denote the total number of episodes collected by Algorithm 3 at round ℓ . For

on the success event of Lemma B.3, we have 2 T ℓ ≥ ¯ T ℓ and

Proof. Recall that T ℓ = 2 ℓ and N ℓ = ⌈ T ℓ /T oed ℓ ⌉ for T oed ℓ = | Π ℓ | . By Lemma B.3, ¯ T ℓ can be bounded as

and T oed ℓ can be bounded as

Note that we can bound

From this it is immediately obvious that 2 T ℓ ≥ ¯ T ℓ as long as (B.3) is satisfied.

To show the second conclusion note that, by our choice of T i = 2 i , we have that T ℓ ≥ ∑ ℓ -1 i =1 T i , so it therefore remains to show that

However, we can bound

so a sufficient condition is

which we see is met when (B.3) holds.

Lemma B.3. Consider running Algorithm 7 with weight matrix H , parameter ˜ N , and confidence δ , and rerunning each policy in Π out N ≤ ˜ N times. Then, under Assumptions 1 and 3, with probability at least 1 -6 δ :

where ̂ A denotes the least-squares estimate of A ⋆ obtained on the data generated by rerunning Π out . In addition, we have

Furthermore, we have

and the total number of episodes collected by this procedure is bounded by NT out +(16+2log( T out )) · T out , for T out = | Π out | .

Proof. By Lemma B.1, the assumptions of Lemma C.6 and Lemma C.7 are met, so we can therefore apply these results in our setting. By Lemma C.6, the event E exp occurs with probability at least 1 -δ . Throughout the remainder of the proof we union bound over the success event of Lemma C.7 and E exp , which together occur with probability at least 1 -4 δ .

Let

denote the features returned by rerunning every policy in Π out N times. By Lemma C.7, we then have that:

Applying Proposition 3 with E the event that the above conclusion holds and Γ := N · ∑ π ∈ Π out ˇ Λ π , we obtain that, with probability at least 1 -δ (using the mapping to the martingale regression setting described in Appendix A.1.1):

for ζ = 26 β 2 λ max ( Γ )tr( Γ -1 ) and β as defined above. Since ∥ ϕ ( x , u ) ∥ 2 ≤ B ϕ under Assumption 1, we have ∥ ˇ Λ π ∥ 2 ≤ B 2 ϕ , so we can upper bound λ max ( Γ ) ≤ NT out B 2 ϕ . By Lemma C.7, we can also bound (using that D = d x HB 2 ϕ by Lemma B.1):

Combining these and using that

as shown in Lemma C.8 (and using our bounds on C R and p R in Lemma B.1), we can therefore bound

Using that tr( H Γ -1 ) ≤ ∥H∥ op · tr( Γ -1 ) , and the bound on tr( Γ -1 ) given above, it follows that

Finally, by Lemma C.7, we can bound

By Lemma C.8, we can bound the total number of episodes collected by Algorithm 7 by (16 + 2 log( T out )) · T out .

Bound on Frobenius Norm Error. By Lemma C.7, we can lower bound

Furthermore, since ∥ ϕ ( x , u ) ∥ 2 ≤ B ϕ , we always have ∥ ˜ Λ ∥ op ≤ NT out B 2 ϕ , which implies ˜ Λ ⪯ NT out B 2 ϕ · I . By Proposition 4, we then have that with probability at least 1 -δ (again using the mapping to the martingale regression setting described in Appendix A.1.1):

Lemma B.4. Under Assumption 3, for any H , H ′ , we can bound

Proof. We have

Under Assumption 3, we know that there exists some ˇ Λ ′ ∈ Ω such that λ min ( ˇ Λ ′ ) ≥ λ ⋆ min . We can then bound

which proves the result.

C Experiment Design in Arbitrary Dynamical Systems

In this section we generalize somewhat the setting of Section 5. In particular, our goal will now be to collect some set of trajectories D = { τ t } T t =1 , which minimize

for some general feature mapping ψ : T → R d , Φ : R d → R , and T = ( X × U ) H ×X the space of possible state-input trajectories, τ = ( x 0 , u 0 , x 1 , . . . , x H -1 , u H -1 , x H ) ∈ T . We will assume that ψ can be decomposed additively as

In Section 5 we considered the special case where ψ h ( x , u ) = ϕ ( x , u ) ϕ ( x , u ) ⊤ ; in this section ψ could instead be any arbitrary mapping.

As before, we will be interested in defining optimal exploration with respect to some set of exploration policies, Π exp . Let

denote the space of expected value of ψ ( τ ) for mixtures of policies in Π exp . To distinguish elements Λ ∈ Ω from elements in Ω ψ , we will let Γ = E π ∼ ω [ E π [ ψ ( τ )]] refer to elements of Ω ψ , and in particular define Γ π := E π [ ψ ( τ )] (where it is assumed that the expectation is collected over trajectories on (5.1)). We will usually denote unnormalized sums of features, e.g. ∑ T t =1 ψ ( τ t ) , with Σ . We also define ̂ Ω ψ to be the space of all possible combinations of ψ ( τ ) :

We generalize Assumption 7 and Assumption 8 as follows.

Assumption 9 (Regularity of Φ ) . We make the following assumptions:

There exists some M < ∞ satisfying

Assumption 10 (Regret Minimization Oracle) . Let cost h ( τ ) = ⟨ Q h , ψ h ( τ ) ⟩ for some Q h ∈ R d , and cost( τ ) = ∑ H h =1 cost h ( τ ) the total cost of trajectory τ . We assume we have access to some learner A R which, in the setting when | cost( τ ) | ≤ 1 for all τ ∈ T , is able to achieve low regret on { cost h ( · , · ) } H h =1 with respect to policy class Π exp . That is, with probability at least 1 -δ :

for some C R > 0 , p R > 0 , and α ∈ (0 , 1) , and where π t is the policy A R plays at episode t .

Algorithm 4 Dynamic Optimal Experiment Design (DYNAMICOED)

1: input : objective Φ , number of episodes T (OR number of iterates N , episodes per iterate K ), confidence δ , regret minimization algorithm A R , exploration policies Π exp

2: Play any policy π exp ∈ Π exp for K episodes, collect trajectories D 0 = { τ 0 k } K k =1 , set Γ 0 ← K -1 ∑ K k =1 ψ ( τ 0 k )

3: for n = 1 , 2 , . . . , N do

4: Set γ n ← 1 n +1

5: Run A R on cost

for K episodes, collect trajectories D n = { τ n k } K k =1 , denote policies run as Π n

6: Γ n ← (1 -γ n ) Γ n -1 + γ n K -1 ∑ K k =1 ψ ( τ n k )

7: return ( N +1) K Γ N , ∪ N n =0 Π n , ∪ N n =0 D n

Wedefine DYNAMICOED as in Algorithm 4. We then have the following generalization of Theorem 3.

Theorem 5 (Full Version of Theorem 3) . Let Assumption 7 hold, and assume that we have access to a learner A R satisfying Assumption 8. Fix N,K > 0 . Then, with probability at least 1 -δ , DYNAMICOED runs for at most ( N + 1) K episodes, and collects a dataset satisfying D = {{ τ n k } K k =1 } N n =0 satisfying

In this work we are particularly interested in the case where ψ ( τ ) ∈ S d ψ + . We encapsulate this in the following assumption.

Assumption 11 (Matrix Experiment Design) . We assume that ψ ( τ ) ∈ S d ψ + and that, for all τ ∈ T , tr( ψ ( τ )) ≤ D for some D > 0 .

The following corollary instantiates Theorem 3 under Assumption 11 with objective Φ( Γ ) = tr ( H ( Γ + Γ 0 ) -1 ) , the objective considered in Algorithm 1.

Corollary 1. Consider the objective

and assume that H ⪰ 0 and Assumption 10 holds with α = 1 / 2 and Assumption 11 holds. Fix N,K , let T := ( N +1) K , and consider running Algorithm 4 on this objective and with these choices of N and K . Then Algorithm 4 will run for at most T episodes, and, with probability at least 1 -δ , will return data satisfying

C.1 Proof of Theorem 3 and Theorem 5

Lemma C.1 (Lemma C.1 of [56]) . Consider running Algorithm 5 with some convex function f that is β -smooth with respect to some norm ∥ · ∥ , assume that y n ∈ Y for some Y and all n , and let

Algorithm 5 Approximate Frank-Wolfe

1: input : function to optimize f , number of iterations to run N , starting iterate x 1

2: for t = 1 , 2 , . . . , N do

3: Set γ n ← 1 n +1

4: Choose y n to be any point such that

6: return x N +1

R := sup z , y ∈Z∪Y ∥ z -y ∥ . Then for N ≥ 2 , we have

Lemma C.2 (Lemma C.2 of [56]) . When running Algorithm 5, we have

Proof of Theorem 3. By our assumption on A R , Assumption 8, we have that, at round n , with probability at least 1 -δ/ 2 N ,

where we have used that, under Assumption 9 and by the definition of cost n h ( τ ) , | cost n ( τ ) | ≤ 1 for all τ ∈ T . This implies that

Furthermore, by Azuma-Hoeffding and under Assumption 9, we have that, with probability at least 1 -δ/ 2 N ,

This implies that

Note that

and that for any Γ ∈ Ω ψ , we have

for some ω . This implies that

By (C.1) above, we have that

is an approximate minimizer of M · sup π ∈ Π exp E π [cost n ( τ )] , with approximation tolerance M ( C R log p R 2 NK δ · K α -1 + √ 8 log(4 N/δ ) K ) . We can therefore apply Lemma C.1 with

to get that

The result then follows since Γ N +1 = 1 K ( N +1) ∑ N n =0 ∑ K k =1 ψ ( τ n k ) by Lemma C.2.

Proof of Corollary 1. By Theorem 5, for any setting of N and K , we have that with probability at least 1 -δ :

which implies

This gives

It then remains to bound R,β, and M . By Lemma D.6 of [56], we have that

We can then compute the second derivative as, using Lemma D.6 of [56]:

Recall that M is any bound on

By the above computation of the gradient, we can bound this as

To bound β , by the Mean Value Theorem it suffices to bound the operator norm of ∇ 2 Γ Φ( Γ ) . Using the expression above, we can bound this as

Finally, it's straightforward to bound R ≤ 2 D . Putting all of this together gives the result.

C.2 Collecting Full-Rank Data

Algorithm 6 Minimum Eigenvalue Maximization (MINEIG)

1: input : scale N , confidence δ , regret minimization algorithm A R , exploration policies Π exp

2: for j = 1 , 2 , 3 , . . . do

3: N j ←⌈ 2 j/ 3 ⌉ -1 , K j ←⌈ 2 2 j/ 3 ⌉ , T j ← ( N j +1) K j , λ j ← T -1 / 18 j , δ j ← δ 4 j 2

4: Σ j , Π j ← DYNAMICOED (Φ , N j , K j , δ j , A R , Π exp ) for Φ( Γ ) = tr(( Γ + λ j · I ) -1 )

5: if λ min ( Σ j ) ≥ 12544 Dd ψ log 2 N (2+32 T j ) δ then

6: break

7: return Π j

In this section, we consider the setting where ψ ( τ ) ∈ S d ψ + , and our goal is to collect { τ t } T t =1 such that ψ ( 1 T ∑ T t =1 ψ ( τ t )) > 0 . For this to be achievable, we need the following assumption, a generalization of Assumption 3.

Assumption 12 (Full-Rank Data) . Consider ψ ( τ ) such that ψ ( τ ) ∈ S d ψ + . Then we have sup Γ ∈ Ω ψ λ min ( Γ ) ≥ λ ⋆ min for some λ ⋆ min > 0 .

̸

Throughout this section we also assume that Assumption 10 is satisfied with α = 1 / 2 (though all results generalize in a straightforward way for α = 1 / 2 ). We have the following result.

Lemma C.3. Under Assumptions 10 to 12, running Algorithm 6 we have that with probability at least 1 -δ , it will terminate after collecting at most

episodes, and return policy set Π such that

Furthermore, if we rerun each policy in Π once, the resulting features Σ will satisfy, with probability at least 1 -δ/N :

Proof. By Lemma C.4 and our choice of N j and K j in Algorithm 6, we have that if λ j ≤ λ ⋆ min 4 d ψ and

then λ min ( Σ j ) ≥ λ ⋆ min 4 d ψ · T j with probability at least 1 -δ j . It follows that, with probability at least 1 -δ j , the if statement on Line 5 will be true once λ j ≤ λ ⋆ min 4 d ψ , (C.3) holds, and

By our choice of λ j = T -1 / 18 j , a sufficient condition to ensure λ j ≤ λ ⋆ min 4 d ψ , (C.3), and (C.4) is

Since T j = ⌈ 2 j/ 3 ⌉⌈ 2 2 j/ 3 ⌉ ∈ [2 j , 4 · 2 j ] , it follows that the if statement on Line 5 will be met after running for at most

episodes.

By Lemma C.5, if λ min ( Σ j ) ≥ 12544 Dd ψ log 2 N (2+32 T j ) δ and we rerun all policies in Π j , then we will collect data ˜ Σ such that λ min ( ˜ Σ ) ≥ 1 2 λ min ( Σ j ) , with probability at least 1 -δ/ 2 N . As the if statement on Line 5 will only be true once this is met, it follows that, with probability at least 1 -δ/ 2 N , rerunning all policies in Π j once, we will collect data Σ which satisfies

The lower bound on λ min ( ∑ π ∈ Π Γ π ) follows analogously from Lemma C.5.

The result then follows noting that the failure probability of running DYNAMICOED is at most

C.2.1 Supporting Lemmas

Lemma C.4. Under Assumptions 10 to 12, consider running DYNAMICOED on the objective

with N = ⌈ 2 i/ 3 ⌉-1 and K = ⌈ 2 2 i/ 3 ⌉ , for some λ > 0 and i . Let T := ( N +1) K . Then if λ ≤ λ ⋆ min 4 d ψ and

with probability at least 1 -δ ,

Proof. Applying Corollary 1 with H = I and Γ 0 = λ · I , we have that, with probability at least 1 -δ :

where the second inequality follows since

which implies N +1 = ⌈ 2 i/ 3 ⌉ ≥ T 1 / 3 / 4 1 / 3 and K = ⌈ 2 2 i/ 3 ⌉ ≥ T 2 / 3 / 4 2 / 3 . If T satisfies (C.6), then we can bound

Furthermore, under Assumption 3 there exists some Γ ∈ Ω ψ such that Γ ⪰ λ ⋆ min · I , so we can upper bound

and we can lower bound

Thus,

It follows that if λ ≤ λ ⋆ min 4 d ψ , then we have

which proves the result.

Lemma C.5. Consider running some policies ( π τ ) T τ =1 , for π τ F τ -1 -measurable, and collecting covariance Σ T = ∑ T t =1 ψ ( τ t ) . Then under Assumption 11, as long as

with probability at least 1 -δ , if we rerun each ( π τ ) T τ =1 , we will collect features ˜ Σ T such that

Proof. This follows from applying Lemma D.7 of [56] to the matrix 1 D Σ T . Note that while [56] considers the setting of linear MDPs, the proof of Lemma D.7 of [56] does not make use of the linear MDP assumption, and the proof therefore extends immediately to our setting. Furthermore, though it is not explicitly stated, the lower bound on λ min ( ∑ T τ =1 Γ π τ ) is also proved in Lemma D.7 of [56].

Furthermore,

Algorithm 7 Learn Minimizing Exploration Policies (LEARNEXP Π )

1: input : H , iterates bound ˜ N , confidence δ , regret minimization algorithm A R , exploration policies Π exp

2: for i = 1 , 2 , 3 , . . . do

3: N i ←⌈ 2 i/ 3 ⌉ -1 , K i ←⌈ 2 2 i/ 3 ⌉ , T i ← ( N i +1) K i , δ i ← δ/ 4 i 2

4: Π i MINEIG ← MINEIG ( ˜ NT i , δ i , A R , Π exp )

6: Φ( Γ ) ← tr( H ( Γ + T -1 i Γ i 0 ) -1 )

5: Run policies in Π i MINEIG ⌈ T i / | Π i MINEIG |⌉ times, set Γ i 0 to collected features

7: Γ i , Π i ← DYNAMICOED (Φ , N i , K i , δ, A R , Π exp )

8: if

fw fw

(C.8)

9: then

10: Γ out ← Γ i fw + Γ i 0 , Π out ← Π i MINEIG ∪ ( ∪ ⌈ T i / | Π MINEIG |⌉ j =1 Π i fw )

11: return Γ out , Π out

C.3 Rerunning Policies

In this section, we build on the analysis of the DYNAMICOED algorithm to show that, not only do the features collected by DYNAMICOED approximately minimize Φ , but that, under certain conditions, if we rerun the policies that DYNAMICOED ran to collect this data, we will collect a new set of features which also approximately minimizes Φ .

In particular, we specialize this argument to objectives of the form Φ( Γ ) = tr( H Γ -1 ) . LEARNEXP Π (Algorithm 7) proceeds by first calling MINEIG to collect full-rank data, using this data as a regularizer of Φ( Γ ) , and the running DYNAMICOED on this objective. After meeting a certain termination criteria, it terminates, and returns the policies it has run over its operation.

Lemma C.6. Let E exp denote the event that, for all i = 1 , 2 , 3 , . . . , the success event of MINEIG and DYNAMICOED occur, and

Then if Assumptions 10 to 12 hold, P [ E exp ] ≥ 1 -δ .

Lemma C.7. Consider rerunning each policy in Π out N ≤ ˜ N times, and let ˜ Γ denote the obtained features. Then, if Assumptions 10 to 12 hold, with probability at least 1 -3 δ , on the event E exp :

and

for T out := | Π out | .

Lemma C.8. On the event E exp , under Assumptions 10 to 12, we can bound

Furthermore, the total number of episodes collected by Algorithm 7 is bounded by (16+2log( T out )) · T out .

C.3.1 Supporting Lemmas and Proofs

Lemma C.9. Under Assumption 11, for any Γ = E τ ∼ ω [ ψ ( τ )] and H ⪰ 0 we can bound

Proof. By Von Neumann's Trace Inequality we can lower bound

By our assumption that tr( ψ ( τ )) ≤ D , we can bound ∥ Γ ∥ op ≤ D , which proves the result.

LemmaC.10. Assume tr( ψ ( τ )) ≤ D for all τ . Let Γ K denote the time-normalized features obtained by playing policies { π k } K k =1 , where π k is F k -1 -measurable. Then, with probability at least 1 -δ ,

Proof. This follows from an argument identical to the proof of Lemma C.4 of [56]. While [56] considers the setting of linear MDPs, we note that the proof of Lemma C.4 of [56] nowhere relies on the linear MDP assumption. The result stated here then follows identically as Lemma C.4 of [56], after normalizing ψ ( τ ) by D .

Proof of Lemma C.6. By Lemma C.3, the failure probability of running MINEIG at round i is δ i = δ/ 8 i 2 , and by Corollary 1 the failure probability of DYNAMICOED at round i is also bounded by δ i = δ/ 8 i 2 . It follows that the total failure probability of running MINEIG and DYNAMICOED is bounded by

Furthermore, by Lemma C.3, we have that rerunning all policies in Π i MINEIG , we will obtain features Γ satisfying, with probability at least 1 -δ i / ˜ NT i :

Repeating this ⌈ T i / | Π i MINEIG |⌉ times and union bounding, we have that

with probability at least 1 -δ/ ˜ NT i · ⌈ T i / | Π i MINEIG |⌉ ≥ 1 -δ/ ˜ N .

Proof of Lemma C.7. Let Π MINEIG , Γ 0 , Π fw , Γ fw denote the policies and features obtained on the round at which Algorithm 7 terminates. Let T fw = | Π fw | denote the number of episodes of DYNAMICOED on the terminating round, and N fw , K fw the corresponding values of N i and K i . Throughout the proof we make use of the fact that at termination of Algorithm 7, all of (C.7)-(C.10) are met.

Proof of (C.11) and (C.12) . By Lemma C.10 we have that, with probability at least 1 -δ :

On E exp , by Lemma C.3, we can bound

and, furthermore, we can lower bound

Since Π MINEIG ⊆ Π out , it follows that

Combining these, we therefore have that, with probability at least 1 -δ :

In addition, also by Lemma C.3, we have that with probability at least 1 -δ/ ˜ NT out · N ≥ 1 -δ , that

Proof of (C.13) . By Corollary 1, on E exp we have that:

Since T fw satisfies (C.8), we can bound

It follows that

By Lemma C.10, we have that, with probability at least 1 -δ :

Since (C.10) is satisfied and | Π i MINEIG | ≤ T i , we have

By Lemma A.2 it follows that

This implies that

Now if

we can bound this all by

where the last inequality follows from (C.14). However, note that (C.15) since (C.9) holds. Finally, note that

where the last inequality follows since (C.7) holds. We can therefore upper bound 4 T fw ≤ 12 T out . Putting this together proves the result.

Proof of Lemma C.8. To bound T out , it suffices to show that (C.7)-(C.10) are satisfied for sufficiently large T i .

On E exp , by Lemma C.3, we can bound

so to ensure (C.7) is met it suffices that

On E exp , we have

Which also implies

Furthermore, by Lemma C.9 we can lower bound

Combining these and using that N i = O ( T 1 / 3 i ) and K i = O ( T 2 / 3 i ) , it is easy to see that (C.8)-(C.10) will be met once

The bound on T out then follows since T i = ⌈ 2 i/ 3 ⌉⌈ 2 2 i/ 3 ⌉ ∈ [2 i , 4 · 2 i ] , so it can be at most a constant larger than the sufficient condition before terminating.

Let i ⋆ denote the round that Algorithm 7 terminates on. Note that at round i , MINEIG runs for at most | Π i MINEIG | , DYNAMICOED runs for at most T i episodes, and we run for an additional ⌈ T i / | Π i MINEIG |⌉ · | Π i MINEIG | episodes on Line 5. In total, then, the number of episodes Algorithm 7 runs for is bounded by

where the last inequality follows since T i ∈ [2 i , 4 · 2 i ] . Now note that, since Algorithm 7 only terminates once (C.7) is met, we will have max j =1 ,...,i ⋆ | Π j MINEIG | ≤ T i ⋆ . This implies that 2 ∑ i ⋆ i =1 | Π i MINEIG | ≤ 2 i ⋆ T i ⋆ ≤ 2 log( T i ⋆ ) · T i ⋆ . Bounding T i ⋆ ≤ T out gives the result.

D Smooth Nonlinear Systems

In this section we restrict to the nonlinear regulator system of (1.1). Our goal will be to show that, under our assumptions, the nonlinear regulator system exhibits certain smooth behavior. As we have assumed

it will be convenient to define J ( θ ; A ) := J ( π θ ; A ) and θ ⋆ ( A ) = θ ⋆ . For the remainder of this section, we will typically use θ in place of π θ . In addition, when considering radius terms such as r θ ( A ⋆ ) and r cost ( A ⋆ ) , to simplify results we assume that r θ ( A ⋆ ) ≤ 1 and r cost ( A ⋆ ) ≤ 1 . Note that this does not change the validity of the result since, for example, if a result holds with A ∈ B F ( A ⋆ ; r ) for some r > r θ ( A ⋆ ) , it also holds for A ∈ B F ( A ⋆ ; r θ ( A ⋆ )) . Throughout this section, we let ∇ x f ( x )[∆] refer to the directional gradient of f ( x ) in direction ∆ .

We first have the following result, which shows that under our assumptions, the controller loss is differentiable.

Lemma D.1. Under Assumptions 1, 2, 4 and 5, for any A satisfying A ∈ B F ( A ⋆ ; r θ ( A ⋆ )) , the controller loss J ( θ ; A ) is four-times differentiable in θ and A . Furthermore, we can bound

In this section, we generalize Assumption 6 to the following.

Assumption 13. We assume there exists some r θ ( A ⋆ ) > 0 such that, for all A ∈ B F ( A ⋆ ; r θ ( A ⋆ )) , θ ⋆ ( A ) satisfies:

∇ θ J ( θ ; A ) | θ = θ ⋆ ( A ) = 0 ,

θ ⋆ ( A ) is three-times differentiable in A , and we can bound ∥∇ ( i ) A θ ⋆ ( A ) ∥ op ≤ L π ⋆ for some L π ⋆ > 0 and i ∈ { 1 , 2 , 3 } .

The first condition requires that θ ⋆ ( A ) corresponds to a stationary point of the loss. This will be met, for example, by choosing θ ⋆ ( A ) to be a minima (local or global) of J ( θ ; A ) . It is not obvious, however, that the first and second condition can be simultaneously satisfied. In the following we show that, assuming ∇ 2 θ J ( θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) is full-rank (which will be the case, for example, when θ ⋆ ( A ⋆ ) is a strict local minimum of J ( π θ ; A ⋆ ) ), there always exists some θ ⋆ ( A ) satisfying both conditions of Assumption 13, with L π ⋆ scaling polynomially in problem parameters, and r θ ( A ⋆ ) scaling inverse polynomially in problem parameters. Note that this definition of π ⋆ ( A ) is general enough to capture settings where the global minimum of J ( π ; A ) cannot be efficiently computed-it suffices to take π ⋆ ( A ) a local minimum of the loss.

Proposition 5. Assume that Assumptions 1, 2, 4 and 5 hold and that λ min ( ∇ 2 θ J ( θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) ) > 0 . Let r θ ( A ⋆ ) > 0 be some value satisfying

Then there exists some function θ ⋆ ( A ) such that, for all A ∈ B F ( A ⋆ ; r θ ( A ⋆ )) :

θ ⋆ ( A ) is three-times differentiable in A ,

and it suffices that we take

While Proposition 5 shows that there exists some θ ⋆ ( A ) satisfying Assumption 13, it does not directly give a recipe for constructing such a map. The following result shows that under a mild additional assumption, the minimizer of the loss satisfies Assumption 13.

Proposition 6. Let

Then under Assumptions 1, 2 and 4 to 6, there exists some r θ ( A ⋆ ) > 0 and L π ⋆ < ∞ such that θ ⋆ ( A ) satisfies Assumption 13.

The scaling of L π ⋆ in Proposition 6 can be shown to match that of Proposition 5, but in general r θ ( A ⋆ ) could be smaller than the value of r θ ( A ⋆ ) given in Proposition 5. In particular, in the setting of Proposition 6, we can only show that r θ ( A ⋆ ) scales with min θ ̸∈B 2 ( θ ⋆ ( A ⋆ ); r ) J ( θ ; A ⋆ ) -J ( θ ⋆ ( A ⋆ ); A ⋆ ) for some r > 0 which scales inverse polynomially in problem parameters. While we can show that J ( θ ; A ⋆ ) -J ( θ ⋆ ( A ⋆ ); A ⋆ ) scales inverse polynomially in problem parameters, including in λ min ( ∇ 2 θ J ( θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) ) , for θ approximately a distance of r from θ ⋆ ( A ⋆ ) , it is possible J ( θ ; A ⋆ ) has some local minimizer θ ′ arbitrarily far away from θ ⋆ ( A ⋆ ) , such that J ( θ ′ ; A ⋆ ) and J ( θ ⋆ ( A ⋆ ); A ⋆ ) are arbitrarily close, in which case ∆ ⋆ , and therefore r θ ( A ⋆ ) , could be arbitrarily small. The failure mode here is that, while θ ⋆ ( A ⋆ ) may be the global minimum of J ( θ ; A ⋆ ) , for A arbitrarily close to A ⋆ , the global minimum of J ( θ ; A ) could instead be near θ ′ , which would render the map θ ⋆ ( A ) discontinuous.

By making further assumptions on J ( θ ; A ⋆ ) which exclude this case, we can obtain a value of r θ ( A ⋆ ) scaling similarly to in Proposition 5. For example, in the following, we show that under the assumption that J ( θ ; A ) is convex, this holds.

Proposition 7. Assume that there exists some r conv ( A ⋆ ) > 0 such that, for all A ∈ B F ( A ⋆ ; r conv ( A ⋆ )) , J ( θ ; A ) is convex in θ , and set

Then we have that θ ⋆ satisfies Assumption 13 with

and it suffices that we take

Note that, if J ( θ ; A ) is µ -strongly convex in θ for all A near A ⋆ , we can lower bound λ min ( ∇ 2 θ J ( θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) ) ≥ µ .

Approximating the Controller Loss. In order to efficiently direct our exploration, it is convenient to derive a quadratic approximation to the controller loss. The following result shows that, under our assumptions, this is indeed possible.

Lemma D.2 (Formal Version of Proposition 1) . Under Assumptions 1, 2, 4 to 5 and 13, for ̂ A ∈ B F ( A ⋆ ; min { r cost ( A ⋆ ) , r θ ( A ⋆ ) } ) , we have

for some tensor M such that

In practice we do not know H ( A ⋆ ) and must estimate it. The following result shows that the distance between H ( A ⋆ ) and H ( ̂ A ) can be bounded.

Lemma D.3. Under Assumptions 1, 2, 4, 5 and 13, and if ̂ A ∈ B F ( A ⋆ ; min { r cost ( A ⋆ ) , r θ ( A ⋆ ) } ) , we can bound

D.1 Proof of Smoothness of Nonlinear System

We let f w ( · ) denote the density of the noise (which, by assumption, is simply an isotropic Gaussian density). We let f A, θ ( · ) denote the density over trajectories induced by playing controller θ on system A . We will overload notation somewhat and let f A, θ ( x h +1 | τ 1: h ) denote the density over x h +1 induced by playing controller θ given trajectory τ 1: h . Note that f A, θ ( x h +1 | τ 1: h ) = f ( x -A ϕ ( x τ , π θ ( τ ))) and

Throughout this section we let x τ h (resp. u τ h ) denote the state (resp. input) at step h of trajectory τ . Under our regularity assumptions (Assumptions 1, 2, 4 and 5) and since the noise is Gaussian, we can swap derivatives and integrals, which we make use of throughout the following proofs.

Proof of Lemma D.1. Let cost( τ ) denote the cost of trajectory τ . Then we have

Let A t := A + t 1 ∆ A 1 + t 2 ∆ A 2 + t 3 ∆ A 3 and θ s := θ + s 1 ∆ θ 1 + s 2 ∆ θ 2 + s 3 ∆ θ 3 , for some ∆ A i and ∆ θ j , which we assume satisfy ∥ ∆ A i ∥ op , ∥ ∆ θ j ∥ op ≤ 1 . Rather than differentiating J ( θ ; A ) with respect to θ or A , we will differentiate J ( θ s ; A t ) with respect to some x 1 , x 2 , x 3 , x 4 ∈ { t 1 , t 2 , t 3 , s 1 , s 2 , s 3 } . Note that, for example,

i.e. the directional gradient of J ( θ , A ) with respect to A in direction ∆ A 1 , and that this similarly holds for gradients with respect to other t i , s j , or higher-order derivatives. Thus, if we can show that J ( θ s ; A t ) is differentiable with respect to any x 1 , x 2 , x 3 , x 4 ∈ { t 1 , t 2 , t 3 , s 1 , s 2 , s 3 } , and this holds for any choice of ∆ A i , ∆ θ j , then we have that J ( θ , A ) is four-times differentiable with respect to θ and A . Furthermore, we can bound the operator norm of ∇ A J ( θ , A ) , by bounding the value of d d t 1 J ( θ s ; A t ) | t = s =0 for all ∆ A 1 satisfying ∥ ∆ A 1 ∥ op ≤ 1 (and we can similarly bound the operator norm of the higher order derivatives of J ( θ , A ) ).

Differentiating this gives

The fourth derivative of J ( θ ; A ) can be similarly calculated by differentiating d d x 3 d d x 2 d d x 1 J ( θ s ; A t ) ; we omit it for brevity. We have

for some C which does not depend on t or s . Given that ϕ ( x , u ) is four-times differentiable in u and π θ s h ( τ 1: h ) is four-times differentiable in x (which hold by Assumption 4 and Assumption 5), it is clear that log f A t , θ s ( τ ) is four-times differentiable in t i or s i , regardless of the choice of ∆ A i or ∆ θ i . This proves the first result.

Norm Bounds on Gradient. Note that

Furthermore, differentiating these expressions further with respect to t j or s j will simply yield higher-order derivates of ϕ ( x , u ) and π θ h ( τ 1: h ) . Using the norm bounds on the gradient of ϕ ( x , u ) and π θ h ( τ 1: h ) given in Assumption 4 and Assumption 5, and the norm bound of ϕ ( x , u ) given in Assumption 1, we can then bound

for i, j ∈ { 0 , 1 , 2 , 3 , 4 } satisfying 1 ≤ i + j ≤ 4 (where we have used the fact noted above that, to bound the operator norm of ∇ ( i ) A ∇ ( j ) θ log f A, θ ( τ ) , it suffices to bound the directional gradient in every direction). It follows that we can bound

where ( a ) follows from Cauchy-Schwarz, and ( b ) follows from Lemma A.1 and Assumption 2, since we have assumed A ∈ B F ( A ⋆ ; r cost ( A ⋆ )) .

Proof of Proposition 5. Existence and Differentiability of θ ⋆ . By Lemma D.1 we have that J ( θ ; A ) is four-times differentiable in its arguments. By the Implicit Function Theorem, since λ min ( ∇ 2 θ J ( θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) ) > 0 by assumption, we have that there exists some r ′ θ ( A ⋆ ) > 0 and unique function θ ⋆ ( A ) defined on B F ( A ⋆ ; r ′ θ ( A ⋆ )) such that ∇ θ J ( θ ; A ) | θ = θ ⋆ ( A ) = 0 , and θ ⋆ ( A ) is three-times differentiable (note that, while the Implicit Function Theorem is typically stated to give that the resulting function is only one-time differentiable, it can be extended to k -times differentiable, assuming the implicit equation is k -times differentiable [60]).

By Lemma D.1 and the continuity of eigenvalues, it follows that for A close enough to A ⋆ , we have λ min ( ∇ 2 θ J ( θ ; A ) | θ = θ ⋆ ( A ) ) ≥ 1 2 λ min ( ∇ 2 θ J ( θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) ) > 0 . We can therefore apply the Implicit Function Theorem as above to any A satisfying this, to get that there exists some unique ˜ θ ⋆ ( A ′ ) defined for all A ′ near A such that ∇ θ J ( θ ; A ′ ) | θ = ˜ θ ⋆ ( A ′ ) = 0 and ˜ θ ⋆ ( A ′ ) is differentiable.

By the uniqueness of θ ⋆ ( A ) on B F ( A ⋆ ; r ′ θ ( A ⋆ )) , it follows that any ˜ θ ⋆ ( A ′ ) defined in this way must be identical to θ ⋆ ( A ) on B F ( A ⋆ ; r ′ θ ( A ⋆ )) (assuming the regions on which they are defined overlaps). We can therefore define θ ⋆ ( A ) to simply be the extension of θ ⋆ ( A ) to all such ˜ θ ⋆ ( A ) , defined for all A near A ⋆ such that λ min ( ∇ 2 θ J ( θ ; A ) | θ = θ ⋆ ( A ) ) ≥ 1 2 λ min ( ∇ 2 θ J ( θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) ) , and will have that θ ⋆ ( A ) is three-times differentiable and satisfies ∇ θ J ( θ ; A ) | θ = θ ⋆ ( A ) = 0 for all such A .

We then choose r θ ( A ⋆ ) to be defined such that, for all A ∈ B F ( A ⋆ ; r θ ( A ⋆ )) , we have λ min ( ∇ 2 θ J ( θ ; A ) | θ = θ ⋆ ( A ) ) ≥ 1 2 λ min ( ∇ 2 θ J ( θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) ) . By Lemma D.1, we know that ∇ 2 θ J ( θ ; A ) is continuous and furthermore we know that eigenvalues are continuous. Using the gradient bounds given in Lemma D.1 to bound the Lipschitz constant of ∇ 2 θ J ( θ ; A ) , it follows that we can take

Bounding Norm of Gradients. Fix A ∈ B F ( A ⋆ ; r θ ( A ⋆ )) . We know that θ ⋆ ( A ) satisfies ∇ θ J ( θ ; A ) | θ = θ ⋆ ( A ) = 0 .

We wish to differentiate θ ⋆ ( A ) with respect to A , and bound the magnitude of up to the third derivative. Similar to the proof of Lemma D.1, we let A t := A + t 1 ∆ A 1 + t 2 ∆ A 2 + t 3 ∆ A 3 for some ∆ A i satisfying ∥ ∆ A i ∥ op ≤ 1 . As noted in the proof of Lemma D.1, we have

(and similarly for higher-order derivatives). Thus, to show the result, it suffices to show that θ ⋆ ( A t ) is differentiable in t 1 , t 2 , t 3 for all ∆ A i , and to bound the magnitude of this derivative for all ∆ A i with ∥ ∆ A i ∥ op ≤ 1 . We have

which implies

which is well-defined since we have assumed that ∇ 2 θ J ( θ ; A ) | θ = θ ⋆ ( A ) is full-rank, and J is differentiable in both its arguments by Lemma D.1. To compute the second derivative of θ ⋆ , we differentiate through (D.1) which gives

Note that G 2 ( A, ∆ A 1 , ∆ A 2 ) involves at most a third-order derivative of J ( θ ; A ) and first-order derivative of θ ⋆ ( A ) , both of which we know exist by Lemma D.1 and what we showed above. This then further implies

which is well-defined since we have assumed that ∇ 2 θ J ( θ ; A ) | θ = θ ⋆ ( A ) is full-rank. Finally, we compute

Note that G 3 ( A, ∆ A 1 , ∆ A 2 , ∆ A 3 ) involves at most a fourth-order derivative of J ( θ ; A ) and secondorder derivative of θ ⋆ ( A ) , both of which we know exist by Lemma D.1 and what we showed above. We therefore have

which is well-defined since we have assumed that ∇ 2 θ J ( θ ; A ) | θ = θ ⋆ ( A ) is full-rank. As each of these expressions is defined for all choice of ∆ A i , the differentiability of θ ⋆ ( A ) follows.

Note that the above expressions for ∇ A θ ⋆ ( A )[∆ A 1 ] , ∇ 2 A θ ⋆ ( A )[∆ A 1 , ∆ A 2 ] , and ∇ 3 A θ ⋆ ( A )[∆ A 1 , ∆ A 2 , ∆ A 3 ] all depend on at most a fourth derivative of J ( θ ; A ) , as well as ( ∇ 2 θ J ( θ ; A ) | θ = θ ⋆ ( A ) ) -1 . The norm bounds are then a direct consequence of Lemma D.1.

Proof of Proposition 6. By Lemma D.1 we have that J ( θ ; A ) is four-times differentiable in its arguments. Since we have assumed ∇ 2 θ J ( θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) ≻ 0 , by the Implicit Function Theorem [60], it follows that there exists some r ′ θ > 0 and mapping ˜ θ ( A ) such that, for all A ∈ B F ( A ⋆ ; r ′ θ ) , ∇ θ J ( θ ; A ) | θ = ˜ θ ( A ) = 0 , and ˜ θ ( A ) is three-times differentiable.

Our goal is now to show that ˜ θ ( A ) = θ ⋆ ( A ) for A close enough to A ⋆ . By the continuity of eigenvalues, J ( θ ; A ) , and ˜ θ ( A ) , we have that there exists some r and r ′′ θ such that, for all θ ∈ B F ( θ ⋆ ( A ⋆ ); r ) and A ∈ B F ( A ⋆ ; r ′′ θ ) , we have ∇ 2 θ J ( θ ; A ) ≻ 0 and, furthermore, ˜ θ ( A ) ∈ B 2 ( θ ⋆ ( A ⋆ ); r/ 2) for all A ∈ B F ( A ⋆ ; r ′′ θ ) . This implies that ˜ θ ( A ) is strict local minimum of J ( θ ; A ) and, in particular, that

̸

Let ∆ ⋆ := min θ ̸∈B 2 ( θ ⋆ ( A ⋆ ); r ) J ( θ ; A ⋆ ) -J ( θ ⋆ ( A ⋆ ); A ⋆ ) and note that, since we have assumed the global minimum of J ( θ ; A ⋆ ) is unique, we have ∆ ⋆ > 0 .

Fix some A ∈ B F ( A ⋆ ; r ′′ θ ) and assume that ˜ θ ( A ) is not the global minimum of J ( θ ; A ) . This implies that θ ⋆ ( A ) , the global minimum of J ( θ ; A ) , is outside of B 2 ( θ ⋆ ( A ⋆ ); r ) . Furthermore, by the continuity of J ( θ ; A ) , we have, for some L, L ′ > 0 ,

This implies that

However, for r ′′ θ small enough, this is a contradiction. Thus, it follows that ˜ θ ( A ) is the global minimum of J ( θ ; A ) , so ˜ θ ( A ) = θ ⋆ ( A ) .

The result then follows since we already have that ˜ θ ( A ) is three-times differentiable and satisfies ∇ θ J ( θ ; A ) | θ = ˜ θ ( A ) = 0 , and by taking r θ ( A ⋆ ) to be the minimum of r ′ θ and r ′′ θ . The boundedness of L π ⋆ follows as in the proof of Proposition 5.

Proof of Proposition 7. Note that, by convexity and the KKT conditions, the solutions to arg min θ ∈ R d θ J ( θ ; A ) are described by

Thus, an equivalent definition for θ ⋆ ( A ) is that it satisfies ∇ θ J ( θ ; A ) | θ = θ ⋆ ( A ) = 0 . Assumption 13 can then be shown to hold by an argument analogous to Proposition 5.

Proof of Lemma D.2. Let A ( t ) = t ̂ A +(1 -t ) A ⋆ and g ( t ) := J ( θ ⋆ ( A ( t )); A ⋆ ) . By Lemma D.1 and under Assumption 5, we have that both J ( θ ; A ) and θ ⋆ ( A ) are three-times differentiable for all A = t ̂ A +(1 -t ) A ⋆ , t ∈ [0 , 1] , so it follows that g ( t ) is three-times differentiable in t . We can therefore apply Taylor's Theorem to expand g (1) about the point t = 0 to get:

where A ′ = A ( t ′ ) for some t ′ ∈ [0 , 1] . Under Assumption 13, we have that ∇ θ J ( θ ; A ⋆ ) | θ = θ ⋆ ( A ⋆ ) = 0 , which implies that, plugging in the definition of g (1) and g (0) ,

We can bound

The expression for ∇ 3 A J ( θ ⋆ ( A ); A ⋆ ) contains up to the third derivative of both J ( θ ; A ⋆ ) and θ ⋆ ( A ) . By Lemma D.1 and under Assumption 13, since A ′ ∈ B F ( A ⋆ ; min { r cost ( A ⋆ ) , r θ ( A ⋆ ) } ) by construction, we can then bound

The result follows by the definition of H ( A ⋆ ) .

Proof of Lemma D.3. Recall that H ( ̂ A ) = ∇ 2 A J ( θ ⋆ ( A ); ̂ A ) | A = ̂ A . To prove this, we will use that this is differentiable by Lemma D.1, and will apply Taylor's Theorem.

First, note that by Taylor's Theorem we have

for A ′ = t ̂ A +(1 -t ) A ⋆ for some t ∈ [0 , 1] . The third derivative of J ( θ ⋆ ( A ); ̂ A ) will involve up to the third derivative of both J ( θ ; ̂ A ) and θ ⋆ ( A ) , so using Lemma D.1 and Assumption 13, since A ′ ∈ B F ( A ⋆ ; min { r cost ( A ⋆ ) , r θ ( A ⋆ ) } ) by assumption, we can bound

Next, we wish to relate ∇ 2 A J ( θ ⋆ ( A ); ̂ A ) | A = A ⋆ to ∇ 2 A J ( θ ⋆ ( A ); A ⋆ ) | A = A ⋆ = H ( A ⋆ ) . Again applying Taylor's Theorem, we have

for A ′′ = t ̂ A +(1 -t ) A ⋆ for some t ∈ [0 , 1] . By Lemma D.1 and Assumption 13, we can bound

The result follows.

Lemma D.4. Under Assumptions 1, 2, 4, 5 and 13, for all A ∈ B F ( A ⋆ ; min { r cost ( A ⋆ ) , r θ ( A ⋆ ) } ) , we can bound

Proof. Recall that H ( ̂ A ) = ∇ 2 A J ( θ ⋆ ( A ); ̂ A ) | A = ̂ A . The bound then follows from Lemma D.1 and Assumption 13.

E High-Probability Regret Bounds in Nonlinear Systems

In this section, we modify the proof the main result of [17] slightly to show a high probability regret bound for LC 3 . For the sake of brevity, we omit details that are identical to the proof given in [17]. We will need the following assumption.

Assumption 14 (Bounded Cost) . We assume that, for all trajectories τ , we have cost( τ ) ≤ c max .

We adopt the notation used in this work, modifying somewhat the notation from [17]. In particular, we let J ( π ; A ) denote the expected cost of playing policy π under system A , and we set

denote the covariates obtained by the first t episodes of LC 3 (plus a regularizer). We let π t denote the policy played at episode t of LC 3 . For a policy set Π , we define regret as

We will also denote π ⋆ := arg min π ∈ Π J ( π ; A ⋆ ) .

In addition to these notational changes, we modify LC 3 slightly to use the parameter

in the construction of the confidence set, BALL t .

Besides the aforementioned changes, in the following proofs we adopt the same notation as [17]. We have the following result.

Theorem 6. Under Assumptions 1 and 14 and with any policy class Π , with probability at least 1 -δ , LC 3 has regret bounded as

for a universal constant C .

Proof of Theorem 6. By Lemma E.2, we have that the event E 1 holds with probability at least 1 -δ . We therefore assume E 1 holds for the remainder of the proof.

By the definition of the confidence set in LC 3 , on E 1 we have that A ⋆ is the in confidence set for all t ≤ T . It follows that on E 1 ,

where ( a ) follows from the optimistic property of LC 3 when A ⋆ ∈ BALL t , and ( b ) follows from Lemma E.1. On E 1 , we have

where the last inequality follows from the definition of BALL t since ̂ A t ∈ BALL t by construction, and by the definition of E 1 . This gives

By Lemma E.3, with probability 1 -δ we can bound this as

By Cauchy-Schwarz, we can bound ( a ) as

We have

where the last inequality uses Lemma B.6 of [17]. Putting all of this together, we have shown that with probability at least 1 -2 δ , we have

It remains to bound β T and log(det( Σ T ) det( Σ 0 ) -1 ) . We have Σ 0 = λI , so det( Σ 0 ) = λ d ϕ . Furthermore, if ∥ ϕ ( x , u ) ∥ 2 ≤ B ϕ , then we can bound det( Σ T ) ≤ ( λ + B 2 ϕ TH ) d ϕ . Putting this together we have

Recalling that

we can similarly bound

Choosing λ = σ 2 w completes the proof.

E.1 Supporting Lemmas

Lemma E.1. Under Assumption 14, we can bound

Proof. Following the proof of Lemma B.3 of [17], and adopting the same notation, we have

Under Assumption 14 we have A h ≤ c 2 max

. Plugging this in gives the result.

LemmaE.2. Let β t := √ λB A + σ w √ 8 d x log 5 + 8 log( T det( Σ t ) det( Σ 0 ) -1 /δ ) and let E 1 denote the event

Then running LC 3 we have P A ⋆ [ E 1 ] ≥ 1 -δ .

Proof. The proof of Lemma B.5 of [17] shows that with probability at least 1 -δ ,

The result then follows from this, since ∥ A ⋆ ∥ op ≤ B A , and a union bound.

Lemma E.3. With probability 1 -δ , we have

Proof. This is an immediate consequence of Azuma-Hoeffding, since ∑ H h =1 min { 2 β t σ w ∥ ϕ ( x h , u h ) ∥ Σ -1 t , 1 } ≤ H almost surely, and from upper bounding

F Lower Bounds on Learning in Nonlinear Systems

In this section, we assume that θ ⋆ and π ⋆ correspond to the global minimizer:

Here we formally state the additional assumptions needed in Section 4.2, and provide a formal version of Theorem 2.

Assumption 15. There exists some r µ ( A ⋆ ) > 0 such that, for all A ∈ B F ( A ⋆ , r µ ( A ⋆ )) , π ⋆ ( A ) is unique and, furthermore, there exists some µ > 0 such that

Assumption 15 will be satisfied in cases where J ( π θ ; A ) is strongly convex in θ , but may hold even when this is not the case. Intuitively, it requires that our controller class is not overparameterizedmoving θ away from its optimal value will cause the loss to increase. We will additionally make the following regularity assumptions on policies in Π exp and their induced covariates set, Ω .

Assumption 16. There exists some λ > 0 such that, for each Λ ∈ Ω , we have λ min ( Λ ) ≥ λ .

Assumption 16 requires that every exploration policy we consider excites all directions in ϕ space (in contrast, Assumption 3 only assumes there exists some distribution over policies in Π exp which excite all directions). We remark that this assumption is relatively mild if Assumption 3 holds. As we show in Appendix C.2, under Assumption 3, a mixture over policies, ω , satisfying λ min ( E π ∼ ω [ Λ π ]) > 0 can be learned using only a number of samples scaling polynomially in problem parameters. Given ω , a policy class Π exp satisfying Assumption 16 can be obtained by simply mixing ω with every other exploration policy. We are now ready to state our main lower bound.

Theorem 7 (Formal Version of Theorem 2) . Under Assumptions 1, 2, 4, 5, 13, 15 and 16 and if π ⋆ is defined as in (F.1) , as long as T ≥ C lb , for any ω exp ∈ △ Π exp , we have

for B T := { A : ∥ A -A ⋆ ∥ 2 F ≤ 5 d x d ϕ / ( λd x TH ) 5 / 6 } , E D T ∼ A,ω exp [ · ] = E π ∼ ω exp [ E D T ∼ A,π [ · ]] denotes the expectation over trajectories generated by running policies π drawn according to ω exp on system A for T episodes, ̂ π any mapping from observations to policies in Π ⋆ , and

Proof of Theorem 7. This proof follows immediately from Lemma F.1, by lower bounding the righthand side of (F.2) by the min over all policies in Π .

Lemma F.1. Under Assumptions 1, 2, 4, 5, 13, 15 and 16 and if θ ⋆ is defined as in (F.1) , as long as

for any ω exp ∈ △ Π

for B T := { A : ∥ A -A ⋆ ∥ 2 F ≤ 5 d x d ϕ / ( λTH ) 5 / 6 } , where E D T ∼ A,ω exp [ · ] = E π ∼ ω exp [ E D T ∼ A,π [ · ]] denotes the expectation over trajectories generated by running policies π drawn according to ω exp on system A for T episodes, and

Proof. This result is a direct consequence of Theorem 6.1 of [2]-to obtain the result we must only verify that the assumptions of this result are met. We verify each assumption below.

Verifying Assumption 3 of [2]. Part 1 of Assumption 3 of [2] is met by Assumption 15 within diameter r µ ( A ⋆ ) . Furthermore, under Assumptions 1, 2, 4, 5 and 13 and by Lemma D.1, the additional parts of Assumption 3 of [2] are also met with diameter min { r cost ( A ⋆ ) , r θ ( A ⋆ ) } and smoothness constant poly( ∥ A ⋆ ∥ op , L ϕ , L θ , L π ⋆ , L cost , σ -1 w , H, d x ) .

Verifying Assumption 4 and Assumption 5 of [2]. Assumption 4 of [2] is immediately met by Assumption 16. Furthermore, Assumption 5 is met by Lemma F.2 with c cov = 1 , L cov ( θ ⋆ , γ 2 ) = H 2 B 3 ϕ σ 2 w · poly( d x ) , and C cov = 0 .

Given that these assumptions are met, the result follows noting that, if we run for T episodes, then the effective horizon is d x TH (using the mapping from the setting of (1.1) to the martingale regression setting described in Appendix A.1.1). Note that the final bound scales with 1 T instead of 1 d x TH as we are able to bring the d x H factor into the ˇ Λ π exp term, since Λ π exp is not normalized by d x H .

Lemma F.2. Under Assumption 1, for any policy distribution ω ∈ △ Π exp and A,A ′ , we have

Proof. We will prove that the desired bound follows for a particular π ∈ Π exp , which immediately implies that it holds for ω ∈ △ Π exp . By definition we have

and

Fix some v ∈ S d ϕ -1 , and note that, given the expression above, we have

It follows that

As in the proof of Lemma D.1, we have, for any ∆ ,

so we can bound

Furthermore, we can also bound ( v ⊤ ϕ ( x τ h , u τ h )) 2 ≤ B 2 ϕ . We therefore have

where the last inequality follows from Lemma A.1. It follows from the Mean Value Theorem that

As this holds for all v ∈ S d -1 , it follows that

G Additional Experimental Details

In this section, we provide additional details on our experimental results presented in Section 6. All experiments were run on a machine with 56 Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz CPUs, and 64GB RAM. All code was implemented in PyTorch .

G.1 Details on Problem Settings and Controller Parameterizations

We first expand on the precise definitions of the systems considered. As noted in Section 6, for the drone and car examples we set H = 50 , and for the system of Section 1.1 we set H = 10 . In addition, for all examples the noise is distributed as w h ∼ N (0 , 0 . 1 · I ) . In all cases we set γ 2 = 10 H (where γ 2 is a bound on E π exp [ ∑ H h =1 u ⊤ h u h ] ), and we therefore let Π exp denote the set of all policies satisfying E π exp [ ∑ H h =1 u ⊤ h u h ] ≤ γ 2 .

G.1.1 System of Section 1.1 (Figure 1)

The dynamics for this system are given by

This then corresponds to a system in the form (1.1) with

For this system, we parameterize our controller class Π ⋆ as, for any π θ ∈ Π ⋆ with parameter θ ,

Note that the form of this controller lets us simply 'match' the parameters of the system, and cancel undesirable parameters. Given this, for this system we let π ⋆ ( A ) be the controller which sets θ 1:11 to cancel the dynamics of the system A , and set θ 12 = c 1 = 10 .

See Appendix G.1.3 for details on the computation of H ( A ) on this system.

G.1.2 Drone System (Figure 2)

The dynamics of this system are given by

Here we interpret [ x ] 1:3 as the x, y, and z positions, respectively, and [ x ] 4:6 as the x, y, z velocities. This system is therefore equivalent to three double integrator systems, with an affine term (which we interpret as 'gravity') affecting only the z coordinate. We set the cost to

This then corresponds to a system in the form (1.1) with

For this system, we parameterize our controller class Π ⋆ as, for any π θ ∈ Π ⋆ with parameter θ ,

where θ fb h ∈ R 3 × 6 is the state-feedback portion of the controller, and θ offset h ∈ R 3 is an offset term. It can be shown that the optimal controller for a system of the form (G.1) can be parameterized in this way [23]. Furthermore, the optimal parameters can be computed in closed-form. As such, for this system we set π ⋆ ( A ) to be with the optimal parameters, computed using this closed-form solution.

In addition to computing the optimal controller in closed-form, we can also compute the cost of a controller, J ( π ; A ) , in closed-form. To compute H ( A ) in this example, we then simply apply the torch.autograd.functional.hessian function to J ( π ⋆ ( A ); A ) .

G.1.3 Car System (Figure 3)

The dynamics of this system are given by

where [ x h ] 5 denotes the 5th element of x h . Here we interpret [ x h ] 1 as the x position, [ x h ] 2 as the y position, [ x h ] 3 as the x velocity, [ x h ] 4 as the y velocity, [ x h ] 5 as the angle of orientation (that is, the direction the car is facing), and [ x h ] 6 as the angular velocity. The first control dimension, then, corresponds to the 'gas', the power given to the car to move forward or backward, and the second control dimension corresponds to altering the direction of the steering wheel. Similar to the drone system, we set the cost to

for some v 1 , v 2 . To write this in the form of (1.1), in order to make the problem more challenging we choose an overparameterized ϕ ( x , u ) :

For the car system, the controller class Π ⋆ is a hierarchical controller parameterized by some θ ∈ R 4 . This controller first uses PD control to compute a 'goal input', the direction we would like to modify the state in, as:

Given the underactuated structure of the system in (G.2), we cannot directly push the state in the direction of u goal ( x ) . Instead, we set u to the following:

Given the complex form of this controller and the dynamics, there does not exist a closed-form way to set θ optimally. Instead, for this system, we rely on a simple random search procedure to compute π ⋆ ( A ) . To find an optimal controller for system A , we randomly sample parameters θ , compute the cost they incur on system A , and then set π ⋆ ( A ) to the randomly generated controller with lowest cost. Note that this procedure is not differentiable, but we require π ⋆ ( A ) is differentiable. To remedy this, in situations where a differentiable π ⋆ ( A ) is needed (in particular, in the computation of H ( A ) ), rather than returning a single controller, we return the softmin distribution over all controllers sampled, weighting each controller by its estimated cost. As the softmin distribution can be differentiated, this parameterization of π ⋆ ( A ) is differentiable.

For this system, there does not exist a closed-form expression for J ( π ; A ) and, as such, to compute J ( π ; A ) , we simply perform many roll-outs of policy π on system A and average the cost. Given this and the search-based implementation of π ⋆ ( A ) outlined above, we found that computing the hessian H ( A ) using the torch.autograd.functional.hessian as in Appendix G.1.2 was very memory-intensive. Instead, we computed the Jacobian G ( A ) := ∇ A ′ J ( π ⋆ ( A ′ ); A ) | A ′ = A , and then, in place of H ( A ) , we use G ( A ) G ( A ) ⊤ . To compute G ( A ) , we use the torch.autograd.functional.jacobian function. While using G ( A ) G ( A ) ⊤ in place of H ( A ) is not justified by our theoretical analysis, if we are in settings where π ⋆ ( A ) is not precisely the minimum of J ( π ; A ) (which will likely be the case here since we are relying on a sampling-based implementation of π ⋆ ( A ) , which will incur some small error), then we argue that this is a reasonable metric to use. In particular, in this setting, the approximation of J ( π ⋆ ( ̂ A ); A ⋆ ) given in Proposition 1 should have an additional first-order term of the form G ( A ) ⊤ vec( A ⋆ -̂ A ) . As we can upper bound

optimizing for the metric G ( A ) G ( A ) ⊤ instead of H ( A ) can be seen as minimizing the first-order Taylor-approximation of the excess loss. Intuitively, this metric quantifies the sensitivity of the loss to particular parameters in A ⋆ , and in practice we found that optimizing this metric produced significant improvements over existing methods. The implementation of the example from Section 1.1 relied on this same approximation.

G.2 Implementation Details

For all methods considered, our implementation follows the basic structure of Algorithm 1: at every epoch, we explore so as to minimize some exploration objective, form an estimate of A ⋆ on the collected data, and then compute π ⋆ ( ̂ A t ) on our estimate. Our main experimental results (Figures 1 to 3) show the loss of π ⋆ ( ̂ A t ) as the time horizon t increases. For each method, to collect an initial set of data, we begin each trial by exploring randomly for some fixed number of episodes (10 for the drone example, 100 for the car example). The first point in each plot then corresponds to the performance after this initial random exploration. Each aspect of our implementation is modular, and any given component can be easily replaced. Below we highlight our implementation of the exploration routine, and choice of exploration objective, for the various approaches we consider.

G.2.1 Implementation of DYNAMICOED

Implementing the exploration procedure, DYNAMICOED, requires access to a regret minimization oracle. While in principle the LC 3 algorithm of [17] could be applied to this problem to give such an oracle, the LC 3 algorithm requires access to a computation oracle which is not clear how to implement in practice. To remedy this, we implement a Thompson Sampling-inspired modification to the LC 3 algorithm of [17].

The primary computational challenge of implementing the LC 3 algorithm is the computation of the optimistic policy:

where J exp ( π ; A ) denotes the exploration cost that is minimized in LC 3 (i.e. the expected cost on the cost function cost n h ( x , u ) ← 1 M · ϕ ( x , u ) ⊤ (Ξ n ) ϕ ( x , u ) set in DYNAMICOED), and BALL t the confidence set for A ⋆ at iteration t .

To avoid solving this optimization, we adopt a Thompson Sampling-inspired variation of this procedure. In particular, at iteration t , we sample ˜ A t ∼ N ( ̂ A t , Λ -1 t ) . Standard Thompson Sampling would then compute arg min π ∈ Π exp J exp ( π ; ˜ A t ) , but even this can be challenging, so we instead rely on a sampling MPC-inspired approach. Given that we are at state x h and have played inputs u 1 , . . . , u h -1 , we aim to approximately solve the following optimization:

To solve this approximately, we sample many possible u randomly, compute the value of the objective of (G.3) on the trajectories induced by these u , and finally choose the input that minimizes this objective. Rather than playing the entire sequence of chosen inputs, however, we simply play the first input in the sequence, observe the new state on the actual system, and re-solve (G.3) on this new state. Note that the implementation of LC 3 used for the experiments given in [17] relies on a similar Thompson Sampling-based approximation to the LC 3 algorithm.

G.2.2 Implementation of Uniform Exploration

The goal of the procedure we have referred to as Uniform Exploration is to collect data which will result in the estimation error, ∥ A ⋆ -̂ A ∥ op , being minimized, the goal of the method given in [1]. It can be shown that this is equivalent to maximizing λ min ( Λ T ) , so this method reduces to choosing

inputs that maximize λ min ( Λ T ) . To implement this procedure, we rely on the same sampling-based MPC approach as we outlined above, with the primary difference being that instead of minimizing the objective of (G.3), we choose the inputs that maximize

where Λ t denote the covariates we have obtained so far at iteration t . While very similar in spirit to the algorithm of [1], the implementation details are somewhat different than the algorithm proposed in that work. We found that in practice our implementation performed better than directly implementing (a sampling-based variant of) the algorithm from [1], and all reported results for Uniform Exploration are therefore on this version.

G.2.3 Exploring via Cost Minimization

A natural point of comparison to our methods would be to forsake the system identification phase entirely, and simply run standard policy optimization algorithms such as TRPO or PPO [61, 62], to obtain a controller ̂ π T . The primary difficulty with these approaches in the settings we consider is that these algorithms are on-policy , meaning that they primarily roll out trajectories using their current estimate of the optimal policy, ̂ π t , and using the collected data to do policy improvement on ̂ π t . In contrast, our setting is off-policy , in the sense that the learner must explore by playing policies in Π exp , but return some policy in Π ⋆ . Since in the settings we consider Π exp = Π ⋆ , on-policy approaches are simply exploring very differently, and therefore cannot be compared with directly.

̸

This is particularly an issue in our setting where stability may come into play. Indeed, it may be the case that some controller in Π ⋆ will destabilize the system, and cause the norm of the state to increase exponentially in h . Inducing such trajectories significantly improves one's ability to perform system identification as the signal-to-noise ratio also then increases exponentially. However, to induce this trajectory with a state-feedback controller, the power of the input played by this controller will also increase exponentially in h . Since we choose Π exp to include only policies with bounded power, this is not a fair comparison (and, furthermore, is likely not an algorithm one would want to run in practice).

While direct comparison with such approaches is therefore not possible, it is possible to compare against algorithms that, instead of collecting data that minimizes or maximizes objectives such as tr( H ˇ Λ -1 ) or λ min ( Λ ) , instead simply aims to play policies minimizing J ( π ; A ) . In principle, such algorithms are similar to approaches such as TRPO in how they perform their exploration-both collect data by aiming to minimize the actual cost we are attempting to find a controller to minimize.

To implement this approach, we rely on a sampling-based MPC algorithm similar to that described in Appendix G.2.1, but where the goal is now to solve

Note that the key difference between this approach and approaches such as TRPO is that we still only play π ∈ Π exp . Using this objective to induce exploration, we then simply estimate A ⋆ on this collected data, and return π ⋆ ( ̂ A ) . The results of this approach on the drone system are given in Figure 4 (with this cost minimization approach denoted as Cost Minimization Exploration ). As this illustrates, this approach is significantly worse than Algorithm 1, and is also outperformed by Uniform Exploration or Random Exploration when the number of episodes is large enough.

G.3 Additional Results

Finally, in this section we present versions of Figures 1 to 3 with error bars in Figures 5 to 7. In all figures, errors bars denote one standard error.

70

Task-Driven Exploration(Ours)

Mean Excess Controller Loss

60

RandomExploration

UniformExploration

50

CostMinimization Exploration

40

20

10

10

15

20

25

30

35

40

Number of Episodes

8

ControllerLoss

６５4

Task-Driven Exploration(Ours)

Random Exploration

Mean Excess

3

Uniform Exploration

2

1

0

30

40

50

60

70

80

90

Number of Episodes

70

Task-Driven Exploration (Ours)

Mean Excess Controller Loss

60

RandomExploration

Uniform Exploration

20

10

10

15

20

25

30

35

40

Number of Episodes

Task-Driven Exploration (Ours)

ControllerLoss

6000

RandomExploration

Uniform Exploration

5000

4000

Mean Excess

3000

2000

1000

100

200

300

400

500

600

Number of Episodes

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of exploration when learning to control non linear systems. In particular, they derive a task-driven exploration method which focuses on learning the system parameters that are relevant for the specific task that a controller is trying to achieve. \n\nTo derive the algorithm, they rely on a quantity which estimates how much the task loss vary, under a certainty-equivalence controller and the current model estimate. This quantity is the key to estimating what parameters in the model are relevant for the task at hand. It can be used by existing exploration algorithms (DynamicOED) as a minimization target.\n\nThe exploration routine is presented as a second contribution which extends an algorithm originally designed for linear MDPs to non-linear systems.\n\nSTRENGTHS:\nThe premises of the article that all parameters of the system might not be useful to know for the task of the controller makes a lot of sense, and the problem of task driven exploration is interesting and relevant. \n\nOverall, there is novelty as they extend a lot of existing concepts to a much broader class of systems (non-linear) and the contribution seems significant. However, it would be quite hard to implement the proposed algorithm from reading just the paper itself. \n\nThe introduction of the motivating example and associated discussion is quite useful to understand the problem space. \n\nThe related work section helps to position the scope of the paper and is mixing both RL and more traditional control.\n\nThe experiment section show that the proposed algorithm outperforms non task-driven exploration methods. The set of baselines is quite limited but as the concept is quite specific to this article, I believe they are a sound choice.\n\nWEAKNESSES:\nAlthough the paper is well structured and written, the main weakness is the presentation. Since the paper introduces a lot of mathematical concepts and assumptions, it is useful to help the reader by describing intuitively what they mean. \n\nThe assumptions presented are justified by citing previous work but some description would be useful to make the paper more self contained. Specifically assumption 3, it would be useful if the authors provide an intuitive justification.\n\nAssumption 8 lacks some general explanation. How does the choice of the underlying algorithm affect the performance? and why is the cost different than the goal $\\Phi$?\n\nA lot of the concepts are derived from reference [2] which essentially solves the problem in the case of linear systems. It makes the contribution more incremental, however the jump from linear to non-linear is still significant. \n\nTheorem 1 and 2 introduce bounds on the exploration loss which are claimed to be novel, it would still be useful to compare with the bounds for algorithms that are not task dependent. Is there a trade-off? \nSince section 5 is presented as a potentially independent contribution, it would have been useful to add more background about the experiment design problem and clearly highlight the difference with previous work.\n\nThe notation in dynamicOED is hard to follow, the explanation helps only a little. It would be quite hard to implement the algorithm just from reading the paper. It refers a lot to the LC3 algorithm which is not explained (this is certainly not my main expertise).\n\nThe systems use in the experiments have fairly small scales (6 dimensions for the state). One could wonder whether the approach scale in the number of dimensions. It is mention a linear dependency with the system parameter for the convergence. What about the computational cost of the algorithm?\n\nIn the experimental section, it is mentioned that MPC sampling based methods are used for exploration. How do we know that they verify assumption 8 which is key for convergence? \n\nMaybe the author could have tried linearizing the systems and applied the linear version of the algorithm that is cited.\n\nQUESTIONS:\n- Is there any physical system that would be like the motivating example?\n\n- What is D-optimal design? What to the optimization targets presented as example represent at line 270 in section 5?\n\n- Why is dynamicOED returning several exploration policies and not just one per step in algorithm 1?\n\n- How is the optimal controller $\\pi_*$ obtained to compute the excess controller loss?\n\nLIMITATIONS:\nThey assume a specific form of non-linear system which could limit the applicability of the solution (at least the theoretical results), however a few references are mentioned indicating that many systems fit into this category.\n\nThe bounds scale polynomially in the number of system parameters, it is not commented whether it is desirable or not. Couldn’t systems have a large number of parameters?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of task-relevant system identification. It starts with the motivation that not all parameters in the dynamics model are relevant to the given task; thus, the exploration should be placed to prioritize the identification of the relevant dynamics parameters, and thus eventually leading to improved efficiency of the model-based policy learning.\n\nThe authors begin with translating the task loss to the dynamics prediction loss, which is weighted by the cost hessian at ground truth dynamics model. The main result of this paper is to show that this dynamics loss can be approximated tightly by DynamicOED method. The proposed method is validated using some toy problems.\n\nOverall, the paper is well-motivated and presented. This paper did a great job in explaining the theorems. Although the interest of parametric dynamics model is restricted to weighted features, the main results of this paper may be of interest to the reinforcement learning community.\n\nSTRENGTHS:\nSee my comments in Summary\n\nWEAKNESSES:\n- In the motivating example, the explanation of task-irrelevant or relevant parameters are confusing. Shouldn't $a_{3:12}$ be irrelevant to learning the optimal controller, while $a_{1:2}$ critical?\n- Please further explain \"In practice, though they may not formally satisfy 290 the guarantee of Assumption 8, deep RL approaches could be used.\"\n- I think Section 5 a bit deviates from the main problem formulation, especially starting with a new system Equ. (5), although I understand the authors are trying to introduce DynamicOED generally for a broad interest of the readers. \n- The authors could give some perspectives on the promise of the results applied to more general dynamics.\n- I strongly suggest adding more or complex tasks to demonstrate the efficiency of the proposed methods over non-optimal exploration model-based policy learning.\n\nQUESTIONS:\nSee my comments in Weaknesses.\n\nLIMITATIONS:\nSee my comments in Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an active exploration algorithm for systems of the form $x_{k+1} = A \\phi(x_k, u_k) + w_k$ with known features $\\phi$ and unknown matrix $A$, where $w_k \\sim \\mathcal{N}(0,1)$. The problem is reduced to minimising $\\mathrm{tr}(\\mathcal{H}(A_\\star)\\check{\\Lambda}_T^{-1})$ where $\\mathcal{H}(A_\\star)$ is the Hessian at the optimal $A_\\star$ (which is iteratively estimated) and $\\check{\\Lambda}_T^{-1}$ is roughly the expected covariance $\\phi\\phi^T$ over $T$ episodes of length $H$. Thorough analysis is presented. The algorithm achieves instance-optimal rate. Examples on simulated systems are provided.\n\nSTRENGTHS:\nVery strong theoretical paper, solving optimal exploration for a class of linear-in-features systems. Originality, quality, clarity, are excellent. Significance is strong, but since the features are assumed known, it limits the applicability of the algorithm.\n\nWEAKNESSES:\n- The class of systems is quite restricted. If one needs to learn the non-linear features $\\phi(x, u)$ first, then one could just as well use the collected data to estimate $A$ afterwards. It would be interesting to see an extension where features are learned as well.\n- The computational complexity is unclear. Does the algorithm run real-time? Is the numerical implementation stable?\n- There is no Discussion/Conclusion/Limitations section.\n- Empirical comparisons are only done with respect to very naive baselines (random and uniform exploration). At least some variance or entropy minimising active exploration could be considered, as e.g., in [1]. But for this paper it is fine, the theoretical contributions are already sufficient, no further experiments are necessary.\n\n[1] Schultheis, M., Belousov, B., Abdulsamad, H., & Peters, J. (2020). Receding horizon curiosity. In Conference on robot learning (pp. 1278-1288). PMLR.\n\nQUESTIONS:\nCould one further improve the algorithm for control-affine systems? I.e., $x' = A\\phi(x) + Bu$ with unknown $A$ and $B$?\n\nLIMITATIONS:\nNo. The authors are encouraged to add at least one paragraph of Conclusion, which also discusses the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper tackles the challenge of controlling unknown nonlinear dynamical systems in reinforcement learning and control theory. The authors propose a novel algorithm, inspired by recent work in linear systems, focusing on the most critical parameters for learning a low-cost controller on the actual system. This method efficiently explores the system to reduce uncertainty in a specific, task-dependent metric. They demonstrate the algorithm's effectiveness on nonlinear robotic systems and provide a theoretical lower bound showing that their approach learns a controller at a near instance optimal rate. This work could have significant implications for reinforcement learning and control theory.

STRENGTHS:
The author presents an algorithm which achieves the instance-optimal rate, with controller loss matching the lower bound on the loss of any sufficiently regular control rule.

WEAKNESSES:
The regularity assumptions in 3.1 should be addressed clearly and the author could give a toy example to illustrate when these assumptions are satisfied.

QUESTIONS:
The formulation of the nonlinear system in (1.1) seems restricted, can you explain more about the formulation?

LIMITATIONS:
The simulations are conducted on low dimension systems and the experiment setting should be stated more clearly. Besides, this work faces a high computational burden.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper seeks to quantify formally in the settings of nonlinear dynamical systems (a) which parameters are most relevant to learning a good controller and (b) the best exploration strategy to minimize uncertainty in such parameters. This paper draws theoretical insights where minimizing the controller loss in nonlinear systems translates to estimating the system parameters in a particular task-dependent metric. In light of this characterization, the paper presents an algorithm that achieves the instance-optimal rate for efficient exploration, with controller loss matching a proven lower bound. Several numerical experiments are conducted on nonlinear systems to validate the analysis and the approach.\n\nSTRENGTHS:\n- **S1** The paper is very well written. The problem definitions and contributions are clear.\n- **S2** For a particular class of nonlinear systems of the form (Eqn. 1.1), the paper characterizes how minimizing the controller loss in nonlinear systems translates to estimating the system parameters in a task-dependent metric, and provides a lower bound on the loss of any control rule.\n- **S3** The paper proposes an algorithm that achieves the instance-optimal rate, with controller loss matching the proven lower bound. \n- **S4** The proposed approah demonstrates improvements compared to the baselines in several numerical experiments.\n\nWEAKNESSES:\n- **W1** The paper could benefit from having stronger experimental results. The authors could include more simulated environments that fit into the particular class of nonlinear systems, both in simulation and the real world, such as [1]. The authors could also consider comparing with more baseline methods, such as [2] and [3], from the deep model-based RL community.\n- **W2** Could the authors elaborate on how theoretical insights from the paper relate to problems in practical applications of the particular class of nonlinear systems (1.1)? I believe this would improve the paper's impact on the broader robotics, control, and RL communities. For instance, could applications in safety-critical scenarios be benefited from these insights (e.g., flight maneuvers in [1])?\n\n[1] O’Connell et al., \"Neural-fly enables rapid learning for agile flight in strong winds.\" Science Robotics, 7(66):eabm6597, 2022.\n\n[2] Kaiser et al., \"Model-based reinforcement learning for atari.\"  arXiv preprint arXiv:1903.00374, 2019.\n\n[3] Hafner et al., \"Dream to Control: Learning Behaviors by Latent Imagination.\" arXiv preprint arXiv:1912.01603, 2020.\n\nQUESTIONS:\n- The authors mentioned in line 378 that they believe the approach will also scale to deep model-based RL settings. Could the authors explain how their approach can be scaled with deep RL?\n\nLIMITATIONS:\n- The authors target a particular class of nonlinear systems characterized in (1.1). This characterization would determine what kind of systems the theoretical insights and the approach can be directly related to.\n- The authors provide experimental analysis on simulated domains. It would be interesting to see the results in real-world environments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}

comments: SUMMARY:
In this paper, the authors propose a method for learning an optimal controller that is able to focus the exploration so as to minimize the estimation error of relevant parameters. They show that the proposed method is indeed optimal with tight upper and lower bounds on the gap. Experiments show that the proposed algorithm performs better compared to the baseline reaching lower loss at any given number of learning episodes.

**After authors' rebuttal:** Thanks for the added clarification. I agree that the paper's contributions are largely theoretical and there's enough here to merit a publication. The authors mention their intention for addition of proof sketches and intuitive discussion which will improve the readability of the paper further. Given these, I am keeping my rating at 7!

STRENGTHS:
The paper is clearly-written, well-motivated and systematically-structured. The assumptions are explicit and the theorems are described to develop intuition. While the idea of certain parameters dictating the system dynamics might make abstract sense, having the motivating example in section 1.1 made the follow-up analysis more grounded for me. Using the second order moment to quantify parameter relevance is neat but that appears to be adopted from previous work. In general, the theory part of the paper is crisp and laid out to facilitate understanding.

WEAKNESSES:
On the other hand, the experiment section is quite thin. However, as a largely theoretical contribution, this is acceptable. I didn’t look through the appendix so I missed out on the proofs, and the paper could have benefitted from some proof sketches without hashing out all the details.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-28