Initial Value Problem Enhanced Sampling for Closed-Loop Optimal Control Design with Deep Neural Networks
Authors: Xuanxi Zhang
Jihao Long
Wei Hu
Weinan E
Jiequn Han
Venue: Submitted to ICLR 2023 (2023)
Keywords: Optimal Control
Deep Learning
Adaptive Sampling
Distribution Mismatch

Abstract: Closed-loop optimal control design for high-dimensional nonlinear systems has been a long-standing challenge. Traditional methods, such as solving the associated Hamilton-Jacobi-Bellman equation, suffer from the curse of dimensionality. Recent literature proposed a new promising approach based on supervised learning, by leveraging powerful open-loop optimal control solvers to generate training data and neural networks as efficient high-dimensional function approximators to fit the closed-loop optimal control. This approach successfully handles certain high-dimensional optimal control problems but still performs poorly on more challenging problems.  One of the crucial reasons for the failure is the so-called distribution mismatch phenomenon brought by the controlled dynamics. In this paper, we investigate this phenomenon and propose the initial value problem enhanced sampling method to mitigate this problem. We theoretically prove that this sampling strategy improves over the vanilla strategy on the classical linear-quadratic regulator by a factor proportional to the total time duration. We further numerically demonstrate that the proposed sampling strategy significantly improves the performance on tested control problems, including the optimal landing problem of a quadrotor and the optimal reaching problem of a 7 DoF manipulator.

Under review as a conference paper at ICLR 2023
INITIAL VALUE PROBLEM ENHANCED SAMPLING FOR CLOSED-LOOP OPTIMAL CONTROL DESIGN WITH DEEP NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Closed-loop optimal control design for high-dimensional nonlinear systems has been long-standing challenge. Traditional methods, such as solving the associ- ated Hamilton-Jacobi-Belle equation, suffer from the curse of dimensionality. Recent literature proposed new promising approach based on supervised learn- ing. by leveraging powerful open-loop optimal control solvers to generate training data and neural networks as efficient high-dimensional function approximators to fit the closed-loop optimal control. This approach successfully handles certain high-dimennsionl optimal control problems but still performs poorly on more chal- lenging problems. One of the crucial reasons for the failure is the so-called distri- bution mismatch phenomenon brought by the controlled dynamics. In this paper, we investigate this phenomenon and propose the initial value problem enhanced sampling method mitigate this problem. We theoretically prove that this sam- pling strategy improves over the vanilla strategy on the classical linear-quadratic regulator by factor proportional to the total time duration. We further numeri- cally demonstrate that the proposed sampling strategy significantly improves the performance on tested control problems, including the optimal landing problem of quadrotor and the optimal reaching problem of 7 DoF manipulator.
INTRODUCTION
Optimal control aims to find a control for a dynamical system over a period of time such that specified loss function is minimized. Generally speaking. there are two types of optimal controls: open-loop optimal control and closed-loop (feedback) optimal control. Open-loop optimal control deals with the problem with given initial state, and its solution is function of time for the specific initial data, independent of the other states the system. In contrast, closed-loop optimal control aims to find the optimal control policy function of the state that gives us optimal control for general initial states.
By the nature of the problem, solving the open-loop control problem is relatively easy and various open-loop control solvers can handle nonlinear problems even when the state lives in high dimen- sions (Betts) 1998 Rao 2009). Closed-loop control is much more powerful than open-loop control since it can cope with different initial states, and it is more robust to the disturbance of dynam- ics. The classical approach to obtaining closed-loop optimal control function is by solving the associated Hamilton-Jacoke Racobi-Bellman (HJB) equation. However, traditional numerical algorithms for HIB equations such the finite difference method or finite element method face the curse of dimensionality (Bellman 1957) and hence can not deal with high-dimensional problems.
Since the work Han F. 2016) for stochastic optimal control problems, there have been growing interest on making use of the capacity of neural networks (NNs) in approximating high- nigh-dimensional
functions to solve the closed-loop optimal control problems (Nakamura- Zimmeerer et al. 202 la b)
2020 Böttcher et al. 2022 E et al.] 2022). Generally speaking, there are two categories of methods in this promising direction. One is policy search approach (Han & E 2016 Ainsworth et al. 2021 Böttcher al. 2022 Zhao et al. 2022), which directly parameterizes the policy function by neural networks, computes the total cost with various initial points, and minimizes the average total cost.
When solving problems with long time span and high nonlinearity, the corresponding optimization

---

Under review as a conference paper at ICLR 2023
problems can be extremely hard and may get stuck in local minima Levinne & Koltun 2014). The
other category of methods is based on supervised learning Nakamura-Zimmerer Zimmerer et al. 202 b
2020 2022). Combining various techniques for open-loop control, one can solve complex high- dimensional open-loop optimal control problems; see BBtts 1998); Rao (2009); Kang et al. (2021) for detailed surveys. Consequently, can collect optimal trajectories for different initial points as training data, parameterize the control function (or value function) using NNs, and train the NN models to fit the closed-loop optimal controls (or optimal values). This work focuses on the second approach and aims to improve its performance through adaptive sampling,
As demonstrated in Nakamura- Zimmerer et al. (2021b); Zang et al. (2022), NN controllers trained by the vanilla and test error on collected datasets are fairly small. Some existing works attribute this phenomenon to the fact that the learned controller may deteriorate badly at some difficult initial states even though the error is small in the average sense. Several adaptive sampling methods regarding the initial points are hence proposed (see Section 4 for a detailed discussion). However, these methods all focus on choosing optimal paths according to different initial points and ignore the effect of dynamics. This is an issue since the paths controlled by the NN will deviate from the optimal paths further and further over time due to the accumulation of errors. As shown in Section 6 applying adaptive sampling only initial points is insufficient solve challenging problems.
learning- based approach can perform poorly even when both the training error
This work is concerned with the so-called distribution mismatch phenomenon brought by the dy- namies in the serviised-eemming-beeeee based approach. This phenomenon refers the fact that the discrepancy between the state distribution of the training data and the state distribution generated by the NN controller typically increases over time and the training data fails represent the states encountered when the trained NN controller is used. Such phenomenon has also been identified in reinforcement learning (Kakade & Langford 2002 Long Han 2022] and imitation learning (Ross& & Bagnell, 2010). To mitigate this phenomenon, we propose the initial value problem (IVP) enhanced sampling method to make the states in the training dataset more closely match the states that the controller reaches. In the IVP enhanced sampling method, we iteratively re-evaluate the states that the NN controller reaches by solving IVPs and recalculate new training data by solving the open-loop control problems starting at these states. Our sampling method is very versatile to be combined with other techniques like a faster open-loop control solver or better neural network structures. The resulting supervised-leaming based approach empowered by the IVP enhanced sam- pling can be interpreted as an instance of the exploration-labed
training (ELT) algorithms (Zhang) al., 2018 E et al. 2021) for closed-loop optimal control problems (see Appendix A for more discussions). At a high level, the ELT algorithm proceeds iteratively with the following three steps: (1) exploring the state space and examining which states need be labeled; (2) solving the control problem to label these states and adding them to the training data; (3) training the machine learning model.
The main contributions of the paper can be summarized as follows. (1) We investigate the distri- bution mismatch phenomenon brought by the controlled dynamics in the pnpervised-lermingg based approach, which explains the failure of this approach for challenging problems. We propose the IVP enhanced sampling method to update the training data, which significantly alleviates the distri- bution mismatch problem. (2) We show that the IVP enhanced sampling method can significantly improve the performance of the learned closed-loop controller on a uni-dimensional linear quadratic control problem (theoretically and numerically) and two figh-dimensiooal problems (numerically), the quadrotor landing problem and the reaching problem of a 7-DoF manipulator. (3) We compare the IVP enhanced sampling method with other adaptive sampling methods and show that the IVP enhanced method gives the best performance.
2 PRELIMINARY
2.1 OPEN-LOOP AND CLOSED-LOOP OPTIMAL CONTROL
We consider the following deterministic controlled dynamical system:
(t(() f(1 (1), u()), t e [to: 7]
(I)
a=(o)) = a
2

---

Under review as a conference paper at ICLR 2023
where xxt) E R" denotes the state, u(t) U C IR' denotes the control with U being the set of admissible controls. f: [] x R" x U + R" is a smooth function describing the dynamics, to [0, [] denotes the iniiil time, and Ø IR" denotes the initial state. Given fixed to [0, 7 and Zo E IR", solving the open-loop optimal control problem means to find a control path u" lo: 7 + U to minimize
['" L(t, (t), u(t) + (x(TT) s.t. (a, u) satisty the system !
J(u; to: 20)
where L: [0.7] X TR" x 4 - R and M: R. + R are the running cost and terminal cost, re- spectively. We use x= (1; ln. 2Dn) and u" (t; ln, 2o) denote the optimal state and control with the specified initial time to and initial state 2g. which emphasizes the dependence of the open-loop op- timal solutions on the initial time and state. We assume the open-loop optimal control problem is well-posed, i.e., the solution always exists and is unique.
In contrast the open-loop control being a function of time only, closed-loop control is a function of the sime-state pair (t, z). Given closed-loop control u: [0, x R" + UI we can induce a family of the open-loop controls with all possible initial time-state pairs (to, zo):
u(t: to æo) u(t, Du (t; to, æo)),
where Tu (t; tu, Do) is defined by the following initial value problem (IVP):
tu (t: to, zu) ft, xu (t; to, æo), ult, Tu (t; to. æo)), e [to.T]
IVP lo T, u))
Tu (fo: to, xo) = Zg-
To case the notation, always use the same character to denote the closed-loop control function and the induced family of the open-loop controls. The context of closed-loop or open-loop control can be inferred from the arguments and will not be confusing. It is well known in the classical optimal control theory (see, e.g. Liberzon (201T) that there exists a closed-loop optimal control function u² [0, x R ll such that for any to E [0, 7" and 26 E IR"
(1; lo. mo) u² (1, x:" (1; lo, mo)),
which means the family of the open-loop optimal controls with all possible initial time-state pairs can be induced from the closed-loop optimal control function. Since IVPs be easily solved, one can handle the open-loop control problems with all possible initial time-state pairs if good closed- loop control solution is available. Moreover, the closed-loop control is more robust to dynamic disturbance and model misspecification, and hence it is much more powerful in applications.
In this paper, goal is find near-optimal closed-loop control á such that for Do X C TR with being the set of initial states of interest, the associated total cost near-optimal, i.e.,
[J ::00 zo): 0, mo) J[u" (-; (,;0, zg): 0, 2x)] small.
2.2 SUPERVISED-LEARRIIG BASED APPROACH FOR CLOSED-LOOP OPTIMAL CONTROL PROBLEM
Here we briefly explain the idea of the supervised-leaming-baass approach for the closed-loop op- timal control problem. The first step is generate training data by solving the open-loop optimal control problems with zero initial time and initial states randomly sampled in X. Then, the training data is collected by evenly choosing points in every optimal path:
D = {ttdd x ()), u hIsSMMIISS.
where M and N are the number of sampled training trajectories and the number points chosen in each path, respectively. Finally, function approximator (mostly neural network, as considered in this work) with parameters is trained by solving the following regression problem:
M
N
min 0
2 u
(3)
MN
i=1
and gives the NN controller u NN
3

---

Under review as a conference paper at ICLR 2023
3 IVP ENHANCED SAMPLING METHOD
Although the vanilla Zimmerer problems Zimmerre et al. [2211a), it is observed that its performance on complex prob- lems is not satisfactory (see Nakamura-Zimnn Zimmerer et al. (202bb) Zang et al. (2022) and examples below). One of the crucial reasons that the vanilla method fails is the distribution mismatch phe- nomenon. To better illustrate this phenomenon, let μo be the distribution of the initial state of interest and u. : [0, IR + l be closed- loop control function. We use (L) denote the distribution of (t) generated by u: dr(1)
based approach can achieve a good performance in certain
f [1, (r), ul,, ax())), 20 ~ Po- Note that in the training process 3). the distribution of the state at time is u (t), the state distribution generated by the closed-loop optimal control. On the other hand, when we apply the learned NN controller in the dynamics, the distribution of the input state of NN time t (n (t). The error between state 2: driven by u² and u N accumulates and makes the discrepancy between Yuu (t) and Hu (f) increases over time. Hence, the training data fails to represent the states encountered in the controlled process, and the error between u² and NN dramatically increases when t is large. See Figures (left) and below for an illustration of this phenomenon.
To overcome this problem, we propose the following IVP enhanced sampling method. The key idea is to improve the quality of the NN controller iteratively by enlarging the training dataset with the states seen by the NN controller previous times. Given predesigned (not necessarily even-spaced) temporal grid points 0 = to < t < ... < tg- = T, we first generate a training dataset So by solving open-loop optimal control problems a set of initial points sampled from (4o, and train the initial model üg. Under the control of üo. the generated trajectory deviates more and more from the optimal trajectory. So we stop at time (1, ie., compute the IVPs using ún as the closed-loop control and points in Xo as the initial points on the time interval [0, t1]. and then on the interval [t,,T] solve new optimal paths that start the end- points of the previous IVPs. The new training dataset S1 is then composed of new data (between t and T) and the data before time 11 in the dataset So, and we train a new model â using S1. We repeat this process to the predesigned temporal grid points 12, l3. until end up with T. In other words, in each iteration, the adaptively sampled data replaces the corresponding data (defined on the same time interval) in the training dataset (the size of training data remains the same). The whole process can be formulated as Algorithm [ and we refer to Figure for an illustration of the algorithm's mechanism. We call this method IVP enhanced sampling method because the initial points of the open-loop optimal control problems are sampled by solving the IVP with the up-to- date NN controller. It is worth noting that the later iterations require less effort in labeling data as the trajectories are shorter and thus easier to solve.
the time interval [o, 7] starting from points in Xo.
space
space
mange 0501
Region
stalc
aage
Region
Rejion
mange
Region
Region 11
range stale
Ragbon
Ragenil
4 first iteration
4
4
1mm
before iteration
second Iteration
Region
Optimal paths
Regian
controlled by Ag
Region
Optimal pathe from the reached by A, at
Regionn
Paths controllad by i
Region
Optimal paths from the states reached by ai; at
The first
policy & trained on
Region
data)
The second policy u trained on { Region
data [O. 1.): U Region III
data
The final
policy R. la trained [ Region data in [0, 1)) U Region III data in Dy U Region V
data
Figure 1:
An illustration of how the proposed IVP enhanced sampling (Algorithm 1
works when
there are two intermediate temporal grid points é and tz.
is worthwhile mentioning that the IVP enhanced sampling method versatile enough to com- bine other improvements for closed-loop optimal control problems, such efficient open-loop con- trol problem solvers (Kang et al. 2021 Zang et al.] 2022) or specialized neural network structures
Naaaamur-/Zmmmett et al. 2020 202 2022). One design choice regarding the network structure
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 IVP enhanced sampling method for closed-loop optimal control design
1: Input: Initial distribution No, number of time points K, temporal grid points 0 = to < t1 < < lK = T, time step number of initial points N.
2: Initialize: S = 0, û_1 (1, a) = O.
3: Independently sample
initial points from po to get an initial point set Xo.
4: for i 0, ... KK do
5: For any 20 E Xo, compute IVP (æ0, ú;-1) according to 2
EExploration
6: X, ( (14; 0, 2.) z Xo)-
7:
For any 2; E X,, call the open-loop optimal control solver to obtain x² (t; 4, æ;) and Labeling
""(G 4, w.) for E T..
8:
Set St = {11, (1; (1:4,, m.), u² (1; (1;1, m.)) : a, E X,, E [ti, T), (1)1// N).
9:
Set S; = S; Uf(t, u) it < ti, (t, x, u) E S1-1)
10:
Train üi with dataset St.
>Training
11:
end for
12: Output: -1
in the IVP enhanced sampling method whether to share the same network among different time intervals. We choose to use the same network for all the time intervals in the following numerical examples, but the opposite choice is also feasible.
4 COMPARISON WITH OTHER ADAPTIVE SAMPLING METHODS
In this section, we review existing literature on adaptive sampling methods for the closed-loop opti- mal control problem.
We start with the methods in imitation learning (Hussein et al. 2017), which aim to learn the ex- pert's control function. Our task can be viewed as an imitation problem if we take the optimal control function as the expert's control. With the same argument, match phenomenon also exists therein. However, there is a key difference regarding the mecha- nism of data generation between the two settings: in imitation learning, it is often assumed that one can easily access the expert's behavior at every time-state pair while in the optimal control problem, it is much more computationally expensive access since one must solve an open-loop optimal control problem. This difference affects algorithm design fundamentally. Take the for- ward training algorithm (Ross & Bagnell 2010), a popular method in imitation learning for mitigat- ing distribution mismatch, as an example. To apply it to the closed-loop optimal control problem, we first need to consider discrete-ting time version of the problem with sufficiently fine time grid: 0 < ti < ... < tk= T. At each time step 4; we learn policy function ü' RR" + l where the state x in the training data are generated by sequentially applying uo ...., u--1 and the labels generated by solving the open-loop optimal solutions with (t,, 2) the initial time-state pair. Hence, the open-loop control solver is called with numbers proportionally to the discretized time steps K', and only the first value on each optimal control path is used for learning. In contrast, in Algorithm|]
know the distribution mis-
[] we can use much more values over the optimal control paths in learning, which allows its temporal grid for sampling to be much coarser than the grid in the forward training algorithm, and the total cost of solving open-loop optimal control problems is much lower.
Another popular method in imitation learning is DAGGER (Dataset Aggregation) (Ross et al. 201 which can also be applied to help sampling in the closed-loop optimal control problem. In DAG- GER, in order to improve the current closed-loop controller ú, one solves IVPs using á over [0, T] starting from various initial states and collect the states on time grid 0 < f1 < ... < stx-1 < T. The open-loop control problems are then solved with all the collected time-state pairs as the ini- tial time-state pair, and all the corresponding optimal solutions are used to construct a dataset for learning new controller. The process can be repeated until a good controller is obtained. The time- state selection in DAGGER is also related to the distribution mismatch phenomenon, but somehow different from the IVP enhanced sampling. Take the data collection using the controller ú in the first iterative step for example. The IVP enhanced sampling focuses on the states the time grid t while DAGGER collects states at all the time grids. ú is still far from optimal, the data collected later time grids may be irrelevant to even mislead training due error accumulation in states.
5

---

Under review as a conference paper at ICLR 2023
In Appendix G reports more theoretical and numerical comparison between DAGGER and IVP enhanced sampling, which indicates DAGGER performs less satisfactorily.
Except for the forward training algorithm and DAGGER, there other adaptive sampling meth- ods for the closed-loop optimal control problems. Nakamura- Zimmerer et al. (2021a) propose an adaptive sampling method that prefers choose the initial points with large gradients of the value function as the value function tends to be steep and hard to learn around these points. [andry et al. (202I) propose sample the initial points on which the errors between predicted values from the NN and optimal values are large. These two adaptive sampling methods both focus on finding points that are not learned well but ignore the influence of the accumulation of the distribution mismatch over time brought by controlled dynamics. We will show in Section 6 that the IVP enhanced sam- pling method can outperform such sampling methods.
THEORETICAL ANALYSIS ON AN LQR EXAMPLE
In this section, we analyze the superiority of the IVP sampling method by considering the following uni-dimensional linear quadratic regulator (LQR) problem:
1
{
min =(0),(tt) T [u()) + (xx(T))²
s.t. (()) = u()), E (to 7), s<(t) =
where 7' is positive integer, to E [0. and zo IR. Classical theory linear quadratic control (see, e.g. Sontag (013)) gives the following explicit linear form of the optimal controls:
u (t: = T(T 10)+1
(open-loop optimal control)
u (t, (), = T(T-t)+1
(closed-loop optimal control)
We consider the following two models approximate the closed-loop optimal control function with parameter 8:
T Model 1: un(t,z) T( T b(t), where 0 (0, [0.JosesT (b(t))
b(t),
(4)
Model 2: up(t,z) = a(t) + b(t), where (0, JosEsT = b(t)
(5)
Since there will be no error in learning linear model when the data is exact, to mimic the errors encountered when learning neural networks, throughout this section, we assume the data has certain noise. To be precise, for any to [0. 7' and zo E R, the open-loop optimal control solver gives the following approximated optimal path:
ú(;; to, zo) = T(T ((T--6)-14 To + eZ,
T(T- a(t; 20) = zo + St, a(t to ro)dt - to)e
where 0 is small positive number indicate the scale of the and Z is a normal random variable whose mean is m and variance is o². In other words, the obtained open-loop control is still constant in each path, just like the optimal open-loop control, but perturbed by random constant. The random variables in different approximated optimal paths starting from different to or zo are assumed be independent.
We compare the vanilla pnpervised-learmiing based method and IVP enhanced sampling method the- oretically for the first model and numerically for the second model (in Appendix [B) In the vanilla method, we randomly sample NT initial points from a standard normal distribution and use corre- sponding optimal paths learn the controller. In the IVP enhanced sampling method, we randomly sample N initial points from a standard normal distribution, set the temporal grid points for sam- pling as < < ... < T < T, and perform Algorithm [ both methods, the open-loop optimal control solver is called NT times in total.
Theorem compares the performance of the vanilla method and the IVP enhanced sampling method under Model 4 The more detailed statement and proof can be found in Appendix ( This theo- rem shows that both the distribution difference and performance difference with respect to the opti- mal solution for the vanilla method will increase when 7 increases, while they are always constantly

---

Under review as a conference paper at ICLR 2023
bounded for the IVP enhanced sampling method. Therefore, compared to the vanilla method, the
IVP enhanced sampling method mitigates the distribution mismatch phenomenon and significantly improves the performance when T is large.
Theorem 1. Under Model I ② let uo. By and ug be the optimal controller, the controller learned by the vanilla method, and the controller learned by the IVP enhanced sampling method, respectively. Define IVPs: (1) = u. (1) = u,(l,, Is ((), Iz = Tinit, O S L. L T,
8 E (0, U, a).
1. If Timi is random variable following standard normal distribution, which is independent of the initial points and noises in the training process. Let (#2, ( the state variables in the training data of the vanilla method and the last iteration of the IVP enhanced sampling method. Then, (t), (t),), zelt) and (t) are normal random variables, E.. (t) = Er, (t), (t) = Exa(t (t) and
VT and { (1)) be
1 [0)2 (1) (1 NT 2
[EE) - Elra(0)11 e.
2. If Tini is a fixed initial point, define the total cost
Jg J== = % (1sa))) S (a
Then, EJg, Jo = (T2 + +1mmm² + NT e2 EJa - Jo s 3(m² + o²
N )e²
THE OPTIMAL LANDING PROBLEM OF QUADROTOR
In this section, we test the IVP enhanced sampling method on the optimal landing problem of quadrotor. We consider the full quadrotor dynamic model with 12-dimensional state variable and 4- dimensional control variable (Bouabdallah et al.) 2004 Madani & Benallegue 2006 Mahony al., 2012). We aim to find optimal landing paths from some initial states To to a target state 27 = 0 with minimum control efforts during a fixed time duration T' = 16. The open-loop optimal solutions are obtained by solving the corresponding two-point boundary value problems with the space-marching technique (Zang al. 2022). See Appendix C and more details.
We sample N 500 initial points for generating training data and and use a fully-connected neural network to approximate the optimal control. The temporal grid points on which we do IVP enhanced sampling is 0 < 10 < 14 < 16. After learning, we use learned models to run the initial value problem at 500 training initial points and show the similarity between paths controlled by the NN controller and their corresponding training data. In Figure 2 the left sub-figure shows the average pointwise distance between data reached by the NN controller and corresponding training data at different times. The right sub-figure shows the maximum mean discrepancy (Borgwardt et al. 2006)
between these two datasets using Gaussian kernel k(x,y/ y) = cxp(- || ). In both figures, there jumps t = 10 and 14 since the NN-controlled path is continuous across time while training data is discontinuous at locations where we do IVP enhanced sampling. It can be seen that without adaptive sampling (after iteration 0), the discrepancy between the states reached by the NN controller and training data is large. With our method, they get closer to each other as the iteration goes.
Average distance
pollcy after Iteration 2 policy iteration policy after iteration D
105
policy after iteration
15-
policy after iteration
10:1
policy after iteration
MMD
10.
10
5-
10-3
0-
0
2
10
12
16
10
122
14
36
Time
Time
Figure 2: Left: the average pointwise distance between the training data and the data reached by controllers at different times. Right: the maximum mean discrepancy (in the logarithm scale) be- tween the training data and the data reached by controllers every time using the Gaussian kernel.
7

---

Under review as a conference paper at ICLR 2023
Next check the performance of the learned NN controller. Figure 3 (left) and Figure (middle) show the cumulative distribution function of the ratio between NN-controlled cost and optimal cost on 500 training initial points and 200 test initial points, respectively. Without any adaptive sampling, the performance of model Vio is poor even on the training set. In contrast, the NN model is very close to the optimal policy after two rounds of adaptive sampling. More statistics of the results are provided in Appendix D
On training inital points
On 200 test istial points
Oi 250 test instial pnints
100%
100%
100%
000
90%
60%
percenzge
60%
60%
percentipe
80%
40%
40%
40%
20%
20%
10
1.1
3.5
2.0
5.0
10.0
policy cost/ rppiiaallooo cost
policy cost/ optimal cost
policy cost motimall cost
Figure 3: Cumulative distribution function of the cost ratio (with the ideal curve being a straight horizontal segment passing ratio = 1. percentage=100)) Left: results of the IVP enhanced method on 500 training iittill points. Middle: results of the IVP enhanced method on 200 test initial points. Right: comparison between different sampling methods.
Then we compare our method with four other methods. As data generation is the most time- consuming part, for the sake of faimess, we keep the number of solving open-loop problems the same (1500) among all the methods (except the last method). The first method is training a model on directly sampled I 1500 optimal paths (called vanilla sampling). The second method is the adaptive sampling (AS) method proposed by Nakamura- Nakamura-Zimmerer et al. (202Ia) that chooses initial points with large gradient norms. This is equivalent to choosing initial points whose optimal control has large norms, and we refer to this method as AS w. large u. With a little modification, the third method, AS w. large v, is to choose initial points whose total costs are large under the latest NN controller. The last adaptive sampling method, AS w. bad v. is a variant of the SEAGuL algorithm (Sample Efficient Adversarially Guided Learning) (Landry al. (021). The original SEAGuL algo- rithm proposes to use a few gradient ascent updates to find initial points with large gaps between the learned values and optimal values. Here we give this method more computational budget to solve more open-loop optimal control problems to find such initial points. The cumulative distribution functions of cost ratios of the above methods are shown in Figure 3 (right), which clearly demon- strate the superiority of the IVP enhanced sampling method. More details about the implementation and results of these methods are provided in Appendix
We also test the NN controllers obtained from different sampling methods in the presence of obser- vation noises, considering that the sensors have errors in reality. The detailed results are provided in Appendix [F] It is observed that when measurement errors exist, closed-loop controllers are more reliable than the open-loop controller and the one trained by the IVP enhanced sampling method performs best among all the considered methods. Finally, grid points in Algorithm | and train networks on the same 500 initial points. The results listed in Appendix E show that our algorithm is robust to the choice of temporal grid points.
test 4 different choices of temporal
THE OPTIMAL REACHING PROBLEM OF A 7-DOF MANIPULATOR
In this section, we consider the optimal reaching problem on 7-DoF torque-controlled manipulator, the KUKA LWR iiwa R820 14 (Kuka) Bischoff et al. 2010). See the figure in appendix F for an illustration of this task. Let x. (g. v) (q. 4) be the state of the system where q, a E R are joint angles and velocities of the manipulator, respectively. Our goal is to find the optimal torque u E U C IR7 that drives the manipulator from 2o (qo.0) 0) to I = (qi, 0) in T = 0.8 seconds and minimizes a quadratic type cost. See Appendix F for details of the problem and the experiment configurations. To obtain training data, we use differential dynamic programming (Jacobson & Mayne 1970p implemented in the Crocoddy library (Mastalli et al. 2020).
We use the QRNet (Nakamura- Zimmerer et al. 2020: 20216) as the backbone network in this ex- ample (see Appendix | for details) and evaluate networks trained in six different ways: four of them
8

---

Under review as a conference paper at ICLR 2023
trained using Algorithm| I with different choices of temporal grid points for adaptive sampling and two of them are trained by vamilla sampling method with 300 (Vanilla300) and 900 (Vanilla900) trajectories separately. All four networks (ASI-AS4) trained by the IVP enhanced sampling have initial training data of 100 trajectories and three iterations (K = 3), i.e., each of them requires solving the open-loop solution 300 times in total for generating training data. The difference of the four networks lies in that they use different temporal grids (f1 and t2) for enhanced sampling. Each experiment has been independently run five times and we report their average results. We plot the cumulative distribution functions of cost ratios (clipped at 2.0) between the NN-controlled cost and optimal cost in Figure 4 (left). More details and results are provided in Appendix F We find that adding more data in the vanilla sampling method has very limited effects on improvement while the IVP enhanced sampling greatly improves the performance. Furthermore, such improvement again robust to different choices of temporal grid points (AS/-AS4).
In addition, we also test the performance of the network trained by our adaptive sampling method in the presence of measurement errors. At each simulation timestep, we sample the disturbances uniformly in [-a, [-o,a]1 for a = 1e-5, le-4, 1e-3 and add them to the input states of the network. See Figure 4 (right) for the result on the best model trained from AS/-AS4. We find that the NN controller performs well at o = 1e-4, 4, and there are more than 60% of cases on which our controller
achieves a ratio less than 2.0 at σ 1e-3.
100%
100%
percentage
percentage
60%
60%
20%-
20%
1.0
2.0
policy optimel
policy optimal
Figure 4: Cumulative distribution functions of cost ratios (with the ideal curve being a straight horizontal segment passing ratio = 1, percentage=100%) under different training schemes (left) and different intensities of measurement noises (right).
8 CONCLUSION AND FUTURE WORK
In this work, we propose the IVP enhanced sampling method to overcome the distribution mismatch problem in the lerariind-earning----
approaches for the closed-loop optimal control problem. Both theoretical and numerical results show that the IVP enhanced sampling method significantly improves the performance of the learned NN controller and outperforms other adaptive sampling methods. There are few directions worth exploring in future work. In the IVP enhanced sam- pling method, one choice we need to make is the temporal grid points for adaptive sampling. We recommend that at each iteration, one can compute the distance between the training data and data reached by the NN controller at different times (see Figure for an example) and choose the time at which the distance starts to increase quickly as the temporal grid for adaptive sampling. We observe that the IVP enhanced sampling method performs well using this strategy. It will be ideal to make this process more systematic. Another direction is to design more effective approaches utilizing the training data. In Algorithm | (lines 8-9), at each iteration, we replace parts of the training data with the newly collected data, and hence some optimal labels are thrown away, which are costly to obtain. An alternative choice is to augment data directly, i.e., setting S, S. S in line 9. Nu- merically, we observe that this choice gives similar performance to the version used in Algorithm I which suggests that so far the dropped data provides little value for training: see Appendix F and
F for details. But it is still possible to find smarter ways to utilize them to improve performance. We also need to evaluate the IVP enhanced sampling method for problems with more features like state/control constraints. Furthermore, the IVP enhanced sampling method can be straightforwardly applied to learning general dynamics from multiple trajectories as the controlled system under the optimal closed-loop policy can be viewed as special dynamical system. It is an interesting direc- tion to investigate its performance in such general settings. Finally, theoretical analysis beyond the I. LQR setting is also interesting and important problem.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Samuel Ainsworth, Kendall Lowrey, John Thickstun, 7aid Harchaoui, and Siddhartha Srinivasa. Faster policy learning with continuous-tima gradients. In Learning for Dynamics and Control, pp. 1054-1067. PMLR, 2021.
Richard E. Bellman. Dynamic Programming. Princeton University Press, 1957.
John T. Betts. Survey of numerical methods for trajectory optimization. Journal of Guidance, Control, and Dynamics, 21(2):1932277 1998.
Rainer Bischoff, Johannes Kurth, Günter Schreiber, Ralf Koeppe, Alin Albu-Schäffer, Alexander Beyer, Oliver Eiberger, Sami Haddadin, Andreas Stemmer, Gerhard Grunwald, et al. The kuka-dlr lightweight robot arm-a new reference platform for robotics research and manufacturing. In ISR 2010 st international symposium on robotics) and ROBOTIK 2010 (6th German conference on robotics), pp. 1-8. VDE, 2010.
Karsten M Borgwardt, Arthur Gretton, Malte Rasch, Hans-Peter Kriegel, Bernhard Schölkopf, and Alex J Smola. Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics, 22(14):e49-e51) 2006.
Lucas Böttcher, Nino
and Thomas Asikis. Al pontryagin or how artificial neural networks learn to control dynamical systems. Nature communications, 13(1):1-9, 2022.
Samir Bouabdallah, Pierpaolo Murrieri, and Roland Siegwart. Design and control of an indoor miero quadrotor. In IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04 2004, volume 5, pp. 4393-4398. IEEE, 2004.
Justin Carpentier and Nicolas Mansard. Analytical derivatives of rigid body dynamics algorithms. In Robotics: Science and Systems, 2018.
Justin Carpentier, Florian Valenza, Nicolas Mansard, et al. Pinocchio: fast forward and inverse 2015-2021.
dynamics for poly-articulated systems.
Justin Carpentier, Guilhem Saurel, Gabriele Buondonno, Joseph Mirabel, Florent Lamiraux, Olivier Stasse, and Nicolas Mansard. The pinocchio c++ library: fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives. In 2019 IEEE/SICE International Symposium on System Integration (SII), pp. 614-619. IEEE, 2019.
Djork-Aré Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In Yoshua Bengio and Yann LeCun (eds.), 4th Inter- national Conference on Learning Representations, ICLR 2016. San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Weinan E, Jiequn Han, and Linfeng Zhang. Machine- learning assisted modeling. Physics Today, 74:36 41, 2021. ISSN 0031-9228. doi: 10.1063/PT.3.4777
Weinan E, Jiequn Han, and Jihao Long. Empowering optimal control with machine learning: A perspective from model predictive control. arXiv preprint arXiv:2205.07990, 2022.
Xavier Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. Journal of Machine Learning Research- Proceedings Track, 9:249-256, 01 2010.
Jiequn Han and Weinan E. Deep learning approximation for stochastic control problems. arXiv preprint wxXiv:1611.07422 2016.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
David H Jacobson and David Q Mayne. Differential dynamic programming. Number 24 in Modern analytic and computational methods in science and mathematics. Elsevier Publishing Company, 1970.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In In Proc. 19th International Conference Machine Learning. Citeseer, 2002.
10

---

Under review as a conference paper at ICLR 2023
Wei Kang, Qi Gong, Tenavi for deep learning and feedback design: A survey. Physica D: Nonlinear Phenomena, 425:132955, 2021.
and Fariba Fahroo. Algorithms of data generation
Jacek Kierzenka and Lawrence F Shampine. A bvp solver based on residual control and the maltab pse. ACM Transactions on Mathematical Software (TOMS), 27(3):299-311 2001.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Kuka. Kuka ag, 2022. URL BtRpsttts:/// www kuka. com/
Benoit Landry, Hongkai Dai, and Marco Pavone. Seagul: Sample efficient adversarially guided learning of value functions. In Learning for Dynamics and Control, pp. 1105-1117. PMLR, 2021.
Sergey Levine and Vladlen Koltun. Learning complex neural network policies with trajectory opti- mization. In International Conference Machine Learning, pp. 829-832 PMLR, 2014.
Daniel Liberzon. Calculus of variations and optimal control theory: concise introduction. Prince- ton university press, 2011.
Jihao Long and Jiequn Han. Perturbational complexity by distribution mismatch: A systematic anal- ysis of reinforcement learning reproducing kernel hilbert space. Journal of Machine Learning vol, 1:1-34, 2022.
Tarek Madani and Abdelaziz Benallegue. Control of quadrotor mini-helicopter via full state back- stepping technique. In Proceedings of the 45th IEEE Conference on Decision and Control, pp- 1515-1520. IEEE, 2006.
Robert Mahony, Vijay Kumar, and Peter Corke. Multirotor aerial vehicles: Modeling, estimation, and control of quadrotor. IEEE Robotics and Automation magazine, 19(3):20-322 2012.
Carlos Mastalli, Rohan Budhiraja, Wolfgang Merkt, Guilhem Saurel, Bilal Hammoud, Maximilien Naveau, Justin Carpentier, Ludovic Righetti, Sethu Vijayakumar, and Nicolas Mansard. Crocod- dyl: An efficient and versatile framework for multi-contact optimal control. In 2020 IEEE Inter- national Conference on Robotics and Automation (ICRA), pp. 2536 2542. IEEE, 2020.
Tenavi Nakamura-Zimin Zimmeerr, Qi Gong, and Wei Kang. QRnet: optimal regulator design with Iqr- augmented neural networks. IEEE Control Systems Letters,
2020.
Tenavi Nakamura-Zimmere Qi Gong, and Wei Kang. Adaptive deep learning for high-dimensional Hamilton Jacobi-Bellaan equations. SIAM Journal Scientific Computing, 43(2):A1221- A1247, 202
Tenavi Nakamura-Zimmerer. Qi Gong, and Wei Kang. Neural network optimal feedback control with enhanced closed loop stability. arXiv preprint arXiv:2109.07466 2021b.
Tenavi Mikamura--iimerra Qi Gong, and Wei Kang. Neural network optimal feedback control with guaranteed local stability. arXiv preprint 225.v2200.003394 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
Lev Semenovich Pontryagin. Mathematical theory of optimal processes. CRC press, 1987.
Anil Rao. A survey of numerical methods for optimal control. Advances the Astronautical Sciences, 135(1):9975528 2009.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. JMLR Workshop and Conference Proceedings, 2010.
11

---

Under review as a conference paper at ICLR 2023
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Proceedings of the fourteenth international con- ference artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
Eduardo Sontag. Mathematical control theory: deterministic finite dimensional systems, vol- ume 6. Springer Science & Business Media, 2013.
Russ Tedrake and the Drake Development Team. Drake: Model-based design and verification for robotics, 2019.
Yaohua Zang, Jihao Long, Xuanxi Zhang. Wei Hu, Weinan E, and Jiequn Han. A machine learning enhanced algorithm for the optimal landing problem. In 3rd Annual Conference on Mathematical and Scientific Machine Learning, pp. 1-20. PMLR, 2022.
Linfeng Zhang, Han Wang, and Weinan E. Reinforced dynamics for enhanced sampling in large atomic and molecular systems. The Journal of chemical physics, 148(12):124113, 2018.
Zhigen Zhao, Simiao Zuo, Tuo Zhao, and Ye Zhao. Adversarially regularized policy learning guided by trajectory optimization. In Learning for Dynamics and Control Conference, pp. 144-857. PMLR, 2022.
A SUDERVISED-LEARNII--BASSEE APPROACH THROUGH THE LENS OF ELT ALGORITHM
As pointed out in the introduction, the waaeeedd-eanning-bae approach empowered by the adap- tive sampling can be interpreted as an instance of the
haaet-ll-abblliiiiiiiiii (ELT) algorithms (Zhang et al., 2018 E et al., 202T) for closed-loop optimal control problems. Through the lens the ELT algorithm, there based approach for the closed-loop optimal control problem:
least three aspects to improve the efficiency of the supervised-learninn
Use the adaptive sampling method. Adaptive sampling methods aim to sequentially choose the time-state pairs based previous results to improve the performance of the NN con- troller. This corresponds to the first step in the ELT algorithm and is the main focus of this work.
Improve the efficiency of data generation, i.e., solving the open-loop optimal control prob- lems. Although the open-loop optimal control problem is much easier than the closed-loop optimal control problem, its time cost cannot neglected and the efficiency varies signif- icantly with different methods. This corresponds to the second step in the ELT algorithm and we refer to Kang et al. (202T) for a detailed survey.
Improve the learning process of the neural networks. This corresponds to the third step in the ELT algorithm. The recent works Nakamura-Zimmenn
al. (2020) 20216 2022) focus on the structure of the neural networks and design a special ansatz such that the NN controller is close the linear quadratic controller around the equilibrium point improve the stability of the NN controller.
B DETAILED ANALYSIS OF THE LQR EXAMPLE
In this section, we give the detailed settings for the comparison in Section 5 and the detailed state- ment and proof of Theorem [ Through this section, all symbols having a hat are open-loop optimal paths sampled for training, e,g. ûj f., út,, .1. Let T denote a single state instead a state trajec- tory. The clean symbol without hat or tilde is the IVP solution generated by specific controllers which are specified in the subscript; e.g. Zo, Fv, Fa are trajectories generated by uor Uy ua which optimal, vanilla, and IVP enhanced controllers, respectively. The positive integer in the sub- script always denotes the index of the optimal path. Symbols with superscript are related to the i-th iteration of the IVP enhanced sampling method.
12

---

Under review as a conference paper at ICLR 2023
For the vanilla method, we first randomly sample NT initial states 1 from standard normal distribution where N is a positive integer (recalling T is a positive integer). Then NT approximated optimal paths are collected starting to 0:
T (t(t) = T T
eZ 2₂(1)= T(T -1)+ T +1 3, + cZ,z
3 +
(6)
where { are i.i.d. normal random variables with mean and variance o², and independent of initial states. Finally, the parameters are learned by solving the following least square problems:
min /
T ['2 (0,(t) Ela((t)
Optimizing O, independently for each have
NT (1,(1)+ + T(T 1) +1 8,(1) b12 or 02 arg min (a.b)
NT
T
0: = arg min
aā₃ b12
for the first and second models, respectively. We will use u, to denote the closed-loop controller determined in this way.
For the IVP enhanced sampling method, we choose K = T and the temporal grid points 4 i for 0 S S K. We first sample initial points { from the normal standard distribution, denote the parameters optimized at i-th iteration as 0 and initialize f-1 = 0. At the i-th iteration (0 S i T 1), we use Ugi to solve the IVPs on the time horizon [0, i]
(t ug. (t,,
2)(0) 2y. 1 s
(8)
テ) (t). Here omit the controller subscript a for simplicity, i.e. z) = za.." We then compute N approximated optimal paths starting from { at t, =
and collect {
(t(() = -( + - + eZj, 2)(() = T(T -) 25 + (t- - 1) t te [ T(T - +1
T
(9)
where (2))osisT-L1SSS are i.i.d. normal random variables with mean m and variance 0², and independent of [ Note that út and ěj are only defined in t E 2"] (for i 2 1), we then fill their values in interval [0, i) with values from previous iteration,
a1 = (1),
= = (1), LE (0,
(10)
Finally, we solve the least square problems to determine 8¹:
min /' N (t - uo (t))
(1I)
We will use un to denote the closed-loop controller ugr 1, the closed-loop controller generated in the (T' - 1)-th iteration by the IVP enhanced sampling method. The theorem below gives the performance of u, and ua when using Model
Theorem 1". Under Model
(4). define the IVPs generated by u, u², uy and un as follows:
(1) (I( "o N. (1));
(0) Tmn:: S
(t) (t) = un, (())
(0) = Tmit 0 0S
(t)t) un (t) = un (t, Ia (t)),
F. Timiss 0 5
1. If Tim a random variable following standard normal distribution, which is independent of the initial points (5 A= j= and noises / (7) 77ibosigT-111s55 in the
training process, the state variables ă; (t) in the training data and (t) in the IVP from the vanilla method follow normal distributions and satisfy:
1
Ei,(t) = Ex, (t), (+) E (t) -0 NT
Je2.
(12)
13

---

Under review as a conference paper at ICLR 2023
On the other hand, the state variables - in the training data and x ra(t) in the IVP from the IVP enhanced sampling method also follow and satisfy:
T---- (1)=-(()=- (t) Ex Ea
-
)
) 020. (13)
2. If Tinit is a fixed initial point, define the total cost
1 T
J.
T
1
/
J. T
[u.(t)) [T)(
1
[ua(t)/dd + (T))
T
Then,
EJ, Jo =172 +1)mm² + a² NT.)e,
(14)
EJg - Jo s 3(m2 + a² N)e
(15)
Proof. We first give the closed-form expressions of u, and ua using Model and (t) given in equation (6), we have
Recalling út
T
T +
(1()) T(T 1) (1(
T(T - 1) + EZj, j NT.
Therefore, recalling u, is learned through the least square problem
we have
T
T2
Ue (t,
x +
22u,
(16)
T(T
+1
T(T - t) +
where
NT
1
2.
Zz.
NT
To compute ua recalling equations 9 and have
10
when 0 D N and t E [i, (, 1),
T
T(T -) i) +
T(T- +
-t) (t)
+ i)eZ3.
(17)
T(T - 1) +1
Therefore,
T(T - 1) +
- T(T -
T -
T(T - +1 EZj
Hence, recalling ua is learned through the least square problem 11], have, when f E [Ł, +
T
T(T - T(T - + 2%
Ui (t,xx))
(18)
T(T +1
where
N Z:
Σ Zj. -
N 1=1
Equation 18 also holds when i T 1 and T.
14

---

Under review as a conference paper at ICLR 2023
We then compute the starting points (=/)ooc57-111jjs in the IVP enhanced sampling method.
By equation (1O), we know that when 1 i F and 0 < < t, 0 = f Together
with equation (8). we know that when 0 - ugr (1, ugr 1 (1, -ua (4, x) for
1 E [ i 1), and oi +1) xz (1) for i E [0, which implies x" + (1)
(i) 2/ (i) .t.. Therefore,
for 1 S - N, when 0 s
2, we have
(t) (t)
ug. (t, a ((,,a=((())) (t)) (t (t)
T(T - i) +1 T(T - 1) +1 (1) + T(T - () +1
T
E LE ( 1]:
2j (1) = =ēj.
Solving the above ODE, we get the solution
+1)= T(T 1) +
T(T - i) +1 (1- LE [1,1+]] 1].
Hence, by definition, for 0 -
- +1 T) 2) +
2 (i+ (i+
Utilizing the above recursive relationship, we obtain, for 0 SSiST-
T(T - -) +1 2
T(T - -) + T(T - k h + k=
T2+1 23 Σ
2k.
(19)
Now we are ready to prove the main results of the Theorem. First, for equation 12 control (16), we have
using the
T
T +1 T(T +
(t)
T(T ()+ +1 [t) +
EZ., (0) Finit.
Solving this ODE gives
(t) T(T - -t)+
Pinit ct
(20)
T +2+1
Combining the last equation with the fact that
T(T- (t T(T - t) +
7 + 3, etZz
J. Pinit and (T NT are independent normal random variables and
Giit NO0, 1), Zz ~ (M(,, we know that (t) and (t) are normal random variables with
Ei (1) (1), [E] R (1) Elr, 02 NT )
Next, we prove the equation for S SisT and f [i, + 1), using the control (8). we have
T
-TT Zi (t) + - - +
T(T - t) +1
Solving the above ODE with the initial condition Fa (0) = Ainit we can get the solution
(1) T(T -1) +1
T(T 1) +1
+2+1 Tinit + Σ
2h + (1 i)e
(21)
- --
k=0
when 0 -iis- and E [ 1). The above equation also holds when i = T 1 and t T.
In this section, we take by convention that summation Σ 2 C+ if n.
15

---

Under review as a conference paper at ICLR 2023
On the other hand, combining equations t e i + or - and =
17 and
19 we know that when 0 i S T 1 and
(1)= T(T +
+ ( 1)
T(T -i) +
- = 7
i-1
T(T - 1) +1
+
Σ
7 (t - i).
T(T - k - +1
k=0
The above equation also holds when equation (21 and the fact that Fiii and {2) -1i independent normal random variables and
7 and
Combining the last equation with
s). iit N(0 1), Zj ~ -M(m,
we know that (t) and va are normal random variables with
1 0². )
F -1 (1) = Ear (1), BBja (1)))=-1()) (1) E/
We then prove equations
14p and 13}. First, with the optimal solution
T
mo(
Dinit
Io(T)
Fint
T
T
have
1
T
1
1
J.
intt
dt
+
Tinit
iiit
T +
T +
T +
Recalling equation
20 and plugging
(0 into (6),, we know that
1
xr (T)
Tinit + +T =T
T2 +
T
u. (t(
Finit + eZ.
T
Hence,
27
2T
1
Ju = e
Tint Zu + 20. 2 + T2 + 1
Timit Zo +
T +
72
+
which gives
EJ, J, -7 1) + a²
NT )²²
On the other hand, recalling equation and plugging 21 into 18) we know that
1
1
Fa (T)
Pinit
2
7 +
T(T 1) +
T
T
U, (t) = T +1
Dinit
2 + eZ
(1,
T(T - k 1) +1
To compute the difference between Js and Jo. we first notice that
[T
1 T
EJJa Jo
T J
Var(ua(t))dt + Var(za(T))
Eug (t)
Ex,
Jo
I lz.
16

---

Under review as a conference paper at ICLR 2023
By the independence of {Z²
T-1 ( =0
know that
o²c²
1-1
T
0²2
T-1
o²2
1
+
NT
[ 1)+1
N
[[T - k 1) + 1 N
o²t²
1-11-1
T
T-1
N
T(T k - 1]
[TT - 1: - 1)
1=0
0=
2²2
T-2 T-1
Σ Σ
T
T-1
1+
N
--
[[ - - 1
k= 0 0i
A
0²2
k +1
[T-
2²2
Tk +1
22
N
Meanwhile, noticing that
T-1
1
1
Ex.(T) =
Ami +
T(T kk 6-1)+ +1 cm,
T +
k=0
T
T
Rua (1)
Pinit
+ em,
T
T(T k 1)
it is straightforward to compute that
2
12
+ + (1, + +1,,
7
1
where
T
T(T - k 11+1 T
T(T k - 1) +1
1=0
T-1
T
T
T(T - k - 1) +1
T(T - 1) +1
0= k=
T(T - 1)
T=
T(T
and
THH 2
4
2 TT
1
ΣΣ
T(T h - 11
T(T k 1)
T(T - k- +1
k=
i=0 k=0
k=
+ -
T-11-1
T(T T(T-i-1)+*
T+T(T-
T-
2
-
O=
1
Tk +1 22
T(T - k 1-1) 1)
k=
17

---

Under review as a conference paper at ICLR 2023
Therefore,
EJ - Jo = =11 11+ 2em.Tinit
0² ))
13 +
+ +1++) 3(m² +
T +
Next we present the numerical results when we use Model to fit the closed-loop optimal control. In the following experiments, we set с = 0.1, m = 0.1 and o² = 1. Figure (left) compares the optimal path (t) with (t) and z, (t), the IVPs generated by the controllers learned by the vanilla method and IVP enhanced sampling method (all three paths start at Timi = 1). In this experiment, set T 30 and N 100. Figure 5 (middle) shows how the time influences the differences of the second order moments between the state distribution of the training data and the state distribution of the IVP generated by learned controllers in the vanilla method and the IVP enhanced sampling method. We set the total time T 100 and N = 100. Figure (right) compares the performance of the vanilla method and IVP enhanced sampling method on different total times T. The performance difference is an empirical estimation of E[J, - Jol and E J.l when Zinit follows standard normal distribution. In this experiment, for each method, we set learn 10 different controllers with different realizations of the training data and calculate the average of the performance difference on 1000 randomly sampled initial points (from a standard normal distribution) and 10 learned controllers.
and
10
enhanced
IVP enhanced sampligg
3.
sampling Opti
Distributton Difterence
10-4
Vanilla samping
Differencee
sampling
3
30F
30-4
30
1.0
10
10
5
15
20
25
20
40
60
243
100
101-
301
1013
15
Time
Time
Total Tima
Figure 5: Numerical results on learning Model 2 (5) Left: the optimal path and the paths generated by the vanilla sampling method and the IVP enhanced sampling method. Middle: differences the second order moments (in the logarithm scale) between the distributions of the training data and the data reached by the controllers at different times. Right: performance differences (in the logarithm scale) of the vanilla sampling method and the IVP enhanced sampling method for different total times (in the logarithm scale).
C FULL DYNAMICS OF QUADROTOR
In this section, we introduce the full dynamics of quadrotor (Bouabdallah al.) 2004 Madani &
Benallegue 2006 Mahony et al.) 2012) that are considered in Section 6 The state variable of quadrotor is æ in farth-fixed coordinates,
2y 17' w E 121 where p = (xc, y, R the position of quadrotor
(Ø. 0, v)) R (roll, pitch, yaw) is the attitude terms of Euler angles in Earth-fixed coordinates, and ws R the angular velocity in body-fixed coordinates. Control
E R is the velocity in body-fixed coordinates.
(8, Tas Ty. 7 E Rd is composed of total thrust s and body torques (Te, Ty. T²) from the four rotors. Then we can model the quadrotor's dynamics as
p R RT(n)) (n) 0b
1 Au 0, Wp x Ub R(n)g + m
r K K(n)wn
th J-lus Juh + J-1 Bu,
with matrix and defined as
0
0
A =
0 0
B
0
1
0
1 0
18

---

Under review as a conference paper at ICLR 2023
The constant mass and inertia matrix J diag( [Jz: Jy: are the parameters of the quadrotor, where Jz, Jy. and J3 are the moments of inertia of the quadrotor in the r-axis, y-axis, and z-axis, respectively. We set m = 2kg and J/ = Jy = J = 1.2416kg m² which are the same system parameters as in (Madani & Benallegue, 2006). The constants g = (0, 0, 0)z denote the gravity vector where ! 9.81m/ 8 the acceleration of gravity on Earth. The direction cosine matrix R(n) E SO(3) represents the transformation from the Earth-fixed coordinates to the body-fixed coordinates:
cos 0 cos vb
Cos 0 sin yb sin 0 sin y sin ø + cos & cos ø sin @ sin 1/ cos Φ cos 1/ sin Φ
sin 0 cos 0 sin Φ cos 0 cos Φ
R(n)
sin 0 cos y sin @ sin eb cos sin I cos cos o + sin 1 sin
and the attitude kinematic matrix K (n) relates the time derivative of the attitude representation with the associated angular rate:
[ sin & tan
cos tan 0 sin Ø cos Ø sec
K(
0
COS Φ sin @ sec
Note that in practice the quadrotor directly controlled by the individual rotor thrusts (Fi, F2, F3, Fa) and we have the relation u EF with
1
-1
E
0
-c
where is the distance from the rotor to the UAV's center of gravity and e constant that relates the rotor angular momentum to the rotor thrust (normal force). So once we obtain the optimal control u", we are able get the optimal F immediately by the relation F =E u.
D PMP AND SPACE MARCHING METHOD
In this section introduce the open-loop optimal problem solver used for solving the optimal landing problem a quadrotor. The solver is based on Pontryagin's minimum principle (PMP) (Pontryagin 1987) and space-marching method (Zang et al. 2022). The optimal landing problem is defined as
min [ L(x(T), u(T))) ((())))++M((())))) (x(T)),
20,00
(22)
a(t) = f(x(t), u(t)), [0,7],
a<0) 2o:
where ætt): [0, + 12 and u(t) [0, + 2 denote the state trajectory and control trajectory, respectively, and f is the full dynamics of quadrotor introduced Appendix C By PMP, problem can be solved through solving two-point boundary value problem (TPBVP). Introduce costate
22
variable E 12 and Hamiltonian
H(r,A,u) L(x, u) + A f(z, u).
The TPBVP is defined as
at() H H A(t), u ut)) A(1) 0% H (2(11, A(1), u v²))
(23)
We have the boundary conditions:
x(0) Fo, A (T) VM (a (T)),
and the optimal control uz (t) should minimize Hamiltonian at each t:
u" (t)
aig (xx(t), X(t), u).
(24)
u
19

---

Under review as a conference paper at ICLR 2023
We use solve_bvp function of scipy (Kierzenka& & Shampine 2001) to solve TPBVP (3 24 and set tolerance to 10 max moxx_ to 5000. We note that when the initial state 26 is far from the target state 2y, solving the TPBVP directly often fails. Thus we the space-marching method proposed in Zang et al. (002)). We uniformly select K points in the line segment from ør to Zn. and denote them { a V6. 2r!5 } according to their increasing distances to æT (a 20)- These K TPBVPs will be solved in order and at every step we use the previous solution as the initial guess to the current problem.
E EXPERIMENT DETAILS AND MORE RESULTS OF THE OPTIMAL LANDING PROBLEM
In this section, we give more details about implementation and numerical results in Section 6 We aim find the optimal controls to steer the quadrotor from some initial states Tp target state = O. The distribution of the initial state of interest is the uniform distribution on the set X [- x/4, क /44), E
{z. y E [-40, 40), E [20, 40), Uz uy: U/= E 1,, 0, E
[- x, T|: w 0). We
consider a quadratic running cost:
L(x, u) = (u (--ua) Qa - uu))
where un = (mg, 0, 0, 0) represents the reference control that balances with gravity and Qu diag(1, 1, 1, 1) represents the weight matrix characterizing the cost of deviating from the reference control. The terminal cost is
M (x) = -""pp=p+o"Quupp + u w Qwfw= = 2²Q2x
where Qp 5/a, Quf 10l3, Qus 2513, Qws 501z. We set the entries in the terminal cost larger than the running cost as we want to give the endpoint more penalty for deviating from the landing target.
We sample N = 500 initial points for training. On every optimal path, we select time-state-action tuples with time step iteration. Note that when solving BVPs and IVPs, we use denser time grids to ensure the solution is accurate enough. The neural network models in all quadrotor experiments have the same structure with 13 13-dimensional input (12 for states and for time) and 4-dimensional output. The networks are fully-connected with hidden layers; each layer has 128 hidden neurons and we use tanh as the activation function. The inputs are scaled to 1) where the upper bound and lower bound are the maximum and minimum of the training dataset. Since the activation fucntion is tanh, we adapt Xavier initialization (Glorot & Bengio 2010 before training. We train the neural network by the Adam (Kingma& & Ba 2015) optimizer with learning rate 0.001, batch size 1000, and 1000 epochs. At every iteration of IVP enhanced sampling method, we train
0.2. Thus the number of training data always 81 500 at every
neural network from scratch. Our model and training programs are implemented by PyTorch (Paszke et al.) 2019).
In the first experiment, use our IVP enhanced sampling method and choose temporal grid points 0 < 10 < 14 < 16. More statistics of the ratios between the NN-controlled costs and the optimal costs during three iterations are shown in Table [ We also test an alternative way to construct dataset during the IVP enhanced sampling as discussed in Section & i.e., setting S, = S, - in Algorithm I line 9. We denote the final policy obtained in this approach as ū and report the statistics of corresponding cost ratios in Table | as well. We have to point out that the ratio of ù has been clipped at 10.0 as there is test path with ratio over 1000. The performance of ū is similar to that of ú2 suggesting that so far the dropped data provides little value for training. Additionally, as we always train networks 1000 epochs and this alternative approach has more training data, the training time of ū is 1.5 times that of others.
We also illustrate the trajectories of states controlled by learned policy and optimal policy in Figure 6 The path controlled by úhg matches the optimal path at the beginning but deviates around = 10. Then the path controlled by vi fits the optimal path more and deviates around path controlled by di matches the optimal path till the terminal time. Note that the cost of three controlled paths is 3296. 18, 119.91, 6.69, respectively, and the optimal cost is 6.32.
14. Finally, the
20

---

Under review as a conference paper at ICLR 2023
Policy
Mean
Std
Max
90 %
75%
Median
do
45.52
55.73
521.98
23.53
20.32
17.32
11
4.41
4.17
39.69
2.24
2.01
1.67
22
1.02
0.03
1.30
1.01
1.01
1.01
1.03
0.04
1.43
1.01
1.01
1.01
(a) on training points
Policy
Mean
Std
Max
90 %
75%
Median
67.90
86. 14
578.05
145.82
74.13
38.21
ú
5.57
7.70
91.63
11.54
6.52
3.74
ú2
1.09
0.25
3.95
1.16
1.08
1.04
1.14
0.64
10.00
1.24
1.10
1.05
(h) On test points
Table 1: The numerical results of the IVP enhanced sampling method training and test points. The results in Table (a) and (b) correspond to Figure 3
(left) and (middle), respectively. dg. ú1, 22 denote the policy after the first, second, and third round of training, respectively. ú denotes the final policy obtained from an alternative approach for constructing datasets during the iteration of the IVP enhanced sampling; see discussion in Section
g
X
-10
10
0
-30
-20
-40
10
15
0.4
0.0
0.2
-0.5
0.0
-1.0
-02
-0.2
-1.5
0.6
-2.0
10
15
0
10
15
5
10
15
time
time
time
path controlled by úp
path controlled by ú
path controlled by ú
optimal path
Figure 6: The optimal path and path controlled by learned controllers. We show the d-dimensional position (xc, 2) and 3-dimensional attitude = (dh, 0, 1b) in terms of Fuler angles in Earth- fixed coordinates.
In the experiments of comparing different methods, the three adaptive sampling methods pro- cessed as follows. The initial network is the same policy as in the IVP enhanced sampling method, i.e., the policy ún after iteration 0 in the IVP enhanced sampling method, which trained on 500 optimal paths. Then we will sequentially add 400, 300, 300 paths to training data so we use 1500 optimal paths to train the final network (solving 1500 open-loop optimal control problems namely).
When sampling new training paths, we will randomly sample 2 initial points, calculate some indices using the latest policy and preserve the preferable one. For AS w. large u, we will calculate the norms of the control variables of these two points at time 0 and choose the one with larger norms. For AS w. large v, we will solve IVPs to obtain the NN-controlled costs and choose the larger one. For AS w. bad v, we will solve 2 TPBVPs corresponding to these two points and choose the one with a large difference between the NN-controlled value and optimal value. Note that in all other methods, the time span of the solved open-loop problem is always T. Nevertheless, in our method, the time span of the optimal paths to be solved is getting shorter and shorter along iterations, which means it takes less time to solve. Specifically, in our experiment, takes about 11.1s to solve
21

---

Under review as a conference paper at ICLR 2023
optimal path whose total time T to 16 iteration 0 and takes about 3.1s to solve optimal path whose total time T t2 = 2 at iteration 2. More statistics are shown in Table 2
Methods
Mean
Std
Max
90 %
75%
Median
IVP enhanced sampling
1.09
0.25
3.95
1.16
1.08
1.04
Vanilla sampling
2.27
1.39
10.11
3.93
2.54
1.86
AS w. large
2.04
1.34
8.70
3.39
2.21
1.57
AS w. large
1.61
0.66
5.56
2.28
1.75
1.37
AS w. bad v
1.32
0.84
8.49
1.54
1.26
1.12
Table Comparison on different sampling methods.
We further test the performance of NN controllers obtained from different sampling methods in the presence of observation noises, considering that the sensors have errors in reality. During simulation, add disturbance e to the input of the network, where e E R (including the disturbance of time) is uniformly sampled from [-a, a) We test o 0.01, 0.05, 0.1 and the numerical results are shown in Figure 7 We also test the performance of the open-loop optimal controller under perturbation where a disturbance é R is added to the input time. Figure 7 shows that when disturbance exists, closed-loop controllers are more reliable than the open-loop controller and the one trained by the IVP enhanced sampling method performs best among all the methods.
ch 0.01
am 0.05
a= 0.1
100%
100%
100%
percantaage
50%
60%
10%
46%
23
28
18
58
policy optipnall
policy ootimal
palicy
onhanced sampling
sampling
large
large
Open loop optimal controller
Figure 7: Cumulative distribution function of the cost ratio between NN controlled value and the optimal value under disturbance.
Finally we consider the impact of different choices of temporal grid points in Algorithm I We test experiments which are all trained using the same 500 initial points. The results listed in Table show that our algorithm is robust to the choice of temporal grid points.
Temporal grid points (T
16)
# of iterations
0
4
8
10
12
14
2
67.90
1.14
67.90
5.57
1.09
4
67.90
37.11
2.34
1.18
67.90
37.66
11.13
1.79
1.32
Table 3: Average cost ratio 200 test points of every model. The first line means that we take 2 iterations and the corresponding temporal grid points for adaptive sampling are < 14 < 16. The average ratio of policy cost optimal cost is 67.90 after iteration 0 and the average ratio after iteration 1.14. The second line shows the same experiment in Figure 3 (middle).
F EXPERIMENT DETAILS AND MORE RESULTS OF THE MANIPULATOR
We can write down the dynamics of the manipulator as,
f (a:, u) (v, axr u)):
22

---

Under review as a conference paper at ICLR 2023
5
3
Figure 8: An illustration of the reaching problem the manipulator. The solid manipulator demon- strates its initial position. We label the end effectors of the five instances of robots by "1,2,3,4.5" to indicator the position of the robot different times t 0.0, lo = 0.2, 13 0.4, 14 = 0.6, ts 0.8.
where u R is the control torque, x (q. v) R44,, q R the joint angles, v = q R is the joint velocities, q a(x, u) E ITT is the acceleration of joint angles.
The acceleration given by the (non-linear) forward dynamics a C)q g(q) +g)) u.
Here (g) is the generalized inertia matrix, C(g, Q)q represents the centrifugal forces and Coriolis forces, and g(q) is the generalized gravity.
The reaching task is to move the manipulator from the initial states near 2o to the terminal states 31. In the experiments, we take æ = (4o. 0), Φ = (q1, 0) with
q0 [1.6800, 1.2501, 2.4428, - 11.2669, - 0.9978, 1. 1.1236, -1.3575 q1 (2.7736, 0.5842, 1.5413, 1.7028. 22.1.22.16666 0.0847, -2.5764 See Figure g for an illustration of the task. Besides, we take the running cost and terminal cost be
L(x, u) = axx u) TQaa(z + (u -u--- (u) 10=-"")Quu Qu(u aaau-us u2))
M(x) = (x x-,-,,)0 T - x1).
where un is the torque to balance gravity at state 21, i.e. a(x), u,) ) 0. Under this setting, (æ1, ui) an equilibrium of the system, i.e. a a , ui) and f( u1) (v), a.) 0.
In the experiment, we take Qa = 0.005 Qu 0.0251,, Q, = 250007 where we use large weights Q, ensure the reaching goal is approximately achieved.
The backbone network for this example is the QRNet (Nakamua-Zimmerrt et al. 2020 2021b). QRNet exploits the solution corresponding the LQR problem at equilibrium and thus improves the network performance around the equilibrium. The usage of other network structures also demon- strates the genericness/verratilli of the IVP enhanced sampling method. Suppose we have the linear quadratic regulator (LQR) u LQQ for the problem with linearized dynamics and quadratized costs at (æ), un) ), the QRNet can formulated as
au QR (t, Φ)
olw OR (t, + u (N, æ; 0) u NN (T, (T, x1));
where u NN, xx; f)) is any neural network with trainable parameters 0, and a is a saturating function that satisfies a (u,) u1 (u(u)) 17. The a used in this example is defined coordinate-wisely as
Umax
Umin
(u) = Umin +
1 -2xpl-22(1 u1)
23

---

Under review as a conference paper at ICLR 2023
where 01 (Mmax u1))(uu 2miaa) C2 (Umax Ummin) ( -2)) (u) ((mii) ] with Umin: Mmax being minimum and maximum values for u. Here u, Umin = 150 and Umaa = 150 are the corre- sponding values at each coordinate of u, Umin Umax respectively.
To get the LQR, we need to expand the dynamics linearly as f(z u) x fø f u1) æ1) + fa un (i)(u= ui),
and the term related to acceleration in the running cost quadratically as
Qualar "ee,uu"Qaaaaa:. a(x, u)
u) La u)² Qa (ar
(ar - 2 a az x1) + (u u1) auuuaaulu u1) u1)
2( x T
a Qu aula u1))
where La Oz ( u. (a) x + an (1), U. )u uI), and we exploit a(æ1, ui) = 0 and f(1 u1) = O. The derivatives boil down to a₂ and a, which can analytically computed in
the Pinocchio library (Carpentier et al.) 2021 2019: Carpentier & Mansard 2018). In the experiment, we solve the LQR by the implementation in the Drake library (Tedrake & the Drake Development Team 2019).
In the simulation and open-loop solver, we take time step At = 0.001 and use the semi-implicit Euler discretization. The initial positions q sampled uniformly and independently in 7- dimensional cube centered at go with side length 0.02. Initial velocities are set to zero. Other than directly applying the open-loop solver collected initial states, we first sample another mini- batch of initial states and call the open-loop solver on it. We then pick one solution of the lowest cost and use it as an initial guess later. This can not only speed up the data generation process but also avoid sampling trajectories that fall into bad local minima. We use the differential dynamic program- ming solver implemented in (Mastalli et al. (020), which is a second-order algorithm that favors a good initial guess. The dataset is then created from the (discrete) optimal trajectories warm-started by the initial guess. Each trajectory has T/ 800 data points that are pairs of 15-dimensional input states including time and 7-dimensional output controls. The validation dataset and test dataset contain 605 and 1200 optimal trajectories, respectively.
Finally, all the QRNets OR are trained by minimizing a mean square error over the training dataset with the Adam optimizer (Kingma & Bal 2015) with learning rate 0.001, batch size 256 and epochs 2000. u NN is fully-connected network with 6 hidden layers; each layer has 128 neurons.
The first three layers use the tanh function as activation while the last three layers use ELU (Clevert ett al. 2016). Network and training implemented in PyTorch (Paszke et al. 2019). During iterations, all networks are trained from scratch, i.e. new network with random weights instead of inheriting weights from the previous iteration. After each epoch of training, we compute the loss on the validation dataset. The network with the least validation loss is then used for data generation in the next iteration or as the final policy (at the last iteration).
We compare the ratio of policy cost over the optimal cost, see Table 4 for the mean ratio over all 1200 trajectories of the test dataset. The ratio has been clipped at 2.0 for each trajectory. The results demonstrate that the IVP enhanced sampling method has great improvement over the vanilla seerrised-learnnng based method. It also shows the IVP enhanced sampling is not sensible to the choices of the temporal grid points. Besides, we also try augmenting the dataset with newly collected data instead of replacing them, as detailed in Section 8 Through the comparison between AS3 and AS3* in Table we find that the alternative approach does not bring further improvement.
G COMPARISON WITH DAGGER
In this section, we give a comprehensive comparison between DAGGER (Dataset Aggregation) (Ross et al. 201T) and the IVP enhanced sampling, in terms of concepts, theoretical results, and numerical results. When referring to DAGGER, we mostly mean the single iteration version of DAGGER (i.e., augmenting dataset once) unless otherwise stated explicitly.
Concept. Both DAGGER and IVP enhanced sampling methods solve IVPs using the policy from the previous iteration to generate time-state pairs as new initial time-state pairs and call the open- loop solver to label the trajectories starting from these pairs. Their main differences follows.
24

---

Under review as a conference paper at ICLR 2023
Temporal grid points (T 0.8)
# of iterations
0.0
0.16
0.48
0.56
0.64
0.72
Vanilla300 Vanilla900
1.9204
1.8717
AS1
1.8104
1.2830
1.1182
AS2
3
1.9971
1.4535
1.0441
AS3
1.9161
1.3678
1.0745
AS3*
3
1.8549
1.4402
1.1491
AS4
1.9552
1.2892
1.1295
Table 4: The mean ratio of policy costs / optimal costs of the optimal reaching problem of the manip- ulator. The ratio has been clipped at 2.0 for each test trajectory. The vanilla300/900 correspond to networks trained 300/900 optimal trajectories, respectively. The choices of temporal grid points for adaptive sampling in the remaining rows can be inferred by the location of columns. For ex- ample, AS1 has temporal grid points < 0.16 0.48 0.8. AS3* has the same temporal grid points as AS3 except that it augments the dataset directly instead of replacing them, as discussed in Section
In the i-th iteration, the IVP enhanced sampling method only solves the IVPs till the i-th time grid and uses the time and the visited states at the i-th time grid as the initial time-state pairs to collect new open-loop optimal data. In contrast, in each iteration, the DAGGER method needs to solve until the penultimate temporal grid fK-1. collect the states on all the time grids 0 < t < ... < f---1 < T, and all the collected time-state pairs as the initial time-state pair generate new open-loop optimal data. In the IVP enhanced sampling method, the later times are only visited by networks trained from later iterations. Hopefully, networks from later iterations perform better and generate relevant time-state pairs at later time grids. In contrast, each network from the DAGGER method acts on the dynamical system until the final grid. For stiff dynamics that accumulate errors fast, the time-state pairs at later time grids generated by network from earlier iterations can deviate much from what the optimal policy will visit. These data might then deteriorate the performance instead, as supported by the numerical results below.
Computation cost. With the same time grids (say 4o 4...... < = T), we argue that the efforts of training the IVP enhanced sampling method and single iteration DAGGER are approxi- mately the same since the cost for data labeling and training are approximately the same, respec- tively.
First, in terms the computation cost data labeling, the IVP enhanced sampling method requires labeling K dataset of M trajectories. The single iteration DAGGER labels the same amounts of data. Though DAGGER requires fewer efforts in solving the IVPs as the IVP enhanced sampling method always solves IVPs from 1 = O. However, for many problems, the time spent in solving IVPs is negligible compared to that in solving the open-loop optimal control problems. Specifically, in our first numerical example of optimal landing, takes about 2.5 hours do the BVP computation in total while it only takes 7 minutes to do IVP integration in total. In the second numerical example of the reaching problem, solve 100 trajectories, takes 308 seconds to solve the open-loop solution through DDP while it only takes 3 seconds to do IVP integration. Therefore, the cost for data labeling is approximately the same.
Second, in terms of training time, let us assume that the initial dataset contains M trajectories and each trajectory contributes time-state-action tuples. The IVP enhanced sampling method needs to train K networks; each network is trained on an enhanced dataset with M trajectories. Then, there are in total MKN time-state-action tuples visited the IVP enhanced sampling method (same data is counted repeatedly when training different networks, the same below). DAGGER trains two networks, one with M trajectories and the other with M K trajectories. The former dataset contains MN tine-stat--cctonn tuples while the latter one contains approximately
T to
T T
T- = (K (to + + ....... +(K-)) T
1
M
T
T
time-state- tuples. Then, for time grids to = < 10 < 14 < 16 = (e.g. the landing problem of quadrotor below), DAGGER visits approximately 16.67% fewer data than the IVP
25

---

Under review as a conference paper at ICLR 2023
enhanced sampling method; for time grids to 0 0.16 < 0.64 < 0.8 T and to = 0.16 < 0.48 < 0.8 = (e.g. the reaching problem of the manipulator below), DAGGER visits the same amounts of data as and approximately 6.67% more than the IVP enhanced sampling method, respectively; Therefore, with the same number of epochs in training each network, the training efforts do not differ much in the two methods.
In the following, we compare the IVP enhanced sampling method with DAGGER. We will see that, for the LQR example in Section [ and Appendix [ the IVP enhanced sampling method surpasses DAGGER both theoretically and numerically, especially for large T. For the landing and reaching problem studied previously (see Appendix | and E). both methods perform similarly well. How- ever. in more difficult settings of both problems, the IVP enhanced sampling method outperforms DAGGER.
Results on the LQR example. We first investigate the performance of DAGGER on the LQR ex- ample. With a slight abuse of notation, we will use 43 (1) and útj (4) to denote the open-loop optimal paths sampled for training, ž to denote the initial states used to generate the training trajectories, wa(t, æ) to denote the closed-loop controller learned by DAGGER, and xa((),, ua(t) to denote the IVP solution generated by the closed-loop controller ud.
In DAGGER, we again choose K = T and the temporal grid points t, = for 0 i K. We first sample N initial points { from the normal standard distribution and then generated N approximated optimal paths starting at to 0:
T
T(T - +)+1 7 x, + etz).
af(() +
29 + +2) (t)
where (29),²= are i.i.d. normal random variables and independent with initial states whose mean is m and variance is 0². We then train the closed-loop controller uo by solving the following least square problems:
min Σ
( ut, dtt
Then, we use up to solve the IVPs on the whole time horizon [0, with initial states {
()(()) (1, 2 ()), =](0)) = =). S N,
(25)
and collect ( o = (i) for 1,2...., 1. At each time step t; i, we then compute N approximated optimal paths starting from {
T a5(() T(T - 1) + 25 ()) T(T -) i) -)) + (t- (t- [i,
T(T -
(26)
where 12/ are i.i.d. normal random variables and independent with { and (2 whose mean is m and variance is a². Finally, we collect the optimal paths {(ū), bossss to train the closed-loop controller us by solving the following least square problems:
min
(1) ug(1, (0,1,55010 (1)
(27)
for k =
Theorem 2. Under Model define IVP generated by ua!
it((1) uall) = ua(l, x. xa(()) Fimit, SIST
(28)
and the total cost:
1 Ja T /-' [dt(t) (xa()))
If Pini is a fixed point, then
mm
To²
EJJa - Jo 2
+
)
4
3N
26

---

Under review as a conference paper at ICLR 2023
Proof.
With the same approach of computing My in 16 have
T
T + 1
(o(t,z) = T(T - 1) + 1
r +
cZa d1
T(T t)+
where
N Σ
2 = N j=1
Recalling the definition of zg in equation 25 we have
T(T - t) + 1
x (t)
29 + ctZ2
T ++
Hence, for 0 SiST 1, we have
T(T - 1) +1
x) =0)
3) + ciZyg
7
Plugging the last equation into equation
26). we have that for E [
T
Ti
(t))
T + 1 20
T(T 1) + z e
T(T - 1) +
iT - 1) +i
(1
+
78 + (1 - 1).
T2 +1
T(T +1
Therefore,
T
-) +1
at(() = T(T -
20++ +
-t +1 zZj
+1 +1++1
We can then compute the least square problem
27
obtain that for 0 Sis - and / (i,i+1). 1),
we have
T
T(T k) +1
ua(t, = T(T - t)
x +1
T(T - -) +1 =2%
i
k=0
where
N
2
Z
j=l
for 0 <isT- 1.
We can then solve the ODE
28
we have that when 0 S < T - 1 and
EE 11
T(T-1)+1
F(I)
xa(t)= = T2+1 +1 Pinit F(k+1)F(K) k +1
F( + (i+ 1)F(i
F(E))
k=0
where F(t) T(T - + 1. Therefore, when 0 SiST and E 1),
T
T
1
ua(1)
iitt
(F(I)
F(k)e
T +1
kk + ((F))(F)((++ +1)
(1+ (i+i)
k=0
Define
T
A F(()) +
1
F(K)
4
(k + 1)F(K) F(k
(i+1)
0=
k=
then
T-1
T
T2
T-1
Ja T 1=0
Σ
Tini + egl2 + /imi
Finit +
T2 +
T2 +
1=0
2 Pinil
12
T-1
mi
T
Piii 12
T
2e Fint
leg12
+
(72 1)2
72
T 1=0
(72
+ 1)2
72
1=0
1=0
Pimi 1
T-1
T-1 Σ
lee12 +1
ex
72 + 1
T
1=0
i=0
27

---

Under review as a conference paper at ICLR 2023
Therefore
T-1
T-1
T-1
EJa- Jo
E
E
2
(E
4 2 +
Var
ei).
(29)
We can then compute that
Σ
ΣE
T-1i-1
(F(
F(k)
(++))(()((+))
(i+1)
==
1=0
1=0 k==
F(k)eZ)
-ΣΣΣ
<TF(I)
(1F()
(k+ +1
1=
F(k)e
T-1
(T-i-1))((
()f(i
K= (++1))(0((+))
F(k)eZz
((+))(()(++))
Therefore,
T-1
F(k)
E
cm
(30)
(111)(()((+1)
1=
1=0 0=
(T 1 1)/2
(1+1)(()0((+1)
T1-1-Ti/2
em
T(T - 1) +
T 7T +2 T-1
1
cm
2T
T(T - i +1
T(T i)+
2m0
T +
1
CmT
cm
2T
T +1
On the other hand, noticing that
T-17-1
F(N)
Σ
ΣΣ
E
(+1))000+1))
1=0
k= T=
we have
T
20² T-1
T-1
Var
c2)
F (k)
(i+ (+)(F((((+ +)
1
(31)
N
1=0
km
imk
2²02 T-1
1
[[T - k) 113
2
NT
T(T ---1))
-)
-)
202 T-1
[T[ - k)
NT
-))
K=
202
202
T(T + 1)(2T +1)
202T
(T K)²
NT
2
3N
Combining equations
29
30
and
31
we can conclude our result.
We then numerically compare the performance of DAGGER with the vanilla method and the IVP enhanced sampling method Model 2 5 In these experiments, again set 0.1, m 0.1
28

---

Under review as a conference paper at ICLR 2023
and 1. We present the paths generated by the optimal controller and the controllers learned
by the vanilla method, the IVP enhanced sampling method, and DAGGER in Figure (left). Figure
p (middle) compares the performance of these three methods different time honzons T. We
also test DAGGER with multiple iterations in Figure / (right). Here we set total time T = 30.
The experiment shows that the performance of the learned controller does not improve with more iterations. The detailed settings are identical to the numerical experiments in Appendix B
onhanced sampting
campling
30-
Porformarce D'ffreence
-00
100
1.5
1.0
10a
15
20
30
10'
10
20
-0
Figure Numerical results on learning Model 2 (5) with DAGGER.
Results on the landing problem. Following the same settings as in Appendix and 14. The results are summarized in Table 5 It shows that DAGGER performs closely to the IVP enhanced sampling method. However, if we decrease the number of trajectories in the initial dataset from 500 to 300 to train controllers using these two methods, we observe more decreases in performance in the DAGGER method, which implies that DAGGER more sensitive to the data amount. The main reason is that DAGGER demands enough data the beginning have a good initial controller explore the state over the whole time interval. However, in complicated control problems, we do not have such privilege and indeed require adaptive sampling improve the controller.
E we train a policy
by DAGGER with additional sampling at time
# of iterations
result
AS DAGGER
1.092
2
1.069
AS-300
1.287
DAGGER-300
2
1.405
Table 5: Average cost ratio on 200 test points of controllers trained by the IVP enhanced sampling method and DAGGER. All models with time grids 0 < 10 < 14 16. Models with the suffix 300 are those trained on the initial dataset with 300 trajectories, whose cost ratios are clipped at 10.0 for each test trajectory.
Results on the reaching problem. For the problem detailed in Appendix F by additionally sam- pling at time 0.16 and 0.64 in one iteration, the DAGGER algorithm achieves a policy cost optimal cost ratio of 1.049 on the test dataset, which is close to that achieved by the IVP enhanced sampling method.
We then increase the difficulty of the control problem by increasing the moving distance. Following the configurations in Appendix we change the center initial position and the terminal position to
4 [1.60, 1.30, 2.70. 0.85, 1.90 0.95, -1.60)
q [2.75, 0.60, 2.00, L.55, -2115, 0.00..2..00.. T.
Besides, as the DAGGER will generate states in a wider range, we modify umin: Umax in QRNet to Mmin 2000, 2max 2000 in order avoid saturation. We also increase the size of the initial dataset from 100 trajectories to 200 trajectories. Each network trained with 1500 epochs. The other settings the same that in Appendix
Each method has been run times independently, and we report their average and best performance. see, the TVP enhanced sampling
The results summarized in Table and Figure TO As
29

---

Under review as a conference paper at ICLR 2023
method is capable of finding closed-loop controller with average ratio between policy cost and optimal cost achieving 1.0155. However, the DAGGER algorithm cannot yield such a satisfactory result.
In DAGGERI, we apply the DAGGER method with time grids 0 < 0.16 < 0.48 < 0.8, an earlier final grid compared to DAGGER2 which has time grids 0 0.16 < 0.64 < 0.8. We
average improvement from 1.8528 to 1.6327. DAGGERI performs similarly to the network trained at the second iteration of AS, which implies that the extra data sampled Furthermore, we conduct an additional iteration of the DAGGER method, which requires solving the open-loop problem to get 400 more trajectories and extra training of the network with 1000 trajectories in total. It performs worse, which also confirms the arguments we made at the beginning that the uncarefully collected data may deteriorate the performance.
0.48 does not help much.
100%
100%
BNGGER2 (NCKERS
80%
80%
percentag
percentge
40%
40%
20%
20%
DAGCER2
1.0
1.2
1.8
1.0
1.6
1.8
2.0
policy cost opttiaal
policy cost optimal
Cumulative distribution functions of average cost ratios over independent experiments (left) and cost ratios of the best controller among independent experiments under the proposed method and DAGGER for the optimal reaching problem with a larger moving distance.
Figure 10:
Temporal grid points (T 0.8)
# of iterations
0.16
0.48
0.64
AS DAGGERI DAGGER2
3
1.6592(1.541)
1.3610 (1.0155)
2
1.6327 (1.4004)
2
1.8528 (1.6364)
DAGGER3
3
1.9293 (1.7478)
Table The mean ratio between policy costs and optimal costs of the reaching problem with larger moving distance. In each cell, the first number is averaged over 5 independent experiments and the number in parenthesis is the average ratio achieved by the best controller among 5 independent experiments. The ratio has been clipped at 2.0 for each test trajectory. Both AS and DAGGER2 have temporal grid points < 0.16 < 0.64 < 0.8. DAGGER3 repeats DAGGER2 for one more DAGGER iteration.
30


Reviews:
[{"review_id": "l5-NaaQCLQF", "reviewer": "Reviewer_9LKe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the initial value problem enhanced sampling method to mitigate the distribution shift issue in applying supervised learning to solve the Hamilton-Jacobi-Bellman equation. The authors theoretically prove that the proposed sampling strategy improves over the vanilla strategy on the LQR problem by a factor proportional to the total time duration. Then they further numerically demonstrate that the proposed sampling strategy significantly improves the vanilla strategy on the optimal landing problem of a quadrotor and the optimal reaching problem of a 7 DoF manipulator.\n", "summary_of_the_review": "I cannot recommend acceptance for this paper since the novelty is not very clear. The LQR setting for the theory study and the two examples for numerical study are relatively simple. It is also unclear how to compare the proposed approach with policy-based approaches. \n\n========================\n\n**Post rebuttal**\n\nThanks for addressing my comments. I increase my score to 6. However, I still think the theory in this paper is relatively weak. The paper will be much more insightful if the theory part can be strengthened.", "strength_and_weaknesses": "Strength:\n1) The proposed approach seems to improve significantly over the vanilla approach on the two examples presented in this paper.\n2) The proof for the theoretical results on LQR seems to be correct.\n\n\nWeaknesses:\n1) The numerical study is not sufficient to demonstrate the competitiveness of the proposed approach. How does the proposed approach compare with policy gradient (TRPO, PPO, SAC) on those problems?\n\n 2) The theory is only provided for the simple LQR problem, and does not provide too much insights for nonlinear control problems, which are the main focus of this paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bhSu-nYbfhn", "reviewer": "Reviewer_gCS8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the problem of learning (neural network) closed-loop controllers using supervised learning based on data that is generated by trajectory optimization.\nTo mitigate the well-known problem of distribution mismatch, the paper proposes an iterative procedure. The paper assumes that a few time steps $t_i$ are specified to split the trajectory into different segments. In the first iteration, trajectory optimization is used to generate trajectories starting at time step $t_0=0$, which form the training data for generate the neural network controller. In the second iteration the training data from [t_0, t_1] are kept, but the data from [t1, T] are replaced by optimal trajectories that start at t_1 from states reached by the NN-controller (which can be found by approximately solving the corresponding initial value problem). This process is repeated until each of the pre-specified time-steps had served as initial time-step.\n\nThis procedure of generating training data for the closed-loop controller seems novel, although similar methods have been proposed in the past. Furthermore, the paper provides a theoretical analysis on a specific LQR problem showing that the iterative data collection leads to smaller state-error. The method is evaluated for learning controllers for landing a quadrotor and for performing a reaching motions for a robotic arm (both in simulation).", "summary_of_the_review": "The paper is clear and seems technically correct. However, the contributions seem rather small, as the provided theoretical, empirical and intuitive justifications are rather weak. The paper could be improved by evaluating the effect of keeping the data of previous iterations, and by evaluating the DAGGER-style procedure (comparisons should be over computational cost/time). ", "strength_and_weaknesses": "Strength\n----------\n- The paper is well-written and clear. The presentation is good.\n- The claims seem correct.\n\nWeaknesses\n-----------------\n- The novelty is rather limited (details below).\n- The theoretical results are weak (details below).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EntIC5kSwQ", "reviewer": "Reviewer_s6p1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new method for learning deep closed-loop control policies from training on data generated by open-loop optimal control solvers. The paper proposes a resampling procedure that iteratively retrains the controller in states that it may not have seen in the initial training data. The approach was tested empirically in two problem domains and was shown to achieve good performance and to be somewhat robust to dynamics noise.", "summary_of_the_review": "This paper identifies a problem in an existing line of work, proposes a solution to that problem, and demonstrates the value of that solution theoretically and empirically. Taken together, I think that makes this a worthwhile read and worthy of acceptance, although I will admit that this specific area of learning closed-loop controllers from open-loop data is new to me, so I would defer to others with more knowledge of the related work for their judgments.\n\nSome comments/questions:\n- The problem setup here involves applying control to a system over some finite interval [0, T]. How would this approach extend to a system that needs to operate continuously (e.g. a robot in a factory) if the premise here is that eventually control quality begins to degrade? Would you need to break up tasks into sub-tasks?\n- The problem formulation here assumes that we have knowledge of the system dynamics. What if those models are inaccurate? I would think that you would end up with a similar problem to the one you used to motivate this work, i.e. that the controls you apply bring you to states you haven't seen in your training data and that degrades performance more and more over time. Given that, do you think this approach could be extended to problems where dynamics models are inaccurate? I feel like this theme was somewhat explored in the experiments where noise was added to the dynamics, but it wasn't explicitly framed in this way\n- In the iterative training procedure do you use the same network throughout and finetune that network during each training stage or do you train a new network during each stage. It may have said this somewhere in the paper and I missed it, but it might be nice to state this more clearly e.g. within Algorithm 1.", "strength_and_weaknesses": "Strengths:\n- Points out a weakness in previous works and proposes a novel solution to it.\n- Writing is intuitive, and a good mix of theoretical and empirical results\n\nWeaknesses:\n- Main concern is the reliance on determining the sampling intervals when training a model. This seems like a critical hyperparameter and I don't have a sense for how easy it is to tune, and hence I'm not sure how easy it would be to extend the approach to other problem domains.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject