Extracting Strong Policies for Robotics Tasks from Zero-Order Trajectory Optimizers
Authors: Cristina Pinneri
Shambhuraj Sawant
Sebastian Blaes
Georg Martius
Venue: ICLR 2021 Poster (2021)
Keywords: reinforcement learning
zero-order optimization
policy learning
model-based learning
robotics
model predictive control

Abstract: Solving high-dimensional, continuous robotic tasks is a challenging optimization problem. Model-based methods that rely on zero-order optimizers like the cross-entropy method (CEM) have so far shown strong performance and are considered state-of-the-art in the model-based reinforcement learning community. However, this success comes at the cost of high computational complexity, being therefore not suitable for real-time control. In this paper, we propose a technique to jointly optimize the trajectory and distill a policy, which is essential for fast execution in real robotic systems. Our method builds upon standard approaches, like guidance cost and dataset aggregation, and introduces a novel adaptive factor which prevents the optimizer from collapsing to the learner's behavior at the beginning of the training. The extracted policies reach unprecedented performance on challenging tasks as making a humanoid stand up and opening a door without reward shaping

Published as a conference paper at ICLR 2021
Extracting Strong Policies for Robotics Tasks from Zero-Order Trajectory Optimizers
Cristina Pinnerf2:2; 2 Sawant Sebastian Blaes', and Georg Martius²
Autonomous Learning Group, Max Planck Institute for Intelligent Systems. Tübingen, Germany "Department of Computer Science, ETH Zurich and Max Planck ETH Center for Learning Systems A "Department of Enginecring Cyberneties, NTNU, Trondheim, Norway
(cpinneri, ssawant, sblaes, gmart ius @etuebingennn. mpg. de
ABSTRACT
Solving high-dimensional, continuous robotic tasks is a challenging optimization problem. Model-based methods that rely on zero-order optimizers like the cross- entropy method (CEM) have far shown strong performance and are considered state-of-the-at the model-basen reinforcement learning community. However, this success comes the cost of high computational complexity, being therefore not suitable for real-time control. In this paper, we propose a technique to jointly optimize the trajectory and distill policy, which is essential for fast execution in real robotic systems. Our method builds upon standard approaches, like guidance cost and dataset aggregation, and introduces novel adaptive factor which prevents the optimizer from collapsing to the learner's 's behavior at the beginning of the training. The extracted policies reach unprecedented performance on challenging tasks like making humanoid stand up and opening door without reward shaping.
Figure I: Environments and exemplary behaviors of the learned policy using APEX. From left to right: FETCH PICK& (sparse reward), DOOR (sparse reward), and HUMANOID STANDUP.
INTRODUCTION
The general purpose based and model-free reinforcement learning (RL) is to optimize a trajectory or find policy that is fast and accurate enough to be deployed real robotic systems.
Policies optimized by model-free RL algorithms achieve outstanding results for many challenging domains (Heess et al., 2017; 17; Andrychowicz et al., 2020), however, in order to converge the final performance, they require large number of interactions with the environment and can hardly be used on real robots, which have limited lifespan. Moreover, real robotic systems are nigh-dimensional and have a highly non-convex optimization landscape, which makes policy gradient methods prone to converge to locally optimal solutions. In addition, model-free RL methods only gather task-specific information, which inherently limits their generalization performance to new situations.
On the other hand, recent advances in model-based RL show that it is possible match model- free performance by learning uncertainty- system dynamics (Chua et al., 2018: Deisenroth & Rasmussen, 2011 1; Du et al., 2019). The learned model can then be used within model- model-ppedicteee predictive control framework for trajectory optimization. Zero-order optimizers are gaining a lot of traction in
"qual contribution.
We acknewledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AT Center (FKZ: O1TS 18039B) and from the Max Planck ETH Center for Learning Systems.

---

Published as a conference paper at ICLR 2021
the model-based RL community (Chua et al., 2018; Wang & Ba, 2020; Williams et al., 2015) since they can be used with any choice of model and cost function, and can be surprisingly effective in finding segh-performance solutions (Pinneri et al., 2020) (close to global optimum) in contrast counterparts, which are often highly dependent on hyperparameter tuning (Henderson al., 2017). One of the most popular optimizers is the Cross-Entropy Method (CEM), originally introduced in the 90s by Rubinstein & Davidson (1999).
to their gradient-
Despite their achievements, using zero-order methods for generating action sequences time con- suming in complex high-dimensional environments, due to the extensive sampling, making it hard to deploy them for real-time applications.
Extracting a policy from powerful zero-order optimizers like CEM would bridge the gap between model-based RL in simulation and real-time robotics. As of today, this still an open challenge (Wang& & Ba, 2020).
We analyze this issue and showease several approaches for policy extraction from CEM. In particular, we will use the sample-efficient modification of CEM (iCEM) presented in Pinneri et al. (2020). Throughout the paper, we will call these optimizers "experts" as they provide demonstration trajecto- ries. To isolate the problem of bringing policy performance close to the expert's one, we consider the true simulation dynamics as our forward model.
Our contributions can be summarized as follows:
pinpointing the issues that arise when trying to distill policy from multimodal, stochastic teacher;
introducing APEX, an Adaptive Policy EXtraction procedure that integrates iCEM with DAgger and a novel adaptive variant of Guided Policy Search;
our specific integration of methods produces an improving adaptive teacher, with higher performance than the original iCEM optimizer;
obtaining strong policies for hard robotic tasks in simulation (HUMANOID STANDUP, FETCH & DOOR), where model-free policies would usually just converge to local optima.
Videos showing the performance of the extracted policies and other information can be found at ht petps://marttin lab. gi github. io/APEX.
2 RELATED WORK
Our objective is to extract high-perforning policies from CEM experts that can operate with a few planning samples to make iterative learning fast. Other kinds of zero-order optimizers have been used to generate control sequences (Williams et al., 2015; Lowrey et al., 2019) but they still have to evaluate thousands of trajectories for each time step. Even simple random shooting has been used as a trajectory optimizer to bootstrap a model-free policy (Nagabandi et al., 2018).
To train policies from optimal control solutions, it shown that the expert optimizers need to be guided towards the learning policy- - known as guided policy search (GPS) (Levine & Koltun, 2013; Levine & Abbeel, 2014). In our work, the expert does not come from optimal control but is the stochastic iCEM optimizer, which we will also refer to as teacher. We apply GPS in model model-predictive control setting, as done in Levine & Koltun (2013); Mordatch & Todorov (2014); Mordatch et al. (2015); Zhang et al. (2016); Kahn et al. (2017); Sun et al. (2018) using local trajectory optimization to generate dataset for training a global policy through imitation learning. These approaches alternate between training policy and creating new data with model-based supervisor guided towards the learner, which was formalized in Sun al. (2018). Stochastic experts require particular guidance strategies, such an adaptive cost formulation that propose here, together with expert warm soum--tarting via distribution initialization and additional samples from the policy, A simple form of warm-starting was already done in Wang & Ba (2020).
Recently, approaches like simple point-to-pe -o-ntont supervised training such as Behavioral Cloning (BC), or Generative Adversarial Network training (GAN) have been explored (Wang & Ba, 2020) for policy distillation from CEM, but only largely sub-optimal policies could be extracted. When the policy is used alone at test time and not in combination with the MPC-CEM optimizer, its performance drops
2

---

Published as a conference paper at ICLR 2021
significantly, becoming almost random for some environments, We argue that the reason behind the difficulty in distilling a policy from CEM expert data is the multimodality of the CEM solution space and its inherent stochasticity due to the sampling, which address below.
Another possibility to train gigher-performance policies from experts is DAgger (Ross & Bagnell, 2010), which is an on-policy method, asking the expert relabel/correct policy actions. Nevertheless, as we will show in the following sections, DAgger alone is not sufficient to extract high-performing policies in our setting. To solve this problem, we use guiding cost in combination with DAgger. combination of GPS and DAgger-style relabeling was proposed in PLATO (Kahn et al., 2017), how- ever to create unbiased training data from iLQG experts. Since unguided DAgger is not appropriate with CEM, PLATO is not successful in our setting either. The components of our algorithm will be explained in the following sections.
METHODS
Trajectory optimization aims to find suitable action sequence ã, = (at; 0.-1...... (c+h) h)) of horizon that minimizes a cost function f(ā,,, st), where 8t is the current state of the system. i.c.
d. arg min f(ā,, 8c).
The cost function f encodes the task. Optimal control is obtained if f is the trajectory cost until step h plus the cost-to-go under the optimal policy for an thimite-horizon problem, evaluated at the last state the finite horizon trajectory. Typically, the cost-to-go is replaced by proxy cost such as the distance to target. In our case, the cost f is given by the sum of negative environment rewards up to the planning horizon.
3.1 IMPROVED CROSS- ENTROPY METHOD FOR MPC
To optimize Eq. 1, we use a variant of the Cross-Entropy Method. Although originally introduced adaptive importance sampling procedure estimate rare-event probabilities, it was recently employed in model-based RL (Chua et al., 2018; 8; Wang & Ba, 2020) as a trajectory optimizer.
A practical implementation of CEM uses a Gaussian proposal distribution over the optimization variables ă. (N o), evaluates the cost function fā)) and refits this distribution iteratively by using the top-k samples. It finds low-cost regions of f and high-performing samples of the variable 7. The cost function can be either expressed through a learned value function or through the dynamics model of the system, which in turn can be learned from data or in an analytical form. In the model predictiv control (MPC) framework, CEM can be used at every time-step generate the optimal action plan, of which only the first action is executed, and the whole procedure repeated for the next steps until task completion.
In this paper, we will make use of iCEM, sample-efficient improvement of CEM by Pinneri et al. (2020) that makes use of colored-noise and memory. In particular, it generates correlated action sequences with a non-flat frequency spectrum, differently from the flat Gaussian noise of CEM. This, together with the memory addition, results in an order of magnitude sample reduction.
3.1.1 USING A POLICY TO INFORM THE OPTIMIZATION: ICEM,
Unguided search for action sequences can be hard, in particular, if the cost function has large flat regions (sparse rewards). Since we aim at extracting a policy T(s), we can expect that after some examples the policy can be used to guide the search into the right region. Thus, we warm-start the mean y of the iCEM Gaussian distribution with the policy actions. In particular, at time = 0 where no prior information exists, the mean is initialized from rolling out the trajectory with the policy until the planning horizon: μ (T()), (ss),... ... (sn)). Whenever the action for a new step is computed, iCEM uses shift initialization of the mean, and only the mean-action action at the end of the horizon is initialized from the policy. Since iCEM is sample-basse based
also provide it with samples from the policy directly. More concretely, the actions performed by the policy are added in the last iteration if iCEM. The policy-informed iCEM algorithm is called iCEM, and is shown in Alg. 1.
3

---

Published as a conference paper at ICLR 2021
Algorithm 1: Improved Cross- Tross-Entropy Method with warm-starting and samples from policy
t, denoted as iCEM, Blue marks policy warm-starting and red is adding policy samples.
Input: f(ā s): cost function: л: policy; N: # of samples: h: planning horizon; k: elite-set size:
3: colored-noise exponent: dini:: noise strength; iter: of iterations; y: reduction factor.
T
for = 0 to 7-1 do
+ get the current state from the environment
i == 0 then
1A0 h-steps rollout of model with starting from
else
L Me shifted M4-1 with last time-step action given by
de - constant vector in [d with values Finit for to iter-1 do
10
N; < max 2.
samples + N, samples from clip(и + (dd, @ of);
with (d, colored-noise
Normal distribution with sxise-xxpoeent 3 and dimension (d, h)
12
add a fraction of the shifted/reused clite-sct to samples
13
if === laxt-iter then
add μ4; to samples add policy actions samples;
15
h-step rollout with policy from current state
16
costs cost function f(ū, s) for ã in samples
17
elite-set, +- best samples according costs
Pe, 0e f fit Gaussian distribution elite-sct, with momentum
19
+ first action of best clite sequence add (s,a) t execute
20
22 return T:
I/ return the trajectory
3.2 OFF- AND ON-POLICY IMITATION LEARNING
How difficult clone a policy from iCEM,? Using a basic off-policy method like Behavioral Cloning (BC), which simply does point-to-point L2 loss minimization, does not work nut-of-theebox This is because during test-time, the policy visits state-space regions for which it never received any expert feedback, and it consequenlyy fails. This problem is also known as "covariate shift". The performance gap (difference in cost to go) between the expert and the policy scales quadratically in time due to the covariate shift (Ross & Bagnell, 2010).
classical way to address this problem is with on-policy imitation learning. DAgger (Dataset Aggregation) by Ross et al. (2011) is the most popular example. DAgger works by iteratively rolling out the current policy, querying the expert for the correct actions (also called relabeling) on the states visited by the policy, and training the policy on the aggregate dataset across all iterations. The covariate shift is alleviated by populating the data with states visited by previous iterations of the policy.
ISSUES OF STOCHASTIC TEACHERS: Nevertheless, some problems arise when considering population- based optimizers as experts, both for BC alone or in combination with DAgger. The optimized action sequence of CEM might converge to different solutions every time it is asked to relabel the policy actions. Naturally, many actions can result in an equally high-performing solution. As an example: in the Fetch Pick & Place task, well before grasping the box, the action of opening or closing the gripper can be arbitrary, as long as the action of opening the gripper is executed on time. To illustrate this problem, we analyze the variance of the relabeled actions produced by the same expert on a fixed state sequence. The variance is high, spanning over a third the action-space, can be seen in Fig. 2(a).
3.3 GUIDED POLICY SEARCH
To lift some of the aforementioned problems, the expert can be guided by the current policy. For high-dimensional control tasks, in addition to the high variance (Fig. 2(a)), zero-order optimizers may
4

---

Published as a conference paper at ICLR 2021
(a)
iCEM
guided iCEM,
(dim=1)
policy
ICEM
actions (dim 1]
policy x
KEM,-GPS fixed
10
20 30
40 50
10
20
30
40
50
steps
steps
Figure 2: Variance of DAgger actions when relabeling 10 times the same trajectory in case of unguided iCEM (a) vs. guided iCEM, (b), for the FETCH PICK& task. The action variance of iCEM is considerably higher than the one of iCEM,-GPS guided by the shown policy (std-dev=0.30 vs 0.13). The policy æ is trained from single expert rollout.
also find multiple different solutions, leading to multi-modal training data. This prohibits successful policy extraction through supervised learning. To address these issues, the cost function f in Eq. becomes the main cost J (task) plus guidance cost to bias the optimizer's ' solution towards the policy:
+ arg min Je (āe, 8t) + ADRL (Tol(a).. a
which is also known as Guided Policy Search (GPS) (Levine & Koltun, 2013). When competing solutions arise, the one closer to the policy is preferred, tackling in this way the problem of multi- modality and creating much more consistent training data. As a remark, in our experiments, we use deterministic policies which reduces the KL-divergence to a squared difference between the actions.
In addition to Eq. 2, we further guide iCEM by warm-starting it from the policy, as implemented in iCEM, (Alg. 1). As consequence, both together lead to reduced action variance, as seen in Fig. 2(b). For an illustrative state sequence, the standard deviation of expert actions drops from 0.30 without guidance to 0.13.
The data for training the policy comes from the expert interacting with the environment, that we call expert rollouts. When combining GPS with DAgger normal expert rollout and for asking the expert to perform action relabeling. This contrasts with PLATO (Kahn et al., 2017), designed for iL.Q experts, where the guidance cost is only used for the expert rollouts. Also, in PLATO the policy is not executed to collect new data, but relabeling of guided expert data is done using the unmodified cost to collect optimal training data. For zero-order optimizers, unguided solutions are not helpful as shown above. Without relabeling the actual policy rollouts, we found PLATO not work in our setting.
the same cost-function (Eq. 2) for the
The hyperparameter À in Eq. 2 is difficult to choose. It might be adapted according to learning progress or other heuristics. However, when using the guided cost in CEM other considerations have to be made which lead to simple adaptation scheme introduced in the next section.
3.4 ADAPTIVE AUXILIARY COST WEIGHTING
The purpose is to trade-off the potential loss in cost by staying close to the policy. Generalizing Eq. 2, phrase the cost function ff in Eq. 1 the sum of the main cost J and set of auxiliary
costs C
o <- arg min Já st) + Σ>, Nyyouux Care st)
(3)
a,
Examples of auxiliary costs are the guidance cost as in Fq. 2 and the action norm || to prefer small action magnitudes. We follow the philosophy that the auxiliary costs are subordinate and should only bias the solutions without causing a large performance drop. This leads to the idea that the auxiliary costs should never dominate the optimization. Since the main costs might vary by orders of magnitude, for instance, in sparse reward settings, we opt for formulation where the auxiliary costs can maximally lead to a fixed fractional loss in performance.
As motivating example, let us consider hard tasks that present flat regions in the cost function: in this case, a non-zeri zero À can lead to a failure of guided CEM to find any good solution. How that possible? Let consider the Fetch Pick & Place task shown in Fig. 1. When none of the sampled
5

---

Published as a conference paper at ICLR 2021
(a) iCEM, GPP with fixed
(b) Success rate iCEM, eexpett
ICEM,-GPS adaptive
espert action policy
iCEM
1.0
(CEM,-GPS food
iCEM,-GPS A-adaptive
1.0
iCEM,-GPS X-fixed
0.5
0.8
success rate
0.0
0.6
-0.5
0.4
0.2
-1.
2
0.0
CEM-iterations
random x
medium
Figure 3: Effect of adaptive À throughout iCEM, iterations and success rate on the FETCH PICK&PLACE task. (a) The action sampling distribution is shown over the iCEM- CEEM-iteaaiions (at predefined time-step) and one of the 4 action-dimensions when guiding with weak policy. Dashed lines indicate the action of the policy and of a high compute-budget iCEM expert. Fixed À shifts the distribution too early, resulting in collapse to the policy behavior and failure to find good solution. (b) Average success rate of iCEM, expert (low compute-budger with 45 samples) over 800 episodes.
action-sequences moves the box (which quite likely in the first CEM iteration) then all sequences are equivalent under the main cost. Elites (Alg. 1 line 17) are only selected based on the auxiliary costs resulting in a quick reduction in sampling variance and preventing good solution to be ever found. To illustrate this phenomenon, Fig. displays the sampling distribution of iCEM, throughout the optimization iterations (line 9 in Alg. 1). Using a fixed (iCEM, -fxed X) typically converges to the policy action which is often unsuccessful in new situation or when the policy is still weak.
We propose the following adaptation of Xy:
R(J)
A = Cj ((Cuuu)) +
(4)
where og is the new hyperparameter, 0 < is small regularization avoid amplifying tiny auxiliary costs, and RC)) represents the cost-range of C' in the elite set:
R(C)
max C' - minn C. eliie-sel
(5)
clite set
This formulation ensures that X; is zero if the main cost cannot be improved by the preliminary elite-set (as in the example above). When an actual improvement is possible and the cost landscape is not flat (R(J)= 0), the auxiliary cost influences up fixed fraction (cy) of the cost-range. Thus, the sampling distribution for the adaptive case will converge only if a good solution is found, as visualized in Fig. 3 (a) (iCEM,-GPP GPS adaptive A) for one auxiliary cost being the KL-term as in Eq.
The new hyperparameter is easier to tune than X. We use the same for all experiments, although the main costs are 3 orders of magnitude apart.
Another aspect of the ratio in Eq. 4 is that, when none of the elite sequences is close to the policy network, then the denominator of Eq. 4 small, which heavily steers the sampling towards the policy, however, only if the numerator is not zero (R(J) * 0), i.e. reward signal from the environment detected. Figure 9 in appendix D shows how the adaptive À parameter evolves over time in the FETCH PICK&PLACE environment for weak and medium policy.
Another effect adaptive guidance in combination with iCEM, is the improvement of the expert/teacher, which is not common nor expected in standard imitation learning. The increased success-rate is shown in Fig. 3(b). As we can see from the plot, the guidance alone is not sufficient to reach the original iCEM expert performance: when the auxiliary cost becomes adaptive, the iCEM benefits from the policy.
TEACHER IMPROVEMENT:
3.5 PUTTING THE PIECES TOGETHER: APEX
Our method, named Adaptive Policy EXtraction (APEX), uses adaptive policy-guided guided iCEM, and DAgger using the same expert to create data for successful training of policies in iterative fashion.

---

Published as a conference paper at ICLR 2021
Algorithm 2: Adaptive Policy EXtraction procedure (APEX)
Input: iCEM,; лe: policy network; # rollouts per iteration init randomly;
2 D 4- 0;
3 [t]
4 while not converged do
f(ā, J 8.) + A (Tallaalloo ()) >2|lā]::
(ā,
II Eqns. and T the resulting trajectory
6 TCEM f n Rollout with iC me);
7
add TCEM to
8
0 train policy on D
Ta +
Rollout with me
ThAgger f- relabel actions Tx with iCEM,, (ā, ma):
DAgger
I1
add TrAgger to D
12
0 < train policy To on D
- ++i+1:
II one APEX iteration
14 return T
The main steps are: create data from the guided expert, update the policy, rollout the policy and relabel the actions using the same expert (DAgger). add the relabeled data to the dataset, and update the policy. This iterates until the desired performance is reached. More details are found in the pseudo-code in Alg.
RESULTS
Can strong policies be obtained from model-based planning and imitation learning? By considering high-dimensional challenging robotics environments, some of them with sparse rewards, we test APEX and find that the answer is yes. In some cases, we considerably improve the sfate-of-the-art currently held by model-free methods.
We perform our experiments on a selection of 4 environments, listed below, which use the MuJoCo (Todorov et al., 2012) physics engine:
HUMANOID STANDUP: (OpenAl Gym Brockman et al. (2016) v2) A humanoid robot is initialized in laying position. The goal is to stand-up without falling, i.c. reaching as high possible with the head. We use a task horizon of 500.
FETCH PICK & PLACE (sparse reward): (OpenAI Gym v1) A robotic manipulator has to move a box, randomly placed on a table, to a randomly selected target location. The reward is only the negative Euclidean distance between box and target location, so without moving the box there is reward.
DOOR (sparse reward): (DAPG project Rajeswaran al. (2018)) A simulated 24 degrees of freedom ShadowHand has to open a door with a handle. The reward the sum of door opening, a quadratic penalty on the velocities, and bounty for opening the door (in contrast to Rajeswaran et al. (2018) where a guidance of the hand to the handle was used).
HALFCHEETAH RUNNING: (OpenAl Gym v3) A half-cheetah agent should maximize its velocity in the positive x- x-direction. In contrast to the standard setting, we prohibit rolling motion of the cheetah by heavily penalizing large absolute angles of the root joint. the standard setting, numeric instabilities in the simulator are exploited by iCEM.
In Fig. 4, we compare APEX against several imitation learning baselines: Behavioral Cloning (BC) (iCEM BC), DAgger (iCEM DAgger), and BC from iCEM, with guidance cost (fixed X) and warm-starting (iCEM,
For reference, we also provide the performance of SAC! (Haarnoja et al., 2018) as a model-free RL baseline, to get idea of the difficulty of the learning tasks. Note that we use partially sparse rewards settings FFTCH TTCK&PI.AC and DOOR (sparse reward) (no hand-to-handa. reward) which makes them particularly challenging. However, for FETCH PICK&PLACE a specialized method for goal-reaching tasks using hindsight relabeling (DDPG+HER) (Andrychowicz et al., 2017) can solve the task. In HUMANOID STANDUP
(only box goal reward)
We took the implementation from https://gitthbb.ccoo www./////wvittt/oooocteoeeo wtw.chrrrrrriiee
7

---

Published as a conference paper at ICLR 2021
HALFCHEETAH RUNNING
HUMANOID STANDUP (500)
return [X 1000)
return (x 1000)
100
75
4-
2-
0-
250
500
750
1000
100
200
300
Iteration (n=1)
Iteration (n=1)
DOOR (sparse reward)
FETCH PICK&PLACE
SUccess rate
0.5
sUccess
0.5
0.0
0.0
100
200
300
200
400
600
iteration (n=10)
iteration (n=25)
APEX
iCEM.- GPS
iCEM DAgger
iCEM BC
SAC
Figure 4: Policy performance the test environments for APEX and baselines. SAC performance is provided for reference.
HALFCHEETAH RUNNING
FETCH PICK&PLACE
1.0
1000)
success rate
0.5
2
0
500
1000
0
200
400
600
iteration [n=1]
iteration (n=25)
APEX
APEX Expert
ICEM Expert
SAC
Figure 5: Interplay between policy and expert. Policy performance (solid line) and expert perfor- mances (dashed lines) on selected test environments for APEX. Due to warm-starting and adding policy samples, experts improve with the policy. For low budgets this effect stronger, see Fig. 8.
the performance of SAC marks the behavior of just sitting. This is local optima and is hard to escape for gradient-based based methods. APEX manages to stand up but cannot balance for long, presumably because there are many ways the robot can fall. Notably, the DOOR (sparse reward) environment with its 24 DoF Shadow hand, is solved by method with high success-rate.
very interesting effect of our approach is that the iCEM, expert working inside of APEX improves with the policy, with the policy performance, even the policies themselves are not very strong yet. Together, APEX able shrink the gap between expert and policy performance and yields strong results, see also the illustrative behaviors in Fig. 1. The policy can achieve significantly higher performance than iCEM, as prominent in the low-budget case in HALFCHEETAH RUNNING presented in Fig. 8.
in Fig. 5. When guided by a policy, the performance raises (dashed blue lines)
4.1 ABLATIONS
Are all components APEX required? We perform several ablations to investigate this question. Figure 6 shows the performance when removing each individual component from APEX, as detailed in Appendix B. Using the adaptive guidance cost instead a fixed X only important the spanse reward environments, expected. In case of dense rewards, the À adaptation does perform identically, Removing murm-starting has drastic effect all environments. APEX without DAgger is also much worse. Not adding policy samples to the optimization has interesting effect. At first glance the policy performance is higher, however, asymptotically the full APEX is better or on par. The reason shown in Fig. 7 where report the expert performance of APEX and APEX without policy
8

---

Published as a conference paper at ICLR 2021
HALFCHEETAH RUNNING
FETCH PICK&PLACE
DOOR (sparse reward)
retum [x 1000)
1.0
1.0
success rate
Success rate
0.5
0.5
0.0
500
1000
1500
200
400
600
0
100
200
300
Iteration (n-1)
Iteration 25)
Iteration (n-10)
APEX
APEX (A-fixed)
APEX w/o policy samples
APEX w/o DAgger
APEX w/o warmstarting
Figure Ablation experiments. We remove different components of the APEX algorithm, see legend. In case of HALFCHEETAH RUNNING, the performance for APEX with X-fixed is not reported as it matches that of APEX.
HALFCHEETAH RUNNING
FETCH PICK&PLACE
retum (x 10001 N &
1.00
rate
0.50
0.25
0
500
1000
1500
0.00
Iteration (n=1)
iteration (n=25)
APEX Expert
APEX w/o policy samples Expert
iCEM Expert
Figure 7: The expert performance for APEX and APEX without adding policy sample. As seen, the expert performance improves with learnt policy the added policy sample directs expert distribution towards a better solution space.
samples. The expert the latter case only marginally improves upon the standard iCEM. Thus, the policy is limited to the iCEM performance, whereas in APEX it can go beyond, see Fig. 5.
CONCLUSIONS
We study the problem of policy extraction from model- based trajectory optimizers using CEM zero-order blackbox method popular in recent model-based RL. Our method (APEX) able to extract strong policies for hard robotic tasks which are especially challenging for model-free RI. methods. This is achieved by imitation learning from a guided CEM expert, where both policy and expert mutually propel themselves to higher performances.
With this work, we want to propose a promising stepping stone towards learning high- gigh-performing policies for real robots, where speed and minimal interaction with the system are more important than asymptotic performance. Model-based RL methods are very sample efficient (Chua et al., 2018; Wang & Ba, 2020) in terms of actual environment interactions, but are too slow to in real-time yet. With the recently proposed improved CEM (Pinneri et al., 2020), the possibility for real-time planning became feasible but still needs huge computational resources. Extracting high performing policies is a promising route to success, where we provide here an important ingredient. Although we are using the simulator as an internal model to study the policy extraction problem isolation, there are good chances that our method will transfer to the model-based RL setup with learned models which we will investigate in future work. One reason for being optimistic about the transfer is that the core problems with stochastic optimizers are identical. Another reason is that a powerful optimizer like CEM ends up finding brittle solutions that are overfitted to the dynamics the simulator and make imitation hard. Learned models can potentially prevent this from happening.
Another route to real robot applications is to pretrain models and policies in simulation and then adapt to the real system and extract policies with APEX. Similarly, the sim-to-real transfer can be done without learned models but using the ground-truth simulations as done here to get strong initialization for fine-tuning on the real hardware. One might ask: why adopting model-based planning method if we are using only a simulator? Because planning with population- based optimizers produce nearly-optima solutions and the extracted policy (if able match expert's 's performance) can beat model- mooee--free baselines like SAC, which can get stuck in local minima in high-dimensional systems, happened in the complicated HUMANOID STANDUP task.

---

Published as a conference paper at ICLR 2021
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pp. 5048-5058, 2017.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl gym. arXiv preprint arXiv:1606.015400 2016.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In NeurIPS, 2018.
Marc Peter Deisenroth and Carl Edward Rasmussen. Pilco: A model-based and data-efficient approach policy search. In International Conference on Machine Learning (ICML), 2011.
Yilun Du, Toru Lin, and Igor Mordatch. Model based planning with energy based models. Conference on Robot Learning (CoRL), 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning. volume 0e///881.012990 pp. 1856-1865 2018.
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: 7777002270022 2017.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters, 2017. cite urxiv:1709.00 709.06660000mmmttt Accepted to the Thirthy-Second AAAI Conference On Artificial Intelligence (AAAI), 2018.
Gregory Kahn, Tianhao Zhang, Sergey Levine, and Picter Abbeel. Plato: Policy learning using adaptive trajectory optimization. In IEEE International Conference on Robotics and Automation (ICRA), 2017.
Sergey Levine and Pieter Abbeel. Learning neural network policies with guided policy search under unknown dynamics. Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 1071-1079. Curran Associates, Inc., 2014.
Sergey Levine and Vladlen Koltun. Guided policy search. In International Conference on Machine Learning (ICML), 2013.
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan online, learn offline: Efficient learning and exploration via model- control. In International Conference on Learning Representations (ICLR), 2019.
Igor Mordatch and Emanuel Todorov. Combining the benefits of function approximation and trajectory optimization. Robotics: Science and Systems, 2014.
Igor Mordatch, Kendall Lowrey, Galen Andrew, Zoran Popovic, and Emanuel V. Todorov. Interactive control of diverse complex characters with neural networks. C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, NeurlPS' 15, pp. 3132-3140. 2015.
A. Nagabandi, G. Kahn, R. S. Fearing. and S. Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning, In IEEE International Conference on Robotics and Automation (ICRA), pp. 7559-7566, May 2018.
10

---

Published as a conference paper at ICLR 2021
Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. Conference Robot Learning (CoRL), 2020.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
https: //tptbgttuu/bb wwt./oo///aaaieee httaaaaaviinnddr haandd hand dapg.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning, In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, 2011.
Reuven Rubinstein and William Davidson. The cross-entropy method for combinatorial and continu- ous optimization, 1999.
Wen Sun, Geoffrey Gordon, Byron Boots, and Andrew Bagnell. Dual policy iteration. In NeurIPS, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks, In International Conference on Learning Representations (ICLR), 2020.
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling. ArXiv, abs/ (bs//1509011 149, 2015.
Tianhao Zhang, Gregory Kahn, Sergey Levine, and Pieter Abbeel. Learning deep control policies for autonomous aerial vehicles with mpe-guided policy search. In IEEE International Conference Robotics and Automation (ICRA), pp. 528-535, 2016.

---

Published as a conference paper at ICLR 2021
APPENDIX
A PERFORMANCE TABLES
We report the performance numbers for experiments in Tab. 1, while the expert/teacher settings are indicated in Tab. 2, and policy settings are Tab. 3. Wall-clock times are Tab. 4.
Table I: Performances for all environments for APEX and APEX policy, Enviroments are abbreviated for space reasons: HC-Run: HALFCHEETAН RUNNING, Hum-Up: HUMANOID STANDUP, and FPP: FETCH PICK&PLACE. We report the cumulative reward (marked with !) and the success rate (marked with 3). SAC and iCEM baseline performances are provided for reference. For an explanation about the budget, see Pinneri et al. (2020).
APEX
Envs
(budget 45)
(budget 100)
(hudget 300)
SAC
Expert
Policy
Expert
Policy
Expert
Policy
HC-Run'
55564241
4847=665
72024510
55961179
93101437
6218:1337
6352
Hum-Up'
149.0k49.6k
73.5k+11.9K
173.7k413.4k
80.5k+14.1k
207.3k44.5k
88.7k412.3k
48.4k
FPP2
0.893
0.0311
0.947
0.770
0.034
DooR²
1.0
0.99
0.02
iCEM (budget 100)
Enys
(budget 45)
(hudget 300)
HC-Run'
3488+119
5235.9=167
7632.54#250
Hum-Up
14..3k113.7k
182.2k
202.4k-51.66
FPP2
0.67
0.81
DOOR²
1.0
Table 2: Expert settings for the considered methods (The values for colored noise exponent for different enviromments are taken from Pinneri et al. (2020))
# elites
Initial std.
Momentum
Decay
Fraction reused clites
Guidance scaling
Horizon
K
Jinst
O
?
constant e
h
iCEM
10
0.5
0.1
1.25
0.3
30
iCEM,
10
0.5
0.1
1.25
0.3
30
APEX
0.5
0.1
1.25
0.3
0.025
30
Warm Start
Add Policy Sample
rollouts per iteration
iCEM
False
False True
iCEM,/
True
(HALFCHEETAH RUNNING, HUMANOID STANDUP) 10 (DooR), 25 (FETCH PICK&PLACE)
APEX
Table Policy settings for iCEM, and APEX
layers
Size
Activation fn
reg.
12 reg.
Optimizer
Learning rate
128
ReLu
le-6
le-5
Adam
5e-4
Batch size
Iterations
# latest rollouts used for training
1024
1000
50 (HALFCHEETAI RUNNING), 100 (HUMANOID STANDUP)
150 25 (FETCH PICK&PLACE), 100 10 (DooR)
12

---

Published as a conference paper at ICLR 2021
Table 4: Wall-clock times for APEX per iteration. Reference machine for wall clock-times is an Intel Xeon Gold 6154 CPU @ 3.00GHz using 32 Cores for parallel simulation models. All times are in minutes.
Wall-clock times per iteration (in min.)
Env
Budget
Expert-only
Dagger-only
Training
Total
HALFCHEETAII RUNNING
45
1.8
2.0
1.4
5.6
100
2.3
2.5
1.4
6.5
300
4.0
4.2
1.5
10.1
HUMANOID STANDUP
45
1.7
2.5
0.9
5.3
300
5.6
6.9
0.9
13.8
FETCH PICK & PLACE
45
7.78
8.7
3.2
20.0
(25 rollouts)
100
12.2
13.5
3.5
29.5
DOOR (10 rollouts)
45
9.7
11.2
3.9
25.1
B ABLATION EXPERIMENTS
In order understand which components of APEX are affecting the performance of the policy extraction and the expert, several ablations were carried out. The results of this ablation studies are shown in Fig. and Fig. 7. In the following, the implementation details of the different ablation studies are discussed.
APEX (A-fixed): Instead of using the adaptive scheme for X that were proposed in Eq. 4, its value is set to a constant value, with value chosen to work well in the respective environment.
APEX w/o DAgger: Instead of using DAgger for policy extracting, plain behavioral cloning is used. In DAgger, states visited by the policy are relabeled with actions from the expert and added in addition the expert data the training dataset of the policy. In our ablation, only the data from the expert added to the training dataset of the policy. In case of DAgger, the policy gets twice the amount of data as in the case of behavioral cloning because each iteration data from the expert and relabeled data from the policy is added to the dataset. In both cases, the same number of gradient-steps are performed during training of the policy.
APEX w/o warmstarting: Instead of initializing the means of all action dimensions along the planning horizon with the actions from the policy at the beginning each rollout, means along action dimensions and the planning horizon are initialized with zero. After each planning step, the mean of action is not initialized with the action from the policy but repeats the last action at time-step t 1.
APEX w/o policy samples: No additional sample trajectory with actions from the policy is added to the other samples during planning (see Alg. 1). To clarify the full version with policy samples one of the random samples overwritten to keep N samples.
C EXPERT AND POLICY INTERPLAY
In Fig. we report the performance of the policy and the experts, for different compute budgets. As discussed in the main paper, it is interesting that the expert inside APEX improves with the policy, The can lead to a higher policy performance than the original iCEM with the same compute budget.
D ADAPTIVE À IN GPS COST
In Fig. 9, report how the adaptive À parameter in the GPS cost of APEX changes during planning for a weak and medium policy, respectively. in case of FETCH PICK&PLACE, In both runs, we fixed the goal and target locations. In case of the week policy, few time steps, meaning the min cost is the same as the max cost among clites. Hence, À is zero allowing the optimizer freely explore without being hampered the weak policy. In case of the
successful solution is found in the first
13

---

Published as a conference paper at ICLR 2021
HALFCHEETAH RUNNING
HALFCHEETAH RUNNING
HALFCHEETAH RUNNING
(budget 45)
(budget 100)
(budget 300)
return (x 1000)
return Ex 1000)
Smmm DX 1000
500
1000
1500
500
1000
1500
500
1000
1500
Iteration (n-1)
Iteration (n-1)
Iteration In-1]
FETCH PICK&PLACE
FETCH PICK TICK&PLACK
(budget 45)
(budget 100)
1.0
1.0
success rate
succese rate
0.0
0.0
200
400
600
800
200
400
600
Iteration (n=25)
(n=25)
APEX
APEX Expert
ICEM Expert
SAC
Figure 8: Same as Fig. 5 but for different compute-budgets. 45, 100 (normal), 300. Notice, that in the case of low and normal budgets in HALFCHEETAh RUNNING the policy outperforms the iCEM expert. In FETCH PICK&PLAS PLACE the policies are able to match the iCEM performance.
better policy, a solution found early on, possibly because of the optimizer being guided by the policy. Thus À becomes non-zero in the second step already.
Weak Policy
Medium Policy
10.0
10.0
0.0015
0.0015
0.0010
0.0010
5.0
0.0005
0.0005
2.5
0.0000
0.0000
5
15
20
25
30
10
15
20
25
30
max cost elite set
min cost elite set
Figure 9: Evolution of the adaptive À parameter during planning. Left for weak, right for medium policy. The light and dark orange curves show the original min/max cost (J) among the clites. The blue curves show how lambda changed due to the differences in the original costs.
14


Reviews:
[{"review_id": "iwFxhDd6IJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach to distill a model-based planning expert into a policy to enable real-time execution on robotic systems. An improved version of the Cross-Entropy Method, iCEM is used to generate trajectories via model-based optimisation where the forward model is the ground-truth simulator dynamics. The expert is further improved by warm-starting using the learned policy. The policy is learned via imitation learning of trajectories from the expert. Several approaches for this step are presented, including the vanilla approach of behaviour cloning (BC), dataset aggregation (DAGGER) to reduce covariate shift and Guided Policy Search (GPS) where the planner is encouraged to keep close to the policy distribution via a trust region KL loss. The paper discusses the merits of each approach and proposes Adaptive Policy Extraction (APEX), an approach that learns the policy from the expert through a combination of DAGGER and GPS where the tradeoff between the cost terms on planner exploration and policy trust region KL is set adaptively. This approach is tested on four continuous control tasks and achieves good performance compared to baselines — the learned policy gets significant improvements compared to the baselines. Additionally, a study that ablates different parts of the APEX algorithm is also presented.\n\nPros:\n1. This paper tackles the problem of distilling model-based planner trajectories into a policy. This is a key problem in model-based RL where planner + policy proposals can often exceed the performance of the policy but the former can be too computationally expensive to run on the real system. The proposed approach takes some steps towards this by combining ideas from prior work (GPS, DAGGER, iCEM) to improve policy learning on challenging continuous control tasks.\n2. The paper is well written and motivated. The experiments are well structured and provide insights into the struggles of BC and other imitation learning methods when combined with stochastic optimisers like CEM. Different ideas such as DAGGER and GPS are introduced and the combination of these together with adaptive weighting is clearly motivated.\n3. The proposed approach leads to significant improvements compared to baseline model-free methods and other imitation learning variants on the presented tasks. Additional experiments including ablation studies also establish further intuition on different parts of the proposed method.\n\nCons:\n1. The proposed approach primarily combines ideas from prior work including GPS and DAGGER. One piece of novelty is the adaptive setting of the \\lambda parameter — this is set to be a fixed scalar times the ratio between the range of the main task cost and the range of the auxiliary costs. When all trajectories get the same or similar main task reward (as can happen in sparse reward tasks) this goes to zero, thereby avoiding collapsing the planner to the policy and leading to premature convergence. As evidenced in the experiments, it is not clear if this adaptation helps anywhere else (and it also comes across as a bit hacky). If this is the case, why not use a simpler adaptation scheme that just sets the lambda to zero when the task reward is uniform and a fixed lambda otherwise? In general though, I find the adaptive lambda not well motivated. It would be useful if there is more analysis on the behaviour of this adaptation throughout learning — maybe a plot of \\lambda throughout learning can shed more light.\n2. The plots in Fig 4 and Fig 5 are quite hard to parse — there are a few places where the colors of the lines are flipped (see eg. Door in both figures, the APEX line is miscoloured). Additionally, there are a few lines that are missing/appear extra in some plots (see eg. Bottom middle plot in Fig 4 where the dashed line is not relevant & bottom right where some lines are missing). It would be great if these can be fixed.\n3. Looking at the results in Fig 4 it looks like the expert (iCEM + Policy proposal) is still ~2x better than the learned policy even with the proposed approach. What is the reason for this? Can this gap be shortened further? It is a bit unsatisfying that the policy cannot match the performance of the planner.\n4. There are no strong model-based baselines provided apart from different variants of imitation learning coupled with iCEM (which are essentially ablations of the proposed method APEX). It would be great if a prior method which shows results on the Cheetah (Wang et al, 2020 from the paper) can be compared with the proposed approach. This should give a better understanding of the relative strengths of the proposed method compared to the state of the art.\n5. As the paper mentions, one key limitation of the approach is the use of a GT dynamics model. A next step could be to extend the approach to train models jointly with the policy. It would be useful to have a longer discussion on this in the paper.\n\nOverall, the paper presents some interesting ideas and the results are promising. More analysis on the adaptation scheme and better baselines are needed to significantly strengthen the paper. I would suggest a borderline accept.", "extra_scores": {}, "extra_text": {}}, {"review_id": "QvokfzMxhw6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In summary, this paper combines GPS and DAgger to learn a policy network by imitating a model-based controller, which uses iCEM as the optimizer. Their approach uses both the iCEM controller and the learned policy with DAgger-like relabeling to collect data and then train the network with behavior cloning. An auxiliary loss inspired GPS, which encourages the consistency between the expert controller and the learned policy, together with an adaptive weighting method, is added to reduce the multi-modal issue. The experiment results show promising results.\n\nStrong points:\n1. Distilling/imitating CEM policy seems to be a promising approach for solving some control problems, especially when the model is known, or we can learn a good model. Given the deterministic model, CEM can find solutions much faster than an RL algorithm, using CEM to search demonstrations and distilling them into a policy network sounds like a very promising approach in the future.\n2. The authors discuss the issue of stochastic teachers and the multi-modal training data and solve them by combing GPS and DAgger. Their approach is simple yet effective.\n3. The adaptive auxiliary cost weighting is novel to me.\n\nWeak points:\n1. The whole pipeline is not very novel to me. The paper simply adds GPS into DAgger by replacing the LGQ with CEM, which is straightforward if one wants to combine optimal control with neural networks.\n2. However, unlike GPS or other model-based DRL papers, this paper requires the ground truth model, which largely limits its application. I agree it's still important to find a way to first solve the problem in the simulation, but I hope the author can give some words about this assumption.\n3. I don't think that the sparse reward and the flat regions matter in the model-based RL setting, where the cost is usually dense and designed by the user. Given the limited search horizon, for example, in PickAndPlace, if the agent can only receive rewards when the object is close to the goal, and the planning horizon is short, it's unlikely for a CEM controller to find a solution. Why not simply changing the reward to avoid those flat regions?\n4. This work studies how to do imitation learning when the expert comes from a stochastic teacher. However, no imitation learning baseline except the simplest BC is considered. Approaches like GAIL are not evaluated. I don't think  Wang & Ba (2020) can prove that GAN is not suitable in this case. As far as I know, they use the learned model instead of the ground truth model, which is different from the setting in this paper. The authors should at least compare with some STOA imitation learning methods.\n\nBased on the previous weakness, like the loss of the novelty, the missing imitation learning baseline, and the limited applications, I wouldn't recommend acceptance given the current version.\n\nMinor issues:\n1. Section 3. \"Optimal control is obtained if f is the trajectory cost until step h plus the cost-to-go under the optimal policy for an infinite-horizon problem, evaluated at the last state in the finite horizon trajectory. \" I believe some words are missing here. Do you have a value network like POLO in your optimization?\n2. Ablation study. APEX without DAgger? What's that? Does this mean no relabeling in Algorithm 2? What are the differences between that and the one without policy samples? How can you train the policy by sampling the trajectories with the policy network while not relabeling it?\n3. What's the time complexity? In Table 1. of the appendix, some results are blank. Does this mean the computation complexity is still too high to finish them in time?\n4. What do the iterations mean for SAC? Is this comparison fair? I think with the same number of environment episodes, SAC has much fewer environment steps. CEM needs extra environment steps to search for a solution.\n5. Presentation is not very clear. For example, what does \"expert rollout\" means in section 3.3? It's hard for me to guess its meaning without reading the Alg 2. in sec 3.5. There are also some grammar mistakes.\n\nPossible ways to improve:\n1. Maybe the authors can combine their approaches with a learned model like PETS-CEM. It would be great if their approach could also benefit in the model-based RL setting.\n2. Provide more imitation learning comparisons.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HG1uoZVFanq", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focusses on how to extract good policies from experts in imitation learning scenarios. The paper proposes a series of trick for policy extraction from Cross-Entropy-Method-based (CEM/iCEM) trajectory optimizers. The algorithm integrates iCEM with DAgger and Guided Policy Search. \nThe extracted policies show good performance versus a model-free baseline (Soft-Actor Critic) and various other extraction baselines (Behavior Cloning, DAgger, Guided Policy Search).\n\nStrengths:\n1. The motivation of the paper is clear. In particular, a pretty thorough analysis of the state of the art and its limitations motivates the current approach\n2. Experiments are conducted on various standard benchmarks (Humanoid Standup, Half Cheetah, DOOR, Fetch Pick& Place) including sparse reward domains\n3. Results are convincing\n4. An ablation study shows how different parts lof the proposed method do interact with different environments\n\n\nWeakness/Comments:\n1. It seems that this is primarily a combination of existing methods. What exactly is the new contribution apart from recombination. \n2. Why does not adding policy samples have a counter-intuitive effect in HALFCHEETAH RUNNING?\n3. Why do you think this will transfer to model-based RL with learned models\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4PmdEIolPdO", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to combine zero order optimizer with imitation learning. Several components are essential for good performance, including policy guided initialization, DAgger and adaptive auxiliary loss.\n\nOverall the paper provides a clear description of the essential components of the algorithms and the result is also quite strong.\n\nSeveral comments below:\n\n(1)The right most figures in Figure 4 is really hard to unpack with so many information and almost similar colors. Honestly I don't understand these figures, it would be nice if the authors make it clearer, either in the caption or make cleaner figures.\n\n(2) For the first figure in Figure 5, the text said \" it performs\ninitially better than with adding the samples, but the experts do not improve with the policy in this\ncase such that the asymptotic performance is lower\". But the figure shows it has the same final performance as Apex, am I missing something here? Also more distinguished color coding can make the figures easier to digest.\n\n(3) Maybe related to the previous question, what does the learning curve shows? Is it performance of the policy or the performance of the ICEM warm starting with the policy?\n\n(4) How important is an accurate dynamics? All the experiments are done with perfect dynamics information, which I also believe is deterministic, which is not possible in real world scenario. Some experiments with dynamics noise added or with learned model in the CEM rollouts can make the paper's claim stronger.\n\n(5) half cheetah is not a difficult task as claimed in the introduction of the paper. The paper itself shoes SAC can already perform very well. Actually this task is a very strange choice compared to other tasks considered. I would recommend something that is more like the others.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to solve high-dimensional, continuous robotic tasks offering a trajectory optimization and a distill policy. The paper is well-written and the work is promising. It is very relevant for the robotics and RL communities.
Decision: Accept (Poster)