Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper focusses on how to extract good policies from experts in imitation learning scenarios. The paper proposes a series of trick for policy extraction from Cross-Entropy-Method-based (CEM/iCEM) trajectory optimizers. The algorithm integrates iCEM with DAgger and Guided Policy Search. 
The extracted policies show good performance versus a model-free baseline (Soft-Actor Critic) and various other extraction baselines (Behavior Cloning, DAgger, Guided Policy Search).

Strengths:
1. The motivation of the paper is clear. In particular, a pretty thorough analysis of the state of the art and its limitations motivates the current approach
2. Experiments are conducted on various standard benchmarks (Humanoid Standup, Half Cheetah, DOOR, Fetch Pick& Place) including sparse reward domains
3. Results are convincing
4. An ablation study shows how different parts lof the proposed method do interact with different environments


Weakness/Comments:
1. It seems that this is primarily a combination of existing methods. What exactly is the new contribution apart from recombination. 
2. Why does not adding policy samples have a counter-intuitive effect in HALFCHEETAH RUNNING?
3. Why do you think this will transfer to model-based RL with learned models

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents an approach to distill a model-based planning expert into a policy to enable real-time execution on robotic systems. An improved version of the Cross-Entropy Method, iCEM is used to generate trajectories via model-based optimisation where the forward model is the ground-truth simulator dynamics. The expert is further improved by warm-starting using the learned policy. The policy is learned via imitation learning of trajectories from the expert. Several approaches for this step are presented, including the vanilla approach of behaviour cloning (BC), dataset aggregation (DAGGER) to reduce covariate shift and Guided Policy Search (GPS) where the planner is encouraged to keep close to the policy distribution via a trust region KL loss. The paper discusses the merits of each approach and proposes Adaptive Policy Extraction (APEX), an approach that learns the policy from the expert through a combination of DAGGER and GPS where the tradeoff between the cost terms on planner exploration and policy trust region KL is set adaptively. This approach is tested on four continuous control tasks and achieves good performance compared to baselines — the learned policy gets significant improvements compared to the baselines. Additionally, a study that ablates different parts of the APEX algorithm is also presented.

Pros:
1. This paper tackles the problem of distilling model-based planner trajectories into a policy. This is a key problem in model-based RL where planner + policy proposals can often exceed the performance of the policy but the former can be too computationally expensive to run on the real system. The proposed approach takes some steps towards this by combining ideas from prior work (GPS, DAGGER, iCEM) to improve policy learning on challenging continuous control tasks.
2. The paper is well written and motivated. The experiments are well structured and provide insights into the struggles of BC and other imitation learning methods when combined with stochastic optimisers like CEM. Different ideas such as DAGGER and GPS are introduced and the combination of these together with adaptive weighting is clearly motivated.
3. The proposed approach leads to significant improvements compared to baseline model-free methods and other imitation learning variants on the presented tasks. Additional experiments including ablation studies also establish further intuition on different parts of the proposed method.

Cons:
1. The proposed approach primarily combines ideas from prior work including GPS and DAGGER. One piece of novelty is the adaptive setting of the \lambda parameter — this is set to be a fixed scalar times the ratio between the range of the main task cost and the range of the auxiliary costs. When all trajectories get the same or similar main task reward (as can happen in sparse reward tasks) this goes to zero, thereby avoiding collapsing the planner to the policy and leading to premature convergence. As evidenced in the experiments, it is not clear if this adaptation helps anywhere else (and it also comes across as a bit hacky). If this is the case, why not use a simpler adaptation scheme that just sets the lambda to zero when the task reward is uniform and a fixed lambda otherwise? In general though, I find the adaptive lambda not well motivated. It would be useful if there is more analysis on the behaviour of this adaptation throughout learning — maybe a plot of \lambda throughout learning can shed more light.
2. The plots in Fig 4 and Fig 5 are quite hard to parse — there are a few places where the colors of the lines are flipped (see eg. Door in both figures, the APEX line is miscoloured). Additionally, there are a few lines that are missing/appear extra in some plots (see eg. Bottom middle plot in Fig 4 where the dashed line is not relevant & bottom right where some lines are missing). It would be great if these can be fixed.
3. Looking at the results in Fig 4 it looks like the expert (iCEM + Policy proposal) is still ~2x better than the learned policy even with the proposed approach. What is the reason for this? Can this gap be shortened further? It is a bit unsatisfying that the policy cannot match the performance of the planner.
4. There are no strong model-based baselines provided apart from different variants of imitation learning coupled with iCEM (which are essentially ablations of the proposed method APEX). It would be great if a prior method which shows results on the Cheetah (Wang et al, 2020 from the paper) can be compared with the proposed approach. This should give a better understanding of the relative strengths of the proposed method compared to the state of the art.
5. As the paper mentions, one key limitation of the approach is the use of a GT dynamics model. A next step could be to extend the approach to train models jointly with the policy. It would be useful to have a longer discussion on this in the paper.

Overall, the paper presents some interesting ideas and the results are promising. More analysis on the adaptation scheme and better baselines are needed to significantly strengthen the paper. I would suggest a borderline accept.
review_title: Distillation of a planner expert into a policy; More analysis and baselines needed.

final_score: rating: 5: Marginally below acceptance threshold
review_content: In summary, this paper combines GPS and DAgger to learn a policy network by imitating a model-based controller, which uses iCEM as the optimizer. Their approach uses both the iCEM controller and the learned policy with DAgger-like relabeling to collect data and then train the network with behavior cloning. An auxiliary loss inspired GPS, which encourages the consistency between the expert controller and the learned policy, together with an adaptive weighting method, is added to reduce the multi-modal issue. The experiment results show promising results.

Strong points:
1. Distilling/imitating CEM policy seems to be a promising approach for solving some control problems, especially when the model is known, or we can learn a good model. Given the deterministic model, CEM can find solutions much faster than an RL algorithm, using CEM to search demonstrations and distilling them into a policy network sounds like a very promising approach in the future.
2. The authors discuss the issue of stochastic teachers and the multi-modal training data and solve them by combing GPS and DAgger. Their approach is simple yet effective.
3. The adaptive auxiliary cost weighting is novel to me.

Weak points:
1. The whole pipeline is not very novel to me. The paper simply adds GPS into DAgger by replacing the LGQ with CEM, which is straightforward if one wants to combine optimal control with neural networks.
2. However, unlike GPS or other model-based DRL papers, this paper requires the ground truth model, which largely limits its application. I agree it's still important to find a way to first solve the problem in the simulation, but I hope the author can give some words about this assumption.
3. I don't think that the sparse reward and the flat regions matter in the model-based RL setting, where the cost is usually dense and designed by the user. Given the limited search horizon, for example, in PickAndPlace, if the agent can only receive rewards when the object is close to the goal, and the planning horizon is short, it's unlikely for a CEM controller to find a solution. Why not simply changing the reward to avoid those flat regions?
4. This work studies how to do imitation learning when the expert comes from a stochastic teacher. However, no imitation learning baseline except the simplest BC is considered. Approaches like GAIL are not evaluated. I don't think  Wang & Ba (2020) can prove that GAN is not suitable in this case. As far as I know, they use the learned model instead of the ground truth model, which is different from the setting in this paper. The authors should at least compare with some STOA imitation learning methods.

Based on the previous weakness, like the loss of the novelty, the missing imitation learning baseline, and the limited applications, I wouldn't recommend acceptance given the current version.

Minor issues:
1. Section 3. "Optimal control is obtained if f is the trajectory cost until step h plus the cost-to-go under the optimal policy for an infinite-horizon problem, evaluated at the last state in the finite horizon trajectory. " I believe some words are missing here. Do you have a value network like POLO in your optimization?
2. Ablation study. APEX without DAgger? What's that? Does this mean no relabeling in Algorithm 2? What are the differences between that and the one without policy samples? How can you train the policy by sampling the trajectories with the policy network while not relabeling it?
3. What's the time complexity? In Table 1. of the appendix, some results are blank. Does this mean the computation complexity is still too high to finish them in time?
4. What do the iterations mean for SAC? Is this comparison fair? I think with the same number of environment episodes, SAC has much fewer environment steps. CEM needs extra environment steps to search for a solution.
5. Presentation is not very clear. For example, what does "expert rollout" means in section 3.3? It's hard for me to guess its meaning without reading the Alg 2. in sec 3.5. There are also some grammar mistakes.

Possible ways to improve:
1. Maybe the authors can combine their approaches with a learned model like PETS-CEM. It would be great if their approach could also benefit in the model-based RL setting.
2. Provide more imitation learning comparisons.

review_title: An interesting paper that combines GPS and DAGGER for imitating CEM's policy

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a method to combine zero order optimizer with imitation learning. Several components are essential for good performance, including policy guided initialization, DAgger and adaptive auxiliary loss.

Overall the paper provides a clear description of the essential components of the algorithms and the result is also quite strong.

Several comments below:

(1)The right most figures in Figure 4 is really hard to unpack with so many information and almost similar colors. Honestly I don't understand these figures, it would be nice if the authors make it clearer, either in the caption or make cleaner figures.

(2) For the first figure in Figure 5, the text said " it performs
initially better than with adding the samples, but the experts do not improve with the policy in this
case such that the asymptotic performance is lower". But the figure shows it has the same final performance as Apex, am I missing something here? Also more distinguished color coding can make the figures easier to digest.

(3) Maybe related to the previous question, what does the learning curve shows? Is it performance of the policy or the performance of the ICEM warm starting with the policy?

(4) How important is an accurate dynamics? All the experiments are done with perfect dynamics information, which I also believe is deterministic, which is not possible in real world scenario. Some experiments with dynamics noise added or with learned model in the CEM rollouts can make the paper's claim stronger.

(5) half cheetah is not a difficult task as claimed in the introduction of the paper. The paper itself shoes SAC can already perform very well. Actually this task is a very strange choice compared to other tasks considered. I would recommend something that is more like the others.
review_title: review1

Meta-review: This paper proposes a method to solve high-dimensional, continuous robotic tasks offering a trajectory optimization and a distill policy. The paper is well-written and the work is promising. It is very relevant for the robotics and RL communities.
Decision: Accept (Poster)