Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: IMITATION LEARNING REVIEW
============================

The gist of the paper is to improve IRL with the so-called IMPLANT algorithm which is supposed to improve control policies in learning controllers.

PROS
====
It would be nice to have a framework for minimizing model estimation errors in imitation learning. This paper presents a framework for mitigating the learned dynamics during imitation learning using an augmented cost fuction at test time. The authors called this IMPLANT> 

CONS
====

I do not see how this algorithm is an MPC framework: where is the action/control law computed in lines 7-16 of Algorithm 1?
What did the authors mean in line 11 when they wrote, "First action a_0^{(i)} ~ \pi_\theta ... ". 

* How was \pi in line 11 computed?

* Is there an upper bound on the length of the horizon H?

* If there is, how does the policy perform for slowly-varying trajectories at lower horizons versus fast-changing trajectories at longer planning horizons?


* I am concerned that the reward function in equation (4) is just a restatement of the optimality principle and do not see why this is being rebranded as a new algorithm.

* Of course, the rollout policy would guarantee better performance. What is the predicate for choosing a *random rollout policy* as the authors mentioned on line 4? What is the point of using a mixture?

* Since you mentioned that the rollouts can be parallel-wise executed, did you give any thoughts to the fact that the control law has to be tightly scheduled in the feedback before the next iteration of the MPC is run?

Grammar errors 
==============

There were a few sentence structures in the paper that could use some revision e.g. "A" popular class of approaches"-->approach
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a method to enhance imitation learning by using MPC on the reward function learned by an imitation learning approach. The chosen MPC approach uses the learned policy to generate candidates for a search process that maximizes the reward. To learn the reward function, the authors propose to use GAIL. Besides showing improved performance in a typical benchmark scenario, the authors compare their method with GAIL in various scenarios where some form of transfer is required. The writing is generally clear and easy to understand.

Historically, IRL methods relied on given or learned transition models as they relied on solving for a policy in an inner loop. Recently, we have seen a handful of methods introduced which do not have that short-coming and with it, we have seen a shift towards model-free methods. It is generally understood in IRL that the learned reward function can be used with any reinforcement learning or with any planning algorithm such as MPC but while some model-based approaches have been introduced, the merits of model-based imitation learning have not been thoroughly compared to recent model-free methods to my knowledge. While this makes the paper relevant, the use of a single existing MPC approach with a learned reward function is a relatively minor contribution. My main issue, however, lies in the evaluation as the dynamics model is generally just given to the agent. While the authors acknowledge that the dynamics model could be learned, that is not the case in any of the experiments and it is of little surprise that a MPC approach with perfect knowledge of the transition dynamics is able to outperform reinforcement learning approaches that have to learn from interaction. Furthermore, the authors claim their approach to be zero-shot in the transfer-experiments in comparison to AIRL. This claim is hardly defensible when IMPLANT is handed perfect knowledge of the new dynamics while AIRL has no knowledge of the new dynamics.

Another major issue with this work stems from an apparent misunderstanding of the role of the discriminator in GAIL. While GAIL treats the discriminator as a reward function in an inner loop, it is not a real IRL method. As the policy converges to the expert’s policy, the learned discriminator can no longer tell the data apart and converges to a constant. On a practical level, the proposed MPC approach uses the learned policy to generate proposals and therefore corresponds more to an iterative step on this policy. This explains why the agent is able to achieve higher scores on benchmark tasks; however, the discriminator by itself is not generally useable as a reward function like the authors suggest. This is underlined by the results that “GAIL - reward only” fails to learn. Instead, the authors should use true IRL methods as the basis for their approach.

Finally, the proposed evaluation is limited as the authors choose 3 of the easiest tasks from the mujoco benchmark. The lack of results on humanoid as the hardest task is problematic.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: #######################################################################

Summary:
 
The paper considers imitation learning problem in the presence of small perturbations and nuisances at test-time. In particular, it proposes Imitation with Planning at Test-time (IMPLANT), a new algorithm for imitation learning that incorporates decision-time planning within an inverse reinforcement learning algorithm. To counteract the imperfection due to policy optimization in RL step, IMPLANT uses the reward function estimated in the IRL step for decision-time planning. The effectiveness of the proposed method has been empirically evaluated on two kinds of setups, i.e., the default ‘no-transfer’ setting and the ‘transfer’ setting where the test dynamics is a perturbated version of the training dynamics. 

#######################################################################

Reasons for score: 
 
Overall, I vote for a weak acceptance. I like the idea of using test-time planning for tackling the test-time perturbation in imitation learning. My major concern is about the clarity of the paper and some additional environments (see cons below). Hopefully the authors can address my concern in the rebuttal period. 
 
#######################################################################

Pros: 
 
1. The paper takes one practical issue of imitation learning: test-time perturbation.  
 
2. For me, the proposed IMPLANT method is novel for the issue of test-time perturbation in imitation learning. Specifically, IMPLANT builds on top of GAIL, and learns an additional value estimation during training, and utilizes the estimated reward function, value function for closed-loop planning based on model-predictive control (MPC). The design is interesting.
 
3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework. In particular, IMPLANT outperforms the BC and GAIL-based baselines in the ‘non-transfer’ setting of imitation with limited expert trajectories, and in the transfer setting with causal confusion, motor noise and transition noise. 

#######################################################################

Cons: 
  
1. For the motivation, it would be better to provide more details about it, which seems not very clear to me. Particularly, it is unclear why the planning is introduced, and why such design can address the issue studied in the paper. Additionally, since this paper claims any IRL methods can be used in the training time, I suggest the authors to showcase and discuss the performance with other IRL methods than GAIL at training time. 

2. For the zero-shot transition setting in section 4.3, IMPLANT is only robust to the noise with small sigma. It would be better to explain why the IMPLANT can achieve this level of robustness and which issue limits the performance in the small perturbation. This might help the authors to further improve the robustness of IMPLANT. 
 
3. Although the proposed method has been evaluated in diverse settings, the environments are limited to three continuous control tasks. It would be more convincing if the authors can provide more cases with both continuous and discrete action space in the rebuttal period. 
 
#######################################################################

Questions during rebuttal period: 
 
Please address and clarify the cons above. 
 

review_title: Interesting idea for tacking test-time perturbation in imitation learning, with limited testing environments.

final_score: rating: 3: Clear rejection
review_content: __Summary__
The paper proposes to combine imitation learning (GAIL) with model predictive control to improve on the policy, especially when the system dynamics are noisier at test time.
Here, MPC uses the reward function and value function learned by GAIL and assumes access to a model of the test-dynamics.
The proposed approach, IMPLANT, is compared to GAIL and behavioral cloning on MuJoCo experiments in a standard imitation learning setting and for three slightly modified test scenarios that add either action noise, transition noise, or causal confusing which adds the last actions to the state space during training and replaces this information by noise at test time.


__Strong points / weak points__
+ (+) The paper is well-written and very easy to follow
+ (+) Using MPC on a reward function that was learned by IRL can be reasonable
- (-) Lack of contribution / novelty
- (-) Using the GAIL reward doesn't seem sound. An actual IRL method should be used
- (-) additional assumptions compared to competitors (access to noisy model, computational tractability of online planning) reduce practicability and are not regarded during the comparisons


__Recommendation__
I recommend rejecting the submission because I can't find a noteworthy contribution.


__Supporting Arguments__
- *Lack of contribution:* I don't want to be gatekeeping, but I really don't see any noteworthy contribution. The standard approach for apprenticeship learning is to learn a reward function by IRL and then optimize this reward function to learn a policy. There is no requirement to use the same algorithm to optimize the reward that used for inferring it. It is not surprising that reoptimizing the reward performs better in face of dynamic perturbations compared to a direct policy transfer; after all, this is one of the main motivations for learning a reward function. It is also not surprising that a model-based MPC approach based on the policy can perform better than directly using the parametric policy, even when there are no changes in the dynamics. So, I actually don't even see the research question here. There can be a small contribution due to the empirical evaluation, however, the chosen baselines perform direct policy transfer and, thus, do not seem that interesting.

- *Using GAIL for IRL:* The paper repeatably refers to GAIL as an IRL approach. However, GAIL does not infer a reward function in accordance with the problem formulation of IRL. The "reward function" used by GAIL is only valid for small policy updates of the current policy (generator). *Maximizing* this reward function will in general not result in any reasonable policy. For optimal policy and discriminator, the reward function would even be constant. So using this reward function for MPC doesn't seem sound. The evaluation did show that optimizing the reward function via MPC can be beneficial when using a small enough horizon and when sampling from the parametric policy, but this is not that surprising since the MPC control will in that case tend to remain close to the parametric policy. 

- *Additional assumptions:* The paper argues that IMPLANT is "zero-shot, unlike the proposed solutions of Fu et al. (2017) and de Haan et al. (2019) which require further interactions" with the test environment. However, I would argue that MPC should be treated like a reinforcement learning method here; it also requires interactions with the test environment (i.e., sampling from the dynamics model). I don't see why it would not be fair to use a policy-parametric RL algorithm (e.g. AIRL + reoptimizing) instead of MPC for optimizing the reward. Actually, I think that the test setting would then still favor MPC since it requires costly online optimization which is not always feasible in practice.


__Questions__
1. Are there problem settings where IMPLANT is applicable, but IRL+RL (e.g. AIRL + reoptimizing) is not?
2. The paper mentions that a random rollout policy for MPC performs worse than using the learned policy. Is this also the case for a very large number of rollouts? Is the resulting control also worse in terms of the learned reward function or only w.r.t. the true reward function? 


__Additional Feedback__
My main concern is that the current submission has too little contribution and, thus, I think that more research needs to be done before publishing a paper on this topic. Performing MPC after IRL seems way too little as a paper story. It's hard to suggest a direction for further research because I do not see the research question to start with. Regarding the combination of IRL and MPC, it might be interesting to investigate using MPC within the IRL loop. For example, it can be challenging to learn reward functions for multimodal demonstration based on a parametric (and usually unimodal) policy. 

The current paper could be improved by using an IRL reward and by comparing the approach with IRL+reoptimizing. However, at least for me, this would not affect my recommendation because these changes do not address my main concern.

I hope this review is not too discouraging; the paper does have strong points in the presentation.
review_title: Good presentation, but its hard to name the contribution

Meta-review: The reviewers highly appreciated the replies and the additional experiments. We also had a private discussion on the paper. To summarize: the replies alleviated quite a few concerns, however the consensus was that the paper still does not meet the bar for a highly competitive conference like ICLR.

The idea of combining MPC (on a 'wrong' model)  with a learned cost function is very interesting and a promising direction. On the downside the reviewers are still not entirely convinced about the contribution and believe that the paper requires a significant re-write to incorporate the discussed points as well as an additional round of reviews.
Decision: Reject