Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method to improve sample efficiency of the model-based imitation learning. The method consists in combining three components:
* Efficient-Zero algorithm
* GAIL-style imitation learning approach which will learn a discriminator and provide it to Efficient-Zero algorithm to specify the reward
* BC algorithm to learn a policy which will regularize the acting distribution of Efficient-Zero

The method demonstrates improved data efficiency (in terms of online rollouts) compared to the baseline, and is conceptually simple.
strengths_and_weaknesses: Strenghts:
* A method which combines different tricks and achieves a competitive performance in a model-based imitation learning setting
* Conceptual simplicity of the method
* The paper studies an important problem - model-based imitation learning

Weaknesses:
* The paper made multiple somehow arbitrary choices which led to a quite an efficient algorithm, but lacks an intuition on why each of the choices was important. For example, the proposed technique (GAIL & BC component) can be plugged-and-played to any other model-based method as well as non model-based method. Is EfficientZero algorithm really crucial for this technique to work? The paper would benefit from more evidence on why each of the proposed components are important. What would happen if instead of model-based algorithm, we use a model-free algorithm with similar acting distribution (mixture between acting and BC policies), having an additional term to optimize for the GAIL-learned reward function?
* Can it be that baseline methods can be significantly improved if EfficientZero algorithm used in these? Can it be that the main benefit in this work comes from using EfficientZero as the main RL algorithm?
* The authors somewhat arbitrary choose the number of expert trajectories. Why exactly these numbers? How do baseline methods perform when more or less expert demonstrations are available?

questions: * Why is it important to use EfficientZero? Can we use any other model-based method?
* Can it be that the main benefit comes from using EfficientZero as the algorithm? What if we use this algorithm in other model-based imitation learning baselines?
* Why did the author choose a specific number of trajectories? How does the method & baselines behave when different number of trajectories are considered?
* Can you provide more intuition / evidence on why each component of the algorithm is important ? (I.e., GAIL, EfficientZero, BC)

limitations: I think the authors adequately presented the limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new IL called EfficientImitate (EI), a planning-based imitation learning method that can achieve high sample efficiency and performance. EI extends AIL to a model-based setting and solve it with MCTS with high sample-efficiency. EI can benefit from BC by using the BC actions in the search. EI provides a natural way to unify the two classes of existing IL methods, BC and AIL. EI uses BC actions as candidates in MCTS. The performance of EI in both state-based and image-based control tasks are amazingly good.
strengths_and_weaknesses: ==Originality==

The proposed EI is novel and interesting. The main idea is to extend AIL to a model-based setting with multi-step losses under the MCTS-based RL framework. Thanks to the planning component of the algorithm, EI naturally unifies two types of previous imitation learning methods (BC and AIL), and shows a significant performance boost. The connections and differences of this work and previous work are well discussed and the related work are cited in the paper.

==Quality==

The proposed method is technically sound. The experiments are conducted in diverse settings with ablations, and the results well supports the claims. Although there are some limitations, this work is a complete piece of work in planning-based IL.

==Clarity==

This paper is clearly written and well organized. The problem is well formulated. The method is clearly motivated and introduced. The limitations and potential impacts are also discussed.

==Significance==

This paper provides an novel sample-efficient IL method that extends AIL into the MCTS-RL.
questions: 1. The base method, MuZero and EfficientZero, also works well in discrete action space. How is the performance of EI in discrete action space?

2. EI uses the Reanalyzed algorithm for offline training, and requires all the samples should be reanalyzed. What is the motivation for using the Reanalyzed algorithm here, and how significant does it contribute to the sample-efficiency of EI? Have you used the Reanalyzed algorithm for other baselines?

3. How to choose the balancing weights $\lambda_d$ and $\lambda_{bc}$ in Eq. (7)? 
limitations: 1. The performance of EI with discrete action space is unknown.

2. There is no discussion on limitations and potential social impacts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents EfficientImitate (EI), which combines Adversarial Imitation Learning (AIL) with model-based planning with MCTS, for sample-efficient imitation learning. During training, an expert buffer is maintained in addition to the standard replay buffer used in EfficientZero. A discriminator is used to predict rewards by contrasting expert demonstrations and agent replays. Next, MCTS is performed each time step using the predicted reward from AIL. In addition, EI unifies BC with AIL by directly including BC actions in the MCTS, which significantly improves the performance. Experiments suggest that EI has achieved significantly better performance than the SOTA methods in various continuous control tasks. 

strengths_and_weaknesses: Strengths: 
Overall, the paper is well-written and easy to follow. Solid experiments have been conducted to support the claims made by the paper. The proposed idea has achieved a very strong performance in experiments, outperforming the state-of-the-art (SOTA) methods by a large margin, with much fewer environment interactions. 

There are some minor weaknesses of the paper. 

1/ The work mainly focuses on the continuous control domains, which has a relatively dense reward and limited planning horizon. However, MCTS is naturally designed for tasks with discrete action spaces, e.g., the game of Go or Atari games, and they are not considered during the experiments. My feeling is that with a sparse reward and long-planning horizon, learning reward from AIL might not be sufficient and may cause extra issues to MCTS. In that case, balancing BC and AIL will be a tricky issue.

2/ The proposed idea is relatively straightforward: it is a simple combination of AIL and MCTS. I think it would be better to dig deeper into the proposed framework. 

questions: 1/ What are the failure modes of the proposed method?

2/ Can the learned reward signal and state embeddings be interpreted in some way? 

3/ How are the reward, value function, and action distribution represented in the MCTS? There are different implementations for Sample Muzero. Some of them are using distributional rewards and values, with discretized action spaces, while others are using continuous actions, e.g., Gaussians, and searching with sampled actions. In addition, how do these design choices affect the final performance of EI?
limitations: I think one limitation of work is that for lots of continuous control tasks, inference speed is critical. However, MCTS is a slow process. Although it achieves a very good performance during evaluation, it might not be appropriate for real-world situations, where inference speed is critical. 

Overall, I still think this is a valuable paper and would like to vote for acceptance.
review_title: null

Meta-review: metareview: This paper introduces a simple approach that improves the sample efficiency of model-based RL for continuous control tasks.  The proposed approach, EfficientImitate builds on EfficientZero and uses a hybrid BC-AIL training scheme.  The contribution is relatively simple and is shown through satisfactory experiments to give a substantial sample efficiency boost.  The paper is clear and appropriately contextualizes its contribution.  All reviewers found the paper to be clear, novel, technically sound, and empirically well validated.  The results show that the innovations constitute a meaningful contribution to a fairly general problem class.  

In initial reviews, two of the three reviewers mentioned that the method is not shown on discrete-action problems.  Personally, I wouldn't have seen this to be a major concern, since continuous control problems are a large problem class.  However, the authors replied with a comment indicating that for lunar lander (a discrete action atari environment), the method works.  It is unclear if this addition was cherry-picked among discrete environments and it is also unclear to me if the authors will add this to the paper.  Nevertheless, as noted, I don't find this to be a major gap in the paper as it currently stands.

Given the sufficiently positive reviews, level of review agreement, and my own reading, I endorse this paper for acceptance.
confidence: Certain
Decision: Accept