Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper takes a model-based approach to "imitation learning". It first learns a (flow) "model" to assign likelihoods to trajectories being expert-like as well as sample expert-like trajectories. This is then combined with an estimate for trajectories being certain goal conditioned, where the goals come from a route planner. Results are shown in the context of autonomous driving in the CARLA simulator with a PID controller tracking the open-loop plan from the planner.

One major issue with the paper is that all the main contribution seem to be in the appendix. If I were to summarize it, the paper introduces an interesting way of integrating two different experts to perform learning from demonstration. On the one hand, you have the A* algorithm as an expert providing what waypoints to follow. On the other hand, you have the expert demonstrating how to drive around on the road. The question becomes, how do we reconcile these "experts" acting at different abstraction levels so that our system works in a more general setting. This kind of a setting is definitely not as general as the paper tries to make it out to be, but nevertheless it's reasonably broad and useful.
The idea to of different subsystems for route planning and path planning are not new, but the way it's done here does seem interesting to me.

Note that Sec 3 related work CILS description seems hard to understand. Moreover, the claim "MBRL can also plan, but with a one-step predictive model of possible dynamics", seems incorrect. One can do multi-step predictions and use those to do model-based RL as well. So it's unfair to claim that this other approach is MBRL and yours is not on that basis. 

Experiments are interesting although quite dense. It's unclear what the initial state distribution look like for these experiments though. I'm guessing CARLA is otherwise a deterministic simulator?

Now I am going to do something that you are not supposed to do as a reviewer. Tell the authors the paper they should have written, rather than the paper that was submitted. Apologies for that.
The terminology of Inverse RL, Imitation Learning and Apprenticeship Learning is a mess in the literature unfortunately and can't blame you not trying to fix it in this paper. However the community would benefit if don't overload these terms. Although all of these fall under learning from demonstrations, it's useful to restrict Inverse RL as trying to figure out the reward function being optimized by an expert, imitation learning as learning the exact behavior of an expert (and therefore not doing better than the expert which are by definition optimal or being applicable to doing something other than the expert) while apprenticeship learning as learning from demonstrations to perform even better at the task even generalizing to other things than the demonstrations. With these definitions under the belt the work in this paper is better positioned as about apprenticeship learning. Learning a model from the expert demonstration about how the world works and then using the same for accomplishing different tasks for which there are _no demonstrations_ doesn't sound like imitation?
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper propose imitative models that learns goal-based probabilistic models of expert demonstrations, and use this to perform test-time planning and control of certain goal-directed behavior. The paper demonstrates extensive and impressive experiments on the CARLA simulator and outperforms existing approaches in the success metric.

The idea is quite simple: given a set of states, learn a probabilistic model that assigns high probability likelihood to expert behavior. After training, inference is performed to optimize the likelihood of a goal according to an expert prior. The paper discusses extensively how goal-based likelihood functions would be designed for autonomous driving, and the architectures for good q(s|\phi) in detail, which are of engineering importance in self-driving applications. While I believe the method described might not be significantly novel technically, I believe the paper made nice contributions in terms of the autonomous driving application.

Minor comments:
	- It seems like the term "state" is used to represent the agent's location on the ground plane, which is also not technically the entire state information?
	- Is the argmax in (1) a strict equality? I guess you would assume q(s|\phi) = p(s|\phi) for this to be always true?
	- How many iterations does it take to converge in Algorithm 2? It seems you would need to updates z{1:T} for all newly encountered goals? 

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: Summary:
- key problem: expert-like probabilistic online motion planning to reach arbitrary goals without reward shaping thanks to off-line learning from expert demonstrations;
- contributions: 1) an imitative planning procedure via gradient-based log-likelihood maximization leveraging "imitative models" q(future states | features), 2) multiple proposals to define flexible goals in this probabilistic framework, 3) a complete implementation for end-to-end navigation in CARLA, 4) an extensive experimental evaluation showcasing the performance, flexibility, interpretability, and robustness of the proposed approach w.r.t. the previous state of the art and several Imitation Learning (IL) and Model-Based Reinforcement Learning (MBRL) baselines.

Recommendation: weak accept (leaning towards strong accept)

Key reason 1: principled probabilistic framework bringing the best of both IL and MBRL worlds.
- this planning as inference method is very succinctly and elegantly described in the paper with enough details in appendix (+ code) to suggest a high chance of reproducibility;
- the flexibility of defining different interpretable goals (6 different types explored in the paper) highlights the versatility of the approach;
- the additional benefits in terms of plan reliability estimation (Appendix E) are significant;
- the paper showcases how powerful and useful a good "imitative model" can be, therefore, reinforcing the interest of the research community in the important topic of off-line learning from large datasets of demonstrations (without requiring costly on-line data collection).

Key reason 2: thorough experimental evaluation with convincing results.
- the experimental protocol used is the standard one on CARLA and the results are state of the art;
- the comparison with related works, including recent ones, is thorough and well explained;
- the additional claims regarding robustness are substantiated by multiple experiments.

Suggested improvements:
- Not needing reward engineering is a major claim of this approach, but it seems that constructing goal likelihoods could be seen as a form of reward engineering, no? Table 3 indeed reports significant performance differences (absolute and relative) depending on how the goals are specified, especially in dynamic environments. As the experiments aim at maximizing the same performance metrics, is there a preferred goal type that works well across all experiments? If not, how is "goal definition" different than "reward engineering"?
- Could the authors please include a variance analysis (using different seeds) in Tables 3 and 4? Previous papers have reported high variance in similar settings (cf., Codevilla et al 2019), and this is a common issue in IL/RL.
- How important is knowing \lambda (traffic light state) perfectly in practice? Can the robustness to noise in \lambda be experimentally assessed? I would also clarify in section 4 and Table 2 that other methods do not use \lambda (the traffic light state), which is a signal very strongly correlated with the "ran red light" metric.
- More generally, what is the robustness of this approach to uncertainty / noise in \phi? Although it is typically available (as the authors mentioned) it is never perfect in practice. Can this be handled in a principled probabilistic way as an extension of the current formulation?
- The current model does not factor the influence of the agent on its environment (\phi := \phi_{t=0}). Is this framework limited to open loop planning, or does this open interesting future research directions towards closing the loop? It seems to be a key open problem to at least discuss in Section 5.

Additional Feedback:
- Figure 5 is confusing, not sure it adds much value to the paper;
- typos in Appendix ("pesudocode", "baselines that predicts", "search search").

 ## Update post rebuttal

Thanks to the authors' excellent replies and my initial inclination towards strong accept, I am happy to bump my score to 8. The authors did an excellent job, their rebuttal is on point, not avoiding hard questions, running additional requested experiments (incl. in a clever way for the most computationally expensive ones), and showing clear insights in future steps. Great job!
review_title: Official Blind Review #3

Meta-review: This paper proposes to build an 'imitative model' to improve the performance for imitation learning. The main idea is to combine the model-based RL type of work to the imitation learning approach. The model is trained using a probabilistic method and can help the agent imitate goals that were previously not easy to achieve with previous works.

Reviewers 2 and 3 strongly agree that the paper should be accepted. R3 has increased their score after the rebuttal, and the authors' response helped in this case. Based on reviewers score, I recommend to accept this paper.
Decision: Accept (Poster)