Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: To manage uncertainty and risk-aware planning in the autonomous driving domain, the authors propose LEADR - a hybrid learning/planning method that solves the POMDP problem with learned behavior sampling distributions. To address the problem of missing critical events sampling in Monte Carlo sampling approaches, LEADER uses a neural network to generate sampling distribution that aims at minimizing the planner's expected return whereas the planner tries to maximize it's return conditioned on the learned sampling distribution. Similar to GAN, a minimax loss is designed for the neural network to learn to  sample critical/unsafe behaviors of other road agents. Evaluation is done in simulated (unregulated) urban driving environments and results suggest improved safety, efficiency and smoothness over comparison methods.
strengths_and_weaknesses: The setup of the method reminds of generative adversarial imitation learning, but incorporation of a belief space planner instead of delegating everything to a neural network can improve sample efficiency (given good priors), safety and explainability. The paper is very well structured with key ideas clearly conveyed. The qualitative and quantitative results are convincing. The video is well-made and helpful in showcasing the capabilities of LEADER.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The proposed method is practical, theory is sound and evaluation results are convincing.
issues: - How does LEADER ensure that the network learns to sample behaviors that are critical but still reasonable (i.e. no abrupt changes in driving direction etc)

- If the sampled behaviors are all the more critical possibilities, does that make LEADER an overly conservative planner?

- Is there mode collapse during training?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses the problem of learning amidst uncertainty in human behaviors.  Monte Carlo (MC) sampling techniques leveraged in addressing the attendant POMDP problem are often used for online planning due to their “anytime” properties.  To reason on risky / critical scenarios, techniques such as IS-DESPOT scheme (introduced by Luo, et. al. [9]), use importance sampling to bias MC sampling toward these risky scenarios, however, to achieve safe performance (e.g. low autonomous vehicle collision rates), the importance sampling distribution can be learned.  The technical contribution of this paper is to learn these importance distributions via a min-max game played between the importance distribution generator and the planner.  Comparisons are drawn between the proposed approach and existing approaches (DESPOT, other RL approaches) in various crowded driving scenarios.
strengths_and_weaknesses: The paper offers a solution to an important problem in planning, which amounts to a combination of importance sampling-based Monte-Carlo planning approach with a learned importance distribution (attention).  While the approach to learning importance distributions is not new (acknowledged by the authors and further highlighted in the references below), the paper is well-written, and offers some ideas on framing the problem, including setting up an actor-critic-like scheme for training attention and critic.  It also provides good insight into the construction of the critic and attention networks (via the supplement).

All the descriptions are fairly complete and the supplement provides sufficient detail for reproducibility.  The limitations section is also very thoughtful and complete.

The problem is this well-explored field; to my estimation, the biggest limitation of the paper is that the related work is limited and lack comparison to relevant approaches that solve essentially the same problem (i.e. using a variation of importance sampling).  Some relevant works are below:

[A] Matthew O'Kelly, Aman Sinha, Hongseok Namkoong, Russ Tedrake, John C. Duchi: Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation. NeurIPS 2018: 9849-9860

[B] Mansur Arief, Zhiyuan Huang, Guru Koushik Senthil Kumar, Yuanlu Bai, Shengyi He, Wenhao Ding, Henry Lam, Ding Zhao: Deep Probabilistic Accelerated Evaluation: A Robust Certifiable Rare-Event Simulation Methodology for Black-Box Safety-Critical Systems. AISTATS 2021: 595-603

[C] Edward Schmerling, Marco Pavone: Evaluating Trajectory Collision Probability through Adaptive Importance Sampling for Safe Motion Planning. Robotics: Science and Systems 2017

[D] Atrisha Sarkar, Krzysztof Czarnecki: A behavior driven approach for sampling rare event situations for autonomous vehicles. IROS 2019: 6407-6414

This gives rise to several basic questions: How do these approaches compare conceptually with the proposed one?  What is novel about the current approach?  Could any of these approaches be added to the empirical comparison?

Another fairly significant weakness relates to the assumption of a finite set of intentions and observations.  Such a representation is quite limiting in practice, as it seems to depend heavily on the chosen concept class (choice of discrete maneuvers / behaviors).  Presumably the simulated agents follow the same discrete behaviors, and hence the approach may not generalize well as a result.  Could the approach work with continuous intents?  What modifications would be necessary?
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: There is still some question I hope the authors can address in their revision surrounding the novelty of the importance sampling-based approach, as this has been extensively studied in the literature already.  Many such relevant works were missing in the paper.  Also, it is not clear whether the approach could apply to real-world driving as it only considers a finite set of discrete intents for agents.  Nonetheless, the idea of solving the problem via a critic-based approach and the techniques used are quite nice and would serve as a contribution to risk-awareness in planning.  I would therefore lean toward accept.
issues: There are two main issues to be addressed here.

1. The authors are strongly encouraged to consider related approaches that were missed in their related work (outlined above), and discuss how their approach contrasts with these.  

2. The assumption of pre-defined discrete intents is limiting in practice.  Is there a principled way of generating these intents?  What is the loss in generality by adopting a particular choice of discrete intents?  What are the limitations when applied to the continuum of actions and intents?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes an importance sampling based planning algorithm in which the importance sampling is learned using neural networks to achieve safe and efficient driving. As opposed to existing work where importance sampling is not time-efficient, the proposed algorithm LEADER learns the distribution for importance sampling. This distribution is trained such that it is biased towards generating samples that are adversarial (against the ego vehicle). A Monte-Carlo based planner then uses these adversarial samples to come up with conservative best driving actions. Simulation experiments show the proposed algorithm outperforms the baselines in terms of safety.
strengths_and_weaknesses: Strengths:
- The paper is very well written.
- The proposed approach makes a lot of sense and is not completely based on "neural network magic" -- it is very intuitive that importance sampling that biases search towards risky scenarios would improve safety in autonomous vehicles and this paper is using deep learning techniques only to accelerate the importance sampling procedure.
- Experiments (despite being simulated) show significant improvements in realistic scenarios.

Weaknesses:
- There is not really a major weakness, but the paper could get even better with some additional analysis on how to use importance sampling. See my detailed comments.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper does a great job in various terms: (i) it is proposing a very intuitive, easy to understand algorithm, (ii) it accelerates the algorithm using machine learning techniques and the way it achieves this (the training procedure) is not trivial, (iii) it has excellent results even though the experiments are all in simulation, which is quite common in autonomous driving research due to high costs and safety issues.
issues: - An additional analysis I suggest authors to consider (possibly as a future work) is about how to use importance sampling. So the attention generator network is able to output the distribution for importance sampling. If my understanding is correct, when the planner tree is constructed, it is only the initial state that is being sampled from this attention generator network. The rest of tree is again populated using the distribution $p(s_{t+1}, z_{t+1} \mid s_t, a_{t+1})$, which is also implied by Equation 4. Now, one could also perform importance sampling for the time steps that are further in the future: for example, instead of just the initial state, one could sample the subsequent two time steps from this distribution. On the extreme, the tree would be constructed such that all transitions come from the importance sampling distribution. I guess this would cause the ego agent to try to plan as if all vehicles will attempt to crash with it, so it would be super conservative. By tuning how many time steps are sampled from the importance sampling distribution, the conservatism of the planner could be tuned. I wonder if this tuning could result in even better performance and/or higher safety.
- I am a little confused about the training procedure for the attention generator. What prevents it from outputting a distribution whose samples are dynamically infeasible transitions, e.g., an exo-vehicle teleporting just in front of the ego vehicle? What is the component (of the network or the loss function) that makes sure the samples from $q$ are dynamically feasible?
- The title of the paper is a little misleading, because there are really works that try to learn the drivers' attention to better predict their trajectories. See, for example, "Social LSTM: Human Trajectory Prediction in Crowded Spaces" by Alahi et al. (even though it is for pedestrians), or "Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction" by Cao et al. (which is for driving).
- I found two typos: (i) In the caption of Figure 1, "The learning components includes" has a singular-plural mismatch, (ii) in Section 7, "This work make a few assumptions" has the same issue.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces a method for planning paths around other agents whose intentions are not directly observable. The contribution is in the use of an attention network to extract some information about the other agents, which is fed into a POMDP planner to select the ego vehicle's actions. Experiments on three simulated traffic scenarios suggest the proposed method outperforms methods based on Monte Carlo POMDP planners, model-based RL and model-free RL in terms of collision rate, distance traveled, etc.
strengths_and_weaknesses: Strengths:
- The problem setting is very important, challenging, and relevant to the CoRL community
- The method seems to outperform the chosen baselines on the simulated driving tasks

Weaknesses:
- It's not clear that the problem has been structured in a way that can directly impact modern frameworks for this problem. Modern systems use advanced trajectory prediction algorithms on other agents (this paper has a relatively simple 3 discrete intentions), plan a detailed vehicle trajectory to be tracked according to the vehicle dynamics (the ego vehicle's 3 actions in this paper are accelerate, decelerate, no-op), and involve collision checking (is it embedded in the reward function? I couldn't find it in the paper), for example.
- The POMDP formulation is written out too generally, with the important details relevant to this problem left out: what are the models used for O, T, R for this specific problem and how do we know these are good models for multiagent collision avoidance?
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The problem formulation is too simplified to demonstrate that the proposed technique is relevant to the real issues here.
issues: A new problem formulation would likely require a completely new algorithm, which seems beyond the scope of the rebuttal phase.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper indroduces an algorithm that uses an adversarial set up to learn autonomous driving behaviours from Monte Carlo-based partially observable reinforcement learning. It uses importance weighting to deal with the issue of rare, but safety-critical, events being missed in sampling. The performance of the approach is evaluated in three simulated driving environments and compared against a number of competing algorithims.
strengths_and_weaknesses: Strengths:
- It deals with an important and challenging problem.
- The adverserial setup appears to lead to robustness.
- The mathematical development seems appropriate to the problem the authors are trying to address.
- It presents results that are competitieve to the state of the art in simulation.
- The discussion of limitations is good.

Weaknesses:
- The advance seems somewhat incremental. The results for LEADER in Table 1 are within a standard deviation of DESPOT-Uniform, so the improvement is probably not statistically significant.
- The evaluations are somewhat superficial in that they do not characterise the impact of the design decisions made in developing the proposed approach, or its sensitivity to the parameters used.
- It is doubtful that the results reported will scale to real world situations with a physical robot.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, the paper is of good quality, and addresses an important problem. It presents a reasonable approach that seems appropriate to the problem in hand. I did not detect any issues with the mathematical development. The results presented show that the approach reaches performance comparable with the state of the art, and possibly slightly better (although the statistical significance of this is questionable). The scalability to real world driving is dubious.
issues: - The curves in Figure 3 only show performance in one (carefully selected? representative?) test run. Please overlay these on a plot showing the average curve with error bars.
- I couldn't understand what the video was meant to show - perhaps because it is missing its sound? Please add a commentary and/or subtitles to explain what is going on with it.
- Please clarify how the test runs were apportioned between each map in Table 1.
- Please specify what 'average' is reported in Table 1. (Is it mean performance?) Moreover, please specify how the errors are calculated. (Is it standard deviation?)
review_title: null

Meta-review: metareview: Strengths
- All the reviewers have found the proposed approach is technically sound. 
- The evaluation has been done in a sufficiently fidelity and comprehensibility. 
- The paper is clearly written. 
- Literature has been strengthened during the rebutal phase. 
- Concern on the problem setting has been well-clarified during the rebuttal phase. 
confidence: 5: The area chair is absolutely certain
Decision: Accept (Oral)