LEADER: Learning Attention over Driving Behaviors for Planning under Uncertainty
Authors: Mohamad Hosein Danesh
Panpan Cai
David Hsu
Venue: CoRL 2022 Oral (2022)
Keywords: Planning under uncertainty
Integrating planning and learning
Autonomous driving

Abstract: Uncertainty in human behaviors poses a significant challenge to autonomous driving in crowded urban environments. The partially observable Markov decision process (POMDP) offers a principled general framework for decision making under uncertainty and achieves real-time performance for complex tasks by leveraging Monte Carlo sampling. However, sampling may miss rare, but critical events, leading to potential safety concerns. To tackle this challenge, we propose a new algorithm, LEarning Attention over Driving bEhavioRs (LEADER), which learns to attend to critical human behaviors during planning. LEADER learns a neural network generator to provide attention over human behaviors; it integrates the attention into a belief-space planner through importance sampling, which biases planning towards critical events. To train the attention generator, we form a minimax game between the generator and the planner. By solving this minimax game, LEADER learns to perform risk-aware planning without explicit human effort on data labeling.

LEADER: Learning Attention over Driving Behaviors for Planning under Uncertainty
Mohamad H. Danesh Oregon State University
Panpan Cai
Shanghai Jiao Tong University cai _aai_paanaaesjee tu. edu.
daueshn@orrggonstatee.ee edu
David Hsu
National University of Singapore
dyhsu@comp.. nus, edu. sg
Uncertainty in human behaviors poses significant challenge to au- tonomous driving in crowded urban environments. The partially observable Markov decision process (POMDP) offers principled general framework for decision making under uncertainty and achieves real-time performance for complex tasks by leveraging Monte Carlo sampling. However, sampling may miss rare, but critical events, leading to potential safety concerns. To tackle this challenge, we propose new algorithm, LEarning Attention over Driving bEhavioRs (LEADER), which learns to attend to critical human behaviors during planning. LEADER learns neural network generator provide attention over human behaviors; it integrates the attention into a belief-space planner through importance sampling, which biases planning towards critical events. To train the attention generator, we form a minimax game between the generator and the planner. By solving this minimax game, LEADER learns perform risk-aware planning without explicit human effort on data labeling. 3
Abstract:
Introduction
Robots operating in public spaces often contend with challenging crowded environment. A representative is autonomous driving in busy urban traffic, where a robot vehicle must interact with many human traffic participants. A significant challenge is posed by the amount of uncertainty in human behaviors, e.g., on their intentions, driving styles, etc.. The partially observable Markov decision processes (POMDPs) [1] offer principled framework for planning under uncertainty. However, optimal POMDP planning is computationally expensive. To achieve real-time performance for complex problems, practical POMDP planners [2, 3] often leverage Monte-Carlo (MC) sampling to make approximate decisions, i.e., sampling subset of representative future scenarios, then condition Deecision-makee on the sampled future. They have shown success in various robotics applications, including autonomous driving [4], navigation [5, 6]. and manipulation [7, 8).
Sampling-based POMDP planning, however, may compromise safety. It is difficult to sample future events with low probabilities, and some may lead to disastrous outcomes. autonomous driving, rare, but critical events often arise from adversarial human behaviors, such recklessly overtaking the ego-vehicle or making illegal U-turns. Failing to consider them would lead to hazards. Earlier work Luo al. [9] indicates that re-weighting future events using importance sampling leads to improved performance. The approach requires an importance distribution (IS) for sampling events. In autonomous driving, an importance distribution re-weights the different intentions of human participants, e.g., routes they intend to take. A higher weight means an increased probability of sampling an intended route, thus examining its consequence more carefully during planning. Prior works have used IS to better verify the safety of driving policies and proposed ways to synthesize
WWrk done when interning at National University of Singapore Corresponding author: Panpan Cai (cai_panpaat @amppan@sjtuu. ssttt.edu.cc)) edu.cn) 3 Code available at: httss//gthhb..ooo/oooooosh//AAAa
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Gritic
Exo-agent's intentions in belief b
After applying
G
Attention Generator
q
intentions
Intentions
V
Environment
Planner
b
Z
concatenation
egoive
(a)
(b)
(c)
Figure I: Overview of LEADER. (a) LEADER contains a learning component (red) and planning component (blue). The learning components include: an attention generator Gy that tries to gen- crate attention q over human behaviors, based on the current belief 6 and observation z from the environment; and a critic Cp that approximates the planner's value estimate, V, based on 6, z and the generated attention, 9. The planning component performs rask- planning using the learned attention q. It decides an action a to be executed in the environment and collects experience data. (b) Attention is defined as an importance distribution over human behavioral intentions. The upper box shows the probability of different intentions of the highlighted exo-agent in green, yellow and red, as well as how the attention generator maps the natural-occurrence probabilities to importance probabilities, by highlighting the most adversarial intention (red). (c) We train LEADER using three simulated real-life urban environments: Meskel Square in Addis Ababa, Ethiopia, Magic Roundabout in Swindon, UK, and Highway in Singapore.
importance distributions offline [10, 11. 12, 13J. However, it is difficult to obtain importance distributions for online planning, due to the real-time constraints.
We propose new algorithm, LEADER, which learns importance distributions both f and for sampling-based POMDP planning. The algorithm uses a neural network attention generator obtain importance distributions over human behavioral intentions for real-time situations. Despite the neural networks' problem terms of interpretability [14], they are quite useful in problems that need to be scaled, such as crowd driving. Next, an online POMDP planner consumes the importance distribution to make risk-aware decisions, applying importance sampling to bias reasoning towards critical human behaviors. To train the algorithm, we treat the attention generator and the planner opponents in a minimax game. The attention generator seeks to minimize the planner's 's value, or the expected cumulative return. This maps to highlight the most adversarial human behaviors by learning from experience. On the other hand, the planner seeks to maximize the value conditioned on the learned attention, which maps find the best conditional policy using look-ahead search. By solving the minimax game, the algorithm learns to perform risk-aware planning, without human labeling. In our experiments, evaluate the performance of LEADER for autonomous driving in dense urban traffic. Results show that LEADER significantly improves driving performance in terms of safety, efficiency and smoothness, compared both risk-aware planning and risk-aware reinforcement learning.
2 Background and Related Work
2.1 POMDP Planning and Monte Carlo Sampling
The Partially Observable Markov Decision Process (POMDP) [1] models the interaction between a robot and an environment as a discrete-time stochastic process. POMDP is written as a tuple (S, A, 7, T, R, O, bo) with S representing the space of all possible states of the world, A denoting the space all possible actions the robot can take, and Z as the space of observations it can receive. Function T(s,a, a, s') p(s'ls, (ss a) represents the probability of the state transition from E S to

---

s' E S by taking action E A. The function R(s, a) defines real-valued reward specifying the desirability of taking action a E A at state s S. The observation function O(s', a. 2) p(z)s' [12ss'.a) .) specifies the probability of observing = E Z by taking action a E A to reach to s' E E.S. E (0. is the discount factor, i.e., the rate of reward deprecation over time. Because of the robot's perception limitations, the world's full state is unknown to the robot, but can be inferred in the form of beliefs, or probability distributions over S. The robot starts with an initial belief bo at = O, and updates it throughout interaction trajectory using the Bayes rule [15], according to the actions taken and observations received. POMDP planning searches for a closed-loop policy, X" B + A. prescribing an action for any belief in the belief space B, which maximize the policy value, Va (bu) E |E 7 R. 8t. π (bb)) [bu. a/. which computes the cumulative reward to be achieved in the current and future time steps, t 2 0, if the robot chooses actions according to policy and the updated belief by, from the initial belief by onwards. Discounting using E [0, 1) keeps the value bounded.
Online POMDP planning often performs look-ahead search, constructing belief tree starting from the current belief and branching with future actions and observations. Enumerating all possible futures, however, is often computationally intractable. Practical algorithms like DESPOT [2] leverage Monte-Carlo sampling and heuristic search to break the computational difficulty. DESPOT samples initial states and future trajectories using the POMDP simulative model. Denote a trajectory as
<= = (su. a1 $1, 21, 0g. $2, 22, .... The initial state Su is sampled from the current belief b. Given any subsequent state st and robot action ag, the next state $141 and the observation 21+-1 are sampled with a probability of p($:+1, Zt+1 St. a4-1) = (s ae, 2x+1 (T de, $t41). A DESPOT tree collates set of sampled trajectories to approximately represent the future. Each node in the tree contains set of sampled future states, forming an approximate future belief. The DESPOT tree branches with all possible actions and then sampled observations under each visited belief, effectively considering all candidate policies under the sampled scenarios. Figure 2(a) shows an example DESPOT tree.
DESPOT evaluates the value of policy using Monte-Carlo estimation:
(b)
Vedg Rz
Σ p(C|b,
ÇEA
where A is a set of trajectories sampled by applying т. Also,
11-1 p(C|b, 1) = b(so) II p( 24 11, ac+1)
(2)
is the probability of a trajectory { being sampled, Ve =- 3a R'ss ax+1) is the cumulative reward along Ç. and H is maximum look-ahead depth, or the planning horizon. By incrementally building a belief tree using sampled trajectories, DESPOT searches for the policy that provides the best value estimate, and outputs the optimal action for when exhausting the given planning time.
2.2 Aisk-Aware Planning and Learning Approaches
There are different techniques for risk-aware planning under uncertainty. Nyberg et al. [16] and Gilhuly et al. [17] proposed two measures for estimating safety risks along driving trajectories, with a focus on open-loop trajectory optimization. Huang et al. [18] modeled risk-aware planning as Srance-constrainee POMDP to compute closed-loop policies that provide a low chance of violating safety constraints. Kim et al. [19] leveraged bi-directional belief-space solvers. It bridges forward belief tree search with heuristics produced by an offline backward solver. Both methods improved the performance of POMDP planning in safety-critical domains, but at the cost of an expensive onf planning stage. Thus, it can hardly apply to large-scale problems. Most relevant to our method, Luo et al. [9] offered IS IS-DESPOT, which improves the performance of online POMDP planning by leveraging importance sampling (IS). It biases MC sampling towards critical scenarios according to an importance distribution provided by human experts, then computes a risk-aware policy under the re- re-weighted scenarios. Manually constructing the importance distribution, however, is difficult for complex problems such driving in an urban crowd. In this paper, we propose a principled approach to learn importance distributions from experience and adapt them with real-time situations.
LEADER is also loosely connected to risk-aware reinforcement learning (RL). Kamran et al. [20] proposed a risk- Q-learning algorithm by punishing risky situations instead of only collision failures. Mirchevska et al. [21] proposed to combine DQN with rule-based safety checker, masking
3

---

out infeasible actions. Eysenbach et al. [22] offered Ensemble-SAC (ESAC) for aisk-aware RL, by additionally learning an ensemble of reset policies to assist the robot avoid irreversible states during training. However, because of low sample efficiency, the above model-free approaches were limited to small-scale tasks like lane changing in regulated highway traffic or controlling articulated robots for simple simulated tasks. Some prior work improved the risk-awareness of model- RI. that plans with learned models. Thomas et al. [23] proposed SMBPO, which learns a sufficiently-laree terminal cost for failure states. Berkenkamp et al. [24] focused on ensuring the stability of control. The model-based methods provided better sample efficiency. However, it is still difficult to learn a model for large-scale, safety-crite crittccliitiaa problems like driving in an urban crowd [25].
2.3 Integrating Planning and Learning
LEADER integrates planning and learning to benefit from both explicit reasoning and past experience. It uses learning assist explicit POMDP planning. which has been inspired by the following prior works. The LeTS-Drive algorithm [26, 27] first proposed to learn heuristics for solving POMDPs, with planner actors and the learner collaborating in closed-loop. Later, Lee et al. [28] proposed a generator-critic framework to learn macro-actions for POMDPs. It uses a neural network critic to approximate the planner's s value function and enable end-to-end training. This work extends the generator-critic framework, applying it to solve a min-max game for learning behavioral attentions.
3 Overview
LEADER learns to attend to the most critical human behaviors for risk-aware planning in urban driving. We define attention over the behavioral intentions of human participants. Assume there N exo-agents (traffic participants) near the robot. Each of them may undertake finite set of intentions, or future routes such as keeping straight, turning left, merging into the right, etc, extracted from the road context by searching the lane network. The actual intention of an ex0-agent is not directly observable. At each time step, LEADER maintains a belief & and an importance distribution q over the intention sets of the N exo-agents. The belief b specifies the natural occurrence probability of exo-agents' intentions. It is inferred from the interaction history. The importance distribution specifies the attention over exo-agents' intentions, determining the actual probability of sampling them during planning. We propose to learn the importance distribution or attention mechanism from the experience of an online POMDP planner. We will use "importance distribution" and "attention" " interchangeably in the remaining.
LEADER has two main components: learner generating attention for real-time situations, and planner consuming attention to perform conditional planning. The generator uses a neural network, q = G (b, 2). to generate an importance distribution q, for the real-time situation specified by the current belief b and observation z. The planner uses online belief tree search to make risk-aware decisions for given b and z, leveraging the importance distribution q to bias Monte-Carlo sampling towards critical human behaviors. To train LEADER, generator and planner form a minimax game:
minmax V. (6, x/q) gEQ TEll
(3)
where Il is the space of all policies, and Q is the space of all importance distributions. In this game, the generator must learn to generate q's that lead to the lowest planning value, meaning to increase the probability of sampling the most adversarial intentions of exo-agents; the planner must find the best policy with the highest value, conditioned on the generated q. We further learn a critic function, C.p (b, 2, 4), another neural network that approximates the value function of the planner, (b, 2()) Rs max V. (b. zlq). to assist pradient-descent training of the generator. Figure la and lb demonstrate the training architecture of LEADER. In the bottom row, the planner plans robot actions using the generated attention and feeds the driving experience to a replay buffer. In the top row, we train the critic and the generator using sampled data from the replay buffer. The critic is fitted to the planner's value estimates using supervised learning: the generator is trained to maximize the planner's 's value, using the critic differentiable surrogate objective.

---

g(.
20
4
Risk
Ug
40
1
dp
41
4g
High
Ro
Low
(n)
(b)
Figure 2:
A comparison of DESPOT and the LEADER planner. (a) The DESPOT tree samples intentions from the belief, b(so). without considering the criticality of the intentions. Some rare critical events might be missed during sampling. (b) The LEADER tree samples intentions from the learned importance distribution, g(8o), which is biased towards adversarial intentions. The tree thus considers more critical events (red), and less safe events (green).
Risk-Aware Planning using Learned Attention
The LEADER planner performs belief tree search to plan robot actions conditioned on the attention over sxo-agents' behaviors, solving the inner maximization problem in Eq. (3). belief tree built by LEADER looks similar to DESPOT tree (Section 2.1). It collates many sampled trajectories, each corresponding to a top-down path in the tree. The tree branches over all actions and sampled observations under each belief node, effectively considers all possible policies under the sampled scenarios. The difference is, however, LEADER biases the tree towards higher-risk trajectories using importance sampling. Figure 2 provides side-by-side comparison of belief trees in DESPOT and LEADER. Appendix introduces the basics of importance sampling, and a theoretical justification on our minimization objective of learning importance distributions.
Concretely, we sample initial states se from the learned importance distribution ((s),, instead of from the actual belief b(su). As a result, the sampling distribution of simulation trajectories is also altered from Eq. (2), becoming:
D-1
q([lb, 2 = g(so) II R t=0
2441/51, at+1),
(4)
where ( is a hypothetical future trajectory. The value of candidate policy n is now evaluated as:
[
p(C16, 2, x)
Va (6)
(b..)) Vids =
Vedç
(b.am)) o([|6, 2,
Here, Vg is the discounted cumulative reward along a trajectory Ç. Eq. (5) first shows the definition of the value of policy и, then applies importance sampling, replacing the sampling distribution p(-lb, 2, T) in Eq. (2) with q( (b, 2, T in Eq. (4). also uses the importance weights p(C b.2.T) to
g({]b,s.r)
unbias the value estimation and ensure the correctness of planning. The value is further approximated using Monte Carlo estimates:
p(Cib,2,T) CEA' g(Clb, 2,
1
p(sO)
V (b)
V₂ A
Vc. (550)
(6)
A
Eq. (6) starts with approximating the value using set of sampled trajectories, A' It then simplifies the importance weights to p(so) as p(Clb, 2, π) and q({(b, m) only differ the probability of o(5o)*
sampling the initial state sp-
The LEADER planner is built on top of IS-DESPOT [9], integrating it with learned importance distributions. Following IS-DESPOT, LEADER performs anytime heuristics search, incrementally building a sparse belief tree when sampling more trajectories. During the search, it maintains for each belief node set of approximate value estimates, and uses them as tree search heuristics. See Luo et al. [9] for more details of the anytime algorithm.
5

---

5 Learning Attention over Human Behaviors
We train the critic and generator neural networks using driving experience from the planner, stored in the replay buffer. The generator is trained to minimize the planner's 's value estimates, solving the outer minimization problem in Eq. (3). uses the critic as a differentiable surrogate objective, which is supervised by the planner's 's value estimates. Appendix B describes the network architectures of our critic and generator.
Critic Network. The Critic network's parameters yp are updated by minimizing the L.2nnorm between the critic's s value estimate and the planner's 's value estimate V using gradient descent, given a sampled tuple of belief observation 2, and attention q:
J(p) (. ()=bC,, 2, - (b,,
where D is the set of online data stored in the replay buffer.
Attention Generator Network. The generator network's s parameters v are updated by minimizing the planner's 's value as estimated by the critic, given a sampled tuple of belief and observation 2:
((,) B(t.))DD (by 2, 2))
where D represents online data stored in the replay buffer. This objective is made differentiable using the reparameterization trick [29], enabling gradient descent training via the chain rule:
(( = (. ( (io, (b, 2, e))1
(9)
The following is the training procedure of LEADER. In each time step, the current belief b and observation z are fed into G. to produce the importance distribution q. Then, the planner takes b, and q inputs to perform risk-aware planning, and outputs the optimal action to be executed in the environment together with its value estimate V. The data point (b, 2, 9, V) is sent a fixedccaacciy replay buffer. Next, a batch of data is sampled from the replay buffer, and used to update C₁p and Ge according to Eq. (7) and (8). The updated Ge is then used for next planning step. Training starts from randomly initialized generator and critic networks and an empty replay buffer. In the warm-up phase, the critic is first trained using data collected by LEADER with uniform attention. This provides meaningful objectives for the attention generator to start with. Then, both the critic and the generator trained with online data collected using the latest attention generator. At execution time, we only deploy the generator and the planner to perform risk-aware planning.
6 Experiments and Discussions
We evaluate LEADER on autonomous driving in unregulated urban crowds, show the improvements on the real-time driving performance, and analyze the learned attention. The experiment task is to control the acceleration of a robot ego-vehicle, so that follows a reference path and drives as fast as possible, while avoiding collision with the traffic crowd under the uncertainty of human intentions. A human intention indicates which path one intends to take. Candidate paths are extracted from the lane network of the map, according to the current location of the exo-agent. Attention is thus defined as joint importance distribution over the intentions of 20 exo-agents near the robot, assuming conditional independence between agents. See details on the POMDP model in Appendix C.
Our baselines include both risk aware planning and risk-aware RL methods. We first compare with POMDP planners using handcrafted attention. DESPOT- Uniform uses DESPOT with uniform attention over human intentions [30]. DESPOT-TTC computes the criticality of exo-agent's intentions using the estimated T'imeTTo-Collision (TTC) with the ego-vehicle. The attention score is set proportional to 1 T. where to is the TTC if the exo-agent takes the indented path with a constant speed. For RL baselines, we first include model-free RL method, Ensemble SAC (ESAC) [22], which learns an ensemble of Q-values for risk-aware training. Then, we include model-based RL method. SMBPO [23], which plans with learned dynamics and risk-aware reward model. Because of the difficulty of real-world data collection and testing for driving in a crowd, we instead used the SUMMIT simulator [30], which simulates high-fidelity massive urban traffic on real-world maps, using realistic traffic motion model [31]. We evaluate LEADER and all basclines on three different

---

LEAGER
Averuge Cumulaaive Reward
Colltzion 1000 Steps
DESPOT Un
DESPOT-TTC
Average Speed
Training lteration (162)
Training lteration (1e2)
Training lteration (162)
(a)
(b)
(c)
Figure 3: Learning curves of LEADER and existing risk-awart askee baselines: (a) average cumulative
reward (b) collision rate per 1000 steps (c) average speed.
Table 1: Generalized driving performance of LEADER and existing risk-aware planning and learning algorithms 400 test runs novel scenes, Errors represent standard deviations.
Algorithm
Cumulative Reward
Collision Rate
Travelled Distance
Smoothness Factor
DESPOT- Uniform
-2.44 + 0.11
0.014 + 0.003
102.12 = 5.2
3.31 + 0.03 3.15 + 0.12
DESPOT-TTC
-2.61 + 0.14
0.016 = 0.002
100.11 + 6.34
ESAC SMBPO
-5.8 0.43
0.05 0.002
19.38 5.94
1.72 0.23
-4.91 0.47
0.038 + 0.002
47.2 12.49
2.16 1.2 0.2
LEADER (ours)
-2.1 0.16
0.007 + 0.001
115.29 + 6.1
4.64 + 0.07
environments: the Meskel square in Addis Ababa, a highway in Singapore, and the magic roundabout Swindon, using different sets reference paths and crowd initialization for each map. Crowd interactions during driving are perturbed with random noise, thus are unique for each episode. See sample driving videos of LEADER here: http://tte..googll.oo//vveee paaeeppeeleee.ererppprprrrrrr pappr...
6.1 Performance Comparison
Learning Curves. Figure 3 shows the learning curves of LEADER and the baseline algorithms in terms of the average cumulative reward, the collision rate, and the average driving speed. LEADER starts to outperform the strongest RL baseline from the 8500th iteration which is 12 hours of training using 12 real-time planner actors. It starts to outperform the strongest planning bascline from the 12500th iteration corresponding to 19 hours of training. At the end of training. LEADER achieves the highest cumulative reward, the lowest collision rate, and the highest driving speed among algorithms.
Generalization. To evaluate the generalization of LEADER, we it with unseen reference paths and crowd initialization for each map with randomness on crowd interactions. Table shows detailed driving performance of the considered algorithms averaged over 400 test runs, equally apportioned among the three maps and map setups. Evaluation criteria include the cumulative reward, collision rate, travelled distance, and smoothness factor. The collision rate measures the average number of collisions per 1000 time steps. The smoothness factor is the reverse of the number of decelerations, Noc: As shown, LEADER outperforms other methods in all metrics, consistent with the learning curves. It drives much more safely and efficiently than RL methods which had a hard time handling the highly dynamic crowd. It also improves driving safety, efficiency and smoothness from the plamning baselines which applied sub-optimal attention.
6.2 The Learned Attention
To provide qualitative analysis on what LEADER has learned, we further provide 2D visualizations of the learned attention Figure 4. We provide three scenes from the Singapore Highway, Meskel Square, and Magic Roundabout, respectively. In each scene, we highlight representative exo-agent, show the of intention paths and visualize the learned attention over the paths using color coding. For other exo-agents, only show the most critical intentions with the highest attention scores.

---

Ego-agent
Ego-sgent
Velceity
Apcelaration
Reference Path
Exo-agent
Exc agonts
Highlihted exo-agent
codec Intendorn
(a)
(b)
(c)
Figure 4: Visualization of the learned attention in: (a) highway (b) Meskel square (c) Magic. We highlighted one exo-agent in blue in each scene. Learned attention over its intentions are color-coded: green, yellow, red, purple, sorted from low attention to high attention. For other exo-agents, we only show the most-attended intention with dotted lines. See more examples here: wtw.:/s.oooolle.oovieieeeeeee
In Figure (a), the highlighted exo-agent blue drives in parallel with the ego-vehicle. It has several possible intentions, including continuing straight and merging neighbouring lanes. The intention of merging right is more likely, as the agent is closer to the right lane. However, the merging left intention is more critical,
the path will interfere with the ego- vehicles's''s. LEADER learns to put more attention on the more critical possibility. As result, the planner decides to slow down the vehicle and prepare for potential hazards. In Figure 4 (b), the blue agent can either proceed left or turn right. Both intentions are equality likely. While the turning right intention gets the agent out of the ego-vehicle's 's way, proceeding left, however, the ego-vehicle. The generator thus assigned more attention to the latter. The planner subsequently decides stop the veh-vll, since the blue agent in front may need wait for its path to be cleared. In Figure 4 (c), all intention paths of the blue agent intersect with the ego-vehicle's ss The generator learned to attend the path closest to the ego-vehicle, which is more hazardous. Consequently, the planner stops the ego-vehicle to prevent collision.
the agent to continue interacting with
7 Summary, Limitations and Future Work
In this paper, we introduced LEADER, which integrates learning and planning to drive in crowded environments. LEADER forms minimax game between learning component that generates attention over potential human behaviors, and planning component that computes risk-aware policies conditioned on the learned attention. By solving the minimax game. LEADER learned to attend to the most adversarial behaviors and perform risk-aware planning. Our results show that LEADER helps improve the real-time performance of driving in crowded urban traffic, effectively lowering the collision rate while keeping the efficiency and smoothness of driving, compared to risk-aware planning and learning approaches.
Limitations of this work are related to assumptions, model emror and data required for training. This work makes few assumptions. First, we assume there is available map of the urban environment, providing lane-level information. However, we believe this requirement can be met for most major cities. Second, in our POMDP model, we focused on modeling the uncertainty of human intentions, and ignored perception uncertainty such as significant observation noises and occluded participants. These aspects will be addressed in future work through building more comprehensive models. Third, LEADER will also be affected by model errors, as the approach relies on the planner's 's value estimates to provide learning signals. But since our approach emphasizes the most adversarial future, i.e.. relies on conservative predictions, it is more robust to model errors than typical planning approaches. Lastly, although we have improved tremendously from the sample-efficincc of RL algorithms, LEADER still requires hours of online training. A promising solution is to "warm up" the critic and generator networks using offline real- world driving datasets, such as the Argoverse Dataset [32] and the Waymo Open Dataset [33], then perform further online training.

---

References
[1] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-133 1998.
[2] A. Somani, N. Ye, D. Hsu, and W. S. Lee. Despot: Online pomdp planning with regularization. Advances in neural information processing systems, 26, 2013.
[3] D. Silver and J. Veness. Monte-carlo planning in large pomdps. J. Lafferty, C. Williams,
J. Shawe-Taylor, R. Zemel, and Culotta, editors, Advances in Neurul Information Processing Systems, volume 23. Curran Associates, Inc., 2010. URI http:: /ptp:////noo/oooo/ae//ooeo ang.uurppngss..uur........ http://////./eeeeeeeeeaaeaaeeeea
Paper. pdf.
[4] R. Hussain and S. Zeadally. Autonomous cars: Research results, issues, and future challenges. IEEE Communications Surveys & Tutorials, 21(2):1275-131) 2018.
[5] Cai, Y. Luo, D. Hsu, and W. S. Lee. Hyp-despot: A hybrid parallel algorithm for online planning under uncertainty. The International Journal of Robotics Research, 40(2-3):558 2021.
[6] A. Goldhoorn, A. Garrell, R. Alquézar, and A. Sanfeliu. Continuous real time pomep to find- and-follow people by humanoid service robot. In 2014 IEEE-RAS International Conference on Humanoid Robots, pages 741-747. IEEE, 2014.
[7] K. Li, D. Hsu, and W. S. Lee. Act to see and see to act: Pomdp planning for objects search in clutter. In 2016 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 5701-5707. IEEE, 2016.
[8] Y. Xiao, S. Katt, A. ten Pas, S. Chen, and C. Amato. Online planning for target object search in clutter under partial observability. 2019 International Conference on Robotics and Automation (ICRA), pages 8241 18247. IEEE, 2019.
[9] Y. Luo, H. Bai, D. Hsu, and W. S. Lee. Importance sampling for online planning under uncertainty. The International Journal of Robotics Research, 382--1) 2019.
[10] M. O' O'Kelly, A. Sinha, H. Namkoong, R. Tedrake, and J. C. Duchi. Scalable end-to-end au- tonomous vehicle testing via rare-event simulation. Advances in Neural Information Processing Systems, 31, 2018
[11] M. Arief, Z. Huang, G. K. S. Kumar, Y. Bai, S. He, W. Ding. H. Lam, and D. Zhao. Deep probabilistic accelerated evaluation: robust certifiable rare-event simulation methodology for black-box safety-critich systems. In International Conference on Artificial Intelligence and Statistics, pages 603- PMLR, 2021.
[12] E. Schmerling and M. Pavone. Evaluating trajectory collision probability through adaptive importance sampling for safe motion plamning. Robotics: Science & Systems, 2017.
[13] A. Sarkar and K. Czamecki. A behavior driven approach for sampling rare event situations for autonomous vehicles. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6407-6414, 2019. doi:10. 1009/000040897720111 20199977..99...0 1967715
[14] M. H. Danesh, A. Koul, A. Fern, and Khorram. Re-understanding finite-state representations of recurrent policy networks. In International Conference Machine Learning, pages 2388- 2397. PMLR, 2021.
[15] A. R. Cassandra, L. P. Kaelbling, and M. L. Littman. Acting optimally in partially observable stochastic domains. In Aaai, volume 94, pages 1023-1028, 1994.
[16] T. Nyberg, C. Pek, L. Dal Col, C. Norén, and Tumova. Risk-aware motion planning for autonomous vehicles with safety specifications. In IEEE Intelligent Vehicle Symposium, 2021.
[17] B. Gilhuly, A. Sadeghi, P. Yedmellat, K. Rezaee, and S. L. Smith. Looking for trouble: Informative planning for safe trajectories with occlusions. IEFE International Conference on Robotics and Automation (ICRA), 2022.

---

[18] X. Huang, A. Jasour, M. Deyo, A. Hofmann, and B. C. Williams. Hybrid risk-aware conditional planning with applications in autonomous vehicles. In 2018 IEEE Conference on Decision and Control (CDC), pages 3608-3614, IEER, 2018
[19] S.-K. Kim, R. Thakker, and A.-A Agha- Mgha-Mohammmdi. Bi-directional value learning for risk- aware planning under uncertainty. IEEE Robotics and Automation Letters, 4(3):2493-2500, 2019.
[20] D. Kamran, C. F. Lopez, M. Lauer, and C. Stiller. Risk-aware high-level decisions for automated driving at occluded intersections with reinforcement learning. 2020 IEEE Intelligent Vehicles Symposium (IV), pages 1205-1212. IEEE, 2020.
[21] B. Mirchevska, C. Pek, M. Werling, M. Althoff, and J. Boedecker. High-level decision making for safe and reasonable autonomous lane changing using reinforcement learning. In 2018 21st International Conference Intelligent Transportation Systems (ITSC), pages 2156-2162. IEEE, 2018.
[22] B. Eysenbach, S. Gu, Ibarz, and S. Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. In International Conference Learning Representations, 2018. URL https: / //tp///evepeeeeee////mmamee net/ forum?id-S1vu-- bCW.
[23] G. Thomas, Y. Luo, and T. Ma. Safe reinforcement learning by imagining the near fu- ture. In M. Ranzato, A. Beygelzimer, Y. Dauphin. P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 13859-13869. Cur- Associates, Inc., 2021. https: //RLLttpp://prrccediigs 73227711166681122222222442222323 htwp://roc.nnnn.....ee.......... wwwrips..c/eeeeeeeeeeeee paarrips..c/pppprr///////ee pdf.
[24] F. Berkenkamp, M. Turchetta, A. Schoellig, and A. Krause. Safe model- reinforcement learning with stability guarantees. Advances in neural information processing systems, 30, 2017.
[25] M. H. Danesh and A. Fern. Out-of-distribution dynamics detection: Rl-relevant benchmarks and results. arXiv preprint arXiv:2107.04982 2021.
[26] Cai, Y. Luo, A. Saxena, D. Hsu, and W. S. Lee. LeTS-Drive Driving in a crowd by learning from tree search. Robotics: Science & Systems, 2019.
[27] P. Cai and D. Hsu. Closing the planning-learning loop with application to autonomous driving crowd. IEEE Transactions on Robotics (Accepted), 2022.
[28] Y. Lee, P. Cai, and D. Hsu. MAGIC: Learning macro-actions for online pomdp planning. Robotics: Science & Systems, 2021.
[29] D. P. Kingma and M. Welling, Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114 2013.
[30] Cai, Y. Lee, Y. Luo, and Hsu. SUMMIT: A simulator for urban driving in massive mixed traffic. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4023-4029. IEEE, 2020.
[31] Y. Luo, P. Cai, Y. Lee, and D. Hsu. Gamma: A general agent motion model for autonomous driving. IEEE Robotics and Automation Letters. (12):399935066 2022.
[32] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hart- nett, J. Kaesemodel Pontes, D. Ramanan, P. Carr, and J. Hays. Argoverse 2: Next generation datasets for self-dri driving perception and forecasting. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Bench- http://///ttttsttteeenaaaaaaaaa procaadddnss.ooooeeioooe.oei neurips.
marks, volume 1, 2021. cc/paper/2021/E1 /fshs//sseeeeeaaa.aaaaaaaaaaaaa. Paw.-eddd--ap-r---rraeeee /ttpa///////////////aaaaaaaaaaaa
[33]
Sun, Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 2020.
10

---

[34] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling, In NIPS 2014 Workshop on Deep Learning, December 2014, 2014.

---

Appendix
A Theoretical Background of Importance Sampling
Importance sampling is variance-reduction technique for Monte Carlo sampling, often used to obtain more reliable estimations from fewer samples. Given a random variable i p(x) and function (z). Suppose we want to estimate the expected value of f(r with sampling:
= -[S(a)] - / [(()p(p)de R 1 Σ (zx))
(10)
where { sampled from p(z), and Σ, f(x;) is the estimation using the samples. Given an importance distribution q(z), importance sampling performed as:
((x)) dx ((x)
1 E (2re)
p(x.) (x;)))
1 = [())p()))))))))))))())))))))[[)
(11)
where (r=)i=1,..... now sampled from the importance distribution ((x).. and p(x.) to as the importance weights, are used to correct the point- baint-based estimation. If we define 1ig
g(z.)* often referred
Σ. f(xx) p/2,) By is an unbiased estimator of It has also been shown that the variance of plg is,
Varljág) - ((())p())- uq(x))² ((x)
1
dx.
(12)
Therefore, the optimal importance distribution q² that offers the lowest variance is:
(x) 1f(x))p(z)
(13)
The above equation suggests increasing the sampling probability of those x with high absolute function values [=))]. This theoretical indication supports core idea of learning emphasize critical events that lead to hazards and thus large negative values.
B Neural Network Details
Figure demonstrates network architectures of the attention generator and the critic. The attention generator network first concatenates numbers in the current belief b and the observation into single vector and feeds it to a feature extractor. The feature extractor consists with ReLU activation. The extracted features are input to Gated Recurrent Unit (GRU) cell [34], which keeps track of history inputs in its latent memory. Based on the latent memory, the network two fully-connected layers and soft-max layer to output the importance distribution q. The critic network has similar architecture. The belief b, observation 2, and the generated importance distribution q first concatenated into single vector. Then, the vector is fed to a feature extractor containing 6 fully-connected layers with ReLU activation, then input to GRU cell for tracking memory. Based the latent memory, the critic network another 3 fully-connected layers to finally output a single decimal number, representing the estimated planner value v. See detailed representations of 2, and g the following section.
fully-connected layers
C
The POMDP Model for Urban Driving
Our POMDP model for driving in an ill-regulated dense urban traffic is defined as follows:
State Modeling: A world state encodes:
the state of the ego-vehicle, 8, = (Per Ve, Okey Pa), where Pc: ve and ar denote its position, velocity, and heading direction, and P. denotes its reference path.
12

---

m9
E
mT
mT
9.x
2.28
8:0
080
6.2
03
(a)
b
8002
080
2
202
F00
9
(b)
Figure 5: Network architectures of the attention generator and (b) the critic function.
observable states of 20 nearest exo-agents. Sewo = {ps, 2/2 a4 dr) where pss U.
a, the position, velocity, and heading direction of the ith exo-agent.
the intention of the ith exo-agent. Suppose an exo-agent has M potential paths to undertake according the lane network, the value of its intention will be taken from (O..... -
hidden states of 20 nearest exo-agents, Oeno - = { where
A belief b thus discrete probability distribution defined over the hidden states intentions exo-agents, assuming probabilistic independence between different participants. is represented using Σi= M, probability values, where M, the number of intentions for the ith exo-agent. An importance distribution q is specified in the same way.
Action Modeling: An action of the ego-vehicle is its acceleration discretized to three values, ACC, CUR, and DEC, meaning to accelerate, keep the current speed, and decelerate. The acceleration and deceleration are 3m/s² and -3m/s²² respectively.
Observation Modeling: An observation 2 from the environment includes all observable parts of the state s and excludes the hidden intentions. Namely, 2 (sc: Sego). Due to perceptual uncertainty, these observations often come with noise. However, in this work, we particularly focused on the uncertainty in human behaviors and ignored perceptual uncertainty, because the latter often has secondary influence on decision-making.
Our transition model assumes the ego-vehicle follows its reference path using a pure-pursuit steering controller and the input acceleration. Exo-agents are not controlled by the algorithm. We assume they take one of their hypothetical intended paths, using the GAMMA motion model [31] to interact with surrounding participants. When simulating exo-agent, the GAMMA model conditions prediction on set of factors, including the intended path, collision avoidance with neighbors, and kinematic constraints. At each time step, all agents are simulated forward by fixed duration of 1/3s. Afterward, small Gaussian noises are added to all transitions to model uncertain human control. So, the transition function of the ego-vehicle and exo-agents can be written
Transition Modeling:
(sclace acc a, steer = PurePursuit(se, Pc)), BicycleModel (s
(14)
where s. is the state of the ego ego-vehicle, Pa its planed reference path, and
GAMMA( SezO) +
(15)
where Sezo represents the states of ego-vehicles, and sampled Gaussian noise.
Reward Modeling:
The reward function takes into account driving safety, efficiency, and smoothness. When the ego- ago-vehicle collides with any exo-agent, it imposes a severe penalty
of Too -20 (v² + 0.5) depending the driving speed To encourage driving smoother, we also add small penalty rsm = -0.1 for the actions ACC and DEC to penalize excessive speed changes. Finally, to encourage the vehicle to drive at speed closer to its maximum speed Uman. we give it penalty of rar v-umaxx at every time step. The above rewards are additive. So. r(s,a) (s) ror(s) + (a)
(16)
13


Reviews:
[{"review_id": "u-1OiY8-eo3", "reviewer": "Reviewer_Djif", "summary": "This paper addresses the problem of learning amidst uncertainty in human behaviors.  Monte Carlo (MC) sampling techniques leveraged in addressing the attendant POMDP problem are often used for online planning due to their “anytime” properties.  To reason on risky / critical scenarios, techniques such as IS-DESPOT scheme (introduced by Luo, et. al. [9]), use importance sampling to bias MC sampling toward these risky scenarios, however, to achieve safe performance (e.g. low autonomous vehicle collision rates), the importance sampling distribution can be learned.  The technical contribution of this paper is to learn these importance distributions via a min-max game played between the importance distribution generator and the planner.  Comparisons are drawn between the proposed approach and existing approaches (DESPOT, other RL approaches) in various crowded driving scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper offers a solution to an important problem in planning, which amounts to a combination of importance sampling-based Monte-Carlo planning approach with a learned importance distribution (attention).  While the approach to learning importance distributions is not new (acknowledged by the authors and further highlighted in the references below), the paper is well-written, and offers some ideas on framing the problem, including setting up an actor-critic-like scheme for training attention and critic.  It also provides good insight into the construction of the critic and attention networks (via the supplement).\n\nAll the descriptions are fairly complete and the supplement provides sufficient detail for reproducibility.  The limitations section is also very thoughtful and complete.\n\nThe problem is this well-explored field; to my estimation, the biggest limitation of the paper is that the related work is limited and lack comparison to relevant approaches that solve essentially the same problem (i.e. using a variation of importance sampling).  Some relevant works are below:\n\n[A] Matthew O'Kelly, Aman Sinha, Hongseok Namkoong, Russ Tedrake, John C. Duchi: Scalable End-to-End Autonomous Vehicle Testing via Rare-event Simulation. NeurIPS 2018: 9849-9860\n\n[B] Mansur Arief, Zhiyuan Huang, Guru Koushik Senthil Kumar, Yuanlu Bai, Shengyi He, Wenhao Ding, Henry Lam, Ding Zhao: Deep Probabilistic Accelerated Evaluation: A Robust Certifiable Rare-Event Simulation Methodology for Black-Box Safety-Critical Systems. AISTATS 2021: 595-603\n\n[C] Edward Schmerling, Marco Pavone: Evaluating Trajectory Collision Probability through Adaptive Importance Sampling for Safe Motion Planning. Robotics: Science and Systems 2017\n\n[D] Atrisha Sarkar, Krzysztof Czarnecki: A behavior driven approach for sampling rare event situations for autonomous vehicles. IROS 2019: 6407-6414\n\nThis gives rise to several basic questions: How do these approaches compare conceptually with the proposed one?  What is novel about the current approach?  Could any of these approaches be added to the empirical comparison?\n\nAnother fairly significant weakness relates to the assumption of a finite set of intentions and observations.  Such a representation is quite limiting in practice, as it seems to depend heavily on the chosen concept class (choice of discrete maneuvers / behaviors).  Presumably the simulated agents follow the same discrete behaviors, and hence the approach may not generalize well as a result.  Could the approach work with continuous intents?  What modifications would be necessary?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "There is still some question I hope the authors can address in their revision surrounding the novelty of the importance sampling-based approach, as this has been extensively studied in the literature already.  Many such relevant works were missing in the paper.  Also, it is not clear whether the approach could apply to real-world driving as it only considers a finite set of discrete intents for agents.  Nonetheless, the idea of solving the problem via a critic-based approach and the techniques used are quite nice and would serve as a contribution to risk-awareness in planning.  I would therefore lean toward accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "There are two main issues to be addressed here.\n\n1. The authors are strongly encouraged to consider related approaches that were missed in their related work (outlined above), and discuss how their approach contrasts with these.  \n\n2. The assumption of pre-defined discrete intents is limiting in practice.  Is there a principled way of generating these intents?  What is the loss in generality by adopting a particular choice of discrete intents?  What are the limitations when applied to the continuum of actions and intents?"}}, {"review_id": "YEHW0orscw", "reviewer": "Reviewer_Fx8v", "summary": "To manage uncertainty and risk-aware planning in the autonomous driving domain, the authors propose LEADR - a hybrid learning/planning method that solves the POMDP problem with learned behavior sampling distributions. To address the problem of missing critical events sampling in Monte Carlo sampling approaches, LEADER uses a neural network to generate sampling distribution that aims at minimizing the planner's expected return whereas the planner tries to maximize it's return conditioned on the learned sampling distribution. Similar to GAN, a minimax loss is designed for the neural network to learn to  sample critical/unsafe behaviors of other road agents. Evaluation is done in simulated (unregulated) urban driving environments and results suggest improved safety, efficiency and smoothness over comparison methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The setup of the method reminds of generative adversarial imitation learning, but incorporation of a belief space planner instead of delegating everything to a neural network can improve sample efficiency (given good priors), safety and explainability. The paper is very well structured with key ideas clearly conveyed. The qualitative and quantitative results are convincing. The video is well-made and helpful in showcasing the capabilities of LEADER.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed method is practical, theory is sound and evaluation results are convincing.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- How does LEADER ensure that the network learns to sample behaviors that are critical but still reasonable (i.e. no abrupt changes in driving direction etc)\n\n- If the sampled behaviors are all the more critical possibilities, does that make LEADER an overly conservative planner?\n\n- Is there mode collapse during training?"}}, {"review_id": "QA0Z0QxbNlF", "reviewer": "Reviewer_KBsD", "summary": "This paper introduces a method for planning paths around other agents whose intentions are not directly observable. The contribution is in the use of an attention network to extract some information about the other agents, which is fed into a POMDP planner to select the ego vehicle's actions. Experiments on three simulated traffic scenarios suggest the proposed method outperforms methods based on Monte Carlo POMDP planners, model-based RL and model-free RL in terms of collision rate, distance traveled, etc.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem setting is very important, challenging, and relevant to the CoRL community\n- The method seems to outperform the chosen baselines on the simulated driving tasks\n\nWeaknesses:\n- It's not clear that the problem has been structured in a way that can directly impact modern frameworks for this problem. Modern systems use advanced trajectory prediction algorithms on other agents (this paper has a relatively simple 3 discrete intentions), plan a detailed vehicle trajectory to be tracked according to the vehicle dynamics (the ego vehicle's 3 actions in this paper are accelerate, decelerate, no-op), and involve collision checking (is it embedded in the reward function? I couldn't find it in the paper), for example.\n- The POMDP formulation is written out too generally, with the important details relevant to this problem left out: what are the models used for O, T, R for this specific problem and how do we know these are good models for multiagent collision avoidance?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The problem formulation is too simplified to demonstrate that the proposed technique is relevant to the real issues here.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "A new problem formulation would likely require a completely new algorithm, which seems beyond the scope of the rebuttal phase.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "B3QIVduKEqv", "reviewer": "Reviewer_4p6j", "summary": "This paper proposes an importance sampling based planning algorithm in which the importance sampling is learned using neural networks to achieve safe and efficient driving. As opposed to existing work where importance sampling is not time-efficient, the proposed algorithm LEADER learns the distribution for importance sampling. This distribution is trained such that it is biased towards generating samples that are adversarial (against the ego vehicle). A Monte-Carlo based planner then uses these adversarial samples to come up with conservative best driving actions. Simulation experiments show the proposed algorithm outperforms the baselines in terms of safety.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is very well written.\n- The proposed approach makes a lot of sense and is not completely based on \"neural network magic\" -- it is very intuitive that importance sampling that biases search towards risky scenarios would improve safety in autonomous vehicles and this paper is using deep learning techniques only to accelerate the importance sampling procedure.\n- Experiments (despite being simulated) show significant improvements in realistic scenarios.\n\nWeaknesses:\n- There is not really a major weakness, but the paper could get even better with some additional analysis on how to use importance sampling. See my detailed comments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper does a great job in various terms: (i) it is proposing a very intuitive, easy to understand algorithm, (ii) it accelerates the algorithm using machine learning techniques and the way it achieves this (the training procedure) is not trivial, (iii) it has excellent results even though the experiments are all in simulation, which is quite common in autonomous driving research due to high costs and safety issues.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- An additional analysis I suggest authors to consider (possibly as a future work) is about how to use importance sampling. So the attention generator network is able to output the distribution for importance sampling. If my understanding is correct, when the planner tree is constructed, it is only the initial state that is being sampled from this attention generator network. The rest of tree is again populated using the distribution $p(s_{t+1}, z_{t+1} \\mid s_t, a_{t+1})$, which is also implied by Equation 4. Now, one could also perform importance sampling for the time steps that are further in the future: for example, instead of just the initial state, one could sample the subsequent two time steps from this distribution. On the extreme, the tree would be constructed such that all transitions come from the importance sampling distribution. I guess this would cause the ego agent to try to plan as if all vehicles will attempt to crash with it, so it would be super conservative. By tuning how many time steps are sampled from the importance sampling distribution, the conservatism of the planner could be tuned. I wonder if this tuning could result in even better performance and/or higher safety.\n- I am a little confused about the training procedure for the attention generator. What prevents it from outputting a distribution whose samples are dynamically infeasible transitions, e.g., an exo-vehicle teleporting just in front of the ego vehicle? What is the component (of the network or the loss function) that makes sure the samples from $q$ are dynamically feasible?\n- The title of the paper is a little misleading, because there are really works that try to learn the drivers' attention to better predict their trajectories. See, for example, \"Social LSTM: Human Trajectory Prediction in Crowded Spaces\" by Alahi et al. (even though it is for pedestrians), or \"Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction\" by Cao et al. (which is for driving).\n- I found two typos: (i) In the caption of Figure 1, \"The learning components includes\" has a singular-plural mismatch, (ii) in Section 7, \"This work make a few assumptions\" has the same issue."}}, {"review_id": "6bi5ZBrwcp", "reviewer": "Reviewer_D38s", "summary": "The paper indroduces an algorithm that uses an adversarial set up to learn autonomous driving behaviours from Monte Carlo-based partially observable reinforcement learning. It uses importance weighting to deal with the issue of rare, but safety-critical, events being missed in sampling. The performance of the approach is evaluated in three simulated driving environments and compared against a number of competing algorithims.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- It deals with an important and challenging problem.\n- The adverserial setup appears to lead to robustness.\n- The mathematical development seems appropriate to the problem the authors are trying to address.\n- It presents results that are competitieve to the state of the art in simulation.\n- The discussion of limitations is good.\n\nWeaknesses:\n- The advance seems somewhat incremental. The results for LEADER in Table 1 are within a standard deviation of DESPOT-Uniform, so the improvement is probably not statistically significant.\n- The evaluations are somewhat superficial in that they do not characterise the impact of the design decisions made in developing the proposed approach, or its sensitivity to the parameters used.\n- It is doubtful that the results reported will scale to real world situations with a physical robot.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper is of good quality, and addresses an important problem. It presents a reasonable approach that seems appropriate to the problem in hand. I did not detect any issues with the mathematical development. The results presented show that the approach reaches performance comparable with the state of the art, and possibly slightly better (although the statistical significance of this is questionable). The scalability to real world driving is dubious.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {"issues": "- The curves in Figure 3 only show performance in one (carefully selected? representative?) test run. Please overlay these on a plot showing the average curve with error bars.\n- I couldn't understand what the video was meant to show - perhaps because it is missing its sound? Please add a commentary and/or subtitles to explain what is going on with it.\n- Please clarify how the test runs were apportioned between each map in Table 1.\n- Please specify what 'average' is reported in Table 1. (Is it mean performance?) Moreover, please specify how the errors are calculated. (Is it standard deviation?)\n"}}]

Author rebuttal:

Decision: Accept (oral)