Is Anyone There? Learning a Planner Contingent on Perceptual Uncertainty
Authors: Charles Packer
Nicholas Rhinehart
Rowan Thomas McAllister
Matthew A. Wright
Xin Wang
Jeff He
Sergey Levine
Joseph E. Gonzalez
Venue: CoRL 2022 Poster (2022)
Keywords: Forecasting
Planning
Partial Observability
Autonomous Driving

Abstract: Robots in complex multi-agent environments should reason about the intentions of observed and currently unobserved agents. In this paper, we present a new learning-based method for prediction and planning in complex multi-agent environments where the states of the other agents are partially-observed. Our approach, Active Visual Planning (AVP), uses high-dimensional observations to learn a flow-based generative model of multi-agent joint trajectories, including unobserved agents that may be revealed in the near future, depending on the robot's actions. Our predictive model is implemented using deep neural networks that map raw observations to future detection and pose trajectories and is learned entirely offline using a dataset of recorded observations (not ground-truth states). Once learned, our predictive model can be used for contingency planning over the potential existence, intentions, and positions of unobserved agents. We demonstrate the effectiveness of AVP on a set of autonomous driving environments inspired by real-world scenarios that require reasoning about the existence of other unobserved agents for safe and efficient driving. In these environments, AVP achieves optimal closed-loop performance, while methods that do not reason about potential unobserved agents exhibit either overconfident or underconfident behavior.

Is Anyone There? Learning a Planner Contingent on Perceptual Uncertainty
Charles Packer UC Berkeley
Nicholas Rhinehart UC Berkeley
Rowan McAllister Toyota Research Institute
Matthew A. Wright UC Berkeley
Xin Wang Microsoft Research
Jeff He UC Berkeley
Sergey Levine UC Berkeley
Joseph E. Gonzalez UC Berkeley
Abstract: Robots complex multi-agent environments should reason about the intentions of observed and currently unobserved agents. In this paper, we present a new learning-based method for prediction and planning in complex multi-agent Our
environments where the states of the other agents are partially- approach, Active Visual Planning (AVP), uses high-dimensional observations to learn a flow-based generative model of multi-agent joint trajectories, including unobserved agents that may be revealed in the near future, depending on the robot's s actions. Our predictive model is implemented using deep neural networks that map raw observations to future detection and pose trajectories and is learned entirely offline using dataset of recorded observations (not ground-truth states). Once learned, our predictive model can be used for contingency planning the potential existence, intentions, and positions of unobserved agents. We demonstrate the effectiveness of AVP on a set of autonomous driving environments inspired by real-world scenarios that require reasoning about the existence of other unobserved agents for safe and efficient driving. In these environments, AVP achieves expert- level closed-loop performance, while methods that do not reason about potential unobserved agents exhibit either overconfident or underconfident behavior.
Keywords: Forecasting, Planning, Partial Observability, Autonomous Driving
Introduction
When navigating in complex multi-agent scenarios, humans reason about the uncertainty of their own perception and how their actions affect their perception [1]. For example, a human driver at a blind intersection depicted in Fig. Ia may inch forward to improve their visibility of oncoming traffic before deciding whether or not to proceed. This type of explicit reasoning is natural human behavior critical to safe and efficient decision making. However, it does not exist in autonomous systems that assume full observability of the environment, nor in systems that assume their limited situational awareness is persistent and unactionable. While it is possible to engineer like behavior for many individual scenarios involving perceptual uncertainty, can we instead develop principled data-driven solution to learn end-to-end planners that incorporates perceptual uncertainty?
The partially observable Markov decision process (POMDP) model admits notion of perceptual uncertainty, where the state of the underlying MDP is hidden due imperfect perception (such as visual occlusions or sensor noise) and must be estimated by the agent in order to construct an optimal policy. POMDPs can be solved exactly for small discrete state-action spaces, but currently cannot be in large or continuous state-action spaces typical of AV planning [13], leading to variety of approximate solutions that require significant human design effort, e.g. state spaces, affordances, sgenario-speciffi heuristics, and/or dynamics models. Reinforcement learning (RI.) can be applied to POMDP problems without requiring human-defined states or dynamics, however, RL adds a significant burden of online data collection. We propose tractable data-driven method to accommodate partial observability, unknown dynamics, raw observations, and offline data.
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

Plaskigg
Lammed
Leamal
Row a wirepe
Ponul
Clodlhop em exstrtiin
evited
rodli
ovenllrr
chory
in cherr
Bajoy
Bi
Oflice
Navs
bvasikke e
Office
Offire
New
Worg al [M]
New
Acric
Way atll (17)
New
Acticy
(b)
Figure 1: (a) Navigating blind intersection requires prediction and planning under perception-inducee uncertainty: expert driver (red) should not only predict that a vehicle (hlue) may emerge from the occluded area, but also understand that inching forward into the intersection will reduce uncertainty regarding oncoming traffic by improving visibility, thus enabling the driver to safely execute a turn. Active Visual Planning (AVP) enables prediction and planning with respect to unobserved agents, including actively planning to reduce future uncertainty due to perception. (b) Comparison of recent planning approaches for autonomous vehicles, focused on data-driven contingency planning (upper section) and planning under perceptual uncertainty (lower section).
In this paper, we present Active Visual Planning (AVP), scalable method for contingency planning under multiple forms of partial observability (uncertainty in existence, positions, and intentions) that accommodates unknown dynamics, raw high-dimensional observations such as camera images or LiDAR scans, and learning from offline data. AVP models discrete uncertainty (uncertainty in existence) and continuous uncertainty (uncertainty in position) by reasoning about the existence of other entities and their future trajectories and planning accordingly. Importantly, AVP incentivizes an agent to take actions that actively reduce uncertainty over the existence of unobserved entities (e.g., moving to a location with a wider field of view) if it improves the quality of the overall plan (only small subset of all uncertainty induced by imperfect perception is relevant to near-term decision making in many real-world scenarios). We evaluate AVP in various multi-agent settings in the CARLA [14] simulation environment. Our main contributions are:
The first fully-learned planner that supports active contingency planning over perceptual uncertainty. A benchmark for evaluating the robustness of planners to real-world examples of perceptual uncertainty in a visually realistic simulator (CARLA). We illustrate that our method outperforms methods that do not properly account for potential occluded vehicles. Supplementary material can be found at https://sites, http://sites.goggleeee www.le.
2 Related Work
POMDPs and belief space planning. A long history of research in POMDP solvers and belief space planning for autonomous driving exists [15, 16, 17]. and recent work has demonstrated success in modern autonomous vehicle (AV) simulators and real-world systems [18, 19, 20, 21, 22, 23]. Simple POMDPs can be solved using brute force enumeration [24), incremental pruning [25], and linear support [26], but in large or continuous state-action spaces typical of AV planning, exact POMDP solutions are generally intractable [13], leading to approximations such as Point Based Value Iteration [27, 28] and Heuristic Search Value Iteration [29], or making simplifying assumptions for tractability such assuming belief distributions are unimodal, as iterative-linear (c.g. iLQG controllers) or matchhng methods [30] do. Sampling based Monte Carlo tree search methods [31, 32, 33] improve on baset solvers and enable scaling to larger POMDPs with continuous state and action spaces. However, approximate methods that have been shown to work on real AV systems require human-defined state spaces, affordances, and scenario-specific heuristics [12] (e.g. augmenting states with affordances such as time since last stop, distance to stop sign, etc). In contrast, our modeling framework (AVP) is data-driven and automatically learns both dynamics model and contextual features from raw sensor observations, and we demonstrate that the same model architecture and planner generalize to several different scenarios without modification. We illustrate these differences in Fig. 1b.
Reinforcement and imitation learning. Although the MDP framework of reinforcement learning (RL) assumes the agent receives state observations, POMDPs can be reduced to MDPs by recording

---

Fan Inseusnt abvrcsikon
Grsdinn-aadd pally plaazer
dwedeer
Powficted futrre inajetario
max L
Genill croskt
Seaset outoat (LIDAR)
Olerrvl past Inglstueties
Trjestrry anadar
Figure 2: AVP uses the perceptual context and robot's past positions to predict both when agents may be detected and where the detected agents and the robot may go. By modeling both the motion of other vehicles and potentially detecting them as dependent on the robot's future position, partial-observabiiity when planning. The perceptual context ig (e.g.. LiDAR) is encoded by CNN and the past positions, &o. are encoded by GRU. Both encodings are input to a GRU decoder which generates predicted distribution of future states 1.T for each agent and detections D for each non-ego agent.
model enables the robot to factor
the action-obserration history (or a sufficient statistic of the history) [34], and in practice deep RL has been shown work well in certain POMDP environments using truncated histories and recurrent neural networks [35]. Both model-free [36, 37, 38, 39] and model-based deep RL [40] have been successfully applied to AV tasks to plan directly from images. While RL requires learning online, our approach is trained fully offline from logged data. Recent work in offline RL [41] aims to use RL on previously collected data, but both offline and online RL require access to a reward function during training. whereas AVP does not. AVP is closely related to imitation learning (IL), which also learns a control policy offine [42, 43]. However, unlike II., AVP only requires expert observations instead of expert demonstrations (i.e. AVP does not require action labels) and is flexible planning new goals test-time, as opposed to IL which directly learns a control-level policy.
Occlusion-aware prediction and planning. In settings with visual occlusions, map knowledge and geometry be employed for occlusion- 10]. For example, forward reachable set (FRS) methods avoid the entire set of future reachable states of hypothetical vehicles placed on the edges of occluded regions. However, such approaches often result in overly conservative plans, not only by assuming the worst case in existence and position, but also by assuming the worst case in intention (i.e., that entities are controlled by adversarial policies). In contrast, AVP learns a probabilistic model of existence, positions, and intention, enabling our planner to weigh risk and reward instead of relying on worst-case analysis. AVP also does not utilize any explicit map knowledge or geometry (such as estimated field of view), and instead learns to use raw sensor observations. Game theory has been applied to occlusion-aware planning improve on worst-case analysis by modeling interactions between vehicles [11]. Although game-theoretic planners are capable of generating expert-level driving policies, in practice, they are difficult to apply to scenarios with more than two vehicles, whereas AVP naturally scales beyond two agents.
prediction [44, 45, 46) and planning [47, 48, 7, 8,
Active Visual Planning
Preliminaries: We consider multi-agent settings comprising an unknown number of agents, which we model as a fixed-size large set of agents, with some potentially not present. We will subsequently refer the agent that is guaranteed to exist as the robot ego, for which we are designing controller, and A other agents as human actors (no direct control). We consider continuous positions in d-dimensional Cartesian space and discrete time, and denote the position of agent at time as x E Rd, and omit the superscript to denote the joint positions of all agents as E F Axd Let t = 0 denote the current timestep, Xo denote the current joint positions, and Xg, denote future joint positions from to t. We say non-ego actors x!:A are observed and only if the actor is detected by the ego. We denote an actor a has been detected by ego at time with binary flag de. and the one-hot vector of all detections as d2 4 di 1 if actor a has been observed in the ego agent's perceptual context i (e.g., LiDAR sensors or camera images). X, denotes an observed joint position from the ego't perspective, where for each agent 0, the true position is revealed (X) x() if dž 1, and is
3

---

masked otherwise. The robot's current observation is perceptual context and subset of the past joint trajectories, denoted o = (R fo). Uppercase denotes random variables, lowercase denotes realized values: Xg and Der represent a stochastic sequence of future positions and future detections.
A fully-observed model (prior work): We base our models on prior work that introduced an autoregressive probabilistic model for multi-agent prediction [49] and planning [6] with several desirable properties: the predictive model can be fully learned from observations using maximum likelihood estimation (no need to specify an approximate dynamics model), it supports multi-modal joint predictions over varying numbers of agents, models co-dependencies between different agents, and enables straightforward yet expressive planning through gradient-based optimization of the model. Importantly, the plans generated through the model are contingent, i.e., the plans explicitly account for (are contingent on) different future outcomes of modelled stochastic events. However, these modelled events did not include the potential appearance of new agents, which precludes explicitly reasoning about such events. In many real-world driving scenarios, full observability is an unrealistic assumption: occlusions frequently block the view of other relevant agents, and the imperfect sensing and perception systems used in modern self-driving cars can often lead low confidence erroneous object detection and tracking. Our model enables prediction and planning with respect to currently-unobberved agents.
The starting point for our behavioral model is a parameteric probabilistic model qe (Xlo) of fu- ture joint trajectories Xgt, conditioned on (recall is perceptual context in and past trajectories xgo). The distribution factorized across both time (1...T) and agents (0...A) into product of autoregressive normal distributions o of present positions / for each agent a, and g is con- ditioned on paramaterized transformation of previous predictions and the context, denoted as 1:A i): <t
s:-1 = fx A:A <t
T A
obs.
0)
II t=1 o=0
(I)
4 = N ua (s:- (((
(2)
where X can be sampled by drawing Z ~ (O, I): X; (s ao (s Zí. This particular
parameterization allows us to express variation in position as probabilistic noise component drawn from a simple base distribution Z; for the ego vehicle, we assume this noise is controllable (and thus fix z while planning), whereas for non-ego actors we sample z I:A ZI:A.
Modeling partial observability: To adapt the model in Eq. 1 to the partially observed setting where many entries of x are often unknown, we model joint trajectories x of detected agents indices pamial-ohs. and model the joint probability of future agent detection as Ug:
D= {tt, : di = 1} as gg
T A
partial-oss, La
1:4 1))
II (t,a)D
(X; (3)
Go D
o) = II 1f (D; ==
(4)
where of is a Bernoulli distribution predicted by neural-network that represents the probability that agent a is detected at time t. Note is observed during training (even when x not). Since we no longer assume access to x, 8t-1 1 is instead transformation of x. Because of is conditioned on the perceptual context, the model can relate visual cues (such as visibly occluded areas of LiDAR point cloud or camera image) with the probability of future detection.
Benefits of modeling detection instead of existence: Eq. (4) models currently undetected agents appearing into view, and also currently detected agents disappearing from view. For example, this behavior is seen in the overtake and intersection environments in Fig. 3: if the ego agent successfully navigates past the other agent, the non-ego agent will eventually be predicted to "disappear" (become undetected) as it drives out of the ego vehicle's 's LiDAR range. By modeling detection instead of existence, we can train partially-observe trajectory model without requiring privileged access to ground truth position information for all occluded agents; a learned model of existence similar to the model of detection in Eq. (4) (via a Bernoulli variable) would require explicitly modeling the positions of agents while undetected. AVP implicitly considers the dynamics between detected and undetected agents and retains aleatoric uncertainty in each distribution as a function of the observed

---

Algorithm 1: Planning with AVP
Require:
Trained predictive model M (consisting of context encoder and antoregressive state encoders and devoder), purtially observed past joint trajectories & go present perceptual context fo. and planning criterion C.
Randomly initialize sequence of robot z's (each z, denoted as x;)
Compute initial encuding No using leurned trajectory and context encuders (&co, To
for each optimization step do
for each sample for each i in horizon T do
Pass the state encoding 8t to the learned trijectory decader to get Dirr, X and RNN hidden state by Sample x ~ [1(0, I) lor every non-controllable (ie.. human) actor to compute x! (use lixed x. compute x() Sample d
D and combine with x to compute the observed positions x
Append d and x to encade s,.
10 Compute the gradient of the joint irajectory loss V Vi(rier gr) using sampled joint irajectories
11
Use V E to update robot's x²
information (perceptual context and positions of detected agents); see the Fig. 3 in the supplement for visual explanation.
To use our learned model for planning, we plan a sequence of z's for the controllable robot agent, while sampling z's for all other human-controlled agents. By planning in z-space, we are effectively planning policies, or contingent plans: fixing z" creates deterministic robot policy from a partial policy x" - dh which is specified by the learned parameters 9 of the model. Because z remains sampled, the overall system remains stochastic; planning over x erroneously implies that all agents are explicitly controllable. Following [6]. planning is derived as maximum a posteriori estimation of the free policy parameters z== conditioned on the initial positions and perceptual context, as well set of high-level goals g (g. G) consisting of target coordinate locations g and constraints G the joint trajectory. In experiments apply minimum distance constraint (8 meters to the nearest vehicle), which corresponds to near-collision penalty. Estimation of posterior p p(z'gT G, o) is relatively lightweight gradient-based optimization procedure, since we optimize a small number of free parameters z while keeping all other parameters of the learned model fixed. For our cost function G, we adopt the same lower bound planning objective as [6], which balances (1) a behavioral prior, (2) a scaled distance to the destination, and (3) a set of indicator functions ōg (x) the optional constraints:
Planning with learned model:
log Lo partial-oba, (XsTlo) +og N (xF:8 B, 1) Hlog (x(()T) ]
C( ZET
E 20.1
(5)
(1) prior
(2) destination
constraint
Optimizing d according to Eq. thus produces a policy that (with high expectation) reaches a destination specified at test-time, while mimicking human-like driving behavior seen in the expert observations. The optimization of Eq. 5 accounts for the detection model because it is function of x (note that decomposes fx, d, i)), which itself is function of the agent detection model (see Eq. 3). See Algorithm for pseudocode describing the planning procedure.
Model architecture and training: We implement autoregressive models ta and Up using encoder- decoder architecture (see Fig. 2) by building on the open-source implementation of [6]. So is implemented as two separate networks: a convolutional neural network (CNN) that encodes the perceptual context, and a recurrent neural network (RNN) trajectory encoder that reads a past trajectory and generates an encoding. qu is implemented as an RNN decoder that takes the encoding and generates a sequence of μ and representing future positions. Similarly, Vo is implemented as an RNN that generates a sequence of Bernouli parameters p. The entire network is jointly trained on the dataset of observations (consisting of observed joint trajectories and perceptual context) using maximum likelihood estimation according to Eqs. (3) and (4). During inference, if the vp does not predict detection hd ~ Ug = 0), then the predicted position (sampled from qe) is ignored. AVP can handle scenarios with varying numbers of agents as long as the model was trained on a dataset that includes the same number of agents, as shown by the threc-agent intersection experiment. Beyond this practical example of AVP's efficacy, theoretically, an agent trained on dataset containing up to N agents will be able to handle up to N agents at test-time, and the variability in number of agents can be handled through the modeling of detection (the majority of N agents will then be predicted as undetected). In our experiments, we use CNN with three convolutional layers (each with 32 filters
5

---

(a) Overtake
(b) Blind summit
(c) Intersection
(d) 3-agent inter.
Figure 3: An overhead visual of each scenario implemented in CARLA, An example joint trajectory is plotted illustrate the expert human policy in the presence of an occluded actor. Trajectories for cach car are visualized using circles and lines (ego in blue, actor in orange, and other moving entities in grey), with the circles evenly spaced in time. The overtake and blind summit scenarios have portions of the map cropped to fit the page.
and an 8 8 kernel) as our context encoders, and GRUs with three layers (256 hidden units each) as our recurrent encoders and decoders. The initial observation (model input) during model training contains 60x360 LiDAR range map and partially observed trajectory with 15 waypoints, sampled at 20Hz. The future trajectory (target sequence) consists of 30 waypoints, sampled at 7.5Hz. Further details can be found in the supplemental material.
Experiments
Scenarios requiring decision-making under perceptual uncertainty are ubiquitous in the domain of autonomous vehicles: occlusions obscuring important objects and agents unavoidable even with state-of-the-art LiDAR, RADAR, and masge-based sensors, additionally, sensor performance can significantly degrade under certain conditions such as inclement weather. To test the capability of our method, we focus on several autonomous driving scenarios where correctly reasoning about the existence of occluded agents leads to clear difference in planner behavior when evaluated in a closed-loop control setting. We implement these scenarios, visualized in Fig. 3, in the CARLA simulator [14]. Our key hypothesis is that in these scenarios, the ability of the AVP predict and plan with respect to unobserved vehicles will result in both safe and efficient control compared to existing approaches that do not properly account for unobserved vehicles.
4.1 Scenarios
Two-lane overtake (a): The ego vehicle attempts to overtake a truck on two-lane freeway and must avoid potential oncoming traffic. An agent that is correctly planning with respect to hypothetical oncoming traffic will make the lane change quickly to avoid getting stuck parallel to the truck with no escape maneuver. An overconfident agent that is oblivious to the possibility of occluded oncoming traffic will attempt the overtake maneuver at a lower speed, opening up the possibility of an unavoidable collision. An underconfident agent will never initiate an overtake.
Blind summit (b): The ego vehicle approaches a steep incline occluding the road past the apex of the hill. Because another vehicle may exist the top of the hill, an expert agent will slow down just enough to avoid an unavoidable collision with hypothetical occluded vehicle. An overconfident agent oblivious to the possibility of hidden vehicle may crash depending on their speed, while an underconfident agent will be overly conservative and proceed with speed far slower than the minimum necessary to avoid a hypothetical collision.
Intersection (c): The ego vehicle approaches an intersection with connecting left turn lane, however the lane is occluded by traffic. An expert agent will slowly proceed through the intersection to avoid a potential collision with an occluded vehicle turning left. An overconfident agent proceeding will cross the intersection faster, but will unable to avoid a potential collision with an occluded vehicle attempting to make turn if one exists. An underconfident agent will become frozen prior to entering the intersection due to the inability
oncoming traffic in the connecting lane.
Three-agent intersection (d): In this variation of the intersection, occluded agents may exist to the right and left of the ego vehicle. The behavior of an expert agent similar that in the two-agent variant: the ego must proceed through the intersection slowly enough to emergency brake and avoid collisions with other turning vehicles that are initially occluded by traffic.

---

4.2 Training and evaluation procedure
To collect observations to train our learned model, we use hand-coded deterministic policies for all agents, which are designed to generate all plausible modes of realistic driver behavior in the scenario (all modes are balanced equally). At the evaluation stage, we use the same hand-coded deterministic policies for the actors, but use planned trajectory for the ego vehicle with a fixed replanning rate for a fixed horizon T. Planned waypoints are turned into explicit low-level actions in the CARLA simulator (steering, throttle and brake) using a P-controller (see the supplement for pseudocode describing the control procedure). We evaluate each method a total of 30 times each scenario with balanced split of modes, and report the fraction of episodes where the planner was able to successfully reach the goal (RG) in Tab. 1. Additionally, we report the fraction of episodes where the planner reached the goal in optimal fashion (RG*): for example, an overly conservative planner may successfully clear an intersection (RG), but take much longer in doing so than an experienced human driver (RG*). Joint trajectories x that satisfy RG* are a strict subset of those that satisfy RG, i.e., RG* + RG. During training and evaluation, we randomize the starting positions of all agents in the scene and randomly drop fraction of each LiDAR point cloud observation. The CARLA simulator is an additional source of non-determinism (e.g., slight changes in lighting and foliage).
Single-agent planning baseline: Though it is possible to plan in multi-agent scenarios using single-agent behavior model, it will often lead to dangerously overconfident behavior, e.g., entering an intersection while disregarding oncoming traffic. We implement single-agent (learned) planner baseline using version of our model trained only on ego data (effectively the same model as in [50]).
Multi-agent fully-observed planning baseline: In partially observed settings, a multi-agent planner that assumes full observability can lead to dangerous overconfident behavior. For example, the ego vehicle may begin to enter an intersection while oblivious to occluded oncoming traffic; once the ego vehicle begins tracking previously occluded cars, it may be too late to reverse course and avoid a collision. To implement our multi-agent fully-observed baseline, we train our behavioral model on ground-truth fully observed joint trajectories, and only begin predicting a joint trajectory once non-ego agent has been detected. Similar planners be created by combining any number of modern trajectory prediction methods (that are flexible to varying numbers of agents) with model- based planning algorithms such as MPC. Our version of the baseline enables planning contingencies over stochastic behavior, however it does not enable planning with respect to undetected agents.
We compare our approach to recent game- theoretic planner for partially-observed AV settings proposed by Zhang and Fisac [11] (SOAP), which plans with respect to occluded vehicles by formulating the driving scenario as a two-player zero-sum game pursuit: the ego vehicle computes an area consisting of all points from which another agent could 'capture' or collide with the ego (the 'danger zone"). The ego vehicle also computes a set of points that vehicle located just beyond the its field of view could reach in the next / timesteps (the 'forward hidden set'). Having computed both these two sets, the ego vehicle can then safely plan by ensuring the danger zone and forward hidden set never intersect. SOAP generates an expert-level policy similar to AVP, however it requires significant setup per scenario, whereas our method can be easily transferred to new scenarios (assuming expert observations exist).
Multi-agent partially-observed planning baseline:
4.3 Results and discussion
In our experiments, AVP clearly outperforms the single-agent and fully oblyy-obeerved multi-agent base- lines, both of which exhibit overconfident behavior leading collisions with occluded vehicles. Since our learned model and planner are robust to noise, the planner either clearly succeeds clearly fails depending on the scenario configuration: in Tab. 1, we can see that the single-agent and multi-agent baselines will consistently fail when another agent is present (due to each scenario's 's construction). On the other hand, both multi-agent partially-observed planners (AVP and SOAP [11J) generate expert-level plans that are contingent on the existence of undetected agents.
Although both AVP and SOAP are capable of generating expert-level plans in two-agent partially- observed settings, AVP is more general solution. While the game-theoretic baseline has no learned
7

---

Ninliceves
Ieernsteecleee agren
Necled
80
BC*1
Dinc
MG1
Tex
8G1
MG*1 1
C.
Tinec
80"
C4
Tiac
Siegle oddiinal (PS)
14:20
16/30
18/3)
1555
18330
13/30
17/30
19.6
19:20
16/36
14230
13.5s
1730
1730
00/30
135-
Nall-gemi (iderrr RR]
11/91
11/31
13/91
13:91
18238
12:40
12.0
7730
(28)
28/30
135-
oloipeel, (1g)
2790
Arpre Vinal Phaning (00))
30/90
30:36
8/90
26/b
54/30
30.30
0/30
245
90/00
20/30
2790
2106
56/30
30/30
1/30
253-
Table 1: Comparison of our Active Visual Planning (AVP) approach to baselines in a closed-loop control setting. Goal-reaching success rate 'RG' indicates the agent reached the designated goal (higher is better). Near-expert goal-reaching success rate 'RG*' indicates the human-level performance, i.e., the agent reached the designated goal as quickly as the human expert (higher is better). 'C' indicates the number of collisions incurred (lower better). 'Time' indicates the average time before the goal was reached, not counting episodes that ended in a collision (lower is better). i indicates results reported in [11] and via correspondence with the first author.
components (and thus does not require an initial training stage), it requires significant setup for each individual scenario: since the planner requires discrete state space, the reachable arcas of the map must first be discretized into lanelet map. Additionally, a road centerline for each agent must be specified a priori, since the state of each agent is represented as vertical and lateral positioning relative to a reference trajectory. Computing the forward hidden set during planning requires a precise measure of the ego agent's FOV, which is done by placing (observed) dynamic obstacles in a 3D occupancy grid [51], which must be also be generated a priori. While AVP requires training behavioral model, assuming a dataset of observations exists or can easily be collected, applying AVP to new scenarios is straightforward. AVP predicts and plans continuous positions in Cartesian space, and does not require reference trajectories at test-time based on provided map knowledge. AVP also does not require geometric representation of FOV to estimate occlusions, and instead learns to recognize occlusions directly from perceptual context, e.g., LiDAR or camera images.
AVP also scales to scenarios with additional agents: the prediction and planning procedures remain the same as the two-agent variant, and the compute required single forward pass through the neural network scales linearly with the number of agents. In contrast, SOAP exhibits overconfident behavior on the three agent scenario, and thus does not plan safe trajectories (as seen in Tab. 1). Because the SOAP planner uses a game-theoretic solver that is limited to two agents, scaling the existing SOAP planner to three or more agents can only be done by solving multiple two-player games independently between the ego and non-ego agents. This independence assumption leads to overconfident driving policies: for example, the ego may exist in a state with a closed-loop escape policy to avoid car 1, but which results in collision with car 2 (and vice versa).
Limitations: AVP models detection instead of directly modeling existence, which has the key advantage of enabling training on partially-observel data (further discussed in Sec. 3). However, disadvantage of this approach is reduced interpretability: AVP cannot be probed to visualize the predicted current positions of occluded agents, instead, the model can only express where occluded agents may appear (become disoccluded) in the future. We anticipate future work will further explore this trade-off between interpretability and tractability for planning under perceptual uncertainty. Because AVP uses a learned model, it requires a method of collecting training data in the form of partially-labelled agent coordinates (for agents that are detected by the sensor system), and synchronous perceptual context (such as LiDAR RGB images). Additionally, the learned trajectory model is object-centric, which (in its current form) does exclude modeling certain phenomena such as objects splitting (e.g., pedestrian getting out of a vehicle is modeled as a pedestrian appearing out of nowhere). Both of these limitations are shared by most learned trajectory- mrodelioy-mmdellngg techniques.
Conclusion
In this paper we present Active Visual Planning (AVP), a scalable method for contingency planning under multiple forms of partial observability (uncertainty in existence, positions, and intentions) that accommodates unknown dynamics, raw observations, and learning from offline data. Because AVP is learned purely from observations, it does not require expert action labels or domain knowledge (such as state-space discretization or specification of an approximate dynamics model), making it easily generalizable new environments and scenarios. We demonstrate the effectiveness of AVP on set of autonomous driving environments inspired by real-world scenarios that require reasoning about the existence of other agents, in which AVP achieves expert-level performance.
8

---

References
M. T. Turvey, C. Carello, and N.-G. Kim. Links between active perception and the control of action. In Synergetics of cognition, pages 269-295. Springer, 1990.
[2] J. F. Fisac, E. Bronstein, E. Stefansson, D. Sadigh, S. S. Sastry, and A. D. Dragan. Hicrarchical game- thcoretic planning for autonomous vehicles. In 2019 International Conference on Robotics and Automation (ICRA), pages 9590 9596. IEEE, 2019. 2
[3] A. Bajesy, A. Siththaranjan, C. J. Tomlin, and A. D. Dragan. Analyzing human models that adapt online. In International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[4] B. Ivanovic, A. Elhafsi, G. Rosman, A. Gaidon, and M. Pavone. Mats: An interpretable trajectory forccasting representation for planning and control. arXiv preprint arXiv:2009, 00500000077777..... 2020. 2
[5] A. Cui, S. Casas, A. Sadat, R. Liao, and R. Urtasun. Lookout: Diverse multi-future prediction and planning for self-driving, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16107-16116, 2021.
[6] N. Rhinchart, J. Hc, C. Packer, M. A. Wright, R. McAllister, Gonzalez, and Levine, Contingencies from observations: Tractable contingency planning with learned behavior models. In IEEE ICRA, 2021. 2, 4, S,
[7] Ö. S. Taş and C. Stiller. Limited visibility and uncertainty aware motion planning for automated driving. In 2018 IEEE Intelligent Vehicles Symposium (IV). pages 1171 1178. IEEE, 2018. 2,
[8] S. G. McGill, G. Rosman, T. Ort, A. Pierson, Gilitschenski, Araki, I.. Fletcher, S. Karaman, D. Rus, and J. J. Leonard. Probabilistic risk metrics for navigating occluded intersections. IEEE Robotics and Automation Letters, 4(4):4322-423) 2019. 2.3
[9] L. Wang. C. F. Lopez, and C. Stiller. Generating efficient behaviour with predictive visibility risk for scenarios with occlusions. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1-7, IEEF, 2020. 2,
[10] H. Andersen, W. Schwarting, F. Naser. Y. H. Eng. M. H. Ang. D. Rus, and J. Alonso Trajectory optimization for autonomous overtaking with visibility maximization. In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (/TSC), pages 1-8. IEEE, 2017. 2,
[11] Z. Zhang and F. Fisac. Safe occlusion-aware autonomous driving via game-theoretic active perception. In RSS, 2021. 2, 3, 7, 8
[12] K. H. Wray, B. Lange, A. Jamgochian, S. J. Witwicki, A. Kobashi, S. Hagaribommanhhali, and D. Ilstrup. POMDPs for safe visibility reasoning in autonomous vehicles. In 2021 IEEE International Conference on Intelligence and Safety for Robotics (ISR), pages 191-195. IARF, 2021.
[13] C. H. Papadimitriou and N. Tsitsiklis. The complexity of Markov decision processes. Mathematics of operations research, 12(3)441-450, 1987. 1, 2
[14] A. Dosovitskiy, G. Ros, E. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator.
In CoRI. pages 1-16. PMLR, 2017. 2, 6
[15] N. Oza. Probabilistic models of driver behavior. In Proc. of Spatial Cognition Conference. Citeseer, 1999.
[16] Forbes, Huang, K. Kanazawa, and S. Russell. The Batmobile: Towards a Bayesian automated taxi. In IJCAI, volume 95, pages 1878-1885, 1995. 2
[17] T. Huang. D. Koller, J. Malik, G. Ogasawara. B. S. Rao, S. Russell, and J. Weber. Automatic symbolic traffic scene analysis using belief networks. In AAAI, volume 94, pages 966-972, 1994. 2
[18] H. Bai, S. Cai, N. Ye, D. Hsu, and W. S. Lee. Intention- aware online POMDP planning for autonomous driving in a crowd. In 2015 ieee international conference on robotics and automation (icra), pages 454-460. IEEE, 2015. 2
[19] W. Liu, S.-W. Kim, Pendleton, and M. H. Ang. Situation- aware decision making for autonomous driving on urban road using online POMDP. In 2015 IEEE Intelligent Vehicles Symposium (IV), pages 1 1126-1 1163. IEEE, 2015. 2

---

[20] S. Brechtel. T. Gindele, and R. Dillmann. Probabilistic deccision-makingg under uncertainty for autonomous driving using continuous POMDPs. In 17th international IEEE conference on intelligent transportation systems (ITSC), pages 392-399. IEEE, 2014.
[21] X. Lin, J. Zhang. Shang. Y. Wang. H. Yu, and X. Zhang. Decision making through occluded intersections autonomous driving. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 1449-2455 JEEE, 2019.
[22] M. Bouton, A. Nakhaei, K. Fujimura, and M. J. Kochenderfer. Scalable decision making with sensor occlusions for autonomous driving. In IEEE ICRA, pages 2076-200 2081. IEEE, 2018. 2
[23] C. Hubmann, N. Quetschlich, J. Schulz, Bernhard, Althoff, and C. Stiller. A POMDP maneuver planner for occlusions in urban scenarios. In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 2172-2179. IEEE, 2019. 2
[24] R. D. Smallwood and E. J. Sondik. The optimal control of partially observable Markov processes over a finite horizon. Operations research, 21(5):1071-1083 1973. 2
[25] N. L. Zhang and W. Liu. Planning in stochastic domains: Problem characteristics and approximation. Technical report, Technical Report HKUST-CS96-3,, Department of Computer Science, Hong Kong 1996. 2
[26] H.-T. Cheng. Algorithms for partially observable Markov decision processex. PhD thesis. University of British Columbia, 1988. URL http:///ww...aaaa.......aaa/laaa items/ 0098855.. 2
[27] J. Pineau, G. Gordon, S. Thrun, et al. Point-based value iteration: An anytime algorithm for POMDPs. In IJCAI, volume 3. pages 1032. Citeseer, 2003.
[28] M. T. Spaan and N. Vlassis. Perseus: Randomized point-based value iteration for POMDPs. Journal of artificial intelligence research, 24:1 2005.
[29] T. Smith and R. Simmons. Heuristic search value iteration for POMDPs. arXiv preprint 177.v:1207.4166. 2012. 2
[30] R. McAllister and C. E. Rasmussen. Data-efficient reinforcement learning in continuous state-action Gaussian-
Neural Information Processing Systems (NeurlPS), pages 2040-2049. 2017.
[31] D. Silver and Veness. Montc-carlo planning in large pomdps. Advances neural information processing systems, 23, 2010. 2
[32] A. Somani, N. Ye, D. Hsu, and W. S. Lee. Despot: Online pomdp planning with regularization. Advances in neural information processing systems, 26, 2013.
[33] Z. N. Sunberg and M. J. Kochenderfer. Online algorithms for pomdps with continuous state, action, and observation spaces, In Tiihty-Eghhth International Conference on Automated Planning and Scheduling. 2018. 2
[34] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-133 1998. 3
[35] Hausknecht and Stone. Deep recurrent q-learning for partially observable MDPs. In 2015 anai fall symposium series, 2015. 3
[36] X. Pan, Y. You, Z. Wang, and C, Lu. Virtual to real reinforcement lcarning for autonomous driving. arXiv preprint arviv: 170.:0392. 2017. 3
[37] M. Toromanoff, E. Wirbel, and F. Moutarde. End-to-end model-free free reinforcement learning for urban driving using implicit affordances. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7153-7162, 2020.
[38] J. Chen, B. Yuan, and M. Tomizuka. Model-free deep reinforcement learning for urban autonomous driving. In 2019 IEEE intelligent transportation systems conference (ITSC), pages 2765-2771. IEEF, 2019. 3
[391 Y. Tang. Towards lcarning multi-agent negotiations via self-play. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019. 3
[40] J. Cao, X. Wang. T. Darrell, and F. Yu. Instance-aware predictive navigation in multi- aulti-agent environments. arXiv preprint 200.22200008993, 2021.
10

---

[41] S. Levine, A. Kumar, G. Tucker, and J. Fu, Offline reinforcement learning: Tutorial, review, and perspec- tives open problems. arXiv preprini arXiv:2005.01643, 2020.
[42] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End end learning for self-driving arXiv preprint wrXiv:660.007316
2016. 3
[43] F. Codevilla, E. Santana, A. M. López, and A. Gaidon. Exploring the limitations of behavior cloning autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13299-3335 2019. 3
[44] X. Ren, T. Yang, E. A. Alahi, Q. Chen. Safety-aware motion prediction unseen vehicles for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15731-15740, 2021. 3
[45] H.-k. Chiu, J. Li, R. Ambruş, and J. Bohg. Probabilistic 3D multi-modal, multi-object tracking for autonomous driving. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 14227-14233. IEEE, 2021. 3
[46] P. Tokmakov, J. Li, W. Burgard, and A. Gaidon. Learning to track with object permanence. arXiv preprint arXiv:2103.14258, 2021.
[47] C. Richter. J. Ware, and N. Roy. High-speed autonomous navigation of unknown environments using learned probabilitics of collision. In 2014 IEEE International Conference Robotics and Automation (ICRA), pages 61 14--11... IEEE, 2014.
[48] C. Richter, W. Vega-Browe and N. Roy. Bayesian learning for safe high-speed navigation in unknown environments. Proceedings of the International Symposium on Robotics Research (ISRR). 2015. 3
[49] N. Rhinchart, R. McAllister, K. Kitani, and S. Levine. PRECOG: Prediction conditioned on goals in visual agent settings. ICCV, pages 2821 2830, 2019.
[50] N. Rhinehart, R. McAllister, and S. Levine. Deep imitative models for flexible inference, planning, and control. In ICLR, 2020. 7,8
[51] A. Hornung. K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard. OctoMap: An efficient probabilis- 3D mapping framework based on octrees. Autonomous Robots, 2013. doi: 10t110..00//11514--000 11-1-19951----9211----1- -0 URL https: petps://octomap gi sithub.r. .o. Software available at https: settps:/octtmmpp github. io. 8


Reviews:
[{"review_id": "wFTVBc7IhtY", "reviewer": "Reviewer_ec4g", "summary": "This paper develops a planner for autonomous driving in a multiagent environment. In particular, the paper contributes a method for addressing an imperfect perception system (e.g., sometimes doesn't detect other agents). A learned model of detection of other agents is fed into a planner that aims to compute collision-free paths that reach the goal quickly. Experiments in CARLA simulations of 4 traffic scenarios suggest the proposed method performs better than three prior approaches.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Important and difficult problem, relevant to this community, well-motivated\n- Modeling partial observability as a Bernoulli variable of whether an agent will be detected makes sense\n- Incorporating this model of perceptual uncertainty into the planner makes sense\n\nWeaknesses:\n- Results section has a lot of words but not much substance. Only a high-level comparison is given on 4 relatively clear scenarios. Since perceptual uncertainty is such an important factor, this section would have been much stronger with more experimental analysis of the detection model, how much uncertainty the detector/planner can handle, how realistic these uncertainties are (maybe using real driving data), etc. Issue with Table 1 described further below.\n- Block diagram in Figure 2 should be re-structured. What is the overall output? Where does the planner connect? I don't understand this diagram.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall this is a cool problem and the approach seems promising, but the experimental results do not provide sufficiently deep insight into what's really going on here.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Table 1 should be re-generated in a more sensible manner. RG and RG* are super vague concepts that throw away a lot of information here. The important metrics here seem to be reaching the goal quickly and not colliding despite the imperfect knowledge. I suggest reporting how each algorithm does on these metrics (and/or other metrics), and then incorporate a human expert either as a normalization factor (e.g., time to goal relative to human expert) or as another \"algorithm\" baseline row. Also, since every algorithm gets 15, 30, or 60 out of 60, it suggests there are a few discrete jumps in the difficulty of the scenarios --> this should be elaborated on because it seems unlikely to be a coincidence.\n- Additional experiments should be done (e.g., suggestions above) to provide a deeper level of analysis/insights into how uncertainty and the planner interact.\n- Block diagram should be improved per comments above.\n- The results section claims AVP scales to scenarios with additional agents, but the experiments show a jump from 2 to 3 agents. This is not a strong demonstration of scalability.\n- Modify overclaiming/inaccurate statement in the conclusion \"Because AVP is learned purely from observations, it does not require expert action labels or domain knowledge (such as state-space discretization or specification of an approximate dynamics model), making it easily generalizable to new environments and scenarios\", since \"we use hand-coded deterministic policies for all agents, which are designed to generate all plausible modes of realistic driver behavior in the scenario.\"\n- It may be worth adding a reference to the work of Charles Richter (papers circa 2015) that deal with planning conservatively around blind corners (a related form of perceptual uncertainty)?\n- The limitations section is not being used as intended. The first sentence describes an advantage of the approach. The third sentence describes future work. Surely there are more limitations than a lack of interpretability."}}, {"review_id": "kWxmG_oArwg", "reviewer": "Reviewer_uLnS", "summary": "This paper proposes an imitation learning like method for autonomous driving. The method outputs contingency plans based on maximum likelihood estimation of ego-agent control parameters in a learned probabilistic multi-agent interaction model. The main novelty of the work is to extend the previous PRECOG model (Rhinehart et al., ICCV 2019) by modelling partial observability via detection events. The method is evaluted in three driving scenarios with the CARLA simulator, showing improved performance over baslines using PRECOG or other alternative approaches.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "** Strengths\n\n- Addresses a highly relevant and interesting problem for autonomous driving: reasoning about future uncertainty and presence of unseen traffic participants, and using this for planning ego-motion.\n- Addressing partial observability via modelling detections is interesting, and has the advantage of avoiding need for ground truth positions, which the paper well argues for.\n- Paper is overall well written and organized.\n\n** Weaknesses\n\n- The evaluation is rather underwhelming. It focuses on the aggregate performance in the three benchmark tasks, but does not provide any insights into qualitative behavior, or any ablations on the proposed model.\n\n- Especially more effort is needed to evaluate the key novelty of the work, modelling of partial observability. It would be interesting to see how well the model succeeds in predicting future uncertainty, which is key to taking appropriate active sensing actions to potentially reduce that uncertainty. It is not clear what knowledge is represented or how it could be examined/introspected, an empirical demonstration would be welcome. Interpretability is important to have in the target application, but is limited in the proposed method (as is appropriately acknowledge in the paper).\n\n- I also wonder whether the benchmark tasks are too easy, as they are solved perfectly by the proposed method? This gives little clues as to the limits of the proposed method.\n\n- Data association (DA) is an issue in cluttered traffic scenes or with a large number of agents: which measurement originated from which target? DA is assumed solved in the paper, and the detection modelling approach relies on this assumption. It seems unlikely the method can handle complex traffic scenes.\n\n- It seems doubtful the proposed method will scale to more than a handful of agents. Firstly, there is the problem of obtaining training data. The paper is not very clear on how this is done, but Lines 232-233 indicate a substantial amount of manual effort. Second, the paper does not say explicitly, but being based on inference in a graphical model, one could expect scaling to be exponential in the size of the model (but please, correct me if I am mistaken). To be fair, this is a common issue in other methods as well but would be worth commenting on.\n\n\n** Other comments\n- The section on POMDPs and belief space planning is otherwise accurate, but it misses the significant stream of sampling-based Monte Carlo tree search inspired methods, see e.g., POMCP (Silver & Veness, NeurIPS 2010), DESPOT (Somani et al., NeurIPS 2013), and POMCPOW (Sunberg & Kochenderfer, AAAI 2018). These methods have far greater scalability than point-based solvers, and are sometimes applicable in continuous problems. These should be discussed as well.\n\n- Line 102: domain of x_t should be R^{(A+1) x d} to account for ego agent\n\n- How training data is collected is not quite clear (Lines 232-233), this should be properly explained in the paper or supplementary material.\n\n- The paper makes some claims about \"closed loop optimality\" in the introduction. However, by Section 4.2. it turns out that what is meant is ability to reach human performance (fine, but that's something different). The terminology should be revised, I don't think it is reasonable to claim optimality without a clearly defined reward function.\n\n- What is the relation of this work to methods such as PHD filtering (e.g., Vo & Ma,\"The Gaussian mixture probability hypothesis density filter\" IEEE Transactions on signal processing)? Is it fair to say the prediction part method approximates a PHD filter with known data association?\n\n**Added after author response**: I suggest the authors also consider the related work Cunningham et al. \"MPDM: Multipolicy decision-making in dynamic, uncertain environments for autonomous driving\", ICRA 2015, which also models agents with distinctive closed-loop behaviors.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a work with some novelty (extending PRECOG to partially observable case), but this novel contribution needs a more thorough evaluation. The appropriateness of the evaluation should be checked, and some further insights into what the method has learned or how the knowledge can be examined would be good to have.\n\n**Added after author response**\n\nI thank the authors for their reply. I have also seen the revised paper and supplementary material. This paper has interesting ideas, most notably the approach of modelling detections instead of ground truth positions of other agents. The author response also addresses all the points raised in my \"other comments\" part well.\n\nHowever, in my evaluation, the following negative points outweigh the positives:\n- Underwhelming evaluation. The evaluation in the main paper simply reports aggregate performance over three scenarios, with perfect scores for the proposed method. This falls short of demonstrating what the limitations of the proposed method are, what are the aspects that are crucial for the proposed method to work (ablation studies), or how it performs with respect to objectives such as maintaining a multi-modal belief over intentions/existence/trajectories of other agents, or actively taking actions to reduce future uncertainty. The revised supplementary material adds qualitative examples, but in my opinion these are key aspects of the proposed method and should be evaluated more thoroughly.\n- Lack of scalability. The training data collection requires a manual definition of a control policy for all agents. All combinations of such policies must be present in the training data. Even with 2 distinct behaviours per agent, for n agents the method requires data from 2^n scenarios. This should also be addressed in the limitations, and I agree with Reviewer ec4g that this section can be improved.\n\nTherefore, I am still leaning towards reject.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I would appreciate commenting on/addressing the weaknesses and other comments mentioned above.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "hOmYv4Ag-4d", "reviewer": "Reviewer_56zH", "summary": "The authors propose Active Visual Planning (AVP), a data-driven approach for automated driving that uses context derived from perceptual observations (specifically, lidar scans) along with past vehicle waypoints to predict the distributions of future states for both an ego vehicle and non-ego vehicles, enabling contingency planning which is capable of addressing the partial observability that arises in common driving scenarios. Learning occurs offline, using a dataset of recorded observations, and hand-coded deterministic policies reflective of human driving behaviors. Per the study conducted using the CARLA simulator, AVP permits the scenarios of interest to be handled with increased safety, relative to single and multi-agent models that plan only according to the information that is observable. The authors claim that AVP is the first fully-learned planner that supports active contingency planning over perceptual uncertainty.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper addresses a compelling, well-motivated topic relevant to robot learning, proposing a novel method for predicting and acting upon the existence, positions, and intentions of other vehicles on the road under partial observability, to safely complete key automated driving tasks.\n\n+ The paper is well-organized and clearly written, the code for AVP is made available, and the supplemental details provided on the accompanying website are helpful.  \n\n+ AVP is compared with three other suitably chosen methods, which include simpler, less conservative frameworks (single and multi-agent models that plan only according to the information that is observable) and a game-theoretic framework that considers all of the possible outcomes of the driving scenarios of interest. \n\n+ The driving scenarios selected for the experimental study are well-motivated scenarios that arise in everyday driving, which also present safety hazards for human drivers. AVP is shown to address these driving scenarios with superior rates of success relative to the selected baselines. \n\n- It would be helpful to provide more detailed, quantitative results characterizing the plans generating by the four competing methods, as well as to show, qualitatively, what the plans look like that were produced by the three baselines selected.\n\n- The paper would be strengthened by providing clarification on the extent to which the driving scenarios encountered in the training process differed (or did not differ) from the scenarios encountered at test-time. This would lend support to the claim made in the paper that AVP is flexible and can be easily transferred to new scenarios. This is especially important because of the additional claims made that one of the baselines (SOAP) requires significant setup for each individual scenario.\n\n- The paper would also be strengthened by speculating on the pathway by which the planners learned for specific driving scenarios could be integrated into a single unified framework for use in real automated driving. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In AVP, the authors have contributed a novel and well-motivated data-driven framework for prediction and planning that is of potential benefit to complex and hazardous automated driving scenarios, offering improved performance with respect to successful task completion. The paper would be strengthened by providing more implementation details, a more detailed exposition of the experimental results, and evidence for the flexibility/transferability of AVP that is claimed by the authors. \n\nPost-Rebuttal Comments: I appreciate the revisions and clarification provided by the authors, and I remain in favor of acceptance. Taking the authors' responses and the CoRL review criteria into account, I have decided to keep my rating of weak accept.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- The authors should provide more evidence to support the claim “our method can be easily transferred to new scenarios”. Perhaps they can do so by providing clarification on the extent to which the driving scenarios encountered in the training process differed (or did not differ) from the scenarios encountered at test-time. For example, what was different about the ego and non-ego trajectories, environments, etc., encountered in training vs. testing. \n\n- Along these lines, the statement “further details can be found in the supplemental material” in the section on Model Architecture and Training gives the reader the impression that more details on the training procedure will be found in the supplement, but there are no additional details provided.\n\n- The term “visual context” is a little confusing, since vision is not one of the sensing modalities being used in this work. Perhaps “perceptual context” could be used instead for greater generality, or “scan context” for greater specificity. If visual context is a term of art commonly used to describe this, feel free to ignore my suggestion. \n"}}, {"review_id": "HlTjz8RyYt6", "reviewer": "Reviewer_D5KJ", "summary": "This paper presents an approach to safety-aware trajectory planning in the presence of both seen and unseen agents, with a focus on simulated self-driving scenarios. Building upon earlier work, in which an RNN is used to estimate trajectories. In the (previous) approach, future trajectories of the human agents are sampled from a normal distribution trained from data collected offline and the trajectories of the agent are chosen via optimization with respect to those sampled trajectories. In this work, the authors train the system in such a way that the other \"human\" agents may not always be visible and so predictions about the true state of the system must incorporate that other agents may be in the scene yet not currently observed. The proposed agent is capable of understanding the impact unseen agents may have on their planning in a handful of common driving scenarios, improving performance with respect to baseline approaches that do not consider this term.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this paper has an interesting (if somewhat small) idea about how to incorporate unseen agents in decision-making. The main contribution seems mostly to be the change in the form of the data and the training process, which now takes into consideration that not all vehicles will be observed at all times and so the unobserved state should not be directly penalized; instead, the latent state used for forward simulation of trajectories during planning consumes the partial state. While this approach is novel, the contribution seems fairly small. The application domain, however, is fairly important and understudied, offsetting this.\n\nThe prose in the paper is very clear. However, the paper was hard to follow in places, due primarily to a notable omission: the pseudocode describing the planning procedure exists only in the supplementary material, yet was important for understanding the method. Without it, the paper is not particularly self-contained and so the authors must spend more time describing what this approach looks like in the main text or move the pseudocode into the text; while the equations listed in the text are enough to define the optimization objective, it is a big leap from following along with those equations. Reading through the pseudocode was essential for my understanding of the proposed approach.\nAdding in the pseudocode would greatly help understanding.\n\nSmaller questions and comments for clarity.\n- Watching the overtake videos in the supplemental material, it appears that (to at least a certain extent), the ability of the agent to succeed during the overtake boils down partially to luck. There does not appear to be sufficient time to successfully overtake if the oncoming car were to arrive sooner. Have the authors experimented with the timing of the arrival of the oncoming car and do they expect this would change the results? (There is a balance here between seeking opportunity and behaving conservatively, and I still believe the results to show relative improvement to the other results. My question touches upon the agent's desire to prioritize safety: is this a tunable parameter, or would it require changing the training data?)\n- The experiments seem to be broken into \"chunks\" of 15, since every result for all planners succeed in 15, 30, or 60 of the 60 trials. Relatedly, is each trial within each experiment identical or are the trials randomized somehow? The authors should include clarity about how these trials are run in the results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While this approach is novel, the contribution seems fairly small. The application domain, however, is fairly important and understudied. The central issue with the paper as it exists right now is in its clarity, and the overall approach was difficult to fully understand without multiple full read throughs of the paper and reading the supplemental material.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "The central issue is the clarity of the paper's description of the overall approach, mainly the omission of the planning pseudocode. As mentioned above, though what is included in the paper is clear, it was difficult to discern how everything came together during planning; the pseudocode helped my understanding of this, though there are perhaps other ways to communicate these ideas as well.\n\nThere are other small questions above that, if included, would also help clarity."}}]

Author rebuttal:

Decision: Accept (poster)