RAP: Risk-Aware Prediction for Robust Planning
Authors: Haruki Nishimura
Jean Mercat
Blake Wulfe
Rowan Thomas McAllister
Adrien Gaidon
Venue: CoRL 2022 Oral (2022)
Keywords: Risk Measures
Forecasting
Safety
Human-Robot Interaction

Abstract: Robust planning in interactive scenarios requires predicting the uncertain future to make risk-aware decisions. Unfortunately, due to long-tail safety-critical events, the risk is often under-estimated by finite-sampling approximations of probabilistic motion forecasts. This can lead to overconfident and unsafe robot behavior, even with robust planners. Instead of assuming full prediction coverage that robust planners require, we propose to make prediction itself risk-aware. We introduce a new prediction objective to learn a risk-biased distribution over trajectories, so that risk evaluation simplifies to an expected cost estimation under this biased distribution. This reduces sample complexity of the risk estimation during online planning, which is needed for safe real-time performance. Evaluation results in a didactic simulation environment and on a real-world dataset demonstrate the effectiveness of our approach. The code and a demo are available.

RAP: Risk-Aware Prediction for Robust Planning
Haruki Nishimura"
Jean Mercat
Blake Wulfe
Rowan McAllister
Adrien Gaidon
Toyota Research Institute, USA
firstname lastname@tri.....mmm..........tt wew.aaamtti.ggooaae..... global
Robust planning in interactive scenarios requires predicting the uncer- tain future make risk-aware decisions. Unfortunately, due to long-tail safety- critical events, the risk is often under-estimated by finite-sampling approximations of probabilistic motion forecasts. This lead to overconfident and unsafe robot behavior, even with robust planners. Instead of assuming full prediction coverage that robust planners require, we propose to make prediction itself risk-aware. We introduce a new prediction objective to learn risk-biased distribution over trajec- tories, so that risk evaluation simplifies to an expected cost estimation under this biased distribution. This reduces the sample complexity of the risk estimation dur- ing online planning, which is needed for safe real-time performance. Evaluation results in didactic simulation environment and on a real-world dataset demon- strate the effectiveness of our approach. The code² and demo² are available.
Abstract:
Keywords: Risk Measures, Forecasting, Safety, Houman-Robor Interaction
Introduction
In safety-critical and interactive control tasks such as autonomous driving, the robot must success- fully account for uncertainty of the future motion of surrounding humans. To achieve this, many contemporary approaches decompose the decision-making pipeline into prediction and planning modules [1-5] for maintainability, debuggability, and interpretability. A prediction module, often learned from data, first produces likely future trajectories of surrounding agents, which are then sumed by planning module for computing safe robot actions. Recent works [6, 7] further propose couple prediction with risk-sensitive planning for enhanced safety, wherein the planner computes and minimizes a risk measure [8] of its planned trajectory based on probabilistic forecasts of human motion from the data-driven predictor. A risk measure is a functional that maps a cost distribution to deterministic real number, which lies between the expected cost and the worst-case cost [9].
Although combining data-driven forecasting and risk-sensitive planning has been shown to be effec- tive, there exist several limitations to this approach. First, accurate risk evaluation of candidate robot plans remains challenging, due to inaccurate characterization of uncertainty in human behavior [10] and smite-sampling from the predictor. Some existing methods that promote diversity of predic- tion (e.g.. [11, 12)) may alleviate this issue, but they are not explicitly designed for reliable risk estimation needed for robust planning. Second, endowing an existing planner with risk-sensitivity often requires non-trivial modifications to its internal optimization algorithm [13-15]. This modifi- cation can be problematic, if, for example, an autonomy stack already has a dedicated and complex (risk- neutrane planner in use and cannot easily modify its internal optimization algorithms.
To address the above limitations, we propose to consider risk within the predictor rather than in the planner. We present risk-biased trajectory forecasting framework, which provides a general ap- proach to making a generative trajectory forecasting model risk-aware. Our novel method augments pre-traine traitre generative model with an additional encoding process. This modification changes the
"The first two authors contributed equally to this work.
hnttpe://github.cc http:/ttabb..oo//TTMMMMM//A
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

output of the prediction so that it purposefully and deliberately over-estimates the probability of dan- gerous trajectories. This "pessimistic" forecasting model gives distributional robustness (e.g., [16]) to the planner against potential inaccuracies of the human behavior model.
We achieve the pessimistic risk-biased distribution using novel prediction loss. This shifts the computational burden of drawing many prediction samples that capture rare events from online de- ployment offline prediction training. The planner can still obtain an accurate estimate of the risk measure in real-time during deployment with fewer prediction samples required from the biased distribution. Furthermore, our approach also eliminates the need for modifications the planner's 's optimization algorithm. Thus, one can achieve enhanced safety by simply replacing conventional probabilistic motion forecaster with the proposed risk-biased model, while still using the same exist- ing nest-llutral planner. This capability is intended for use in robotic applications where misestima- tion of risk could lead to injury, including autonomous vehicles and home robots that must operate safely in close proximity to humans.
Specifically, our contributions in this work are as follows:
We propose a risk-biased trajectory forecasting framework, which makes forecasts more useful for the downstream task and leads plans that are robust distribution shifts,
Our risk-biased model off-loads the heavy computation of risk estimation from online plan- ning, providing lisk-wwareness to a generic risk-neutral planner.
We extensively evaluate our proposed approach in simulation with planner in the loop and offline with complex real-world data.
2 Related Work
Trajectory forecasting from data. Early trajectory forecasting approaches defined hand-crafted dynamics models [17, 18], and incorporated rules that induce obstacle avoidance behavior [19] or mimic the overall traffic flow [20, 21J. More recently, data-driven, learning-based methods have gained popularity for their ability to better capture the complexity of human behavior [22], and typically use neural networks defining multi-modal trajectory distributions [12, 23-38].
Significant effort is directed toward increasing the coverage, or diversity, of motion forecasting mod- els [11, 12, 33 33-41] in order to ensure that no critical events are missed. Diversity can be explicitly encouraged using best-of best-of-many loss [25], by replacing meen-ssuared loss with a Huber loss [40], by choosing trajectory samples that maximize the distribution coverage [34], or by setting diverse anchors or target points [36-38]. Another strategy to increase mode coverage takes advantage of the latent distribution of CVAEs [5, 11, 41] or GANs [12]. Cui et al. [5] argue that besides coverage, sample efficiency is also an important factor. The authors trained road-scene motion forecasting model to produce predictions of other agents that induce diverse reactions from the given robot plan- ner. Similarly, McAllister et al. [42] train a model with a weighted loss giving a low weight to the predictions that do not affect the planner. Huang et al. [27] train a forecasting model that allows simple optimization procedure to select the safest among set of plans generated by planner. While prior work considered awseensssssess or planner- awareness, to the best of our knowledge, we are the first to use risk as a proxy to make forecasts more useful for the downstream task.
Subjective probability and prospect theory. Our pessimistic risk-biased prediction can be interpreted as a model of subjective probability (e.g.. [43]), which is closely related to risk- awareness [44]. For instance, prospect theory [45] studies how humans make risk-aware deci- sions and introduces the notion of probability weighting [46]. Under this model, the distribution is "warped"" so that the probabilities of unlikely events are always wver-weig Recent robotics literature has leveraged prospect theory to better model risk-awareness in human decision making, for example, in collaborative human-robot manipulation [47] and driver behavior modeling [48].
Prospect theory is descriptive model of human decision making, which differs from our goal of designing risk-aware robots. Moreover, our model only overestimates the probability of events that incur high-cost for the robot, unlike probability weighting that overestimates any unlikely outcome.
Risk-sensitive planning and control. Risk-sensitive planning and control date back to the 1970s, as exemplified by risk-sensitive
Cutroonnnniii--uuaaaaiii-------- [49, 50] and risk-sensitive

---

Markov Decision Processes (MDPs) [51]. More recent methods include ress-vens nonlinear MPC [6, 52], Q-learning [44, 53], and actor-critic [54, 55] methods, for various types of risk mea- sures. Refer to a recent survey [56] for further details. Unlike those methods in which the policy directly optimizes risk-measure, we propose to instead bias the prediction so that risk-sensitivity can be achieved by risk-neutral planner that simply optimizes the expected value of the cost.
3 Background
3.1 Generative Probabilistic Trajectory Forecasting
Let z and be the past and the future trajectories of an agent, and Yl, denote the random variable of the future trajectory conditioned on the observed past trajectory æ. We would like to fit the distribu- tion of p(Y lxx given dataset D of i.i.d. samples of (x, y) pairs. To p(Yl)) we maximize the like- lihood of future trajectories w.r.t. the model parameters 0, d: maximizeg,6 [ L(0, di y ylx),, One method to fit this distribution to learn a conditional variational auto-encoder, CVAE [57]. We focus on this approach because it produces a structured latent representation. The CVAE conditions its likelihood estimation on a latent random variable Z|.,, with a posterior 4o (z)2.,,), or Z|> with an inferred prior gø (#|z) used in the joint likelihood (2(2) =l The marginal likelihood of the future trajectory (or "model evidence") ") is pe (z),,)), and can be rewritten as:
where L( Φ: vl..) is the likelihood of the sample knowing
C(0, vlu) f po po((la,))dz = f
= polula..) Jon (2l2.0) dz Joi (2/2.0)
[Po w. =1x)
Bus (#laxx))
(1)
900 900 (2)12.))
Using Jensen's 's inequality, the logarithm of (1) is lower bounded by
T(O0, de xr,) v) = (sla.)) [In(po (vl=,)))] KT.(go, (=la.,))loo (=lz))
(2)
called the evidence lower bound (ELBO). We model dø and pe using neural networks. The encoders assume a Gaussian prior with independent elements to produce the inferred prior fo, : (œ) + (ulz, diag(=) (x)), and the posterior føz : (xx v) + (Kz.y, diag(Ele..)) The decoder makes the forecast ge: : (x, 2) + Every term in (2) can be either computed or estimated with Monte-Carlo
sampling as established in [57, 58].
3.2 Risk Measures
A risk measure is defined as a functional that maps a cost distribution to a real number. In other words, given random cost variable C with distribution p, risk measure of yields deterministic number r called the risk. In practice, we often consider a class of risk measures that lie between the expected value E,,[C] and the highest value sup(C). The former corresponds to the risk- evaluation of C, while the latter gives the worst-case assessment. Such risk measures often take a user-specified risk-sensitivity level
additional argument, which determines where the risk value r is positioned between 1E,1( [C] and sup(C'). Formally, let us define a risk measure as R2 (C, σ) - E [Ep[C], su)( Examples of such risk measures include entropic risk [50]: R2
R
Rentropic (C, O) = log Ep [p]cex((C)) as well as CVaR [59]:
RVaR (C. = inf SER
E. E. [max((0 C 01) (0ll
(3)
The rest of the paper assumes CVaR (3) as the underlying risk measure, but note that the proposed approach is necessarily bound to this particular choice. For CVaR, the risk value given risk- sensitivity level E (0, 1) can be interpreted as the expected value of the right (1 -t-ii of the cost distribution [60]. Thus, (C, tends E,[C] + and to sup(C)) as a 1.
Another intriguing property of CVaR is its fundamental relation to distributional robustness. CVaR belongs to a class of risk measures called coherent measures of risk [61] with the following dual characterization ([61], Theorem 4a):
RC, = sup E,IC], gEe
(4)
where Q is a uniquely-deternined. non-empty and closed convex subset of the set of all density functions. This suggests that CVaR is equivalent to worst-case expectation of the cost C when
3

---

the underlying probability distribution is chosen udversarially from Q. Therefore, an autonomous robot optimizing CVaR (or coherent measures of risk in general) obtains distributional robustness, in that the objective accounts for robustness to potential inaccuracies in the underlying probabilistic model. this context, the set Q is often referred as an ambiguity set in the literature [62, 63].
Problem Formulation
Suppose that a robot incurs cost under a planned policy т or trajectory. This cost is given by a function J such that C = (Y) with Y being the human future trajectory random variable, which the robot predicts probabilistically. We assume that J is known and differentiable in for each æ. One can design such cost function that Jd is high when the robot collides into the particular trajectory Y = y of a human. Supplementary material E defines the cost function used this work.
We begin with pre-trained generative model, as defined in Section 3.1, that gives a predictive distribution pY [x) = J p(Y|r,> (e..))((=))d through an inferred latent distribution (Z) This latent is mapped to the trajectory space by generator decoder y g(2, x). Under this unbiased model, the risk is given by r = R, (J (g x)). σr) using the risk measure introduced in Section 3.2.
Given the unbiased model and the risk measure, we are interested in finding another distribution qu (Z) in the latent space with learnable parameters v, under which simply taking the neutrll expectation of the cost will yield the same risk value as given above. This can be achieved by enforeing the following equality constraint on this biased distribution (Z):
Eg [J (g((,,))))
R2 (g) o))
(5)
We show that such a distribution exists in Section A. of the supplementary material. Comparing both sides in (5), we note that such q should be dependent on the risk-sensitivity level a. We propose to optimize the parameters v of the risk-biased distribution go (2)..)) In general, many distributions g can satisfy (5). We propose to pick particular that additionally minimizes the KL divergence from the prior p, to prevent the biased distribution from becoming too different from the original unbiased distribution. This leads to the following constrained optimization problem:
minimize KL (qp (Z))))lp((2)) subject to Egg [J (g(Z,z)) = R2 (J" ()(Z.)))).. (6)
In general, we cannot guarantee uniqueness of the solution to the optimization problem (6). How- ever, in the supplementary material A, we provide further analysis of (6) along with a sufficient assumption under which the solution would be unique (Proposition A.3).
Connection to importance sampling. Importance sampling has been employed in rare-event sim- ulation for accelerated safety verification of autonomous systems [64-66], which yields a pessimistic sampling distribution similar to our risk-biased model. However, a crucial difference of our approach that it estimates more general risk measure instead of an expected value. Given desired risk- sensitivity level, unweighted samples from the proposal g will directly yield the risk estimate (5). This removes the need compute the importance weights.
Connection distributional robustness. When coherent measure of risk is chosen as the un- derlying risk measure (such as CVaR), the right-hand side of (5) is always equivalent to a worst-case distribution g chosen out of an ambiguity set Q (4). In general, it is difficult verify if the optimal distribution qu- is in Q, since the specifics of Q depend on the choice of the risk measure as well as the risk-sensitivity level σ. Nevertheless, it holds true that any feasible distribution go for (6) yields the same worst-case expected cost as the most adversarial distribution from Q. Therefore, planner relying on de instead of p will possess distributional robustness. We demonstrate this crucial capability via an empirical evaluation in Section 6.3.
5 Implementation Details
Section B of the supplemental defines a usual (unbiased) CVAE trajectory forecasting model that learns two encoders, defining the Gaussian latent variables Z/z and Z and one decoder, predict- ing Y 1 We propose to solve problem (6) by learning third neural network encoder to define

---

Algorithm Proposed Risk-Biasing Loss Estimation
Input: Trajectory (ar, y) ~ D, risk level a ~ p(a), KL-loss weight 8, risk weight at robot motion Mrobot
1: for & E (1,.... ......KK K1} do
2: Sample latent 2x (V(l,,, 21,) with prior parameters (1122. E1,) fer (x)
3: Decode risk-neutral predictions yı (o =xl=)
4: Compute risk using (ya:... and Junen with Monte Carlo estimation (c.g., [68])
5: for E (1,..., K2} do
Sample biased latent #% ~ N(1 52(0) with risk risk-hiased parameters (n! ,.(01) = fula, a, yrobo))
7: Decode bik-ed predictions Dh = (00 (
8: Compute expected cost f Kg Σ k- Hioto (gu)
Compute risk loss Lrss p(r r) and prior loss Lpiie KL (11
Output: Loss value Liisk + Lgrior to train to (0 and Φ1 are fixed)
a biased latent distribution that, in combination with the pre-trained decoder, produces biased fore- casts. This biased encoder takes the past trajectory z, risk-level ø, and the robot future trajectory Brabot It outputs the parameters of Normal distribution y()) and log(diag(E(()) (+))).
In practice, soften the hard constraint (5) by using the penalty method [67], which progressively increases the weight a of the risk-loss during training. We also leverage a user-defined sampling distribution p(ar) sample different risk-sensitivity levels during training, so that the risk estimate remains accurate at any reasonable value of a at inference time. Finally, we encourage the model to overestimate the risk rather than underestimate it scale by the positive value and define an asymmetric risk-loss that penalizes linearly the underestimation of the risk and logarithmically its overestimation:
p(x) = |||
if sz <
[og(s2)), otherwise.
We obtain the following loss function with a and |3 controlling the relative importance of the losses: C(+b) Bo) [a (E [J (z())(z))) R, (J" (g()) + KL (go (Z (z,,. lIp())))(()))))
The expected values and the risk measure are approximated by Monte Carlo sampling. For comput- ing CVaR (Rp (J (g(Z, æ)). ). we use the estimator proposed by Hong et al. [68]. Consistency and asymptotic normality of this estimator hold under mild assumptions [68].
Algorithm lays out the procedure for training our proposed risk-aware prediction. It relies on a fully trained CVAE with the encoder f the distribution of Y| from a dataset. We train a new latent-biasing encoder fØ : 2; 0, Vrbox > (1()), (()) to bias the latent distribution while keeping the rest of the CVAE fixed. The risk-level o
[[x) and decoder go I,
that fits
randomly sampled on [0, 1] during training and chosen by the user test time.
Experiments
6.1 Biasing forecasts in a didactic scenario
0-
1.0
Figure 1: Top-down view simulated scene. The robot red moves to right down road as pedestrian in blue is crossing. The color of the depicted pedestrian trajectory samples indicates their corresponding Time- To-Collision (TTC) cost for the robot. The slow mode in red is more costly than the fast mode in green.
We created the didactic simulation environment in Fig. I where a red robot drives at constant speed along straight road with a stochastic pedestrian. The pedestrian either walks slowly or quickly, yielding bimodal distribution over their travel distance. We collected dataset in this environment where the initial position and orientation of the pedestrian are set at random. We used it to train risk-biased CVAE model according to the method presented in sections and 5. Fig. 2b shows
5

---

the risk-neutral prediction (a = 0) of the pedestrian's travel distance in a specific scene. As can be seen, the model captures both of the equally-likely modes. In contrast, the risk-biased case (a = 0.95), the model predicts the slower mode with much greater frequency because, in this scene, the pedestrian walks slowly will collide with the robot. If, alternatively, the pedestrian walks quickly, the vehicle will pass behind it safely without collision. In other words, the risk-biased model pessimistically predicts collisions with a greater probability than does the risk-neutral model. With 0.95, pessimism is high that the safer mode falls in the tail of the distribution. In supplemental D we explore the latent representation of this model.
3000
Number of samples
1000
Number of samples
1200
Number of samples
2500
1.50
1000
800
2000
1.25
600
1500
600
0.75
400
200
500
0.25
0
0.00
1.0
10
10
Travel distance
Travel distance
Trvel distance
(a) Ground truth data distribution.
(b) Risk-ncutral prediction.
(c) Risk-biased prediction = 0.95.
Figure 2: Histograms of the pedestrian travel distances at the end of the 5-sccond cpisode in the defined scene. Each bar is colored with the average Aime-To-Colliiiee (TTC) cost of the bin.
6.2 Planning with biased prediction
The previous experiment demonstrates the ability of our approach to bias predictions towards dan- gerous outcomes. The experiment in this section evaluates whether this ability can benefit online planning using a model-based trajectory optimization algorithm. In this setting, predictions are used to evaluate the risk of various candidate robot trajectories, Vrobot, in order to select the best one.
We generated a new dataset in which the robot's initial speed and per- per-timestep accelerations are sampled randomly, as opposed to the constant velocity model used previously. This variation ensures that the robot trajectories generated by the planner are within the training distribution. We modified the biasing encoder to account for the changing robot trajectories by adding y/mhor to its inputs. This allows our new model to achieve pessimistic forecasting with respect to a particular y/ohox.
The online planner controls the longitudinal acceleration of the robot, which is modeled as a double integrator system. We employed the cross entropy method (CEM) [69, 70] as the underlying opti- mization algorithm. CEM is stochastic optimization method that locally optimizes ylobot based on a given initial yrobor In each episode, CEM first draws samples risk-biased prediction samples given Blotor and observed pedestrian motion x, and uses the samples to produce a locally optimal yronne This process involves a linear complexity in Nsamples- We incorporated a quadratic trajectory tracking cost the planner's 's objective so that the robot encouraged to continue moving at a constant speed when the risk of collision is deemed negligible. See supplementary material C.1.2 for details.
We evaluated the performance of the combined risk-biased predictor and CEM planner across 500 episodes. Fig. 3 shows that CEM using the risk-biased predictor consistently produces Fu with low TiCello-Col (TTC) cost values, even when sampling few trajectories. We compared with a baseline approach in which risk-sensitive version of CEM performs planning using an unbiased CVAE predictor. We obtained this risk-sensitive planner by replacing the Monte Carlo expectation with the CVaR estimator [68]. This baseline is an instance of the conventional risk-sensitive planning with data-driven human motion forecasting [6. 71, which evaluates the risk within the planner rather than in the predictor. Fig. 3 shows significantly higher TTC cost of Vjobot for the baseline when CEM uses fewer than 16 samples. This is because the collision risk underestimated with few samples, and thus the planner over-optimistically optimizes trajectory tracking to the detriment of safety.
6.3 Robustness to out-of-distribation pedestrian behavior
For this last didactic experiment, we evaluated the distributional robustness of the overall prediction- planning pipeline to test-time change of the pedestrian stochastic behavior model. Specifically, we used the same dataset and planner as in Section 6.2, but we reduced the overall average speed of the pedestrian by 25% only at test time, after training. Other factors such as bi-modality were

---

Risk 0.95
Risk level 1.0
CAKE Sensiive CEM RAP & cTM (oun)
Ground-Tuuth TTC Cose
0.55
0.25
0.125
0.125
0105
0.105
0.106
22' 101
Nom. Prediction Samples
Num. Prediction Samples
Num. Prediction Samples
Figure 3: Ground -rrund--ruthh TTC cost of the optimized Viatees averaged over 500 episodes (lower the better). Ribbons
show 95% confidence intervals of mean. Our risk- biased predictor (RAP) coupled with CEM consistently achieves low cost regardless of the number of prediction samples that CEM draws online from the predictor.
Table 1: Ground-trute TTC cost of the optimized Fioiae under different test-time pedestrian behaviors, averaged
over 500 episodes (lower the better).
Predictive Model
# Prediction Samples
Planner
Pedestrian Behavior
TTC Cost
Unbiased CVAE
64
Risk-Neutral CEM
NA
same as training
0.23 =0.01
Unblased CVAE
64
Risk-Neutral CEM
NA
25% reduced speed 0.44 10.02
Unhiased CVAF
64
Risk-Sensitive CEM
0.95
25% reduced speed 0.37 20.02
Biased CVAB (RAP)
64
Risk-Neutral CEM
0.95
25% reduced speed 0.34 20.02
Unbiased CVAE
Risk-Sensitive CEM
0.95
25% reduced speed 0.46 20.02
Biased CVAB (RAP)
1
Risk-Neutral CEM
0.95
25% reduced speed 0.34 10.02
kept the same as at training time. From the robot's perspective, reducing the average speed of the pedestrian in the test scenario (as exemplified in Fig. 1) results in adversarial distribution shift. We studied how robust the risk-aware robot is under this out-of-distribution pedestrian behavior that the predictor did not witness during training.
Table summarizes the results of this experiment. The first row shows the nominal in-distribution risk-neutral case. The second row shows that the risk-neutral robot is not robust at all to the distri- bution shift, resulting in the doubling of the nominal TTC cost. This is expected from an autonomy stack without risk- awaeensssssss. Rows 3 and suggest that risk- ass-eeeresssss improves robustness given sufficient samples from the predictor. Finally. the last two rows show that our proposed framework remains robust to the distribution shift even with a single prediction sample, whereas the conven- tional risk-sensitive predictiio-planniig approach (row 5) does not show robustness any more. This demonstrates that, when the computation budget for online planning is limited, distributional robust- ness is better achieved within the predictor rather than in the planner.
6.4 Application to real-world data
We applied our risk-biased forecasting framework real-world data from the Waymo Open Motion Dataset (WOMD) [71]. Following a similar approach to [27], we selected the annotated scenarios that cover interesting interactions between two agents. We randomly selected one of the two inter- acting agent as the ego and the other as the agent to predict. We input other agent tracks and the map as additional conditioning information account for the interaction with the environment and the other agents. Then, we trained a biased CVAE model as described in Section 5. In this experiment we only conditioned the biased-encoder on the ego past trajectory, not its future trajectory, in order to avoid ground truth information leakage. This means that the -ensed- is making an implicit forecast of the ego future, which leads to the failure mode presented in Fig. 4d wherein the wrong implicit forecast leads to incorrectly-biasee distribution.
Table 2 shows the results of this experiment. First note the large difference between the minFDE and FDE values of the unbiased model, which illustrates that the predictions are diverse. This qualitatively supported by Fig. 4, which shows wide diversity of predicted trajectories. As expected, our biased CVAE model (RAP) with a risk level o = 0 shows results that match the ones from the unbiased model. As the risk-level increases, the predictions increasingly differ from those of the unbiased distribution as well as the ground truth trajectory. This is reflected by the larger minFDE and FDE values. At a 1, the biased prediction distribution collapses to the mode that the model estimates to be the most costly, yielding minFDE close to FDE. The risk estimation error the average difference of the mean cost under the biased prediction and the risk estimated using a large number of samples from the unbiased prediction. Its mean value shows the risk estimation bias of the proposed approach while its mean absolute value shown in the next column shows the average error that is made in either direction. Up to = 0.95, the risk estimation is nearly unbiased.
7

---

Aguet
(a) Restrnl
(b) Risk-awaare 0.95
(c) Risk-neutral
(d) Risk-aware, 0.95
Figure 4: Visualization of WOMD scenes and forecasts. The lane centerlines represented in gray, the past observations in black. the ground truth futures in green, and the 16 forecast samples in red. The ego blue and the agent to predict in green. In figure (b) the risk-aware forecasts turns towards the cgo. In figure (d) the risk-aware forecasts would be costly if the ego went straight ahcad. This is a failure case where the forecasts are biased towards an expected ego trajectory that did not occur.
Table 2: Motion forecasting
and risk estimation error on the WOMD validation set. minFDE (K): mini- mum final displacement error over K samples, risk error (K): mean value of signed difference between the average cost of the biased forecasts over K samples and the risk estimation using the unbiased forecasts, risk
|error] (K): mean value of the absolute values of the risk estimation error.
Predictive Model
minFDE(16)
FDE (1)
Risk error (4)
Risk errror (4)
Unbiased CVAF
NA
3.82 20.04
13.06 20.01
NA
NA
Biased CVAE (RAP)
0
3.81
=0.04
13.07
#0.02
0,00 +0.00
0.12 #0.00
Biased CVAE (RAP)
0.3
4.32
20.05
11.89 #0.02
0.02 10.00
0.13 #0.00
Biased CVAE (RAP)
0.5
5.32 =0.06
12.05 10.02
0.02 10.00
0.16 $0.00
Biased CVAE (RAP)
0.8
7.78 =0.07
13.53
20.02
0.01 20.00
0.26 $0.00
Biased CVAE (RAP)
0.95
10.13 #0.09
15.33
#0.02
0.03 #0.01
0.43 $0000
Biased CVAE (RAP)
11.58 #0.09
16.29 #0.02
-0.22 #0.01
0.60 #0.01
At = the risk estimation is slightly under-estimated. Usage of real data in this section limits our ability to provide accurate safety statistics for our approach. Instead, we provide extensive qualitative results. Section C of the supplement gives additional experimental details and results. We also provide extra experiments, figures and animations on our project website Finally, the model can be tested directly on several hundred samples, with any risk-level, on our online demo".
7 Limitations
A first limitation of our approach is that the constrained optimization problem (6) is difficult to solve in practice due to challenges presented in Section 4. The constraint relaxation and the neural network optimization method yield a sub-optimal risk-aware predictor that may still underestimate risk when the risk- sensiivvit level is close to 1. Therefore, conservative behavior is desired, or in the case of extremely low probability but catastrophic events. Second, in the real data application, our risk-aware prediction is not conditioned plan which may lead the forecast to collapse onto a mode that is not the most critical. Finally, we only forecast marginal agent behavior instead of jointly predicting the behaviors of several agents in the scene. This neglects potential risk-avoiding interactions in the future and leads to overly pessimistic biased forecasts.
method might be inadequate when extremely
specific robot
Conclusion
This paper proposes a risk-aware trajectory forecasting method for robust planning in human-robot interaction problems. We present novel framework to learn pessimistic distribution offline that simplifies online risk evaluation to expected cost estimation. Our experimental results show that this method leads to safe robot plans with reduced sample complexity. We additionally demonstrate the effectiveness of our approach in real-world scenarios with low risk-estimation error and strong qualitative results. In future work, we intend to evaluate our approach in a realistic simulator, and also improve the accuracy of risk estimation on real-world data by conditioning biased prediction on potential robot plans.
'nttpe://sites. http://eeee.eooeeeoo////o/////aa
'attps://hugginggace.oo http::::cc.epppcceceeeeeeee http::///gggggaaoooooo/o//eeeeaa http:///rrsssaaaeeeeerriitttttit
8

---

References
[1] E. Schmerling, K. Leung, W. Vollprecht, and M. Pavone. Multimodal probabilistic model- based planning for human-robot interaction. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 3399-3406. IEEE, 2018.
[2] H. Fan, F. Zhu, C. Liu, L. Zhang, L. Zhuang, D. Li. W. Zhu, J. Hu, H. Li, and Q. Kong, Baidu Apollo EM motion planner. arXiv preprint 1807. 000000000,8 2018.
[3] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang. S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8660-8669, 2019.
[4] B. Ivanovic, Elhafsi, G. Rosman, Gaidon, and M. Pavone. MATS: An interpretable tra- jectory forecasting representation for planning and control. In Conference on Robot Learning, pages 2243-2256. PMLR, 2020.
[5] A. Cui, S. Casas, A. Sadat, R. Liao, and R. Urtasun. Lookout: Diverse multi-future prediction and planning for self-driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16107-16116, 2021.
[6] H. Nishimura, B. Ivanovic, A. Gaidon, M. Pavone, and M. Schwager. Risk-sensitive sequential action control with multi-modal human trajectory forecasting for safe crowd-robot interaction. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 11205-11212. IEEE, 2020.
[7] R. S. Novin, A. Yazdani, A. Merryweather, and T. Hermans. Risk-aware decision making for service robots to minimize risk of patient falls hospitals. 2021 JEEE International Conference Robotics and Automation (ICRA), pages 3299-33000 3305. IEEE, 2021.
[8] A. Shapiro, D. Dentcheva, and A. Ruszezynski. Lectures on stochastic programming: model- ing and theory. SIAM, 2021.
[9] A. Majumdar and M. Pavone. How should robot assess risk? towards an axiomatic theory of risk in robotics. In Robotics Research, pages 75-84 84. Springer, 2020.
[10] R. Cheng, R. M. Murray, and J. W. Burdick. Limits of probabilistic safety guarantees when considering human uncertainty. In 2021 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 3182-3189. IEEE, 2021.
[11] Y. Yuan and K. Kitani. Diverse trajectory forecasting with determinantal point processes. arXiv preprint arXiv: 19Xxvv1907.049677 2019.
[12] X. Huang, G. McGill, J. A. DeCastro, L. Fletcher, J. J. Leonard, B. C. Williams, and G. Ros- man. Diversitygan: Diversity- vehicle motion prediction via latent semantic sampling. IEEE Robotics and Automation Letters, 5(4):5089-5066, 2020.
[13] A. Shaiju and I. R. Petersen. Formulas for discrete time LQQR, LQG, LEQG and minimax LQG optimal control problems. IFAC Proceedings Volumes, 41(2):8773-8778, 2008
[14] N. Bäuerle and J. Ott. Markov decision processes with average-value-at-risk criteria. Mathe- matical Methods of Operations Research, 74(3):361-379, 2011.
[15] Y. Chow, A. Tamar, S. Mannor, and M. Pavone. Risk-sensitive and robust decision-making: CVaR optimization approach. In Advances in Neural Information Processing Systems, vol- ume 28, 2015.
[16] H. Rahimian and S. Mehrotra. Distributionally robust optimization: A review. arXiv preprint arXiv:1908.05633 2019.
[17] J. Mercat, N. E. Zoghby, G. Sandou, D. Beauvois, and G. P. Gil. Kinematic single vehi- cle trajectory prediction baselines and applications with the ngsim dataset. arXiv preprint 10X1::908.11472.2 2019. 2019.

---

[18] C. Schöller, V. Aravantinos, F. Lay. and A. Knoll. What the constant velocity model can teach
us about pedestrian motion prediction. IEEE Robotics and Automation Letters, 5(2):1696 1703, 2020.
[19] D. Helbing and P. Molnar. Social force model for pedestrian dynamics. Physical review E, 51 (5):4282, 1995.
[20] M. Treiber, Hennecke, and D. Helbing. Congested traffic states in empirical observations and microscopic simulations. Physical review E, 62(2):1805, 2000.
[21] A. Kesting, M. Treiber, and D. Helbing. General lane-changing model mobil for car-following models. Transportation Research Record, 1999(1):86--)) 2007.
[22] S. Lefèvre, D. Vasquez, and C. Laugier. A survey on motion prediction and risk assessment for intelligent vehicles. ROBOMECH journal, 1(1):1-14, 2014.
[23] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. Social GAN: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2255-2264, 2018.
[24] N. Deo and M. M. Trivedi. Multi-modal trajectory prediction of surrounding vehicles with maneuver based lstms. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1179-1184. IEEE, 2018.
[25] A. Bhattacharyya, B. Schiele, and M. Fritz. Accurate and diverse sampling of sequences based on "best of many" sample objective. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 8485-8493, 2018.
[26] Ngiam, B. Caine, V. Vasudevan, Z. Zhang. H.-T. L. Chiang, J. Ling, R. Roelofs, Bewley, C. Liu, A. Venugopal, et al. Scene transformer: A unified multi-task model for behavior prediction and planning. arXiv e-prints, pages arXiv-2106, 2021.
[27] X. Huang, G. Rosman, A. Jasour, S. G. McGill, J. Leonard, and B. C. Williams, TIP: Task- informed motion prediction for intelligent systems. arXiv preprint arXiv:2 10.08875, 2021.
[28] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Synamically-feasttee trajectory forecasting with heterogeneous data. In European Conference on Computer Vision, pages 683-700. Springer, 2020.
[29] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. Gohome: Graph-oriented heatmap output for future motion estimation. arXiv preprint 1xXv:22009.08227 2021.
[30] T. Gilles, S. Sabatini, D. Tsishko,, B. Stanciulescu, and F. Moutarde. Thomas: Trajectory heatmap output with learned multi-agent sampling. arXiv preprint arXiv:2 10.06607, 2021."
[31] A. Sadeghian, V. Kosaraju, A. Sadeghian, N. Hirose, H. Rezatofighi, and S. Savarese. So- phie: An attentive GAN for predicting paths compliant to social and physical constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1349-1358, 2019.
[32] J. Mercat, T. Gilles, N. El Zoghby, G. Sandou, D. Beauvois, and G. P. Gil. Multi-head attention for multi-modal joint vehicle motion forecasting. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2020.
[33] N. Rhinchart, K. M. Kitani, and P. Vernza.. R2P2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), pages 772-788, 2018.
[34] T. Gilles, S. Sabatini, D. Tsishkou, B. Stanciulescu, and F. Moutarde. Home: Heatmap output for future motion estimation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 500-507. 2021,
[35] J. Amirian, J.BB. Hayet, and J. Pettré. Social ways: Learning multi-modal distributions of pedestrian trajectories with gans. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0-0, 2019.
10

---

[36] H. Zhao, J. Gao, T. Lan, Sun, B. Sapp. B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, al. TNT: Target-driven trajectory prediction. arXiv preprint arXiv:2008.0822 08294, 2020.
[37] S. Narayanan, R. Moslemi, F. Pittaluga, B. Liu, and M. Chandraker. Divide-and-conquer for lane-aware diverse trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15799-15808, 2021.
[38] B. Varadarajan, A. Hefny, A. Srivastava, K. S. Refaat, N. Nayakanti, A. Cormman, K. Chen, B. Douillard, C. P. Lam, D. Anguelov, et al. MultiPath++:: Efficient information fusion and I.. 2021.
trajectory aggregation for behavior prediction. arXiv preprint 1.
[39] N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. Torr, and M. Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 336-345, 2017.
[40] S. Casas, C. Gulino, S. Suo, K. Luo, R. Liao, and R. Urtasun. Implicit latent variable model for sceme-consistent motion forecasting. In European Conference on Computer Vision, pages Springer, 2020.
[41] Y. Yuan and K. Kitani. DLow: Diversifying latent flows for diverse human motion prediction. In European Conference on Computer Vision, pages 346-364. Springer, 2020.
[42] R. McAllister, B. Wulfe, J. Mercat, L. Ellis, S. Levine, and A. Gaidon. Control-aware predic- tion objectives for autonomous driving, arXiv preprint arXiv:2204. 13319, 2022.
[43] L. J. Savage. The foundations of statistics. Courier Corporation, 1972.
[44] Y. Shen, M. J. Tobia, Sommer, and K. Obermayer. Risk-sensitive reinforcement learning. Neural computation, 26(7):1298-1328, 2014.
[45] A. Tversky and D. Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5(4):297-323, 1992.
[46] N. C. Barberis. Thirty years of prospect theory in economies: A review and assessment. Journal of Economic Perspectives, 27(1):173-96, 2013.
[47] M. Kwon, F. Biyik, A. Talati, K. Bhasin, D. Losey, and D. Sadigh. When humans aren't op- timal: Robots that collaborate with risk-aware humans. In 2020 15th ACM/IEEE International Conference on
Interaction (HRI), pages 43-52. IEEE, 2020.
[48] L. Sun, W. Zhan, Y. Hu, and M. Tomizuka. Interpretable modelling of driving behaviors in interactive driving scenarios based cumulative prospect theory. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 4329 4335. IEEE, 2019.
[49] D. Jacobson. Optimal stochastic linear systems with exponential performance criteria and their relation to deterministic differential games. IEEE Transactions on Automatic control, 18(2): 124-131, 1973.
[50] Whitrle. Risk- sensisves linearquuadratieCCausiia control. Advances in Applied Probability, 13(4):764-777, 1981.
[51] R. A. Howard and E. Matheson. Risk-sensitive Markov decision processes. Management science, 18(7):356366, 1972.
[52] V. Roulet, M. Fazel, S. Srinivasa, and Z. Harchaoui. On the convergence of the iterative linear exponential quadratic gaussian algorithm to stationary points. In 2020 American Control Conference (ACC), pages 132-137. IEBE, 2020.
[53] S. Borkar. Q-learning for risk-sensitive control. Mathematics of operations research, 27(2): 294 311, 2002.
[54] Y. Chow and M. Ghavamzadeh. Algorithms for CVaR optimization in MDPs. Advances in neural information processing systems, 27, 2014.

---

[55] J. Choi, C. Dance, J.-e. Kim, S. Hwang, and K.-s. Park. Risk-conditioned distributional soft actor-critic for risk-sensitive navigation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 8337-8344. IEEF, 2021.
[56] Y. Wang and M. P. Chapman. Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control. arXiv preprint (rXxv:2109.08947, 2021.
[57] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
[58] D. P. Kingma and M. Welling. Auto-encoding variational Bayes. arXiv preprint
1rXiv:1312.6114, 2013.
[59] G. C. Pflug. Some remarks on the value-at-risk and the conditional value-at-risk. In Proba- bilistic constrained optimization, pages 272-281. Springer, 2000.
[60] A. A. Trindade, S. Uryasev, A. Shapiro, and G. Zrazhevsky. Financial prediction with con- strained tail risk. Journal of Banking & Finance, 31 (111):3524-3555 2007.
[61] R. T. Rockafellar. Coherent approaches to risk in optimization under uncertainty. In OR Tools and Applications: Glimpses of Future Technologies, pages 38-61. Informs, 2007.
[62] B. P. Van Parys, D. Kuhn, P. J. Goulart, and M. Morari. Distributionally robust control of con- strained stochastic systems. IEEE Transactions on Automatic Control, 61(2)4300442, 2015.
[63] S. Samuelson and I. Yang. Data-driven distributionally robust control of energy storage to manage wind power fluctuations. In 2017 IEEE Conference on Control Technology and Appli- cations (CCTA), pages 199-204, IEEE, 2017.
[64] D. Zhao, X. Huang, H. Peng, H. Lam, and D. J. LeBlanc. Accelcrated evaluation of automated vehicles in car-following maneuvers. IEEE Transactions on Intelligent Transportation Systems, 19(3):733-744, 2017.
[65] B. Wulfe, Chintakindi, S.-C. T. Choi, R. Hartong- A. Kodali, and M. J. Kochen- derfer. Real-time prediction of intermediate-horizon automotive collision risk. In Proceedings of the 17th International Conference on Autonomous Agents and MulliAgent Systems, pages 1087-1096, 2018.
[66] M. O' Oelllyy A. Sinha, H. Namkoong, R. Tedrake, and J. C. Duchi. Scalable end-to-end au- tonomous vehicle testing via rare-event simulation. Advances in neural information processing systems, 31, 2018.
[67] M. J. Kochenderfer and T. A. Wheeler. Algorithms for optimization. Mit Press, 2019.
[68] L. J. Hong, Z. Hu, and G. Liu. Monte carlo methods for value-at-risk and conditional value-at- risk: a review. ACM Transactions on Modeling and Computer Simulation (TOMACS). 24(4): 1-37, 2014.
[69] K. Chua, R. Calandra, R. McAllister, and Levine, Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018
[70] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[71] S. Ettinger, S. Cheng, B. Caine, C. Liu, H. Zhao, S. Pradhan, Y. Chai, B. Sapp, C. R. Qi, Y. Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The Waymo open motion dataset. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9710-9719, 2021.
[72] M. Bahari, S. Saadatnejad, A. Rahimi, M. Shaverdikondori, A. H. Shahidzadeh, S.-M. Moosavi-Dezfoole and A. Alahi. Vehicle trajectory prediction works, but not everywhere. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17123-177 17133, 2022.
[73] S. Khandelwal, W. Qi, Singh, A. Hartnett, and D. Ramanan. What-if motion prediction for autonomous driving. arXiv preprint arXiv:2008. 10587, 2020.
12


Reviews:
[{"review_id": "jThLOgLzYYd", "reviewer": "Reviewer_zkWD", "summary": "This paper tackles risk sensitive planning for vehicle prediction. Existing approaches often fail make diverse predictions suitable for risk aware planning. Additionally, adapting an existing risk-neutral planning pipeline to risk aware one may be costly. \n\nInstead of targeting downstream risk-sensitive control, this paper encodes risk into upstream prediction components. The authors' key insight is that a risk sensitive measure of cost is the same as the risk-neutral expectation of cost under a distribution of trajectories with risky trajectories up-weighted. This means they can use VAEs to learn an importance sampler which biases towards risky parts of the trajectory space, implicitly learning a risk-sensitive measure via proxy.\n\nExperiments are given on a small pedestrian proof of concept scenario, as well as larger experiment on the waymo open dataset. They demonstrate usefulness of their method as compared to an unbiased CVAE. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper is easy to follow, with good mathematical layout and comprehensive existence proofs accompanying each of the measure formulae.\n- The contribution of the paper itself is a novel synthesis of Variational Autoencoder methods, Cross-Entropy importance sampling, and risk sensitive measures. This combination provides a valuable tool for adding robustness to the predictive layers of an perceptual-control pipelines.\n- The results and figures given (in particular, the histograms of sampling for the biased vs unbiased experiments) do a good job showing the intuitive operation of their algorithm, and how it effectively biases predictive distributions towards risky parts of the space.\n\n## Weaknesses\n\n-  There is an odd statement towards the end of section 4 about proving that the solution to their optimization problem is unique. This is under the assumption that the decoder is \"Volume Preserving\" and they even provide a citation for a method for implementing volume-preserving neural networks. However, they then state that they do not attempt to meet this assumption, yet still present it as evidence their problem is well posed. I'm not sure I follow this! I think for the statement to be of any value, I would as least like to intuitively understand what it would take to bring their method in line with the volume preservation assumption.\n- In Section 5.2, there are a few user-defined constants ('s', 'alpha', 'beta'). It would be nice to see some analysis of how altering these values affects the end result.\n- The work is well situated within most of the related literature it deals with. However, the methods used also draw heavily on importance sampling and rare-event validation. I would have liked to see more engagement with this work. For instance, how does the work fit alongside things like (https://ieeexplore.ieee.org/abstract/document/7933977) or (https://proceedings.neurips.cc/paper/2018/hash/653c579e3f9ba5c03f2f2f8cf4512b39-Abstract.html)? Additionally, how do their methods relate to concepts like the proven ideal optimal zero-variance sampler from the importance sampling literature?\n- The authors mention in their limitations section that, while their method works for risk-aware planning at the 95% level, it may not be suitable for much rarer events. I agree, but I think this point is understated: For the AV domain in which their experiments take place, *extremely rare collision events are, in fact, the norm*, with safety requirements several magnitudes stronger than the ones given in the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "On the basis of the strengths and weaknesses outlined above, I think this is a good candidate for publication. My suggestions for additional analysis and literature are minor issues, stemming from a desire to see the work expanded, rather than anything preventing the work from achieving publication quality.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "No major issues. As mentioned in the strengths/weakness section, any insights you could provide into how this work fits into the context of existing rare-event sampling literature, and intuitions about how some of the hyper-parameters in the experiments section affect performance, would help broaden my understanding of the work."}}, {"review_id": "dzwYsbVGJHO", "reviewer": "Reviewer_vmFo", "summary": "There are several recent works that aim to enhance robustness of planning against various sources of uncertainty. This is typically accomplished by  designing risk-aware planning methods. The issue in these methods is that is risk is approximately computed which may lead to unsafe behaviors. This paper follows a different approach and instead of robustifying planning, it robustifies prediction by making it risk-aware. Particularly, a new prediction objective is proposed to learn risk-biased distribution over trajectories of agents that exist in the environment. This approach reduces sample-complexity during online planning which facilitates real-time implementation. The proposed method is supported by simulation results. \n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe idea of proposing a risk-biased trajectory forecasting framework is quite interesting and novel. This pessimistic forecasting allows the planner to be distributionally robust, i.e., robust to un-modeled sources of uncertainty.\n\nThe proposed method does not require modification of the planner and in fact it seems that it can be integrated with various planners (e.g., risk-aware planners and risk-neural planners).\n\nA biased distribution  is constructed from which samples are drawn to forecast future states of agents. This biased distribution allows to accurately predict future states with only few samples.\n\nWeaknesses\n\nThe evaluation is somewhat weak. It should be improved to demonstrate more clearly the distributional robustness of the proposed method. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the authors propose a novel and quite interesting approach to robustify predictions. The proposed risk-aware forecaster can be coupled with existing planners, whether they are risk-aware or risk-neural, and it can design safer (but more conservative) paths than existing risk-sensitive planners. The developed algorithm is supported by nice simulation results. Nevertheless, the evaluation should be improved to demonstrate distributional robustness in a more convincing way.\n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "The authors should address the following issues:\n\n- The evaluation needs to be improved to demonstrate distributional robustness in a more convincing way. For instance, what happens if the human behaves differently between training and test time (section 6.2)? In this scenario, how does the proposed method perform and also how is its performance compared against risk-sensitive planners?\n\n- Figures demonstrating the robot trajectories generated by the proposed method as well as competitive approaches would be helpful (Section 6.2). In fact, this would help visualize how conservative the designed paths are.\n\n- In the simulation studies, the TTC metric needs to be defined. Also, how are the prediction samples defined (e.g., in Fig. 3)? Do they refer to sample trajectories using to train the predictor?\n\n- In the abstract, the authors state :\"This reduces sample complexity during online planning for safe real-time performance\". This statement is confusing. How is sample complexity related to online planning? Samples are collected offline to train a risk-biased predictor. Given, this trained predictor, a planner is used online to design trajectories. Even if the offline training process is not sample efficiency, this has nothing to do with online planning, since samples are not used online for training. The authors should re-write this sentence more clearly.\n\n\n"}}, {"review_id": "ZNsn6K7hO5E", "reviewer": "Reviewer_QxWd", "summary": "This paper proposes to learn a risk-aware prediction model. The key idea is to learn an extra encoder model of the latent random distribution minimizing a cost function biasing the latent random variable to generate risk-biased trajectories. The article makes two main contributions:\n1 - a risk-aware prediction model\n2 - a novel training objective to learn a risk-biased distribution of the latent random variable", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "First, this paper tackles an extremely challenging and important problem to the robotics community. Secondly, the proposed idea is interesting and the proposed approach novel. Overall, the paper is well written, the method is described clearly and the presented results support the claimed contributions.\n\nHowever, this paper has the following main limitations\n1 - It only considers simple scenarios with a maximum of 2 agents\n2 - the risk-aware prediction do not depend on the ego's plan\n\nTherefore, in future work and in a journal of this work I recommend the authors to:\n1 - Consider complex scenarios with a high number of agents.\n2 - Evaluate and discuss how this approach would scale to higher number of agents\n3 - Will over-conservative prediction lead to the freezing-robot problem? How to address this issue?\n\nThe presented results can also be strengthened:\n1 - Present more qualitative results in the form of a video to clearly show the how the predictions vary with the risk value\n2 - How computationally demanding is to train such model? What is the training time? How much time it takes on training iteration? \n3 - How does this model generalizes to new unseen scenarios?\n\nFinally, I recommend the author to revise their citations. It is good that they did such broad literature review but there are several sentences where the citation does not support the written sentence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper tackles and important problem and it proposes an interesting new method. The presented results only consider very simple scenarios but I am ok with those for a conference paper as they clearly demonstrate the advantage of the proposed approach. Adding more experimental results, in a video format for instance, would improve the contribution of the article. Moreover, the limitations section needs to address more issues of the proposed approach. Overall, i think it is a good paper, well written and it has a clear contribution which the results support. Therefore, I recommend its publication.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Presented results only consider simple scenarios with only 2 agents.\nTherefore, in future work and in a journal of this work I recommend the authors to:\n1 - Consider complex scenarios with a high number of agents.\n2 - Evaluate and discuss how this approach would scale to higher number of agents\n3 - Will over-conservative prediction lead to the freezing-robot problem? How to address this issue?"}}, {"review_id": "GxyZYlXIqR", "reviewer": "Reviewer_UeEy", "summary": "The paper considers risk estimation of predictors used by planners for robots. Robustness of plans is achieved by pushing the risk estimation to prediction, rather than computing it at planning time. This has two advantages: (a) planners do not need to be updated when considering new predictors and risk measures; and (b) it simplifies planning using expectations over biased (risk-aware) distributions. The paper shows the approach on didactic and real-world datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The paper is clear and well written.\n+ It has good explanations and arguments for all design decisions and theory.\n+ The literature review and framing is good.\n\nweaknesses\n- An explicit contributions statement at the end of the introduction would strengthen the paper.\n- The paper could highlight the significance and benefits of the \"simplification\" more, e.g., what is the impact on runtime performance at deployment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper introduces an approach and ideas with significant potential impact on robot deployment. It is well written and technically sound. The results are shown empirically on small (didactic) and real-world datasets.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "In Sec. 5.2, what is $s$? Should $s = \\sigma$ or is $s \\geq 1$?\nMoreover, the penalty $\\rho(x)$ is discontinuous in $x$. How does it impact performance? Also, is the problem well posed?  Would $s \\cdot (1 + \\log(x) )$ work instead of $\\log(sx)$?"}}]

Author rebuttal:

Decision: Accept (oral)