CausalAgents: A Robustness Benchmark for Motion Forecasting Using Causal Relationships
Authors: Rebecca Roelofs
Liting Sun
Benjamin Caine
Khaled S. Refaat
Benjamin Sapp
Scott Ettinger
Wei Chai
Venue: Submitted to ICLR 2023 (2023)
Keywords: robustness
motion forecasting
self-driving cars

Abstract: As machine learning models become increasingly prevalent in motion forecasting systems for autonomous vehicles (AVs), it is critical that we ensure that model predictions are safe and reliable. However, exhaustively collecting and labeling the data necessary to fully test the long tail of rare and challenging scenarios is difficult and expensive. In this work, we construct a new benchmark for evaluating and improving model robustness by applying perturbations to existing data. Specifically, we conduct an extensive labeling effort to identify causal agents, or agents whose presence influences human driver behavior in any way, in the Waymo Open Motion Dataset (WOMD), and we use these labels to perturb the data by deleting non-causal agents from the scene. We then evaluate a diverse set of state-of-the-art deep-learning model architectures on our proposed benchmark and find that all models exhibit large shifts under perturbation. Under non-causal perturbations, we observe a 25-38% relative change in minADE as compared to the original. We then investigate techniques to improve model robustness, including increasing the training dataset size and using targeted data augmentations that drop agents throughout training. We provide the causal agent labels as an additional attribute to WOMD and release the robustness benchmarks to aid the community in building more reliable and safe deep-learning models for motion forecasting. 


Under review as a conference paper at ICLR 2023
CAUSALAGENTS: A ROBUSTNESS BENCHMARK FOR MOTION FORECASTING
Anonymous authors Paper under double-blind review
ABSTRACT
As machine learning models become increasingly prevalent in motion forecasting for autonomous vehicles (AVs), it is critical to ensure that model predictions are safe and reliable. However, exhaustively collecting and labeling the data necessary to fully test the long tail of rare and challenging scenarios is difficult and expensive. In this work, we construct new benchmark for evaluating and improving model robustness by applying perturbations to existing data. Specifically, we conduct an extensive labeling effort to identify causal agents, or agents whose presence influences human drivers' behavior in any format, in the Waymo Open Motion Dataset (WOMD), and we use these labels to perturb the data by deleting non-causal agents from the scene. We evaluate diverse set of state-of-the-arr deep-learning model architectures on our proposed benchmark and find that all models exhibit large shifts under even non-causal perturbation: we observe a 25-38% relative change in minADE as compared to the original. We also investigate techniques to improve model robustness, including increasing the training dataset size and using targeted data augmentations that randomly drop non-causal agents throughout training. Finally, we release the causal agent labels as an additional attribute WOMD and the robustness benchmarks to aid the community in building more reliable and safe deep-learning models for motion forecasting (see supplementary).
INTRODUCTION
Machine learning models are increasingly prevalent in trajectory prediction and motion planning tasks for autonomous vehicles (AVs) (Casas et al., 2020; Chai et al., 2019; Cui et al., 2019; Ettinger et al., 2021; Varadarajan et al., 2021; Rhinehart et al., 2019; Lee et al., 2017; Hong et al., 2019; Salzmann et al., 2020; Zhao et al., 2019; Mercat al., 2020; Khandelwal et al., 2020; Liang et al., 2020). To safely deploy such models, they must have reliable, robust predictions across diverse range of scenarios and they must be insensitive to spurious features, or patterns in the data that fail to generalize to new environments. However, collecting and labeling the required data to both evaluate and improve model robustness is often expensive and difficult, in part due to the long tail of rare and difficult scenarios (Makansi et al., 2021).
In this work, we propose perturbing existing data via agent deletions to evaluate and improve model robustness spurious features. To be useful in our setting, the perturbations must preserve the correct labels and not change the ground truth trajectory of the AV. Since generating such perturbations requires high-level scene understanding as well as causal reasoning, we propose using human labelers to identify irrelevant agents. Specifically, we define a non-causal agent as an agent whose deletion does not cause the ground truth trajectory of a given target agent change. We then construct robustness evaluation dataset that consists of perturbed examples where we remove all non-causal agents from each scene, and we study model behavior under alternate perturbations, such as removing causal agents, removing a subset of non-causal agents, or removing stationary agents.
Using our perturbed datasets, we then conduct an extensive experimental study exploring how factors such as model architecture, dataset size, and data augmentation affect model sensitivity. We also propose novel metrics to quantify model sensitivity, including one that captures per-example absolute changes between predicted and ground truth trajectories and another that directly reflects how the model outputs change under perturbation via IoU (intersection-over union)) without referring to the ground truth trajectory. The second metric helps address the issue that the ground truth trajectory is one sample from a distribution of many possibly correct trajectories. Additionally, we visualize scenes with large model sensitivity understand why performance degrades under perturbations.

---

Under review as a conference paper at ICLR 2023
AV ground truth AV pred ction Agent ground truth
AM prediction
truth
(a) Original Scene (minADE 0.282m)
(b) Perturbed Scene (minADE 4.17m)
Figure 1: Trajectory prediction is sensitive to removing non-causal agents. We show a top-down view of a scene from the WOMD (left) and a perturbed version of the scene where we delete all cou-causal agents (right). The AV and its predicted trajectories via the Scene Transformer model (Ngiam et al., 2021) are shown in blue, the ground truth trajectory of the AV is grey, and the ground truth of other agents is green. The perturhation causes large shift in minADE hecause the model fails to predict the ground truth mode (a right turn,, which indicates the brittleness of the model to such perturbations.
Our results show that existing motion forecasting models are sensitive to deleting non-causal agents and can have pathological behavior dependencies on faraway or distant agents. For example, Figure 1 illustrates an original (left) and perturbed (right) scenes with non-causal agents removed. In the perturbed example, the model's prediction misses the right-turn mode, which corresponds to the ground-truth trajectory. Such brittleness could lead to serious consequences in autonomous driving systems if we rely on deep-learning models without further safety assurance from other techniques such optimization and robotics algorithms. The main contributions of our work are follows:
1.
We contribute a new robustness benchmark for the WOMD for evaluating trajectory predic- tion models' sensitivity to spurious correlations. We release the causal agent labels from human labelers as additional attributes to WOMD so that researchers can utilize the causal relationships between the agents for robustness evaluation and for other tasks such as agents relevance or ranking (Refaat et al., 2019; Tolstaya et al., 2021).
2. We introduce two metrics quantify the robustness of motion forecasting models to perturbations, including absolute per-example change in minADE and a trajectory set metric that measures sensitivity without using the ground truth as reference.
3. We evaluate the robustness of several state-of-the-arr motion forecasting models, includ- ing Multipath++ (Varadarajan al., 2021), Wayformer (Nayakanti et al., 2022). and SceneTTransfrre (Ngiam et al., 2021). We show that the absolute per-example change in minADE can range from 0.07-0.23 m (a sigmificant 25 38% change relative to the original minADE). We find that all models are sensitive to deleting non-causal agents, and the model with the best overall performance (in terms of regular metrics used to quantify the trajectory prediction performance such as minADE) is not necessarily the most robust.
4. We show that increasing training dataset size and targeted data augmentations that remove non-causal agents can help improve model robustness.
Overall, this is the first work focusing on the robustness of trajectory prediction models to perturba- tions based on human labels. Such robustness is critical for models deployed in a self-driving car where the reliability and safety requirements are utmost importance. Ultimately, our goal is to provide a robustness benchmark which can aid the community to better evaluate model reliability, detect possible spurious correlations in deep-learning based trajectory prediction models, and facili- tate the development of more robust models or other mitigation techniques such optimization and traditional robotic algorithms as complementary solutions to minimize safety risks.
RELATED WORK
Robustness evaluation on perturbations. Machine learning models are known to have brittle predictions under distribution shift. Across multiple domains, researchers have proposed robustness evaluation protocols that move beyond a fixed test set (Recht et al., 2019; Biggio & Roli, 2018; Szegedy et al., 2013; Hendrycks & Dietterich, 2019; Gu al., 2019; Shankar et al., 2019; Taori
2

---

Under review as a conference paper at ICLR 2023
et al., 2020). Evaluation can be broadly categorized into three types: (i) slicing, i... existing test data is sliced over multiple dimensions, (ii) perturbations, i.. existing test data is modified via transformations, or (iii) dataset shift, i.c. new test data is drawn from different distribution. Our work focuses on perturbations, which have previously been explored in both computer vision and NLP. In computer vision, researchers perturb images via pixel level noise corruptions (Geirhos et al., 2018; Hendrycks & Dietterich, 2019), spatial transformations (Engstrom et al., 2019; Fawzi & Frossard, 2015), and adversarial modifications (Biggio et al., 2013; Szegedy et al., 2013). Such synthetic shifts are easy to apply to arbitrary images, but limited in that they do not test model invariance to more complex modifications such as deleting or modifying irrelevant parts of the image. In trajectory prediction, perturbations are potentially more valuable, since the models train on discrete inputs, namely, the agents and the roadgraph. Because of the structure of the problem, is easier to reliably construct perturbations that do not modify the ground truth labels. This situation mirrors that of NLP, where sentences composed of discrete words can be modified in ways that do not change the prediction task, and indeed, such transformations have proven valuable for testing the robustness of models and identifying possible biases (Dhole et al., 2021).
Robustness evaluation for trajectory prediction. The three types of robustness evaluation (slicing, perturbations, and dataset shift) described above also characterize the trajectory prediction literature. Slicing. The most common approach is to slice model performance along different hyperparameters and buckets, such as duration of the historical trajectories (Radwan et al., 2020), size of the training data (Huynh & Alaghband, 2019; Ngiam al., 2021), sampling frequency (Bera et al., 2016), number of agents in the scene (Rhinehart et al., 2019; Ngiam et al., 2021), criticality/ interactivity of the scenarios (Kooij et al., 2019; Ettinger et al., 2021), and speed of the AV (Ngiam et al., 2021). Perturbations. Another thread of related work focuses on the robustness of the algorithms to perturbations in both training and test data. For example, (Bera et al., 2016) introduced synthetic sensor noises into both the training and test process to evaluate the model's accuracy against sensor noises. (Han et al., 2019) introduced 30% anomalies into the training data (with extra labels). and evaluated the robustness of the algorithm to anomalies in the training process. Dataset shift. Less work has focused on dataset shift due to the difficulty of collecting. annotating, and releasing entirely new data. Examples include training and testing in different locations or routes (Schöller et al., 2019; Sun et al., 2021), weather, time of day, and sensor noise (Sun et al., 2021).
A related body of work has studied adversarial robustness for trajectory prediction. In particular, Cao et al. (2022b) propose an adversarial training framework for trajectory predictions as well as domain-specific data augmentations and show that both empirically improve robustness to adversarial attacks. Additionally, researchers have generated more realistic adversarial attack models and benchmarked models against them: Cao et al. (2022a) generates adversarial realistic trajectories using a differentiable dynamic model, Zhang et al. (2022) perturbs existing trajectories to maximize prediction error, Saadatnejad et al. (2022) perturbs trajectories to result in agent collisions, and Wang et al. (2021) simulates directly from sensor data to modify trajectories in a physically plausible manner. Unlike prior work, we evaluate model robustness to "non-causal" domain shifts instead of hard (e.g- weather, location) or adversarial domain shifts. Because these non-causal perturbations are closer to the original validation dataset than the hard domain shifts and do not assume worst-case behavior or presence of an adversary, the discrepancies we observe are in some ways more immediate priority for improving model robustness.
Agent relevance. Since trajectory prediction in AV systems must reason about other agents in the scene, researchers have attempted efficiently rank agents according to their impact on the AV. The main motivation is to determine which agents to allocate real-time computational resources to. In particular, Aksoy al. (2020) proposed driver's saliency prediction model which incorporates an attention mechanism to understand salient features for driving context. Refaat et al. (2019) approximated an agent's influence by looking at the difference between two plans when given agent is accounted for versus not. However, removing one agent at a time does not account for certain situations where multiple agents may be influencing the car in the same way e.g. two pedestrians are blocking the path of the car and removing one of the pedestrians has no influence on the car. In similar work, Tolstaya et al. (2021) quantifies interactivity using a deep learning model which can suffer from the same robustness issues. More generally, algorithmically defined importancc/rellevnnce or interaction scores can be unreliable, especially in scenes with complex interactions between the AV and surrounding agents. In this work, we human labeling decide which agents are important,
3

---

Under review as a conference paper at ICLR 2023
and our motivation is to these labels to test model robustness. In the future, our causal agent labels can be used to verify algorithmic definitions of agent importance or relevance.
Causal reasoning in autonomous driving. In similar line of work, Ramanishka al. (2018) collect causal annotations using human labelers for the Honda Research Institute Driving Dataset and they slice performance of an object detector over scenes with varying causal attributes. Our work instead uses causal labels to evaluate model robustness for trajectory prediction on WOMD, and we provide more fine-grate ine-grained per-agent measurements of causality.
3 METHODS
3.1 LABELING CAUSAL AGENTS IN WOMD
The objective of the labeling task is to identify all agents moss--arrn cars, cyclists, pedestrians that are causal the AV at any time during a driving segment. Although we are more interested in removing non-causal agents from each scene, we ask labelers to identify causal agents since there are typically fewer of them and they tend to be closer to the AV, making them easier for labelers to identify.
Data. We focus labeling the WOMD validation data because our primary goal is evaluate the robustness models trained on the original dataset. Each example in WOMD is 9.1 seconds in length (91 steps at 10Hz) and is generated in overlapping windows from 20-second video segment. We label the 20-second segments to give labelers access to longer time horizon and to not waste resources on labeling overlapping scenes. Moreover, both the regular and interactive WOMD validation sets are generated from the same 20-second segments of data, hence, our causal labels can be used for both.
Figure Camera images from randomly chosen scene the labeling UL. The causal agents are circled.
Labeling policy and UI. Causality is an inherently subjective label since human drivers may vary in their judgements of which agents in the scene affect their decisions. Therefore, want to be overly conservative and identify as many causal agents as possible to maximize the likelihood that removed agents are actually non-causal. If human labelers we instruct them to include it as causal. We emphasize that false positives (identifying an agent causal when it is truly non-causal) are acceptable to certain extent, but we should avoid false negatives (failing to identify a truly causal agent). (Appendix A includes the exact instructions given labelers.) That said, in ambiguous situations, we did not expect labelers to reason about chained causality relationships. For example, if the AV is driving behind a queue of cars and the first car were brake, it could eventually cause the car front of the AV to brake. In this situation we would only expect the labeler to identify the car directly in front as causal.
unsure if agent is causal not,
The labeling UI a web-based 3D view of the AV and its surroundings in the 20-second segmented videos. An example is shown in Fig. where the camera images from randomly selected scene overlaid with the causal annotations provided by the human labelers.
Human annotations. To maximize coverage and avoid false negatives, each scene is annotated by 5 human labelers and we designate causal agents as all agents that any labeler identified as causal, Appendix shows the distribution over causal agents for the number of human labelers who selected the agent as causal. The majority of causal agents are selected by all labelers, but a significant portion (24%) are selected by only labeler.
3.2 CAUSAL AGENT STATISTICS
To understand the properties of causal agents, we compute several statistics of causal agents in the WOMD validation dataset, including the percentage of causal agents (Figure 3a), the distribution of the relative distance between the AV and the causal agents versus all surrounding agents (Figure 3b), and the breakdown of causal versus all surrounding agents by agent type (vehicle, pedestrian, or cyclist) (Figure 3c). Figure 3a shows that the majority of agents are actually non-causal: on average,
only 13% of the total agents in the scene labeled causal, and 93% of scenes have less than 30%
4

---

Under review as a conference paper at ICLR 2023
33%
17.5%
25000
agentis
% Causal Agents
Num agents
Causal agents
20000
12.5%
-5000
7.5%
10000
i B
11%
2.5
5000
0.0%
20%
40%6
6075
50%
200%
20
% Agents Labeled Causal
Distance (m)
Vehicle
Ped.
Cyclist
(a) Causal agent frequency
(b) Distance from AV
(c) Agent type
Figure 3: Causal agent statisties. Causal agents are less frequent than non-causal agents (on average 13% of
agents are causal), and, compared to typical agents, they tend to be closer to the AV. Cyclists are relatively more likely to be causal agents than pedestrians vehicles.
of the agents labeled as causal. Figure 3b shows that causal agents are typically closer to the AV than non-causal agents; causal agents are average distance of 28.4m from the AV, compared to an average of 49.4m over all agents. Figure 3c shows the likelihood that an agent of a given type (Vehicle, Ped, Cyclist) is causal. Surprisingly, cyclists are more likely to be causal agents than any other agents, and vehicles are more likely to be causal than pedestrians. We hypothesize that this is because cyclists usually share the road with the AV and have strong prior of not respecting road boundaries like a car, whereas there are many parked cars that are not necessarily interacting with the AV and similarly pedestrians be off the road on sidewalks.
3.3 PERTURBED DATASETS
In this work, we consider perturbations that modify the scene by deleting agents. While is possible to create more complex perturbations, such as adding noise the xyz position of the agents, start with deletion since it directly reflects the models' robustness regarding the causal relationships of agents in the scene. Object track states in the WOMD consist of the object's states (e.g.. 3D center point, velocity vector, heading), as well as valid flag to indicate which time steps have valid measurements. To delete an agent from the scene, we set its valid mask false throughout all time steps (and we double check for each model implementation that all agent state is ignored if the valid bit false). We consider four different perturbations:
1. RemoveNoncausal: Removes all non-causal agents in the dataset.
2. RenoveNonccausaEquul. Removes an equal number of randomly selected non-causal agents there are causal agents in the scene. For example, if scene has 5 causal agents, we randomly remove non-causal agents. RemoveNoncaussalquull is meant to be a less aggressive form of RemoveNoncausal since it deletes fewer agents and it allows to compare to RemoveCausal when controlling for the number of agents deleted.
3. RemoveStatic: Removes agents whose xyz positions do not change above a certain thresh- old (e.g. parked cars). We use threshold of 11 m on the L2 distance of the agent's 's xyz state account for sensor noise. Not all static agents are non-causal.
4. RemoveCausal: Removes all causal agents; the complement of RemoveNoncausal.
Among them, we categorize both RemoveNoncausal and RemoveNoncaaualEEqua as "non-causal" perturbations. Specifically, to define non-causal perturbations, let us assume X is scenario represen- tation, Y is the ground truth trajectory of the AV, and f is the ground-trum model that gives the relation- ship between X and Y. If perturbation AXX satisfies f(X AX) fXX Y, define it non- causal perturbation since it does not impact the relationship between X and Y. We define a deep learn- ing model f to be robust to non-causal perturbations if f(X + +AX) f(x) ) = A non-causal AX, where Y is the predicted trajectory from the model. Additionally, we consider RemoveStatic as an important baseline that does not require the human labels. can thus apply it to the training dataset, which we explore in Section 4.3, Finally, include the RemoveCausal perturbation as sanity to ensure models are sensitive to deleting causal agents.
3.4 EVALUATION
Since we only have camera and LiDAR data from the AV perspective, we only collect causal labels and evaluate model predictions for the AV trajectory. We report the average minADE (Ettinger et al., 2021), or minimum Average Displacement Error, which computes the L2 norm between the ground truth trajectory and the model's closest prediction, over 3, 5 and 8 seconds both the original
5

---

Under review as a conference paper at ICLR 2023
and perturbed datasets. We measure minADE in units of meters. In all instances, we use the top 6 trajectories for each model (K=6).
Robustness Metrics. Since we found in our results that the perturbed minADE often improves for large fraction of the examples, averaging over examples cancels out some of the effects we would like to measure. Thus, we introduce metric measure the per-example absolute change in minADE:
Abs - - perturbed
original minnA
We report Abs(A), the standard deviation of Abs( A), and the the relative percentage change in Abs(A) with respect to the original minADE. Finally, since the ground truth may represent only one of several correct ways to drive, in Section 4.2 we also consider pairwise differences between the original and perturbed predictions to measure model sensitivity.
Model
Architecture
Coord. System
# Params
MultiPath++
LSTM factorized attention transformer early fusion attention transformer
agent-centric global agent-centric
125M
Scene Transformer
15M
Wayformer
42M
Table 1: We evaluate on diverse set of models.
The IoU-based metric. The loU-based trajectory metric is computed as follows: given two predicted trajectory sets (with and without perturabtion), we first upsample all predicted trajectories (6 of them in each set) to 100Hz, and then voxelize them into 2D top down grid with resolution of 0.5 meters. We then count the number of voxels both sets occupy, divided by the total number of voxels either output set occupies. To simplify computation, we explicitly ignore the probabilities and speeds of trajectories. This measure quantifies "how geometrically different the trajectories look". An IoU of means the trajectories did not meaningfully change, and an IoU of O means the trajectories do not overlap at all. While more complicated versions of this metric could be computed (e.g. earth movers distance), found this metric intuitive and useful for finding interesting shifts due to perturbation.
Models. We select three representative deep learning models for evaluation: MultiPath++ (Varadara- jan et al., 2021), Scene Transformer (Ngiam et al., 2021), and Wayformer (Nayakanti et al., 2022). Importantly, we only consider non-ensembled models (Multipath++ reports ensemble results in their paper and the WOMD leaderboard). Table I reviews the architectural differences and parameter counts of the models. Since we only evaluate on the AV, we typically only train the models to predict the AV, but for MultiPath++ and CceneTransforme we also train models on all agents (which we indicate by appending -All to the model name). Additionally, for Scene TeeeneTranformer-eee both the marginal and joint models (these models are the same when training on only the AV.)
we include
4
RESULTS
4.1
MODEL SENSITIVITY TO NON-CAUSAL PERTURBATIONS
5
RemoveCausal
RemeveNoncausa
RemnaveNorcausaliquaa
RemoveStatic
10a
Perturbed MINADE
Perturbed minADE
Perurbed minAGE
Perturbed minADE
+
10x
2
N
0
1 3
Original minADE
Original minADE
Original minADE
Original minADE
Figure 4: Model sensitivity to different perturbation types. We plot the per-example perturbed versus original minADE for all perturbations for the MP+ model. The example frequency is shown with log color scale where yellow is the most frequent. The majority of examples show minimal change from the perturbation and lie close to the y=v axis. However, across all perturbation types, there long tail of examples that show relatively large change in minADE(> 1m). Surprisingly, even for the RemoveCausal perturbation, the model performance often improves on the perturbed examples. Comparing RemoveNoncausal and MaanoveNoncausaEEquee indicates that the model is more sensitive to removing larger numbers of non-causal agents).
In order to understand model sensitivity on a per-example level, Figure 4 plots the perturbed versus original MinADE across each perturbation for MultiPath++ (see Appendix H for other models). For
6

---

Under review as a conference paper at ICLR 2023
each perturbation type, we observe that the majority of examples show minimal change (i.e. clustered around the y=z axis), but a long tail of outlier examples experience a large change (>1m in minADE). Among perturbation types, the model is most sensitive to RemoveCausal, which is expected since removing causal agents can change the correct ground-truth trajectory. Interestingly, models are significantly more robust GemoveNonccausllEqua than RemoveNoncausal, which means removing more agents increases model sensitivity. When comparing RemoveCausal and GemoveNoncauusalEgual,, which controls for the number of agents removed, we see that the model is significantly more sensitive to removing causal agents than removing non-causal agents.
Surprisingly, across all perturbation types, including RemoveCausal, the model sees a large portion of examples where minADE improves (i.e. the minADE itself becomes smaller under the perturba- tion): 42.7% of examples show an improvement under the RemoveCausal perturbation, 43.0% for RemoveNoncausal, 49.6% for RemoveNoncausaa is counter-intuitive and motivated us to measure model sensitivity in terms of Abs(A), defined in Equation 1. Across all models, the average Abs( A) is 0,1450 for RemoveCausal, 0.131 for RemoveNonncausal 0.051 for CemoveNoncausaluqque reports Abs( A) for each individual model and perturbation type.
and 51.0% for RemoveStatic. This finding
and 0.089 for RemoveStatic. Appendix C
Comparing models. Focusing on RemoveNoncausal, in Table 2, we evaluate each model and report the original minADE, perturbed minADE, Abs(A), the standard deviation of Abs(A), and Abs(A) (see Appendix C for other perturbations.) Scene TranfTrrafrrmmrr Marginal shows the lowest average absolute sensitivity, while MultiPath++AAll shows the lowest sensitivity relative to original minADE. In general, Abs(A) decreases with the original minADE, but there is no clear relationship between relative Abs(A) and minADE. Unexpectedly, the marginal Scene Transformer is more robust than the joint (we hypothesize that jointly modeling agents in the scene causes the model to pay more attention to non-causal agents). In Appendix G. we report minFDE, overlap rate, miss rate, and mAP.
minADEoi
Model comparison, RemoveNoncausal, minADE, = Perturbed - Original
Model
Original
Perturbed
Abs(A)
Std. Abs(A)
minADEor. (%)
Abs(A)
MultiPath++
0.376
0.395
0.141
10.21
37.5%
Scene Transforofrr Marginal
0,250
0.265
0.067
#0.12
26.8%
Wayformer
0.393
0.406
0.101
10.16
25.7%
MultiPath+++All Trenefranfoommr-AA1 Joint
0.900
0.945
0.226
#0.32
25.1%
0.493
0.504
0.170
+0.26
34.5%
CceneTransformer-AAl Marginal
0.305
0.328
0.081
+0014
26.6%
Table 2: Model sensitivity to the RemoveNoncausal perturbation. The SceneTransformer Marginal model shows the lowest average absolute sensitivity to the perturbation, while the MultiPath+++AI model shows the lowest sensitivity relative to original minADE. Original and Perturhed are the average minADE across the whole dataset. Abs( A) is the average per-example absolute difference between perturbed and original minADF.
Slicing the robustness metric. We further slice the robustness the models (Abs(A)) along several dimensions: AV's current speed, the percentage of removed non-causal agents (the number of removed non-causal agents divided by the number of all context agents), and the minimum distance from the AV to removed non-causal agents. The full results are given in Figure 12 in Appendix E. We see that, across all models, model sensitivity increases when we drop larger fraction of non-causa agents and when the speed of the AV is greater. We also see that model sensitivity typically decreases when we drop agents that are farther away from the AV, though the Scene Transformer models have much noisy robustness measurements when dropping far away agents.
Visualizing examples. We also visualize some examples with the largest output changes under the RemoveNoncausal perturbation in Appendix F.. The findings are discussed in Section 5.
4.2 SENSITIVITY VIA AN IOU-BASED TRAJECTORY SET METRIC To directly measure the magnitude of model output changes with and without perturbation, we use an loU innerreetion-over--niioo to different perturbations. The results of three AV-only models under perturbations RemoveCausal, RemoveNoncausal and RemoveNoncaasallRuull are shown in Figure We find that models are least sensitive to RemoveNoncausaa causal. This is consistent with our finding in Section 4. indicating the model is more sensitive to large perturbations since there are more non-causal agents than causal ones in most examples.
based metric (defined in Section 3.4) to compare model sensitivity
and much more sensitive to RemoveCausal and RemoveNon-
7

---

Under review as a conference paper at ICLR 2023
RempreCausal (mie) i 0,75)
RemoveCCusal (miou: 74)
RemoveCausal Imiel - 0.69)
Remevctioncuuall imtoÚ 0.22)
Remrove nemooveioocauall (iioO - 0.78)
Removeltencaush (miol i 0.83)
Rempvetionneueee (mlol)
Memovelioneeuseee.uu omiou 0.931
RamoveNontouaaeBuu ImioU 0.861
Probabiity Densty
Probeeiiyy Danaky
Probaciily
0.4 Number of Scenes
0.0
0.2
0.3
1.0
0.0
0.2
0.4
0.6
Number at Scenes
Number of Scenes
(a) Scene Transformer Marginal
(b) Wayformer
(c) Multipath++
Figure 5: Density distribution the per scene trajectory set IoU values AV-only models under perturbations (RemoveCausal, RemoveNoncausal. and RemoveNoncausalEqual models are least sensitive to RemoveNon- causalFqual, and more sensitive to RemoveCausal and RemoveNoncausal.
4.3 TRAINING WITH DATA AUGMENTATIONS IMPROVES MODEL ROBUSTNESS
We experiment with two types of data augmentation: 1) data augmentations that heuris- tic definition of non-causal agents, such as randomly dropping any static context agent', and 2) robustness- stareeted data augmentations that directly drop only non-causal agents using labeled portion of the val set.
Heuristic data augmentation. The benefit of using heuristic definition of non-causal agents for data augmentations is that it can be applied without collecting causal labels. We implement 2 types of heuristic- based data augmentation in the training set of WOMD: Drop Context (randomly dropping context agents) as baseline, and Drop Static Context (randomly dropping static context agents). We use the MultiPath++ model and we set the probability of dropping an agent to 0.1 (the best one among 1, 0.5, and 0.8). Table 3 summarizes the results for the RemoveNoncausal perturbation (for the per-scene distribution of model sensitivity, see Appendix K). Models with data augmentation show less sensitivity to the perturbations, and, in particular, Drop Static Context shows significant improvement minADE and Abs( over Drop Context. We hypothesize that Drop Static Context does better because the static context agents are less likely to be causal. Overall, the results for Drop Static Context imply that dropping non-causal agents via data augmentation in training can improve model robustness to such perturbations test time.
Heuristic Data Augs, RemoveNoncausal, minADE A = Perturbed Original
Model
Original
Perturbed
Abs(A)
Std. Abs(A)
minADEga (%)
Abs/
MP++-AIl
0.900
0,945
0.226
#0.32
25.1%
MP++-All Drop Context
0.948
0.988
0.209
#0.31
22.0%
MP++-All Drop Static Context
0.819
0.837
0.183
+0.26
22.3%
Table 3: Heuristic data augmentations. We compare the MP++-All baseline model to the same model trained
with either dropping context agents or dropping static context agents, finding that data augmentations that drop agents that are more likely to be non-causal can improve robustness.
Non-causal data augmentations. Motivated by our results that dropping static context agents improves model robustness, we further explore using non-causal perturbations as data augmentation strategy during training. We randomly sample approximately 70% of the original validation dataset (i.e. 30k scenes), perturb multiple copies of them via the causal labels, and add the perturbed versions into the training dataset. We leave the remaining 30% of the validation set as a holdout for evaluation. We then train baseline model on the new training dataset as well non-causal agents (when possible) with probability 0.1. In Table 4, we see that similarly dropping
model that randomly drops
non-causal seents Angs, &e mID eoncaauaal as mood cobustins Srrrrrrbee Original
Model
Original
Perturbed
Abs(A)
Std. Abs(
Absl.))
(%)
minADFoi
MP++ Baseline
0.395
0.408 0.389
0.150
+10.226
38.0%
MP++ Drop Non-causal
0.373
0.138
1.4.194
37.0%
Table Noncausal data augmentation. We fold a portion of WOMD validation dataset into the original training dataset and apply data augmentations that drop non-causal agents. On held-out validation data, we find significant improvements in model robustness across all three Abs(A) metrics.
Context agents agents for which prediction is required in the WOMD leaderboard.
8

---

Under review as a conference paper at ICLR 2023
4.4 LARGER DATASET SIZE IMPROVES MODEL ROBUSTNESS
We also evaluate model robustness with increasing training data. We randomly select 10%, 20%, 50%, 80% of the training dataset and train separate models on each split. We sample three datasets for each split and average the performance and robustness of each model. Appendix D summarizes the results. As we increase the training data size, the model performance improves (minADE decreases) and both the absolute and relative robustness improves. Interestingly, previously when varying model architectures, we found that the model with the lowest minADE did not always have the best relative robustness. Here, we see a strong trend: for fixed model architecture, lowering the minADE by increasing the training data results in lower relative sensitivity.
5 DISCUSSION
We now discuss few hypotheses and initial supporting evidence for why models are not robust to the non-causal perturbations.
Overfitting. One reason models may fail to generalize to the non-causal perturbations is that they overfit to spurious correlations in the training data (i.e. features that correlate with certain ground truth trajectories but fail to generalize). In our experiments, we observe that models that overfit on the original training dataset (as measured by increasing minADE on the original validation dataset) more sensitive to the non-causal perturbations (see Fig. 11 in Appendix E). Thus, the more the model overfits to spurious features like the number of parked cars in a faraway parking lot, the less well it generalizes to examples where these features are absent. Data augmentation and increasing dataset size may improve robustness by protecting against overfitting.
Distribution shift. Models may fail generalize to perturbations that are significantly different from any data seen during training. In our results, we observe that the more non-causal agents we remove, the less robust models are. Perhaps certain types of scenes with few agents are relatively rare in the training dataset and the model does not generalize well to the distribution shift. By evaluating on the perturbations, we essentially expose the model to rare scenarios not seen in training. One reason that training with data augmentations via dropping (static) context agents or non-causal agents improves robustness could be that it exposes the model to similar scenes during training.
Over-reliance on agents instead of roadmap. A third possible reason that models fail to generalize is that they utilize the non-causal agents to infer the drivable areas instead of using the mapping information in the input (we serve high-definition maps and traffic control signals imput features for all models). Our evidence comes from visualizing examples where dropping non-causal agents creates predictions that disobey the roadgraph rules (see Fig. 10 in Appendix E).
Data-dependent modes. Finally, many of the state of the art models (e.g. (Ngiam et al., 2021; Varadarajan al., 2021)) utilize modes (e.g. straight, left, u-turn, etc.) learned from the data distribution, where the input data influences how the model will utilize its K predictions to minimize its loss function. While effective at minimizing minADE-like metrics, these methods provide no coverage guarantees, and can encourage the model to predict multiple speed profiles for the same mode instead of diverse modes. When we triage examples (see Fig. in Appendix E), we find some of the largest failures come from agent deletions that influence which modes the model predicts, demonstrating weakness of this approach and the metrics they perform well on.
6 CONCLUSIONS
We establish a benchmark and metrics for evaluating the robustness of several state-of-the- models for trajectory prediction for autonomous driving. We find that most tfate-of-the-art models (with different model architectures and coordination systems) show significant levels of sensitivity to perturbations that remove non-causal agents, with higher sensitivity when removing greater number of them. While most examples show minimal change in minADE (< 0.1 m), there is long tail of examples that can have large changes (2 Im and sometimes up 8m). Surprisingly, removing either causal or non-causal agents can cause a significant fraction of examples to improve their minADE. We also find that increasing dataset size and data augmentation can help improve the model robustness. Overall, our results indicate that current machine learning models for trajectory prediction may not be reliable enough on their own, and careful thought needs to be given to how to integrate such models with non-learning components to make safe system. Finally, we will publish the causal agent labels as complementary attributes to the WOMD to aid future researchers in building more robust models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ekrem Aksoy, Ahmet Yazıcı, and Mahmut Kasap. See, attend and brake: An attention-based saliency map prediction model for end-to-end driving. arXiv preprint arXiv:2002.11020, 2020.
Aniket Bera, Sujeong Kim, Tanmay Randhavane, Srihari Pratapa, and Dinesh Manocha. Glmp- realtime pedestrian path prediction using global and local movement patterns. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 5528-5535, IEEE, 2016.
Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning.
Pattern Recognition, 2018.
Battista Biggio, Igino Corona, Davide Maiorea, Blaine Nelson, Nedim Sındić, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 382-402. Springer, 2013.
Yulong Cao, Chaowei Xiao, Anima Anandkumar, Danfei Xu, and Marco Pavone. Advdo: Realistic adversarial attacks for trajectory prediction. In European Conference on Computer Vision, pp. 36-52. Springer, 2022a.
Yulong Cao, Danfei Xu, Xinshuo Weng, Zhuoqing Mao, Anima Anandkumar, Chaowei Xiao, and Marco Pavone. Robust trajectory prediction against adversarial attacks. arXiv preprint arXiv:2208. 000044 2022b.
Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urtasun. Spagnn: Spatially-aware graph neural networks for relational behavior forecasting from sensor data. In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 August 31, 2020, pp. 9491-9497. IEEE, 2020.
Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction. In Conference on Robot Learning, 2019.
Henggang Cui, Vladan Radosavijevic, Fang- Chich Chou, Tsung Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djuric. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In 2019 International Conference on Robotics and Automation (ICRA), pp. 2090-2096. IEEE, 2019.
Fang-Chich
Kaustubh Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, et al. NI-augmenter: A framework for task-sensitive natural language augmentation. arXiv preprint 11222.21 2021.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. In International Conference Machine Learning, pp. 1802- 1811. PMLR, 2019.
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, et al. Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9710-9719, 2021.
Alhussein Fawzi and Pascal Frossard. Manitest: Are classifiers really invariant? arXiv preprint
arXiv: 1507006335 2015.
Robert Geirhos, Carlos RM Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. Advances in neural information processing systems, 31, 2018.
Keren Gu, Brandon Yang. Jiquan Ngiam, Quoc Le, and Jonathon Shlens. Using videos to evaluate image model robustness. arXiv preprint arXiv: 1901.994000000 /0076, 2019.
10

---

Under review as a conference paper at ICLR 2023
Yutao Han, Rina Tse, and Mark Campbell. Pedestrian motion model using non non- par-metraaacc trajectory clustering and discrete transition points. IEEE Robotics and Automation Letters, 4(3):2614-2621) 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
Joey Hong, Benjamin Sapp. and James Philbin. Rules of the road: Predicting driving behavior with a convolutional model of semantic interactions. In CVPR, 2019.
Manh Huynh and Gita Alaghband. Trajectory prediction by coupling scene-lstm with human movement Istm. In International Symposium on Visual Computing, pp. 244-259, Springer, 2019.
Siddhesh Khandelwal, William Qi, Jagjeet Singh, Andrew Hartnett, and Deva Ramanan. What-if motion prediction for autonomous driving. arXiv preprint arXiv:2008. 1000:0008.10887, 2020.
Julian FP Kooij. Fabian Flohr, Ewoud AI Pool, and Dariu M Gavrila. Context-based path prediction for targets with switching dynamics. International Journal of Computer Vision, 127(3):239-28) 262, 2019.
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 336-345, 2017.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting. In European Conference on Computer Vision (ECCV), 2020.
Osama Makansi, Özgün Cicek, Yassine Marrakchi, and Thomas Brox. On exposing the challenging long tail in future prediction of traffic actors. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 13147-13157, 2021.
Jean Mercat, Thomas Gilles, Nicole Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Gil. Multi-head attention for joint multi-modal vehicle motion forecasting. In IEEE International Conference on Robotics and Automation, 2020.
Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. arXiv preprint MrXiv:2207.05844 2022.
Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao- Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified multi-task model for behavior prediction and planning. arXiv e-prints, pp. arXiv-2106. 2021.
Noha Radwan, Wolfram Burgard, and Abhinav Valada. Multimodal interaction-awarr motion prediction for autonomous street crossing. The International Journal of Robotics Research, 39(13): 1567-1598, 2020.
Vasili Ramanishka, Yi-Ting Chen, Teruhisa Misu, and Kate Saenko. Toward driving scene under- standing; A dataset for learning driver behavior and causal reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7699-7707, 2018.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? International Conference on Machine Learning, pp. 5389-5400, 2019.
Khaled S Refaat, Kai Ding, Natalia Ponomareva, and Stéphane Ross. Agent prioritization for autonomous navigation. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2060-2067. IEFE, 2019.
11

---

Under review as a conference paper at ICLR 2023
Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction con- ditioned on goals in visual multi-agent settings. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2821-2830, 2019.
Saeed Saadatnejad, Mohammadhossein Bahari, Pedram Khorsandi, Mohammad Saneian, Seyed- Mohsen Moosavi-Dezfooli, and Alexandre Alahi. Are socially-aware trajectory prediction models really socially-aware Transportation research part C: emerging technologies, 141:103705, 2022.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone, Trajectron++:: Dynamically- feasible trajectory forecasting with heterogeneous data. arXiv preprint arXiv:2001.03093 2020.
Christoph Schöller, Vincent Aravantinos, Florian Lay, and Alois Knoll. The simpler the better: 5(6):7, 2019.
Constant velocity for pedestrian motion prediction. arXiv preprint
Vaishaal Shankar, Achal Dave, Rebecca Roelofs, Deva Ramanan, Benjamin Recht, and Ludwig Schmidt. Do image classifiers generalize across time? arXiv preprint arXiv:1906.022 0200000268, 2019.
Liting Sun, Xiaogang Jia, and Anca D Dragan. On complementing end-to-end human behavior predictors with planning. In Proceedingg of the Robotics: Science and Systems, 2021.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2013.
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18599 2020.
Ekaterina V. Tolstaya, Reza Mahjourian, Carlton Downey, Balakrishnan Varadarajan, Benjamin Sapp, and Drago Anguelov. Identifying driver interactions via conditional behavior prediction. 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 3473-3479, 2021.
Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S Refaat, Nigamaa Nayakanti, Andre Cormman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, et al. Mul- tipath++: Efficient information fusion and trajectory aggregation for behavior prediction. arXiv preprint arXiv:2111.149733 2021.
Jingkang Wang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9909-9918, 2021.
Qingzhao Zhang. Shengtuo Hu, Jiachen Sun, Qi Alfred Chen, and Z Morley Mao. On adversarial robustness of trajectory prediction for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15159-15168, 2022.
Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang. and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In CVPR, pp. 22126-12134 2019.
12

---

Under review as a conference paper at ICLR 2023
**For review purposes, provide copy of the causal labels annotations in the supplementary materials. The causal agent labels are released as a TFRecord of causal labels protos (see the file 'causal labelproto" in the supplementary material). The proto maps scenario id to labeler id to a list of agent ids identified by that labeler.
13

---

Under review as a conference paper at ICLR 2023
A
LABELING POLICY
Below is the exact text given to labelers to define causal agents:
The objective is identify all agents cars, cyclists, pedestrians that are causal to the AV at any time. A causal agent is one whose presence would modify or influence human driver behavior in any way. Causality is an inherently subjective label. If you are unsure if an agent is causal or not, please err on the side of including false positives (identifying an agent as causal when it is truly non-causal) are okay, but we should avoid false negatives (failing to identify truly causal agent).
other words,
If the behavior of a human driver would be modified because of a poten- tial action that agent is likely to take, then that agent should be causal. On the other hand, if the human driver would drive the same regardless of whether the agent is there not, the agent is non-causal.
The labeling policy also included several examples scenarios with causal agents identified such as Figure 6.
Figure 6: Example from the labeling policy. Causal agents are circled in green and a subset of non-causal agents are circled in red.
14

---

Under review as a conference paper at ICLR 2023
B LABELER AGREEMENT STATISTICS
41%
35
30
de Agens
25
24%
20
15
14 %
11%
10%
10
1
2
5
Num labelers who selected agent
Figure 7: Labeler agreement statistics. We plot the distribution of labeler agreement (i.c. number of labelers who selected given agent) across agents the WOMD validation set. The majority agents are selected by more than one labeler.
C MODEL SENSITIVITY TO VARIOUS PERTURBATION TYPES
In this section, we summarize the robustness metrics across different model architectures for each of the perturbation types (RemoveCausal, RemoveNoncausal, RemoveNoncaausilEqual RemoveStatic). We report the model's original minADE, perturbed minADE, average absolute difference between perturbed and original minADE computed per-example (Abs(A) standard deviation of Abs( and the relative % change (Abs(A) divided by the original minADE). Each table below shows the results for different perturbation dataset.
RemoveCausal minADE,
Perturbed Original
Model
Original
Perturbed
Abs(A)
Std. Abs(A)
Abs(A) (%)
MP++
0.376
0.425
0.153
=0.55
40.6%
ST Marginal
0.250
0.272
0.068
10.14
27.
Wayformer
0.393
0.423
0.122
=0.20
31.1%
MP++-AIl
0,900
0.968
0.231
10.34
25.7%
ST-All Marginal
0.305
0.341
0.091
10.16
29.7%
ST-All Joint
0.493
0.540
0.207
00.1
42.0%
Table 5: Model sensitivity for RemoveCausal, min
RemoveNoncausal minADE, A = Perturbed Original
Model
Original
Perturbed
Abs(A)
Std. Abs( ()
AbslA) (%)
On
MP++
0.376
0.395
0.141
+0.21
37.4%
ST Marginal
0.250
0.265
0.067
+0.12
26.8%
Wayformer
0.393
0.406
0.101
-0.16
25,7%
MP++-All
0,900
0.945
0.226
=0.32
25.1%
ST-AIl Marginal
0.305
0.328
0.081
=0.14
26.5%
ST-AIl Joint
0.493
0.504
0.170
+0.26
34.5%
Table Model sensitivity for RemoveNoncausal, minADE.
15

---

Under review as a conference paper at ICLR 2023
RemoveNoncaausalRqual minADE, A
Perturbed Original
Model
Original
Perturbed
Abs(A)
Std. Abs(A)
AbslA) (%
On
MP++
0.376
0.378
0.062
10.12
16.4%
ST Marginal
0.250
0.252
0.023
=0.55
9.3%
Wayformer
0.393
0.395
0,042
10.10
10.6%
MP++-All
0.900
0,907
0.103
=0.20
11.5%
ST-AIl Marginal
0.305
0.308
0.025
+0.05
8.2%
ST-AIl Joint
0.493
0.495
0.051
#0.11
10.3%
Table 7: Model sensitivity for RemoveNoncausallgual, minADE.
RemoveStatic minADE. A = Perturbed Original
Model
Original
Perturbed
Abs(A)
Std. Abs(
AbslA) (%)
On 25.0%
MP++
0.376
0.387
0.094
+0.17
ST Marginal
0.250
0.249
0.043
+0.07
17.3%
Wayformer
0.393
0.400
0.054
-0.12
13.9%
MP++-AIl
0.900
0.927
0.161
10.23
17.9%
ST-All Marginal
0.305
0.291
0.063
=0.08
20.8%
ST-All Joint
0.493
0.462
0.118
=0.18
24.0%
Table Model sensitivity for RemoveStatic, minADE.
16

---

sensitive removing Under review as a conference paper at ICLR 2023
To make it easier to compare across perturbation types, we also report the average Abs(Perturbed. Original) minADE for each model and perturbation type in Table 9.
Abs(Perturbed Original) minADE
Model
R.Causal
R.Noncausal
R.NoncausalEEual
R.Static
MP++
0.153
0.141
0.062
0.094
ST Marginal
0.068
0.067
0.023
0.043
Wayformer
0.122
0.101
0.042
0.054
MP++-All
0.231
0.226
0.103
0.161
ST-All Marginal
0.091
0,081
0.025
0.063
ST-All Joint
0.207
0.170
0.051
0.118
Average
0.145
0.131
0.051
0.089
Table 9: Abs(Perturbeed-Orignnal across different perturbation types and models. report the average absolute difference between the per-exampll perturbed and original minADE for each model and perturbation type. The model sensitivity for RemoveCausal and RemoveNoncausal is similar, with RemoveCausal resulting in slightly larger average absolute change. However, when control for the number of agents and compare RemoveCausal to RemoveNoncausalEquu, causal agents than non-causal agents.
that the model significantly
17

---

Under review as a conference paper at ICLR 2023
D INCREASING DATASET SIZE
Train %, minADE, RemoveNoncausal, A = Ptb Ori
Train(Si
Original
Perturbed
Abs(A)
Std. Abs(A)
Abs(A) (%)
On
10%
1.222
1.309
0.448
+0.69
37.0%
20%
1.039
1.117
0.386
+0.53
37.2%
50%
0.947
0,996
0.266
10.45
28.0%
80%
0.901
0,925
0,236
=0.32
26.2%
100%
0.900
0.945
0.226
#0.32
25.1%
Table 10: Increasing training data improves robustness.
18

---

Under review as a conference paper at ICLR 2023
E EXAMPLE VISUALIZATION
E FAILURE (NON-ROBUST) CASES UNDER NON-CAUSAL. PERTURBATION
We have triaged several top sensitive examples under the RemoveNoncausal perturbation. Among these examples, we have found three failure patterns: 1) predictions under the perturbation violate traffic rules, as shown in Figure 8; 2) predictions under the perturbation missed to capture the ground- truth mode, shown in Figure 9; and 3) predictions under the perturbation violates the causality, for instance, unnecessary slows down when the road becomes more empty due to the removal of non-causal agents, such as the bottom example in Figure 11. Meanwhile, we also have identified examples where the predictions under the non-causal perturbation becomes better, as shown in Figure 10.
ground truth AV prediction truth
groundi truth AV crediction ground truth
T0M
Figure sensitive example from MP++ under non-causal perturbation: side inference on the original validation data, and right side is inference on the RemoveNNonausal data, where all non-causal agents are removed from the scene, but some of predicted outputs weirdly turn right in the middle of straight road.
trg
AV ground truth AV prediction
ground truth
prediction
Agent ground truth
Figure 9; An example from marginal Scene Transformer under non causal perturbation: Left side is inference the original validation data, and right side is inference the RemoveNoncausal data, where all non causal agents removed from the scene. It shows where the model performed worse under perturbation, entirely missing correct mode.
19

---

Under review as a conference paper at ICLR 2023
ground
ground
prediction Agent ground truth
AV prediction Agent ground truth
Figure 10: sensitive example marginal Scene Transformer under non non-causal perturbation: Left side inference on the original validation data, and right side is inference on the RemoveNoncausee data, where all non-causal agents are removed from the scene. The model outputs under perturbation actually improved by capturing the ground-truth mode. The original model missed mode where it should have driven forward, potentially because of spurious correlation with the non-causal agent front of it. After removing that (and other agents), correctly predicted that mode. However, there one mode showing too wide right turn under the perturbation, which is highly unlikely in human driving. This might due to the removal of the static agents from the cross traffic confuses the model about the drivable area.
E.2 AN EVIDENCE EXAMPLE FOR XON-ROBUSTNESS DUE TO OVERFITTING
In this section, show example scenario that showcases overfitting is one potential reason for poor robustness. We have trained the MP++ with 1M iterations, which overfitted at 210k iterations. We then visualize the predictions of iteration and another at IM. The results are shown in Figure 11. We can see that the robustness of the model under non-causal perturbation becomes bad when the model over-fits.
example with two different checkpoints,
at 210k
20

---

Under review as a conference paper at ICLR 2023
ground AV pred ction Agent ground
ground
AM predi ction
Agent ground truth
ground truth
ground truth Agont ground
Agont ground truth
300
20/0
I0b0 20/10 B110
Figure 11: An example from MP++ indicating over-fitting is one possible reason for poor robustness: Left side is inference the original validation data, and right side is inference on the RemoveNoncaussa data, where all non-causal agents removed from the scene. The top shows the performance of the model 210k iteration, while the bottom row on the validation set. We can see that unnecessarily slows down. Note that in this plot, we only visualize the top-1 predictions for hetter visualization. We also only visualize the predictions of the AV in the bottom row.
that IM where observe over-fitting based on minADE iteration, the top-1 prediction under the non-causal perturhation
21

---

Under review as a conference paper at ICLR 2023
F SLICING RESULTS
We further slice the robustness of the models along several dimensions, including the AV's current speed, the percentage of removed non-causal agents (the number of removed non-noncausal agents divided by the number of all context agents) in the scenarios, and the minimum distance from the AV removed non-causal agents, Results are given Figure 12. We found that:
Along the percentage of removed non-causal agents, we found all models are more sensitive if larger fraction of agents are removed. Across them, Marginal and Wayformer are the most robust models. Compared to Wayformer, ST marginal is less sensitive when more than 40% of the context agents are non-causal and removed from the data (Figure 12 left).
Along the AV's speed, ST Marginal is more robust when the AV's speed is slower than 45mph (Figure top-left). Note that the high fluctuations at high speed (> 45mph) is because we have fewer examples there (the count of examples in each bin is provided in Appendix).
Along the minimum distance between the removed non-causal agents to the AV, Wayformer is the most robust onc, particularly when the minimum distance is larger (i.e., all removed non-causal agent relatively far away from the AV, Figure 12 right). Such results indicate that Wayformer learns to not pay too much attention to far-away non-causal agents. On the contrary, we noticed that ST models tend to be more sensitive far-away non-causal agents. We hypothesize that this might be because the global coordination that ST models are used makes it more sensitive large coordinate values.
RemoveMoonausel Agents
Perr Derrovelorccurrlloees
Remmeehorccauallieenn
0.22
6.30
AbsiPurturaed Drignall
0.20
6.25
0.00
Sbuieeuureed
6.3
30"
0.00
0.20
6.0
22
64
0.
8.0
1.3
10
50
20
(A of noncausal agents) 206 cost agents)
AV spreed Impga))
min distance of nancausal agents to AV (m)
MP++
+-I++AII
Marginal
ST-AIl Marginal
ST-All Joint
Pathformer
Figure 12: We slice the average Abs(Perturbed Original) minADE along i) ratio of removed non-causal agents context agents (left), ii) AV speed (mph) (middle), and iii) minimum distance (m) between the removed non cou-causal agents and the AV (right).
22

---

Under review as a conference paper at ICLR 2023
G AGGREGATE RESULTS FOR ALTERNATE METRICS
In this section, we report aggregate results for minFDE, overlap rate, miss rate, and mAP on the RemoveNoncausal perturbation.
minFDE, RemoveNoncausal
Model Name
Original
Perturbed
MultiPath++
0.853
0.895
Scene Transfrraform Marginal
0.487
0.516
Wayformer
0.848
0.885
MultiPth+++AAll
1.430
1,551
Transffroooll Joint
1.170
1.176
SceneTransformer-AN Marginal
0.622
0.674
Table 11: minFDE RemoveNoncausal results.
Overlap Rate, RemoveNoncausal
Model Name
Original
Perturbed
MultiPtth++
0.188
0.191
Scene Transfrranf Marginal
0.211
0.210
Wayformer
0.187
0.194
MultiPat++-AAll TreeforaanformerAl11 Joint
0.167
0.178
0.191
0.202
SceneTranssfrmer- All Marginal
0.198
0.206
Tahle 12: Overlap Rate RemoveNoncausal results.
Miss Rate, RemoveNoncausal
Model Name
Original
Perturbed
MultiPath++
0.064
0.067
Transfrrane Marginal
0.059
0.064
Wayformer
0,059
0,063
MultiPath++-All Trensfsraoormer-AA Joint
0.142
0.157
0.283
0.280
Scene Transformer-All Marginal
0.098
0.111
Table 13: Miss Rate RemoveNoncaussal results.
mAP, RemoveNoncausaa
Model Name
Original
Perturbed
MultiPath++
0.554
0.524
SceneTransformer Marginal
0.475
0.447
Wayformer
0.546
0.539
MultiPath++-All
0.268
0.243
Scene Tranerfoooollll Joint
0.227
0.223
SceneTrransformerAAll Marginal
0.395
0.367
Table 14: mAP RemoveNoncausaa results.
23

---

Under review as a conference paper at ICLR 2023
H
PER-EXAMPLE SCATTER PLOTS
RemoveCausal
RemoveNoncausa
RerneveNoncauaalqqua
RemoveStatic
10a
Perturbed MINACE
Pertubeed minADE
Perurbed minAGE
Perturbed minADE
4
10t
1
3
Original minADE
Original minADE
Orgiinal minADE
Original minADE
RemoveNoncause Figure 13: MP++
RemoveCaausal
Rereveeloncausaaaguu
RemoveStatic
10a
Perturbes mInADE
Perturbed minADE
Perurboe minADE
Perturbed mirADE
10a
0
1
3
4
Original
Original
Original
Original minADE
Figure
Marginal
10a
Removeeausal
RemoveNoncaüse
RemoveStatic
Perturbed mInADE
Perturbed mInADE
Perturbidd minADE
Perturbed minADE
4
1
3
Original
Original
Original
Original minADE
Figure 15: Wayformer
RernoveCausel
RemoveNoncausa
5
ReireveeNonnausalEgge
RemoveStattcc
10a
Perturbes MInADE
Pertrbed minADE
Perurbad minADE
Perturbed minADE
10*
0
1
3
Original
Original
Original minADE
Original minADE
Figure MP++-AI
RemeveCausal
Remove Moncaue
RemoveNNoncane auusaeel
RemoveSttatic
10a
Perturoee mInAGE
MInADE
Perturaed minAOE
Perturbed minADE
0
1
3
Original minADE
Original
Original minADE
Figure 17: ST-AIl Marginal
24

---

Under review as a conference paper at ICLR 2023
RemoveCausal
RemaoveNoncuuall
quinoveNoncaasallquaall
10a
Perturbed mInADE
Perturbed minADE
Perturbed minADE
counts
10t
0
0
1
3
Original minADE
Original minADE
Original minADE
Original minADE
Figure 18: ST-AIl Joint
I COMPARISON ACROSS MODELS
RemoveNoncausallApens
Nemoveeloncauss/oennttuuu
RemoveStaticAgents
Example Density
Examole Density
Example Density
15
15
15
10
10
10
5
0.2
-0.1
0.0
0.1
0.2
-0.2
-0.1
0.1 Original minADE
0.0
0.2
0
-0.1
0.0
0.1
0.2
Perturbed Original minADE
Perturbed
Perturbed Original minADE
MP++ MP++ All
ST Marginal
ST-All Marginal
ST-AIl Joint
Pathformer
per-scene difference hetween perturhed and original minADE for various models and perturhation types. Models that are less sensitive the perturbation have higher example density at O difference. All models show sensitivity to the non-causal perturbations, which either increase decrease the perturbed minADE relative the original minADE. The models are more sensitive to RemoveNoncaasallgeen than GemooeeooncuualAggnntttoo implying that removing more non-causal agents increases model sensitivity. Among the models, Wayformer and Scene Transformer Marginal show the least sensitivity to the perturbation.
Figure 19: Models are sensitive non-causal perturbations. We plot distribution
J COMPARISON ACROSS PERTURBATION TYPES
Figure 20 shows the sensitivity of the Wayformer AV Only, ST Marginal AV Only, and MultiPath++ All Agents models each of the perturbation types. The perturbation can either increase decrease the minADE. On average, it increase the minADE but this depends on the pertubation type (Remove- CausalAgents causes the strongest increase; see Appendix C). The models are most sensitive both perturbations.
the RemoveCausalAgeent and RemoveNonccausll/
The Aesaeeta has the largest effect on the model, producing the most outliers that increase the difference between the perturbed and original minADE, followed closely by RemoveNoncausalA: Surprisingly, the sensi- is close to that of RemoveCausal (exact numbers TODO).
gents, then RemoveStaticAgente and then LammoveeooncasallAgensEE.aa tivity of MaanoveNoncausalAgeee However, when we change the number non-causal agents removed (in RemoveNoncausalAgensse qual) to be the same the number of causal agents removed (in RemoveCausalAggmn the sensitivity much less.
K DATA AUGMENTATIONS
25

---

Under review as a conference paper at ICLR 2023
Wayformer SDC Only
ST Marginal SDC Only
MultiPath++ + SDC Only
Example Density
16
Example Density
20
Example Density
10
15
10
6
0.2 -0.1 0.0 0.1 0.2
-.22
-0.1 0.0 0.1 0.2
-0.2 -0.1 0.0 0.1 0.2 Perturbed Original minADE
Perturbed Original minADE
Perturbed Original minADE
RernoveNoncausallgents
Remnoveloonccusslkeenttqquuu
RemoveCausaaAgeett
RemoveStaticAgents
Figure 20: Models are sensitive to non-causal perturbations. For models, we plot the distribution the per-scene difference between perturbed and original minADE for various perturbation types. The models are least sensitive the RemoveNoncauuallggesttta RemoveCausalAgents and RemoveNoncauusalAgnnt
perturhation, and most sensitive (almost equally so) to
RernoveNencausallppent
DemoveeNoncauusllggnnnEuu
RemoveStaticAgent:
Example Density
10-
Example Density
10
Density
10
8
8
6
0.2
-0.1
0.C
0.1
0.2
0.1
0.1
0.2
00
-0.1
0.0
0.2
Perturbed Original minADE
Perturbed Original minADE
Perturbed Original minADE
MP++-AIl
MP+
Drop Context
MP++-AIl Drop Static Context
Figure 21: Targeted data augmentations can improve model robustness. Dropping static context agents as opposed to context agents has greater effect on reducing model sensitivity to non-causal perturbations.
26

---

Under review as a conference paper at ICLR 2023
L TRAJECTORY SET METRICS (CONTINUED)
Since A(Pib-Ori) on minADE only quantifies how the perturbations impact the models' robustness in terms of the distance between ground truth and the closest predicted trajectories, does not directly reflect the difference between the two predicted trajectory sets (w/ and w/o perturbations). We thus introduce two trajectory set metrics to capture such difference: an IoU based metric as given in Section 3,4 in the main context and trajectory set minADE defined below. Ideally, a model's predicted trajectory sets would not be sensitive to dropping non-causal agents, meaning we expect low difference on the trajectory set metrics.
minADE between trajectory sets (TS_minA (0.mmiAA).. Let Peert, represent the i-th predicted trajec- tory in the predicted trajectory sets w/ and w/o perturbation, respectively. We define TS TS_minADE- min (Porgg perrt) j.j 1,2, ......... N where N is the number of the predicted trajectories of the model. Hence, a smaller TS minADE means that two predicted trajectory sets are more similar.
The results for all the models given in Table 15. We can that most of the models sensitive to the RemoveNoncausal perturbation. The Wayformer least sensitive, which is good. However, it is also least sensitive to RemoveCausal, which indicate that the model is less sensitive to agent removal in general.
Trajectory set minADE for RemoveNoncausal and RemoveCausal
Model
RemoveNoncausal
RemoveCausal
MultiPath++
0.037
0.024
ScemeTrrnnsorrmrr Marginal
0.103
0.094
Scene Transformer Marginal-All
0.140
0.141
Scene Transformer Joint
0.156
0.195
Wayformer
0.0140
0.0164
MP++ All Agents
0.114
0.114
Table 15: The trajectory set minADE for models evaluated on the perturbations of RemoveNoncausal and RemoveCausal
27


Reviews:
[{"review_id": "eOL76XWInhT", "reviewer": "Reviewer_3msf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper constructs a novel benchmark for evaluating the robustness of trajectory prediction models by deleting non-causal agents in scenes. The causal agents in WOMD’s validation data set (around 42k scenes, as indicated in the paper) are labeled with human labelers. This paper further proposes two metrics to evaluate the robustness of models. To improve the model robustness, this paper considers data augmentation methods that show improvements over baselines. \n", "summary_of_the_review": "There are several key clarifications that need to be addressed before I could fully evaluate the paper. I would recommend rejection for now. ", "strength_and_weaknesses": "Strength:\n - This paper proposes a novel perturbation which is deleting agents in the scene to emphasize the causal relationship between agents. \nThe experiments are conducted in realistic datasets. \n\nWeakness\n - The agents in the scene would affect the autonomous vehicle at different timesteps. In this paper, the causal agents are detected for the whole scene and are masked out during the whole scenario in the perturbed data. Is it more reasonable to label the timestep that each agent takes effects on the AV and mask accordingly? \n - Considering that this is a benchmark paper, it would be interesting to replicate the experiments in section 4.3 on other methods, such as SceneTransformer and WayTransformer. \n\nThere are several points that need further clarification.\n - For a better presentation of the two proposed metrics mentioned in the intro, It would be better to describe the IoU based metric in section 3.4 instead of in the result section. \n - It would be better to describe the definition of minADE instead of referring to the original paper, considering that minADE is quite important in evaluating the methods. \n - In section 4.2, the paper mentions that “but a long tail of outlier examples experience a large change ( > 1m).” What does “(?1m)” mean? Is the statement related to Figure 4?\n - In section 4.2, the paper mentions that “the model sees a large portion of examples where minADE improves”. What does the improvement mean? Which metric is used for measuring the improvements?\n - For figure 5, it would be better to put labels on the y-axis to make a clear comparison.  \n - Typo: In page 1, “data augmentation effect model sensitivity.” -> “data augmentation affect model sensitivity.”\n - Typo: In page 5, “there are causal agents in the scene For example,” -> “there are causal agents in the scene. For example,”\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bTM5DaJgi6u", "reviewer": "Reviewer_vJxd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designs a benchmark based on the Waymo Open Motion Dataset for evaluating the robustness of trajectory prediction algorithms. They construct perturbed examples by removing non-causal agents, removing causal agents, removing a subset of non-causal agents, or removing stationary agents. Their experiment results show several state-of-the-art prediction algorithms have performance drops when evaluated on their dataset. They also propose two methods, i.e., increasing the training dataset size and using targeted data augmentation, to increase the robustness.", "summary_of_the_review": "My biggest concern about the proposed benchmark is the correctness of the causal label. If the correctness of the underlying causality cannot be guaranteed, the evaluation results will be meaningless. In addition, compared to using non-causal agents to evaluate robustness, I think adversarial perturbation is more useful and applicable in real-world AV systems. Overall, I suggest rejecting this paper.", "strength_and_weaknesses": "### **Strength**\n\n* Evaluating the robustness of trajectory prediction algorithms is an interesting topic. A standard benchmark is also urgently demanded. \n \n* The structure of the paper is good and the writing is easy to follow.\n\n\n### **Weaknesses**\n\n* **Correctness of non-causal label.** The correctness of the non-causal label cannot be verified or guaranteed. In particular, the causality of sequential data is usually more complex than expected. For example, Figure 1 does not look correct to me. I find that some important agents (a pedestrian and two vehicles in the opposite direction) are removed. I think calling them “non-causal agents” is not reasonable. Actually, I think it is really hard to identify the non-causal agents unless the underlying causality behind the traffic scenario is explicitly shown. Or interventions are allowed to obtain Randomized Controlled Trials.  \n\n* **Missing literature.** Related works about adversarial robustness of trajectory prediction are missing: [1][2][3][4]. Although this paper does not focus on adversarial robustness, I feel that the robustness mentioned in this paper can also be improved by adversarial training. \n\n* **Motivation.** Based on the last point, I question the motivation for evaluating robustness with the non-causal agents. I doubt that the performance drop of prediction algorithms may be caused by the bias and low diversity of the dataset. In contrast, I think adversarial trajectory perturbation may be more fundamental for robustness. \n\n---\n\n[1] Cao, Yulong, Danfei Xu, Xinshuo Weng, Zhuoqing Mao, Anima Anandkumar, Chaowei Xiao, and Marco Pavone. \"Robust Trajectory Prediction against Adversarial Attacks.\" arXiv preprint arXiv:2208.00094 (2022).\n\n[2] Cao, Yulong, Chaowei Xiao, Anima Anandkumar, Danfei Xu, and Marco Pavone. \"AdvDO: Realistic Adversarial Attacks for Trajectory Prediction.\" arXiv preprint arXiv:2209.08744 (2022).\n\n[3] Zhang, Qingzhao, Shengtuo Hu, Jiachen Sun, Qi Alfred Chen, and Z. Morley Mao. \"On adversarial robustness of trajectory prediction for autonomous vehicles.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15159-15168. 2022.\n\n[4] Wang, Jingkang, Ava Pun, James Tu, Sivabalan Manivasagam, Abbas Sadat, Sergio Casas, Mengye Ren, and Raquel Urtasun. \"Advsim: Generating safety-critical scenarios for self-driving vehicles.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9909-9918. 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DXFpollJjY", "reviewer": "Reviewer_oUDJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the robustness of motion forecasting models and makes two main contributions: 1. additional labels for the Waymo Open Motion Dataset with causal labels, 2. evaluation of the robustness of several state of the art models using these \"causal\" labels. The causal labels include agents whose presence influences human drivers’ behaviour in any format.  The paper shows that state of the art models exhibit large shifts in performance under even non-causal perturbation: a 25-38% relative change in minADE.", "summary_of_the_review": "The paper is an interesting read. However, the labelling process needs to be discussed in more detail especially the effect of false positive \"causal\" agents and its effects on the conclusions of the paper. Finally, prior works need to be discussed in more detail.", "strength_and_weaknesses": "Strengths,\n* The paper addresses an important problem and is well written.\n* The proposed \"causal\" labels for Waymo Open Motion Dataset is an interesting and novel idea.\n* The paper reports interesting findings: e.g. Figure 1, where there are very large changes in performance even when \"non-causal\" agents are perturbed.\n* The paper includes extensive and fine grained evaluation on the robustness of motion forecasting models.\n* The paper discusses the underlying causes of the lack of robustness of motion forecasting models in detail.\n\nWeakness,\n* The labelling process as described is prone to errors: as described each frame is annotated by only 1 annotator, this can introduce significant errors as the definition of \"causal\" in Appendix A subjective. E.g. in Figure 6, is it not clear why the while sedan on the right is causal? Moreover, the policy of \"please err on the side of including\" can introduce a significant number of false positives. This labelling policy would have a direct impact on the experiments in Section 4.1 \"RemoveCausal\".\n\n* Important prior work is not discussed \"Are socially-aware trajectory prediction models really socially-aware?, Saadatnejad et. al.\"  This work shows that small perturbations on the trajectories of agents can cause large changes in the predicted future trajectories. In the light of these finding, the findings of this paper are not very surprising.\n\n* The paper does not propose any method to improve robustness: the utility of the paper would increase manifold if the proposed labels could be used to improve robustness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6W6iUwpBz1-", "reviewer": "Reviewer_Chzn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a benchmark and a set of studies about the sensitivity of trajectory prediction models to the removal of non-causal agents.\n\nThe authors had human labelers label the agents that influence the ego behavior in the WOMD validation set. They then used those labels to perturb the dataset. They considered four perturbations, removing all non-causal agents, removing some non-causal agents that equaling to the number of causal agents, removing all causal agents, and removing all static agents.\n\nThey evaluated three representative trajectory prediction models (MultiPath++, SceneTransformer, and Wayformer) under those perturbations. The results show that all those models are sensitive to the removal of causal agents. For example, the relative mADE delta is around 25%, and the prediction errors even decrease in more than 40% of the samples.\n\nThe authors hypothesized that a lack of training data and overfitting could be the cause of this sensitivity. They also found data augmentation (by randomly removing some static agents) could improve the robustness of the models.", "summary_of_the_review": "This is an interesting paper. However, my main concern about this paper is that the studies presented in this paper are based on a prediction model whose performance might not be representative. And there is not much novelty in the proposed method.", "strength_and_weaknesses": "--- Strengths\n\n- Robustness is essential to the trajectory prediction task. This paper presents many interesting results about how state-of-the-art prediction models perform under perturbations.\n\n- The causal labels that the authors released to the WOMD validation set will be valuable.\n\n\n--- Weaknesses\n\n- This is an interesting paper. However, my main concern about this paper is that the studies presented in this paper are based on a prediction model whose performance might not be representative. This paper mainly used the MultiPath++-All model for its studies. However, in Table 2, the performance of the MultiPath++-All model is clearly an outlier. It has more than double the prediction error compared to the other models in the table, and it's even a lot worse than MultiPath++. If I understand correctly, the only difference between MultiPath++-All and MultiPath++ is that MultiPath++-All was trained on all agents, so I will expect the performance of MultiPath++-All should be close to or better than MultiPath++ since it's trained on more agents. I hope the authors provide a convincible explanation about why the *-All models have worse performance. The authors used the outlier MultiPath++-All model for most (if not all) of the other studies, including data augmentation and dataset size analysis. I am concerned that the conclusions might not apply to the other state-of-the-art models since the performance of MulthPath++-All seems to be an outlier.\n\n- There are many interesting analyses presented in the paper, but there is not much novelty in the proposed mitigation method. The main method presented in the paper to improve the model's robustness is to augment the training data by randomly removing some static agents. It is not surprising at all that removing some agents will make the model more robust to the removal of agents at test time.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1vwKRriOPO", "reviewer": "Reviewer_UcfZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new benchmark for evaluating the robustness of models for motion forecasting of autonomous vehicles. The authors collect human-annotated labels of causal agents on the Waymo Open Motion Dataset, and perform various perturbations of the data using these labels. This data is used to evaluate the robustness of state-of-the-art forecasting methods. Towards this end, the authors also propose two metrics to quantify robustness. The takeaway messages are that all models exhibit a large shift in performance under even non-causal perturbations, and that increasing the training dataset size and targeted augmentation can improve model robustness.\n\n", "summary_of_the_review": "The paper is largely well-written and the analysis is extensive. The problems I find lie in dealing with the nature of the problem in setting up the human annotations. To be fair to the authors, if I find I cannot suggest any easy way of fixing the problems, I have not penalized my rating, and instead suggested discussing the problems as a means of increasing awareness for future researchers which is still valuable. ", "strength_and_weaknesses": "### STRENGTHS:\n\n1. The paper is largely well-written. \n2. The empirical analysis is extensive and the paper discusses relevant aspects of model performance. \n\n### WEAKNESSES:\n\nThe main weaknesses I find are in dealing with the nature of the problem and the annotation methodology, which I consider are unfortunately crucial towards this paper's contributions. \n\n#### 1\\. On Causality and Perturbation:\n\nThe community has had a larger discussion on the relationship between perturbations and causality. In explainable AI, perturbations are used to obtain sensitivity maps that are then perceived to indicate causal relationships, a methodology that has received some criticism [R1]. More directly, there has been some effort in addressing the issues for the forecasting tasks [R2].\n\nI find that many of the similar problems listed in these papers apply here: specifically that the perturbed samples may no longer lie on the actual data manifold of the real-world data. Simply ignoring the state of an agent to be removed tampers with the validity of the rest of the agents behaviors in the scene as well. This is perhaps the most egregious in the `RemoveNoncausalEqual` perturbation where an equal number of randomly selected non-causal agents are removed. What guarantees that the scene is still valid for the model to make a prediction? \n\nStated differently, if the perturbation results in a different scene configuration that a model has seen before, even a perfect model might change its predictions but that would not constitute a lack of robustness. For instance, multiple valid predictions are possible for a given sequence. Why is Fig. 1 considered non-robust when there might be other examples of a left turn that seems plausible?\n\nThis in theory would not be a problem with the way causality has been mathematically defined, but I believe the problem comes from the way the definition of a causal agent is operationalized, as I describe later in point 3. \n\n#### 2\\. Accounting for the possibility of multiple valid futures:\n\nThe larger body of work of pedestrian forecasting deals with the possibility of multiple valid futures for an observations [R3, R2]. (Also see [R3, 8.4.2] for a discussion on robustness and a link to \"Lasota and Shah 2017\" in that paper which might be useful.) Here the notion of causality does not incorporate this possibility. The models have been trained on a large number of examples, where an agent behavior that causes a change in one example may not cause a change in another. I believe it is possible that this is what explains the large variation under non-causal perturbations as well. [R2] seems to do this in the counterfactual sense as well, but in terms of observed factors that change the 'uncertainty' over possible futures. Here the instructions do somewhat account for this by stating that a casual agent is `one whose presence would modify or influence human driver behavior in any way`.\n\nTo be fair to the authors I cannot see a directly easy way to fix this methodologically. So at the very least making this discussion explicit for future researchers would help minimise the misinformation in this space coming in from XAI methods. The linked papers might help in this regard.\n\n#### 3\\. Operationalization of causal agents and change in causality over the video:\n\nThe paper defines non-causal agents as:\n```\nwe define a non-causal agent as an agent whose deletion does not cause the ground truth trajectory of a given target agent to change.\n```\n\nIs a causal agent then the inverse or contrapositive of this statement? i.e. is an agent causal when their presence causes a trajectory to change, or if their presence *may or may not* cause a trajectory to change?\n\nThe annotator instructions indicate it's closer to *may or may not*:\n\n```\nIf the behavior of a human driver would be modified because of a potential action that an agent is likely to take, then that agent should be causal.\n```\n\nDriving as an activity inherently involves forecsating. From this definition, any potential agent that can influence one's own future is causal.\n\nIn Fig. 6. if the agent on the left (marked as not causal) rushes ahead, say under a red light, they would switch to being causal right?\n\nBut that's not how the annotation seems to be setup. The same agent can go from being causal to non-causal in principle. But I don't believe that is the case in the annotations? In the annotation statistics the authors say that \"cyclists are relatively more\nlikely to be causal agents than pedestrians or vehicles\" which seems odd as an overall statement seeing as cars around the driver ought to always be causal. If this is simply computed as `(# of agents marked as causal / # of agents in data) for each type`, this is a misleading message. \n\n#### 4. Annotation Methodology:\n\n#### 4.1. Annotator Agreement ility and `true' causality of agents\n\nSince improving the reliability of models is one goal of this work, it would help to look at an inter-annotator agreement/reliability metric to see how much the annoatators agreed on an agent being causal, to understand the nature of the problem better.\n\nThe paper assumes there is an underlying truth in whether an agent is causal or not, while simultaneously acknowledging the subjectivity in the perception of the causality:\n\n``` We emphasize that false positives (identifying an agent as causal when it is truly non-causal) are acceptable to a certain extent, but we should avoid false negatives (failing to identify a truly causal agent). ```\n\nThis subjectivity is reflected in that 24% of the labels are selected by a single annotator. The method leans towards treating false positives as accceptable, but there is no way of knowing whether these 24% would qualify as false positives or not. Traditionally in such subbjective annotations (e.g. perceived emotions) the agreement indicates wheter there is consensus in what would change a trajectory. Such a metric would provide more insight into the nature of the problem.\n\n#### 4.2. Whose perception is being modeled?\n\nWhen setting up annotation collections of this nature, the perception of annotator is asked. Here the instructions ask the annotator to simulate a hypothetical driver (Appendix A). Were all annotators drivers? Why not just ask for whether they themselves would be influenced by an agent?\n\n\n[R1] Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning - Atrey et al. \n[R2] Why Did This Model Forecast This Future? Closed-Form Temporal Saliency Towards Causal Explanations of Probabilistic Forecasts - Raman et al. \n[R3] Human motion trajectory prediction: A survey - Rudenko et al. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject