Robust Trajectory Prediction against Adversarial Attacks
Authors: Yulong Cao
Danfei Xu
Xinshuo Weng
Zhuoqing Mao
Anima Anandkumar
Chaowei Xiao
Marco Pavone
Venue: CoRL 2022 Oral (2022)
Keywords: Adversarial Attack
Trajectory Prediction
Autonomous Driving

Abstract: Trajectory prediction using deep neural networks (DNNs) is an essential component of autonomous driving (AD) systems.  However, these methods are vulnerable to adversarial attacks, leading to serious consequences such as collisions. In this work, we identify two key ingredients to defend trajectory prediction models against adversarial attacks including (1) designing effective adversarial training methods and (2) adding domain-specific data augmentation to mitigate the performance degradation on clean data. We demonstrate that our method is able to improve the performance by 46\% on adversarial data and at the cost of only 3\% performance degradation on clean data, compared to the model trained with clean data. Additionally, compared to existing robust methods, our method can improve performance by 21\% on adversarial examples and 9\% on clean data. Our robust model is evaluated with a planner to study its downstream impacts. We demonstrate that our model can significantly reduce the severe accident rates (e.g., collisions and off-road driving).

Robust Trajectory Prediction against Adversarial Attacks
Yulong Cao*l Danfei Xu Xinshuo Weng", Z. Morley Mao', Anima Anandkumar Chaowei Xiao² and Marco Pavone
University of Michigan NVIDIA
Georgia Institute of Technology
California Institute of Technology
Arizona State University
Stanford University
Abstract: Trajectory prediction using deep neural networks (DNNs) is an essen- tial component of autonomous driving (AD) systems. However, these methods are vulnerable to adversarial attacks, leading to serious consequences such as col- lisions. In this work, we identify two key ingredients to defend trajectory predic- tion models against adversarial attacks including (1) designing effective adversar- ial training methods and (2) adding domain-specific data augmentation to mitigate the performance degradation clean data. We demonstrate that our method is able to improve the performance by 46% on adversarial data and at the cost only 3% performance degradation on clean data, compared the model trained with clean data. Additionally, compared to existing robust methods, our method improve performance by 21% on adversarial examples and 9% clean data. Our robust model is evaluated with a planner to study its downstream impacts. We demonstrate that our model can significantly reduce the severe accident rates (e.g.. collisions and off-road driving)
Introduction
Trajectory prediction is critical component of modern autonomous driving (AD) systems. It allows an AD system to anticipate the future behaviors of other nearby road participants and plan its ac- tions accordingly. Recent trajectory prediction models built Deep Neural Networks (DNN) have demonstrated state-of-the-ar performance on large-scale benchmarks [1-7], showing promising path towards learning-basee -ased trajectory prediction for AD systems. As trajectory prediction plays an important role in AD systems, accurate predictions are required for making safe driving decisions. It is crucial to understand how unknown scenarios will affect trajectory predictions and then bolster the robustness of such trajectory predictions in return.
To achieve this goal, adversarial attacks [8-10] are often used proxy to measure the worst-case performance of the model when facing unseen scenarios. Similarly, we use a standard adversarial attack setup [11] for trajectory predictions. As illustrated in Fig. 1, an adversarial agent (red vehicle) aims to cause a traffic accident. drives along a carefully designed trajectory (i.e., adv history) to influence the trajectory prediction model of the Autonomous Vehicle (green vehicle). Such an ad- versary can critically compromise the predicted trajectories of all other agents by altering its route in inconspicuous ways. By fooling the trajectory prediction models, it can further affect downstream planning of the AV systems and cause serious consequences. Using the adversarial attack as the proxy, this work aims to develop effective techniques bolster the robustness of trajectory predic- tion models against adversarial attacks and improve the AD's safety under uncertain scenarios.
* Work done during intersship NVIDIA Our project wehsite is at
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Diourd
Predesed
Collision
Safe
Adh ages
History
Adv history
Adv hictory
atlack
Under attack RobusiTrij
Figure I: Overview of RobustTraj preventing Autonomous Vehicle (AV) from collisions when its trajectory prediction model is under adversarial attacks. When the trajectory prediction model is under attack, the AV predicts the wrong future trajectory of the other agent turning right (yellow vehicle). This results in AV speeding up instead of slowing down, and eventually colliding into the other vehicle.
At the same time, adversarial robustness for machine learning widely-studied area, but most works focus on classification tasks [12-23]. Among the proposed techniques, adversarial training [8] remains the most effective and widely used method defend classifiers against adversarial attacks. The general strategy of adversarial training is to solve min-max game by generating adversarial examples for model each training step and then optimizing the model to make correct predictions for these samples. However, directly applying adversarial training to trajectory prediction presents a number of critical technical challenges.
First, most trajectory prediction methods employ probabilistic generative models to cope with the uncertainty in motion forecasting [2-7]. As will show in this paper, the stochastic components of these models (c.g., posterior sampling in VAEs) can obfuscate the gradients that guide the adversar- ial generation, making naive adversarial training methods ineffective. Second, adversarial training on trajectory prediction task aims to model joint data distribution of future trajectories and adver- sarial past trajectories. However, the co-evolution of the adversarial sample distribution and the prediction model during the training process makes the joint distribution hard to model and often destabilizes the adversarial training. Finally, prior work [23] shows that adversarial training often leads to degraded performance on clean (unperturbed) data, while retaining good performance in benign cases is crucial due to the critical role of trajectory prediction for AVs. Hence, an effective adversarial training method must carefully balance the benign and the adversarial performance of model.
Our approach. We propose an adversarial training framework for trajectory predictions named Ro- bustTraj. by addressing the aforementioned challenges. First, to address the issue of an obfuscated gradient in ad adversarial generation due to stochastic components, we devise deterministic attack that creates a deterministic gradient path within a probabilistic model to generate adversarial samples. Second, to address the challenge of an unstable training process due to shift in adversarial distri- butions, we introduce a hybrid objective that interleaves the adversarial training and learning from clean data anchor the model output stable clean data distribution. Finally, to achieve balanced performances on both adversarial and clean data, we introduce a domain-specific data augmentation technique for trajectory prediction via dynamic model. This data augmentation technique generates diverse, realistic, and dynamically-feaat feasible samples for training and achieves a better performance trade-off on clean and adversarial data.
We empirically show that RobustTraj can effectively defend two different types of probabilistic tra- jectory prediction models [4, 7] against adversarial attacks, while incurring minimal performance degradation on clean data. For instance, RobustTraj can increase the adversarial performance of AgentFormer [4], state-of-the-arr trajectory prediction model, by 46% at the cost of 3% perfor- mance drop on clean data. To further show impacts of our method on robust trajectory prediction model into planner and demonstrate that our model reduces serious ac- cidents rates (e-g., collisions and oof-road driving) under attacks by 100%, compared to the standard non-robust model trained using only clean data.
AD stack, we plug our

---

2 Related Work
Adversarial attacks and defenses on trajectory prediction. A recent work began to study the adversarial robustness of trajectory prediction models [11]. Zhang et al. [11] demonstrated that perturbing agents' observed trajectory can adversarially impact the prediction accuracy of a DNN- based trajectory forecasting model. To mitigate the issue, Zhang al. [11] proposed several defense methods such as data augmentation and trajectory smoothing. However, these methods are less effective when facing adaptive attacks [24]. In our work, we propose to use adversarial training which provides the general adversarial robustness that can resist adaptive attacks.
Adversarial scenario generation. A few recent studies work on generating adversarial traffic sce- narios such that the autonomous driving systems fail to make safe driving decisions [25, 26]. How- ever, generating, realistic traffic scenarios is challenging and the generated adversarial scenarios can be unrealistic and violate traffic rules by directly optimizing the latent vectors of the traffic model Rempe et al. [26]. In this work, we consider defending against realistic adversarial scenarios grounded on the scenarios from a dataset. Wang et al. [25] perturb the raw input data to mislead the full stack AV system. However, in this work, our primary goal is to study and improve the robust- ness of trajectory prediction models. To obtain salient and unambiguous insights, we minimize the conflating factors in our analysis without considering the perception model.
Ad Adversarial training. A variety of adversarial training methods have been proposed to defend DNN-based models against adversarial attacks [8, 19, 12-23]. The most common strategy is to de- sign min-max game with the inner maximization process and outer minimization process. The inner maximization process generates ad adversarial examples that maximize an adversarial objective (e.g., make wrong prediction). The outer minimization process then updates the model parameters to minimize the error on the adversarial examples. Several recent works also propose to mix clean data and adversarial examples for improving robustness (27, 28] and performance on clean data [18]. Al- though there exists a large body of literature in studying adversarial robustness for machine learning, most focus on the problem of discriminative model (c.g.. object recognition), leaving other problem domains (e.g.. conditional generative models) largely unexplored. In this work, we develop novel adversarial training method for trajectory prediction models, where most statc-of-the- trajectory prediction models are generative and probabilistic, by addressing number of critical technical chal- lenges.
3 Preliminaries and Formulation
Trajectory prediction. The goal is to predict future trajectory distribution P of N agents conditioned on their H history time states X = -x- -1+1 ,...,,") x)), and other environment context such as maps to predict T time-step future trajectories Y =
For observed time steps < O, we denote the agent states as X (zi...., 2/ ........ x.y). where x is the state of agent i at the time step f. Similarly, Y (vi..... y denotes the states of N agents at future time step I 0). We denote the ground truth and the predicted future trajectories Y and Y, respectively. We denote the history information encoded by function f as the decision context C = =(().
YT) (Y!.....,
Probabilistic trajectory prediction models. In this work, focus defending generative, prob- abilistic trajectory prediction models, as they have demonstrated superior performance in model- ing uncertainty in predicting future motions [2-7]. We consider the most popular types of generative models: conditional variational encoders (CVAEs) and conditional GANs (cGANs), both can be viewed as latent variable models. We define latent variables Z = {EI,..., Zis ..., 2N) where zi represents the latent variable of the agent i. CVAE formulates the generative problem [ Po (Y Z) po [Z|X dZZ, where (Z X) is a conditional Gaussian prior
as: Po (Y(X) (N(r) (Z[),, (Z) ) 1 (Z) with mean 1 (Z and standard deviation p (Z[X); PY|Y Z) is a
conditional likelihood model. The model usually trained through optimizing a negative evidence lower objective [4]:
Lonal Letho Laiverary
Ey (x)_goo
Y|Z,X)] (L(9, (Y] po ([X)) + min (R) Y 12
(I)
For simplicity, we ignore contextual information.
3

---

where go (Z X) is an approximate posterior parameterized by o. pe (Z]X) is a conditional Gaus- sian prior parameterized by 6, and Pol Y X) is a conditional likelihood modeling future trajectory
Y via the latent codes Z and past trajectory X. Additionally, Laiversity = ming || Y
2 is a
diversity loss, which encourages the network to produce diverse samples. Given each past trajectory X, the model generates K sets of latent codes (Z(1, Z1h), ,'.. ( from the conditional
Gaussian prior (rr (Z ()) pg (2(X)), where Z(k) " ; resulting in K future trajec- tories (
Similarly, in a conditional Generative Adversarial Net (cGAN)-based model (e.g., Social-GAN [1], it uses loss function follows:
Luaat
Lyn Ldiversity
Ex llog Do (o(()))) -Bx-pl (lo(1 Do Do
(YJX, min || (7(k) - Y 11?
(2)
where G represents the generator and D represents the discriminator. Y = G X, (()) is one
of the predicted trajectories in which (6) is randomly sampled from (0, 1). During the training, Egan is maximized to train D and Liotal is minimized to train G.
Threat model. We follow the setup in prior work [1I] and adopt an idealized threat model, where the adversary alters its observed history X by adding a perturbation 5 bounded by the adversarial set S = {6] || ك <). where is the perturbation bound. The perturbation aims to mislead the prediction Y. A naïve adversarial attack is find the perturbation through adversarial objective 5 arg ma (min |I po (Y + 8, (()) Y [|²), where pe Po (Y + 6, (()) is the predicted trajectory conditioned on the random variable (b)) and adversarial history trajectory 5--6
Naïve adversarial training. Adversarial training formulates a min-max game with an inner maxi- mization process that optimizes the perturbation 8 generate adversarial examples for misleading the model at each training iteration, and an outer minimization process that optimizes the model pa- rameters make correct predictions for these examples. We follow the standard adversarial training formulation [8]:
min max 0,0
Listal X + + Y).
4 RobustTraj: Robust Trajectory Prediction
As stated earlier, applying adversarial training for trajectory prediction presents three critical chal- lenges: (1) gradient obfuscation due to model stochasticity, (2) unstable learning due to changing adversarial distribution, and (3) performance loss in the benign situation. In this section, we describe each challenge more detail and present the corresponding solutions in our RobustTraj method.
Improve adversarial generation with Deterministic Attack. Since trajectory prediction is inher- ently uncertain and there no single correct answer, probabilistic generative models are usually used to cope with the stochastic nature of the trajectory prediction task. Such stochasticity will obfuscate the gradients that are used generate effective adversarial examples in the inner maxi- mization process of adversarial training. The naive attack mentioned in section 3 is a straightfor- ward way to achieve this goal. However, this optimization involves stochastic sampling process (kk) (pp (Z|X), p& (Z(x)). Such stochastic process will obfuscate the gradients for finding the optimal adversarial perturbation 8, making the outer minimization (robust training) less effective. In order to sidestep such stochasticity, we propose the deterministic attack that creates determin- istic gradient path within the model to generate the adversarial perturbation. Ż. Specifically, we use deterministic latent code by replacing the sampling process (k)) (Z|X), Pă (Z(x)), with the maximum- maximum-likelihood sample (here, i.e = p/ (Z..)). The objective for generating the adversarial perturbation is thus:
5 = arg max Lade X K60 Y) arg max II Po ( x 8) - Y 1| where 2 =12( (4) dE3
SEB
We empirically show that gradients from this deterministic gradient path can effectively guide the generation of adversarial examples. We name our attack as Deterministic Attack.

---

7 2
2
6
4
9
9 9
7
3
7
4
4
4
S 5
S
8
S
S
S
S
2
4
4
4
9
4
9
2
2
6
6o
6
6
6
O
4
9
9
4
9
7 2
4
4
4
4
5
2
4
3
6
4
2
6
88
8
O
3
2
0
0
(a) Clean
(b) Salt and pepper
(c) Adv noise
(d) Correlation
Figure 2: Visualizations of the CVAE models trained with clean (a) data, Salt and pepper noise (b), and adversarial perturbations (c); Quantitative results of the correlation between the label of the generated images and conditioned images at different noise levels (d).
Stabilize adversarial training with bounded noise and hybrid objective. During the adversarial training process, the distribution of the perturbed input X + coevolves with the training process as ó is calculated via an inner maximization process at each training iteration. Although ó is bounded by the adversarial set Sp. the resulting latent condition variable C f(x + 6) can be arbitrarily noisy since the Lipschitz constant of neural network layers (f) is not bounded during training (See Lemma I. in Appendix A). Since C = f(X + 5) is noisy, is a less informative signal compared to the deterministic signal X. Thus, modeling (Y| + 5, Z) becomes substantially harder. In an extreme case that can degenerate to model (Y()))) resulting in the undesirable worse performance on the clean data.
s(X + 5) super noisy and contains no information, the training process
To further validate the above hypothesis that it is hard model pe (Y| + 6, Z) with a changing data distribution of X +6, we conduct an additional experiment. For simplicity, we use MNIST [29] as the dataset. As shown in Fig. 2, we divide each digit image into four quadrants. We take the top-left quadrant as the condition X and the remaining quadrants as the output Y. We train a CVAE (Po + 8, Z)) to model Y by using clean data (X) or noisy data (X- + 5), where represents salt and pepper noise [30] or adversarial noise [8] resulting in Fig. 2 (a), (b), (c) respectively. The top-left region of each image in the first row is the conditional variables X. The rest of rows are the generated images with different Z. Each column in the same row uses the same Z. We observe that the model trained on clean data successfully captures the conditional distribution (i.e., the generated image highly depends on X) while the model trained with adversarial noise degenerates and ignores the condition (i.c., each row generates images of the same digit). This result shows that the condi- tional generative model fails to learn from X. To provide a quantitative analysis, we measure the correlation between the label of the generated images and the label of their conditioned image quad- rants, resulting in Fig 2 (d). More details how to calculate the correlation are in the Appendix A. We observe that the correlation drops as the noise level increases for both adversarial nose and salt and pepper noise. Adversarial noise is more effective to degenerate the conditional generative model. Therefore, we conclude that (1) the noises in the conditional data lead to degenerated con- ditional generative model (i.e., from CVAE to VAE); (2) the level of degeneration depends on the noise levels.
Based on the analysis result, to better learn a robust trajectory prediction model, we need to bound |f(x + 6) - f(X) to reduce the noise level. Hence, we propose the following regularization loss Lieg:
Lagg = d(f(x + 6), f(X).,
(5)
where d is a distance function (c.g., we use L2 norm as the distance metric).
In addition, because the clean data has fixed distribution. simultaneously learning from the clean data during the adversarial training process anchors the conditional distribution on a stable clean data distribution. Specifically, we propose the following hybrid objective:
Lelemn (X, Y) Liotal (X, Y),
(6)
where Lioaal could be the loss in Eq. 1 for CVAE-based model or Eq. 2 for cGAN-based model.
Protect benign performance using data augmentation. Adversarial training often leads to per- formance degradation on clean data [23]. However, trajectory prediction is a critical component for safety-critical AD systems and its performance degradation can result in severe consequences (e.g., collisions). Thus, it is important to balance the model performance on the clean and adversarial data when designing adversarial training algorithms.
To further improve the performance on clean and adversarial data, we need to address the overfit- ting problem of the min-max adversarial training [31]. Data augmentation is shown to be effective
5

---

in addressing the problem in the image classification domain [32]. However, data augmentation in trajectory prediction is rarely studied and non-trivial. To design an effective augmentation algo- rithm, Rebuffi et al. [32] argues that the most important criterion is that the augmented data should be realistic and diverse. Thus, we design a dynamie-model based data augmentation strategy A. shown in the Appendix A. By using the augmentation, we can generate diverse, realistic multi-agent trajectories for each scene and construct mug.
RobustTraj. In summary, our adversarial training strategy for trajectory prediction models is formu- lated as follows:
8 = arg max Ladv X + 6, Y), where X, Y) E D Daug
dEB
0, Φ = arg min Goral X + 8, Y) + Celean (X Y) + Log X, X
8,4
where ID, Daug are the training data and augmented data; Cadv is adversarial loss to generate effective adversarial examples in Eq. 4; Liotal is the loss in Eq. 1 or Eq. 2 to train a robust model against adversarial examples; Ligg and Cetean are loss shown in Eq. 5 and Eq. to provide stable signal for training. 39 is hyper-parameter for adjusting the regularization.
5
Experiments and Results
5.1 Experimental setup
Dataset and models. We follow the setting in prior work [4, 3] and use the nuScenes dataset [33] for evaluation. For the trajectory prediction models, we select the representative conditional generative models based on CVAE (AgentFormer [4]) and cGAN (Social-GAN (1)). AgentFormer is a state- of-the-art model based on CVAE and Social-GAN is a classic model based on cGAN. We report the final results for all three models: AgentFormer (AF), mini-AgentFommer (mini-AF) and Social- GAN. More details are shown in the Appendix B.
For the adversarial training, we choose a 2-step Projected Gradient Descent (PGD) attack for the inner maximization and choose 3 = 0.1. We train 50 epochs and 100 epochs for AgentFormer and Social-GAN respectively. For other hyperparame- ters during training, we follow the original settings for AgentFormer and Social-GAN. The details for choosing these hyperparameters can be found in the Appendix B. All experiments are done on the NVIDIA V100 GPU [34]. We consider various baselines, including naïve adversarial training (naïve AT) and four defenses proposed by Zhang er al. [11]: data augmentation with adversarial ex- amples (DA), train-time smoothing. test-time smoothing, DA + train-time smoothing and detection test-time smoothing.
Training details and hyperparameter choices.
Attack and evaluation metrics. For the adversarial attack, we choose a 20-step PGD attack (an ablation study on step convergence can be found in the Appendix B). Without loss of generality, we use Loe as the attack threat model so that S = {8) || I1 e}. We select e = (0.5. -00-meter, where the 1-meter deviation is the maximum change for standard car without shifting another lane [11]. We use four standard evaluation metrics for the nuscenes prediction challenge [33]: aver- age displacement error (ADE), final displacement error (FDE), off road rates (ORR), and miss rate (MR). We evaluate the model's performance on both clean and adversarial data. For convenience, we use ADE, FDE, ORR, MR to represent the performance on the clean data and Robust ADE, Ro- bust FDE, Robust ORR, Robust MR to represent the performance under attacks. We compute these metries with the best of five predicted trajectory samples, i.e., K = 5.
5.2 Main results
Here, we present our main results of RobustTraj. We compare it with the baselines including model trained with clean data (Clean) and naive adversarial training (Naive A7'), and existing defense methods for trajectory prediction [11]. The results have been shown in Table 1.
We observe that our method achieves the best robustness and maintains good clean performance for most cases. For instance, with e = 0.5 attack on AgentFormer model, our method is able to reduce
46% prediction errors 5.099 -2.73 under the attack at cost of 2.6% (991- 1.86, clean performance
5.09
1.86
degradation on ADE, compared to the model trained with clean data at 0.5. Compared to the

---

Table 1: ADE and Robust ADE on different defense methods and models. The 1-st and 2-nd lowest errors are colored.
Model
mini-AF
AF
SGAN
Method
ADE
Robust ADE
ADE
Robust ADE
ADE
Robust ADE
0.5
0.5
1.0
0.5
1.0
0.5
Clean
2.05
2.05
6.86
11.53
1.86
1.86
5.09
8.57
4.80
4.80
10.52
[8]
2.75
8.34
DA [11]
2.31
2.32
5.54
9.32
2.10
2.08
4.35
7.22
5.41
5.40
8.85
17.25
Train-time Smoothing [11]
3.14
3.07
5.67
9.31
2.11
2.13
4.19
6.79
5.50
5.47
8.74
16,51
Smoothing [11]
2.97
4.96
8.50
2.40
7.44
6.16
6.17
9.05
DA + Train-time Smoothing [11]
2.41
2.39
5,48
9.00
2.17
2.13
4.14
6.62
5.63
5.61
8.60
16.14
Detection Test Smoothing [11]
2.31
2.28
5.91
2.08
2.03
4.45
7.59
5.35
5.37
9.28
17.39
RobustTraj
2.14
2.11
3.69
3.82
1.91
2.73
2.86
4.95
5.07
5.20
6,94
Collision
Collision
mntlinm
(a) Benign case
(b) Adv Attack
(c) w/ defense [11]
(d) defense (ours)
Figure 3: Impacts to a MPC-basd downstream planner. (a) is under the benign case while (b), (c) and (d) are under the adversarial attacks. The blue car and the red car represent the AV and the adversarial agent respectively.
existing methods, our method also significantly outperforms in terms of the robustness. For instance, with < = 1.0 attack on AgentFormer model, our method achieves 45% better robustness with 9% better clean performance on ADE compared to the best results from existing methods [11].
Impacts downstream planners. To further study the downstream impact of robust trajectory model in the AD stack, we plug it into a planner. We select a MPC-based planner and evaluate the collision rates under the attack. To perform the attack on closed-loop planner, we conduct attacks on sequence of frames with the expectation over transformation (EOT) [35] method. We follow We choose AgentFormer model since it has
the setting from Zhang al. [11] and choose the most competitive performance. As result, we observe that, while AgentFormer model trained on clean data leads 10 collision cases under attack, the robust trained model with the proposed RobustTraj is able to avoid all the collisions. As shown in Fig. 3, we demonstrate that the proposed RobustTraj able to avoid the collisions (Fig. (d)) while the DA + Train-time Smoothing method proposed by Zhang et al. [11] is not (Fig. 3 (c)).
5.3 Component analysis
In this section, we analyze the effectiveness of the three components. We use the mini-AgentForree model since has competitive performance and lightweight for fast adversarial training process.
6.0
50
Effectiveness of the Deterministic Attack. To demonstrate the importance of the Deterministic At- tack, we compare with competitive alternatives, La-
deterministic latent
tent Anack and Context Attack, which also construct the deterministic path. However, they only attack a partial model as opposed to our end-to-end full model attack. More details about these attacks are the Appendix A. We evaluate their attack effectiveness by attacking a normally trained trajectory prediction model (without robust training). In Fig, 4, demon-
35
10-
context
30
25
20
25
PGD
Figure 4: Peformance of different attacks in mini-AgentFormer..

---

strate that Deterministic Attack is the most effective attack among all. Additionally, we embed them into the whole adversarial training pipeline and evaluate the adversarial robustness. The results are shown in Table 2. We observe that the model trained with the Deterministic Attack achieves the best robustness in terms of ADE. More results with other metrics and the other 6 are in the Appendix
Effeet of additional loss functions. We evaluate the performance of the models trained with additional loss terms: Lelean and Leg. In Table 2, we can see that the regularization term Creg improves robustness of the mod- els and achieves better clean perfor- mance. shows that the regulariza- tion of the introduced noises on con- ditional variables help the model to stabilize the training procedure. By adding the clean loss Lelean. we ob- serve that both the robustness and clean performance are improved fur- ther, which means the benign data in- deed anchors the model output on clean data distribution and provides a stable signal for the better robust training for generative models.
Table 2: ADE and robust ADE for different methods on mini- Mini--ggetFormer, The lowest error in bold.
Method
ADE
Robust ADE
0.5
1.0
0.5
1.0
Clean
2.05
2.05
6.86
11.53
Latent Attack
2.55
2.70
4.10
4.71
Context Attack
2.47
2.59
3,94
4.78
Deterministic Attack
2.61
2.55
3.88
4.35
Deterministic Attack
Lres
2.29
2.31
3.76
4.28
+ Letein Legg
2.23
2.19
3.71
3.83
+ Letem + Ligg + Aug
2.14
2.11
3,69
3.82
Effect of domain-specifie augmentation. To demonstrate the effectiveness of the domain-specinne augmentation, We also combine it with all of the above components to validate its effect. The results are shown in Table 2. We observe that achieves a better performance on clean and adversarial data.
6 Limitations
In this work, we identified the challenges of applying adversarial training on trajectory prediction models based probabilistic generative models since they could cope with the natural uncertainty of motion forecasting. Though the probabilistic generative model is the main-stream architecture for the trajectory prediction task, there are other architectures (e.g., LSTM [1, 36), fow-based method [5. 6] and RL-based method [37]) for generating multi-modal predictions. Additionally, only study the adversarial set with the threat model of Lo perturbation trajectories instead of other types of threat models (e.g., optimization on the latent space [26], perturbation on raw sensor data [25]). Moreover, the primary goal of this paper is to study and improve the robustness of tra- jectory prediction models. To obtain salient and unambiguous insights, we minimize the conflating factors in analysis without considering the perception model in our pipeline. We leave these as future work for building robust trajectory prediction models.
7 Conclusion
In this paper, we aim to study how to train robust generative trajectory prediction models against adversarial attacks, which is seldom explored in the literature. To achieve this goal, we first iden- tify three key challenges in designing an adversarial training framework to train robust trajectory prediction models. To address them, we propose an adversarial training framework with three main components, including (1) deterministic attack for the inner maximization process of the adversar- ial training, (2) additional regularization terms for stable outer minimization of adversarial training, and (3) Nomain-sppecife augmentation strategy to achieve better performance trade-off on clean and adversarial data. To show the generality of our method, we apply our approach two trajectory prediction models. including (1) a CVAE-based model, AgentFormer, and (2) cGAN-base model, Docial-GAN Our extensive experiments show our method could significantly improve the robust- ness with slight performance degradation on the clean data, compared to the existing techniques and dramatically reduce the severe collision rates when plugged into the AD stack with planner. We hope our work can shed light on developing robust trajectory prediction systems for AD.
8

---

Acknowledgement
This work was supported by NSF under the National AI Institute for Edge Computing Leveraging Next Generation Wireless Networks, Grant # 2112562, as well NSF grant CNS-193004] and CMMI-2038215.
References
[1] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L.. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-971, 2016.
[2] B. Ivanovic and M. Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2375-2384, 2019.
[3] T. Salzmann, Ivanovic, Chakravarty, and M. Pavone. Trajectron++: Dynamically-feat feassblee trajec- tory forecasting with heterogeneous data. In European Conference on Computer Vision, pages 683-7 700. Springer, 2020.
[4] Y. Yuan, X. Weng, Y. Ou, and K. Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
[5] N. Rhinehart, K. M. Kitani, and P. Vernaza. R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), pages 772-788, 2018.
[6] N. Rhinehart, McAllister, K. Kitani, and S. Levine. Precog: Prediction conditioned on goals visual multi-agent settings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2821-2830, 2019.
[7] V. Kosaraju, A. Sadeghian, R. Martin- Martin-Martin, Reid, H. Rezatofighi, and S. Savarese. Social Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. Advances Neural Information Processing Systems, 32, 2019.
[8] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 7706006883 2017.
C. Szegedy, W. Zaremba, L. Sutskever, J. Bruna, D. Erhan. Goodfellow, and R. Fergus. Intriguing properties neural networks. arXiv preprint 13..iv1312..... 132669999 2013.
[10] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 99- IEEE, 2017.
[11] Q Zhang, S. Hu, Sun, Q A. Chen, and Z. M. Mao. On adversarial robustness of trajectory prediction autonomous vehicles. arXiv preprint arXiv:2201.0544 05055,, 2022.
[12] A. Jeddi, Shafice, and A. Wong. simple fine-tuning is all you need: Towards robust deep learning via adversarial fine-tuning. arXiv preprint arXiv: /3628, 2020.
[13] D. Liu, R. Yu, and H. Su. Extending adversarial attacks and defenses deep 3d point cloud classifiers. In 2019 IEEE International Conference on Image Processing (ICIP), pages 2279-2283. IEEE, 2019.
[14] N. Pupernot, P. McDaniel, Wu, S. Jha, and Swami. Distillation as defense to adversarial per- turbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pages 582-597. IEEE, 2016.
[15] N. Papernot and P. McDaniel. Extending defensive distillation. arXiv preprint 17X1v:1005.05264, 2017.
[16] A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, Dickerson, C. Studer, L. S. Davis, Taylor, and Goldstein. Adversarial training for free! arXiv preprint ArXiv:1904.120 12843, 2019.
[17] E. Wong, L. Rice, and J. Z. Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.0394 2020.

---

[18] C. Xie, M. Tan. B. Gong, J. Wang. A. L. Yuille, and Q. V. Le. Adversarial examples improve image recognition. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 319-828, 2020.
[19] C. Xie and A. Yuille. Intriguing properties of adversarial training scale. In International Conference on Learning Representations, 2020. URL https: / h/tps////ep/eeeeeeeeee/eee wwwpeew.eeeeeoooeeeeeeeeeeeeeeee weet//oorumddddyxxxaiee
[20] C. Xie, M. Tan, B. Gong, A. Yuille, and Q V. Le. arXiv:2006. 14536, 2020.
Smooth adversarial training.
arXiv preprint
[21] W. Xu, D. Evans, and Y. Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint urXiv:170.001 2017.
[22] Y. Yang. G. Zhang, D. Katabi, and Z. Xu. Me-net: Towards effective adversarial robustness with matrix estimation. arXiv preprint arXiv: 1wt.119955.191,,,. 2019.
[23] H. Zhang, Y. Yu, Jiao, Xing, L. El Ghaoui, and M. Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference on Machine Learning, pages 7472-7482, PMLR, 2019.
[24] A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense security: Cireumventing defenses to adversarial examples. In International Conference on Machine Learning, pages 274-283. PMLR, 2018.
[25] J. Wang, A. Pun, J. Tu, S. Manivasagam, A. Sadat, S. Casas, M. Ren, and R. Urtasun.
[26] D. Rempe, Philion, Guibas, S. Fidler, and O. Litany. Generating useful accident- prone driving scenarios via a learned traffic prior. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[27] S. Lee, H. Lee, and Yoon. Adversarial vertex mixup: Toward better adversarially robust generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 272- 281, 2020.
[28] J. Tu, T. Wang, J. Wang, S. Manivasagam, M. Ren, and R. Urtasun. Adversarial attacks on multi-agent communication. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7768 7777, 2021.
[29] L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142) 2012.
[30] Wikipedia contributors. Salt-and-pepper noise Wikipedia, the free encyclopedia, 2022. URL https: //en. wikipedia.oorg org/u/ index.idxx/i phppteieeppptttttelaalsaa and- pepper_ pupperr_nosseeooddddt00888111111 [On- line; accessed 16-June-2022).
[31] L. Rice, E. Wong, and Kolter. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pages 8093-8104, PMLR, 2020.
[32] S.-A. Rebuffi, S. Gowal, A. Calian, Stimberg, Wiles, and T. Mann. Fixing data augmentation to improve adversarial robustness. arXiv preprini urXiv:2103.01944 2021.
[33] H. Caesar, Bankiti, A. H. Lang, Vora, V. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1 11621-11631, 2020.
[34] J. Choquette, W. Gandhi, O. Giroux, N. Stam, and R. Krashinsky. Nvidia a100 tensor core gpu: Perfor- mance and innovation. IEEE Micro, 41(2).29-33 35, 2021. doi: 109/MM.0021.066111 2021.3069933
[35] K. Eykholt, Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno. and D. Song. Ro- bust physical- world attacks on deep learning visual classification. In Proceedings of the IEEE Conference Computer Vixion and Pattern Recognition, pages 1625-1634, 2018.
[36] H. Xue, D. Q. Huynh, and M. Reynolds. Ss-Istm: A hierarchical Isum model for pedestrian trajectory prediction. In 2018 IEEE Winter Conference Applications Computer Vision (WACV), pages 1186- 1194. IEEE, 2018
[37] N. Deo and M. M. Trivedi. Trajectory forecasts in unknown environments conditioned grid- gridd plans, arXiv preprint arXiv:2001.0075: 2020.
10


Reviews:
[{"review_id": "rc-Bgo6oby", "reviewer": "Reviewer_zQGj", "summary": "The paper proposes several techniques to improve the robustness of NN-based prediction models (e.g., AgentFormer and cGAN). Specifically, the authors claim that there are several keys to defend against adversarial attacks meanwhile avoiding the performance degradation on benign data: (1) use deterministic attack to avoid graident obfuscation, (2) stablize AT with bounded noise and hybrid objective; (3) use data augmentation to enhance the performance on benign cases. Through experiments they find that their approach outperforms some existing baselines (naive AT, data augmentation, training/testing time smoothing, etc), and results in better adversarial robustness and better performance on clean data (although the clean performance is still worse than no AT).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n[S1] The paper is well motivated - trajectory prediction is critical to AV and there are only a few works that study the adv robustness on this problem. Figures and text were clear and easy to understand.\n\n[S2] I like the toy examples on training with clean/adv noise on MNIST. It demonstrates the Lemma 1 and hypothesis on worse performance on clean data. The above analysis leads to the proposal of regularization term. The ablation studies on loss functions and augmentations are insightful.\n\n[S3] The downstream evaluation on MPC-based planner is appreciated while the scenarios are still over simplified. There are only a few actors involoved in the simulator.\n\n**Weakness:**\n\n[W1] The adversarial attack setting is not pratical - therefore it is less interesting to the robotics community. Specifically, this paper studies the L_infty attack (0.5 or 1 m deviation is the maximum change for a standard car without shifting to another lane), this limit the actor behavior a lot and will not find interesting scenarios (lane change, more advanced interactions between agents etc) that can happen in the real world. \n\nMoreover there are no contraints on the vehicle trajectories that are commonly used in other papers [1-3] (e.g., kinematic bicycle models, drivable area in map, constraints on acceleration, velocity, etc). Another major concern is this paper assumes an overly simplified setting - knowing the ground truth states (positions, etc) of all agents instead of taking the raw inputs (LiDAR points and camera images). This assumption on perfect perception results might lead to a large domain gap. I would suggest the authors to discuss more on the limitation section.\n\n[1] AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles. Wang et al., 2021. \\\n[2] Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior. Rempe et al., 2021. \\\n[3] KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients. Hanselmann et al., 2022.\n\n[W2] Limited technical novelty. The several key ingredients proposed in this paper is known in adv ML community while validated in other applications. For instance, mixing benign data and adversarial examples to avoid too much degration on clean results. Obfuscation gradients on stochastic model. Also, some previous works [1,2,3] showed that proper mixing of adv and clean examples can lead to better performance on both benign and adv examples. While in general I do appreciate the empircal validation on the prediction task, I feel a bit underwhelmed and incremental.\n\n[1] Adversarial Examples Improve Image Recognition. Xie et al, 2020. \\\n[2] Adversarial Vertex Mixup: Toward Better Adversarially Robust Generalization. Lee et al, 2020. \\\n[3] Adversarial Attacks On Multi-Agent Communication. Tu et al, 2021.\n\n**Others:** \n- It is better to define the history state for agents explicitly (\\emph{i.e.}, (x, y, t))\n- Limited discussions on realted work \n- The quanlitative figures are less insightful. Could we show all trajectories in a clearly way (e.g., BEV) and show the difference with no defense and other baselines. \n- Fig 2 (d) font size too small \n- Evaluation on PGD attack only. During the adversarial robustness evaluation on defensed models, it would be better if more attacks are tested (e.g., AutoAttack - both white and black boxes).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "See Strengths and Weaknesses above.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Main weakness\n- The problem setting is not very practical. \n- Limited technical contribution.\n\nOthers:\n- The related work is too short and less helpful. Many existing adversarial approaches on self-driving are omitted."}}, {"review_id": "biypC256ah1", "reviewer": "Reviewer_Bth3", "summary": "The paper consider adversarial attacks against self-driving motion forecasting (prediction) models and demonstrates that adversarial training can be used to train robust prediction models. The authors highlight some challenges that are specific to training robust multi-modal prediction are discussed - namely circumventing gradient obfuscation due to stochastic futures and unstable adversarial training due to very noisy latent variables. Furthermore the authors propose balancing adversarial training with data augmentation to protect against regression on clean data. Experiments are presented to demonstrate the effectiveness of the method and highlight how the method overcomes the challenges.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper studies an important problem of robust motion forecasting for self-driving vehicles\n- The paper also presents a robust trajectory prediction algorithm \n- The authors consider performance regression on clean data and balance the adversarial training with data augmentation\n- Presentation is very clear and easy to understand\n- Experiments are well designed to clearly demonstrate the effectiveness of the approaches\n\nWeaknesses\n- $\\delta$ has box constraints. However for a purely waypoint based parameterizations this could suffer from poor realism, i.e. zig-zagging trajectories that are kinematic ally infeasible. Therefore, some of the attacks generated can be very unrealistic and out of distribution wrt real world data, and this could be a reason why vanilla adversarial training induces a regression in performance on clean data. I am very curious about enforcing realism into the threat model (for example using a bicycle model parameterization) and seeing if there is an improvement in the performance.\n- Based on my understanding, the authors propose the deterministic attack as an alternative to sampling a random latent in the inner minimization problem of the threat model. It could also be important to consider to possibility of optimizing the latent to make the attack faithful to the formulation. \n- Impact on downstream planners was considered in a close-loop setting however only qualitative examples were shown.  I believe the results would be much stronger if quantitative results were presented for downstream planning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a very important problem and motivated it  very well. Furthermore presentation was very clear, experiments were thorough, and the methodology was well formulated. The results are also very convincing. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "No major issues, just a few points discusses in the strengths and weaknesses section. Mainly the point about the realism of the threat model.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "FxJiXjXqjB4", "reviewer": "Reviewer_N91T", "summary": "The main contribution of this paper is an adversarial training framework for making trajectory prediction models more robust to adversarial examples. In this setting, the adversary perturbs the trajectory of other vehicles in the scene, with the aim of reducing the accuracy of the model's predictions. This work focuses on adversarial training of conditional generative models, in particular CVAE and cGAN, which achieve state-of-the-art performance on trajectory prediction. Such models may be used in, for instance, autonomous driving systems.\n\nThere are several challenges to performing adversarial training in this setting: generating adversarial examples for stochastic models, inherent instability of adversarial training, and overfitting leading to worse performance on clean data. The proposed approach addresses all three challenges. Empirical evaluations (on the nuScenes dataset) show strong performance on adversarial inputs, while maintaining good performance on clean inputs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n* The proposed approach is clearly explained and well-motivated, in terms of how each component addresses one of the challenges of adversarial training.\n* An experiment on MNIST clearly shows how naive adversarial training can lead to degenerate learning, where the generative model just learns P(Y|Z) rather than P(Y|X, Z), in other words ignoring the input.\n* The evaluation is thorough, including a comparison to a recent approach (Zhang et al.) on defending against adversarial examples in trajectory prediction.\n* Includes experiments on ablations of the framework, showing how each component is necessary for obtaining the best results on both adversarial and clean inputs.\n\n### Weaknesses:\n* I'm not sure about the relevance of this work for real-world deployment. The challenge for the adversary is much more difficult in that setting, since it is no longer about just perturbing an existing trajectory in a dataset. If we imagine an adversarial driver in the scene, they would need to plan how to drive the vehicle in a certain way (which might also depend on how the other vehicles / cyclists / pedestrians in the scene will move) in order to mess up the predictions of the target vehicle.\n* Some parts of the setting and experimental setup are still unclear to me - I've listed these in the Issues section below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work is well-executed and tackles an important problem of training robust trajectory prediction models. However, I am not sure about the applicability of this work to real-world robotics, and thus am hesitant to give a whole-hearted acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I would appreciate the authors' response to the main weakness I highlight above. In addition, I have a few questions about the setup.\n\n### Questions:\n* Is there a training set and a test set? Or is the adversarial training on the same dataset as it's evaluated on?\n* Are the adversarial attacks ensured to be physically plausible? i.e. Are the adversarial perturbations restricted so that a car could physically follow the perturbed trajectory\n* Is the adversary only allowed to change the trajectory of a single vehicle in the scene or any/all vehicles?\n\n### Typos / unclear wording:\n* Page 3, line 101: \"We consider two most\" → \"We consider the two most\"\n* Page 3, footnote 1: \"For simplify\" → \"For simplicity\"\n* In equations 1 and 2, shouldn't the last term be subtracted rather than added? Because a higher value of this term means higher diversity.\n* In equation 2, shouldn't the first two terms be subtracted? Because we want to minimize negative log likelihood, not minimize log likelihood.\n* Page 4, line 136: heading should start on a new line\n* Equation 6 doesn't make sense to me. Aren't the clean loss and total (adversarial) loss two different things? Is this equation instead supposed to be a weighted sum of the two?\n* Page 6, line 227: \"shifting to another lanes\" → \"shifting to another lane\"\n* Page 8, Figure 4 caption: \"AgenetFormer\""}}, {"review_id": "DD5tB4AUdw6", "reviewer": "Reviewer_xfRa", "summary": "This work aims to study how to train robust generative trajectory prediction models against adversarial attacks.\nThe authors list 3 major challenges for this task and correspondingly design 3 components for RobustTraj:  (1) deterministic attack, (2)training with bounded noise and hybrid objective  (3) data augmentation.\nRobustTraj is applied on two trajectory prediction models. Results on nuScenes shows that the proposed method could significantly improve the robustness with small degradation on the clean data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## strengths\nThe authors clearly describe the challenges for adversarial training on trajectory prediction, and propose several reasonable solutions to address these issues.\n\nAccording to the experimental results, the proposed method could significantly improve the robustness with only small degradation on the clean data. I hope this work can help build more robust trajectory prediction work\n\nThe authors  also plug it into a planner and demonstrate that the robust trained model is able to avoid all the collisions.\n\n## Weakness\nLack of generalizability? The proposed adversarial training method may not be feasible/necessary for trajectory prediction methods like LSTM which directly predict future trajectory without sampling.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper is generally convincing with thorough experiments. The improved robustness shown in the supplementary material looks attractive.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "If the authors can address my concern in the weakness. I may be wrong.\nIt would be more strong, if it can be tested in ArgoVerse motion forecasting dataset too.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (oral)