HalentNet: Multimodal Trajectory Forecasting with Hallucinative Intents
Authors: Deyao Zhu
Mohamed Zahran
Li Erran Li
Mohamed Elhoseiny
Venue: ICLR 2021 Poster (2021)

Abstract: Motion forecasting is essential for making intelligent decisions in robotic navigation. As a result, the multi-agent behavioral prediction has become a core component of modern human-robot interaction applications such as autonomous driving. Due to various intentions and interactions among agents, agent trajectories can have multiple possible futures. Hence, the motion forecasting model's ability to cover possible modes becomes essential to enable accurate prediction. Towards this goal, we introduce HalentNet to better model the future motion distribution in addition to a traditional trajectory regression learning objective by incorporating generative augmentation losses. We model intents with unsupervised discrete random variables whose training is guided by a collaboration between two key signals: A discriminative loss that encourages intents' diversity and a hallucinative loss that explores intent transitions (i.e., mixed intents) and encourages their smoothness. This regulates the neural network behavior to be more accurately predictive on uncertain scenarios due to the active yet careful exploration of possible future agent behavior. Our model's learned representation leads to better and more semantically meaningful coverage of the trajectory distribution. Our experiments show that our method can improve over the state-of-the-art trajectory forecasting benchmarks, including vehicles and pedestrians, for about 20% on average FDE and 50% on road boundary violation rate when predicting 6 seconds future. We also conducted human experiments to show that our predicted trajectories received 39.6% more votes than the runner-up approach and 32.2% more votes than our variant without hallucinative mixed intent loss. The code will be released soon. 

Published as a conference paper at ICLR 2021
HALENTNET: MULTIMODAL TRAJECTORY FORE- CASTING WITH HALLUCINATIVE INTENTS
Deyao Zhu', Mohamed Zahran Li Erran Li Mohamed Elhoseiny!
King Abdullah University of Science and Technology
2 Udacity
Alexa AI, Amazon and Columbia University
(deyao. zhu, mohamed. e elhoseiny wewaosssiyekuuus......o.maaa...e @kkaust.ed. [mohammed, zahran) aadacitte..iy..... com [erran1 1 @mm 1. com
ABSTRACT
Motion forecasting essential for making intelligent decisions in robotic naviga- tion. As a result, the multi-agen behavioral prediction has become a core compo- nent of modern human-robot interaction applications such as autonomous driving. Due to various intentions and interactions among agents, agent trajectories can have multiple possible futures. Hence, the motion forecasting model's ability to cover possible modes becomes essential to enable accurate prediction. Towards this goal, we introduce HalentNet to better model the future motion distribution in addition to traditional trajectory regression learning objective by incorporat- ing generative augmentation losses. We model intents with unsupervised discrete random variables whose training is guided by collaboration between two key signals: A discriminative loss that encourages intents' diversity and a hallucina- tive loss that explores intent transitions (i.e., mixed intents) and encourages their smoothness. This regulates the neural network behavior to be more accurately predictive on uncertain scenarios due the active yet careful exploration of pos- sible future agent behavior. Our model's learned representation leads to better and more semantically meaningful coverage of the trajectory distribution. Our experiments show that our method can improve over the faate-of-the-ate aht-art trajectory forecasting benchmarks, including vehicles and pedestrians, for about 20% on av- erage FDE and 50% on road boundary violation rate when predicting 6 seconds future. We also conducted human experiments to show that our predicted trajec- tories received 39.6% more votes than the runner-up approach and 32.2% more votes than our variant without hallucinative mixed intent loss.
INTRODUCTION
The ability to forecast trajectories of dynamic agents is essential for variety of autonomous systems such as self-driving vehicles and social robots. enables an autonomous system to foresee adverse situations and adjust motion planning accordingly to prefer better alternatives. Because agents can make different decisions at any given time, future motion distribution is inherently multi-modal. Due to incomplete coverage of different modes in real data and interacting agents' combinatorial nature, trajectory forecasting is challenging. Several existing works focus on formulating the multi-modal future prediction only from training data (e.g.. (Tang & Salakbutdinov 2019 Alahi et al.) 2016 Casas et al., 2019 Deo & Trivedi 2018) (Sadeghian et al. [2018] Casas et al. 2012 (allmann et al, 2. This severely limits the ability of these models predict modes that
covered beyond the training data distribution, and some of these learned modes could be spurious especially where the real predictive spaces are not or inadequately covered by the training data. To improve the multi- modal prediction quality, our goal is to enrich the coverage of these less explored spaces, while encouraging plausible behavior. Properly designing this exploratory learning process for motion forecasting as implicit data augmentation approach is at the heart of this paper.
WWrk done prior Amazon.

---

Published as a conference paper at ICLR 2021
Most data augmentation methods are geometric and operate raw data. They also have been mostly studied on discrete label spaces like classification tasks (c.g. (Zhang et al. 2017: Yun et al.)
2019 Wang al.) 2019; Cubuk et al., 2019; Ho al.] 2019; Antomiou al.] 2017]] Elhoseiny & Elfeki 2019, Mikotajczyk & Grochowskij 2019; Ratner et al. (017)). In contrast, we focus on
multi-agent future forecasting task where label space for each agent is spatial-temporal. To
knowledge, augmentation techniques are far less explored for this task.
(e.g.[Tang Our work builds recent advances in trajectory prediction problem (e.g.. Tang & Salakhutdinov
(2019): Salzmann et al. (2020), that leverage discrete latent variables to represent driving behav- ior/intents (e.g. Turn left, speed up). Inspired by these advances, we propose HalentNet, a sequential probabilistic latent variable generative model that learns from both real and implicitly augmented multi-agent trajectory data. More concretely, we model driving intents with discrete latent variables 2. Then, our method hallucinates new intents by mixing different discrete latent variables up in the temporal dimension to generate trajectories that are realistic-looke looking but different from training data judged by discriminator Dis to implicitly augment the behaviors/intents. The nature of our augmentation approach is different from existing methods since it operates on the latent space that represents the agent's 'ss behavior. The training of these latent variables is guided by collabora- tion between discriminative and hallucinative learning signals. The discriminative loss increases the separation between intent modes; we impose this classification loss that recognizes the one-hot latent intents corresponding to the predicted trajectories. We call these discriminative latent intents as classified intents since they are easy to classify an existing one-hot latent intent (i.c., low en- tropy). This discriminative loss expands the predictive intent space that we then encourage to explore by our hallucinated intents' loss. As we detail later, we define hallucinated intents as a mixture of the one-hot classified latent intents. We encourage the predictions of trajectories corresponding to hallcuinated intents to be hard to classify to the one-hot discrete latent intents by hallucinative loss but, in the meantime, be realistic with real/fake loss that we impose. The classification, hallucina- tive, and real/fake losses are all defined on top of a Discriminator Dis, whose input is the predicted motion trajectories and the map information. We show that all these three components are necessary to achieve good performance, where we also ablate our design choices.
Our contributions are summarized as follows.
We introduce new framework that enables multi-modal trajectory forecasting to learn dynamically complementary augmented agent behaviors.
We introduce the notion of classified intents and hallucinated intents in motion forecast- ing that can be captured by discrete latent variables We introduce two complementary learning mechanism for each to better model latent behavior intentions and encourage the novelty of augmented agent behaviors and hence improve the generalization. The classified intents 2 is defined not to change over time and are encouraged to be well separated from other classified intents with a classification loss. The hallucinated intents ân, on the other hand, changes over the prediction horizon and are encouraged to deviate from the classified intents as augmented agent behaviors.
Our experiments demonstrate most 26% better results measured by average FDE com- pared to other state-of-the-an methods on motion forecasting datasets, which verifies the effectiveness of our methods. We also conducted human evaluation experiments showing that our forecasted motion is considered 39% safer than the runner-up approach. Codes, pretrained models and preprocessed datasets are available at pettps://githee com/ Vision-CAAn HAARR/HalentNet
2 RELATED WORK
Trajectory Forecasting Trajectory forecasting of dynamic agents has received increasing atten- tion recently because it is core problem to number of applications such as autonomous driving and social robots. Human motion is inherently multi-modal, recent work (Lee et al. 2017 Cui et al.
2018 Chai et al. 2019; Rhinehart al. 2019 Kosaraju et al. 2019 Tang & Salakhutdinov 2019
Ridel et al. 2020 Salzmann et al., 2020; Huang et al., 2019: Mereat et al. (2119) has focused on
learning the distribution from multi agent trajectory data. (Cui et al. [0188] Chai al. 2019 Ridel
ct al. 2020 Mercat et al. 2019) predicts multiple future trajectories without learning low dimen-
sional latent agent behaviors. (Lee al. 2017 Kosaraju et al. 2019 Rhinehart et al. 2019 Huang
2

---

Published as a conference paper at ICLR 2021
et al. 2019) encodes agent behaviors in continuous low dimensional latent space while (Tang & Salakbutdimov 2019; Salzmann et al., 2020) uses discrete latent variables. Discrete latent variables succinctly capture semantically meaningful modes such as turn left, turn right. (Tang & Salakhut- dinov] 2019: Salzmann et al. 2020) learns discrete latent variables without explicit labels. Built on top of these recent work, we hallucinate possible future behaviors by changing agent intents. As the forecast horizon is a few seconds, these are highly plausible. We use a discriminator to encourage augmented trajectories to look real.
Data augmentation Data augmentation is popular technique to mitigate overfitting and improve generalization in training deep networks (Shorten & Khoshgoftaar 2019). New data is typically gen- erated by transforming real data samples in the original input space. These transformations range from simple techniques (c.g. random flipping, mirroring for images, mixup (Zhang et al., 2017) and Cutmix (Yun et al. 2019)) to automatic data augmentation techniques (e.g. AutoAAument (Cubuk et al., 2019g) and class identity preserving semantic data augmentation techniques (Ratner et al., 2017 (e.g. changing backgrounds of objects). Recently data augmentation via semantic transfor- mation in deep feature space (Liu et al. 2018, Wang et al. 2019: Li et al. (020a) has also been proposed. ISDA (Wang et al. 2019) proposes loss function to implicitly translate training samples along with semantic directions in the feature space. For example, a certain direction corresponds to the semantic translation of maaee-esppectacled When a person's 's feature without glasses is trans- lated along this direction, the feature may correspond to the same person but with glasses. MoEx (Li et al. 2020a) proposes a new augmentation method that leverages the first and second moments extracted and re-injected by feature normalization. Specifically, replaces the moments of the learned features of one training image by those of another and interpolates the target labels. Our data augmentation is also in the latent space, which represents agent behavior.
imaginnaive/Hallucinatiee models. GANs (Goodfellow et al. 2014 Radford et al. 2015) are a powerful generative model, yet they are not explicitly trained to go beyond the training data to improve generalization. Inspired by the theory of human creativity (Martindale) 1990), recent ap- proaches on generative models were proposed to encourage novel visual content generation in art and fashion designs. In (Elgammal et al., (017), the authors adapted GANs to generate uncon- ditional creative content (paintings) by encouraging the model to deviate from existing painting styles. In the fashion domain, (Sbai et al. 2018) showed that their model is capable of producing non-existing shape like "pants to extended arm sleeves" that some designers found interesting.
The key mechanism in these methods is the addition of deviation loss, which encourages the generator to produce novel content. More recently, (Elhoseiny Elfeki 2019) proposed a method for understanding unseen classes, also known zero-shot learning (ZSL), by generating visual representations of synthesized unseen class descriptors. These visual representations are encouraged to deviate from seen classes, leading better generalization compared earlier generative ZSL methods. (Zhang et al., (201)) and (Li et al. 2020b) introduced methods to generate additional data based on saliency maps and adversarial learning for few-shot learning task, respectively. In the field of navigation, (Xiao et al., (020a) and (Xiao et al., (2200)) utilized geometric information to hallucinate new navigation training data. In contrast these earlier methods, our work has two key differences. First, our work is sequential probabilistic generative model focusing on motion forecasting requiring time-series prediction in continuous space. Second, the deviation signal in (Elgammal et al.) [2177] Sbai et al. 22018 Elhoseiny & Elfeki, 2019) is based on defining labeled discrete seen styles and seen classes, respectively. In contrast, we model the deviation from a discrete latent space guided by a deviation signal to help the model imagine driver intents without supervision signal. Similar to MoEx (Li et al. (220a). augmented trajectories are dynamically generated during training. We believe we are the first to propose data augmentation method in latent space for trajectory forecasting.
3 METHOD
Problem Formulation We are aiming at predicting the future trajectory ya: of specified agent given the input states x, which contains the historical information like positions and heading angles of the agent itself and the surrounding agents, and a semantie map patch m, which offers context information like drivable region, by generating a distribution Pyy|x, m) to model the distribution of real future trajectory ytt-
3

---

Published as a conference paper at ICLR 2021
m))
122
real/fake
8
classify z
Discriminator
x
P(y
low entropy
Generator
Classified latent Intent/ Behavior
P(([k,m) m)
m
real/fake
2.2'
classify
Discriminator
Encoder
Decoder
P X,
high entropy
Generator
Carotes Frint
Hallucinative Latent Intent/Behavior
Figure I: Overview of our architecture. The generator is trained to infer the behavior intents z and forecast the future trajectories. addition to GAN loss and prediction loss like MLE, we propose classified latent intent behavior that classifies the latent code behind trajectories, and hallucinative learning that generates novel and plausible trajectories by mix two latent codes. White and color points denote the ground truth and generated trajectories, respectively.
Our Model Motion forecasting the real world is multi-modal task. There are usually multiple possible futures given the same state. To accurately model this diversity, define latent code to represent different intents the predicted agent inspired by literature (c.g.. (Tang & Salakhutdinoy 2019 Salzmann et al. 2020).. We denote the input state as local map m, and the correspond- ing ground truth future as ygt- The possible behaviors are modeled by the distribution of latent code conditioned the input state and the map P(z|x, m). Then, the predicted trajectory distribution is calculated by conditioning on both input state and the latent code P(y 2 x, m). For motion fore- casting tasks, we use maximum likelihood estimation (MLE) loss on the ground truth future as the learning objective L = log P(y gt) m). Note that we do not have the label for the latent code in the dataset. Similar to (Tang & Salakhutdinov, 2019 Salzmann et al.) 2020), we represent latent code as discrete random variable. The learning objective can be rewritten as follows.
L = - log P pyyglx m) =- log Σ [[Zz M) 2, x)]
(I)
Hence, we obtain an unsupervised latent code that captures some uncertainty of the future without knowing its label. The distribution P(z[x, m) can be modeled by any model that outputs cate- gorical distribution. P(y/z,x. x. m) is usually modeled by models that output multivariate Gaussian distribution. An overview of our model can be found in Fig It consists of sub-networks, generator module and a discriminator module described in the following paragraphs.
Generator The generator is the prediction model that produces the future trajectory distribution P(y|x, m) given agent states x and local map m. As the possible future is multi-modal, the output distribution should model this uncertainty. As we discussed earlier, model distribution P(z|x, m) and P(y z, x, m) by neural networks. We use a discrete random variable to represent the latent code The uncertainty of the future trajectory can be factorized hierarchically into intent uncertainty and control uncertainty (Chai et al. 2019). The intent uncertainty reflects different intents or behavior modes of the agent. Furthermore, the control uncertainty covers other minor noise. As the simple Gaussian distribution P(y|z, x, m) is not expressive enough to model the complex uncertainty of multi-modal behaviors, this framework encourages the latent code distribution P xx, m) to cover more the intent uncertainty. We denote the modules that generate the latent code distribution and the trajectory distribution as encoder Enca and decoder Dec, with parameter 0 and ø, respectively. In addition, Enc, also encodes agent states and the local map m into feature vector e, which is part of the decoder's 's input. Note that our method does not introduce further restrictions for the model structure. Any model that fits this framework et al., 2020). In our experiments, we select Tra-
be used
generator like MFP &
Salakhutdinov 2019) and TTajectron++(SSllmaaal jectron++ our generator. Its original learning objective Tunjtt is shown in Appx In summary,
4

---

Published as a conference paper at ICLR 2021
the process of the generator can be represented by the following equations.
P(z|x, m), e Enco (x, m) 2 P(z x, m)
P x, m) = Deca (2
Discriminator The discriminator Diss with parameters v takes either a real trajectory Ygt or generated one y sampled from our predicted distribution together with the local map m as input to judge whether the trajectory is real or generated following GAN framework (Goodfellow et al.) 2014). This helps the decoder inject map information into the learning signal and alleviate the vio- lation of road boundaries in prediction. Besides,
add classification head the discriminator. When the input data is generated, this head needs to recognize the latent code & the generator used for creating the input trajectory. In this way, the generator is forced to increase the difference among latent codes and give us more distinct and semantically meaningful driving strategies. This fur- ther discussed in the following paragraphs. The following equation describes the function of our discriminator.
D(y), P(zly, m) ( m)
(3)
Trajectory y is either the ground truth future yt the sample from predicted trajectory distribution y. D(y) is the score to indicate whether y is real or synthetic. P(zly. m) is the classified distri- bution. Our discriminator is modified from the one in DCGAN (Radford al. (2015) by adding fully-connected classification head at the end to classify the latent code. Trajectories y are trans- formed into the format that convolutional layers can handle via differentiable rasterizer trick (Wang) ct al. 2020) and stacked together with the local map as the input for the discriminator as described Appx
Learning Methods Our architecture can be trained by GAN learning objective, together with original learning loss of the generator module depends on the model we choose to combine with. To learn better behavior representation and improve the quality of predicted trajectory distribution, we introduce two new methods Classified latent Intent Behavior and Hallucinative Latent Intent for training.
Classified latent Intent Behavior In the real world, humans can recognize different behavior in- tents by looking at the trajectories. To encourage the latent code contain more information about the intent uncertainty and less about the control uncer- tainty, we mimic this phenomenon and let the dis- criminator classify the latent code behind the gener- ated trajectories. The classification function can be trained by a cross-entropy loss.
Algorithm 1: Training Process
Initialize Enceg Decep Disg; Initialize learning rate (3; while not converge do
// discriminator
Ygt: x Dataset
Sample normal prediction ÿ
LD (11011100 +2) Le
Le = Σ
2, log Zi, where 2~ Prz/x, m)
0 aV
// generator
(4)
Dataset
zi denotes the i-th dimension of the vector is the latent code under the input trajectory, which is a one-hot vector sampled from the multinoulli dis- tribution P(z|x, m). ž denotes the classified cat- egorical distribution generated by our discrimina- tor. Minimizing this loss encourages the decoder to widen the difference among predictions from differ- ent z to reduce the classification difficulty for the dis- criminator. Therefore, we reduce the overlap among output distributions from different latent codes and sharpen them to increase accuracy. Since model trained to classify trajectory into I, we name å classified intent. Note that this loss only applied for generated trajectories since do not have the latent code for ground truth trajectory.
Generate hallucinated trajectory 9n.
P(y êh, x. m)
La (D($) - 1) Le
La (D($n) -
1)2 1
La
0- VVo(ALo,
A)La,)
Not. Dataset
VV // Trajectron++ loss end
5

---

Published as a conference paper at ICLR 2021
Hallucinative Learning Latent codes trained to model intents in the training data. Each predicted trajectory y is calculated from single latent code 2 for all the prediction steps. Assume the predicted horizon is T, y [0t, 9/2. .... .r.. 9. each step is generated conditioned on the same 2. And our discriminator is trained to recognize 2 given the synthetie trajectory y by the classification loss. Besides, the MLE loss encourages synthetic trajectories to be similar the training data. Therefore, the discriminator implicitly classifies the training data into one of the latent code z.
We propose novel way utilize this property and learn beyond the training data by encouraging the model to generate trajectories from unfamiliar driving behaviors. This is done by first sampling second different latent code 2' in addition to the original one å and randomly selecting a time step to The prediction until time step ty in this case ((ūi..... ..... Dt, D is conditioned on the first latent code and we switch to 2' for the remaining steps ((),, +1. D. By this way, we hallucinate new intent by stacking 2 learned intents in the temporal dimension. We denote this mixed hallucinated intent as 2,, and name it hallucinated intent. The predicted distribution from such a intent denoted as P(y|2n, x, m). We aim to encourage the hallucinative trajectories Yn to be plausible but dif- ferent from the training data. To achieve this, we minimize the cross entropy between the uniform distribution and our intent class distribution.
1 - E
Ln= =
log &
N indicates the number of latent codes. Σ, is the i-th dimention of the classified distribution ž. encourages the hallucinative trajectory to be hard to be classified into any latent code z, and therefore, to be different from the training data. The plausibility of the hallucinative trajectory is encouraged by the additional GAN loss. In this way, we implicitly apply data augmentation in the latent space to train more powerful discriminator and improve the generator prediction quality. We call this method hallucinative learning inspired from literature (c.g., Hariharan & Girshick (017)).
Training We use LSGAN(Mao et al., 2017) loss with spectral pormalization et al. 2018 as our GAN learning objective. We also keep the original Trajectron++ learning loss Luaj++ to maintain the performance in case Trajectron++ is our generator. The combination of GAN learn- ing, training of the original generator, classified latent intent behavior, and hallucinative learning is demonstrated in Alg (Detailed version in Appx We use hyperparameter À to balance the training between classification learning and hallucinative learning for the generator by adjusting the weighting of the learning loss.
4 EXPERIMENTAL RESULTS
We compare the performance of our method with state-of-the-art models. To demonstrate our method's performance in complex scenarios, we focus on evaluating the nuScenes dataset (Cae- sar et al.] 2019a) which contains about 1000 driving scenes in 2 cities (Boston and Singapore) with dense traffic. Each scene of them has annotations for pedestrians and vehicles, sampled at rate of 2 Hz, and about 20 seconds long (40 frames). Besides, both cities include maps, which are required in our method. In addition, we also evaluate our method on widely-used pedestrian datasets ETH PPellegrini et al. 2009) and UCY (Leal-Taixé et al. 2014).
Evaluation Metries We use average le displacement error (ADE) and final l2 displacement error (FDE) to evaluate the prediction performance. Each of them contains some sub-versions. ADE- ML/FDEMLL is the ADE/FDE calculated using the most likely predicted trajectories. minADE- k/minFDE-k, select k candidate trajectories for each prediction and use all candidates' minimal value as the final score. ADE-Ful//DDEFuu represents the quality of output distribution. To com- randomly sample 2k trajectories and calculate the average score.
pute ADE-Full/FDE-Full,
Model Setting Our models are trained in two different scenarios. In the first scenario, we train the model totally from scratch, and in the second one, we finetune train the discriminator from scratch. The number of latent code z is set as 25 latent codes following (Salzmann al. 2020). Our method is trained for 23 epochs with the pretrained generator and 35
pretrained generator and

---

Published as a conference paper at ICLR 2021
Table nuScenes: Detailed comparison with Trajectron++. A% is relative improvement.
Table 1: nuScenes: Vehicle prediction
Model
1s
2:
FDE
2,
46
Corst Velecity
Metric
Model
Is
36
4n
S4
6x
0.16
1.52
4.55
CAR-Na
0.38
1.21
2.17
344
SpAGNNN
+43
+20
+25
0.07
045
0.01
203
2:50
FDE (ML))
Ours
Ira++
0366
0.05
0.41
0.40
1.06
1.00
2,06
1.88
3.06
3.45
5.29
4.52
A3
+16
+2
+6
+9
+12
+14
Table 2: nuScenes: Pedestrian prediction
mind DE-10
Di-FDE-10
Trat+
0.05
0.04
0.31
0.65
0.65
0.99
1.15
1.49
1.75
2.57
Model
1s
2s
FDE
3s
4s
10
Ours
Traje+
030
0.05
0.15
0.28
0.42
0:71
0.61
0.89
A%
-16
0
+3
-10
-14
+13
Tmj++
Ours 0.02
0.01
0.17
0.17
0.35
0.62
0.57
RR. Viol.
Traje
6.34%
0.25%
0.45%
0.37%
1.30%
2335
3.21%
7.04%5
12339%
6.00%
1972P5
9.22%
A%
epochs from scratch for vehicles. The training with a pretrained model lasts about 16 hours with single NVIDIA V100 graphic card and about 24 hours from scratch.
Comparison Methods We compare our contribution to state-of- oftte- methods. S-LSTM (Alahi et al. (016)) uses LSTM to predict trajectories and pool the hidden states among agents to model their interaction. CSP (Deo & Trivedi 2018) discretizes behaviors into a fixed number of classes and predict the best possible behaviors. CAR-Net (Sadeghian et al. 2018) utilizes visual atten- tion mechanism to encodes the surrounding environment and SpAGNN (Casas et al.) 2019) detects agents first from LIDAR and semantic map. Then, graph neural network decoder interactively pre- dicts their trajectories. Trajectron++ (Salzmann et al., 2020) encodes surrounding vehicles using a graph neural network model and infers the behavior intents to produce multi-modal prediction. As Trajectron++ is the best model among these baselines, we perform an extensive comparison with using the released pretrained model.
nuScenes Dataset We run extensive experiments the nuScenes dataset (Caesar et al. 2019b)
to evaluate and analyze our trajectory forecasting performance and verify model ability to learn dy- namically complementary augmented agent behaviors. In this task setting, the model forecasts seconds future with maximal 4 seconds of history information during training. However, the predic- tion horizon for evaluation up to 6 seconds to demonstrate our model's generalization capacity. NuScenes dataset contains many agent categories like adult pedestrian and truck. We group them into two semantic classes vehicle and pedestrian, train individual models on them and report the performance separately, following (alzmann et al. 2020).
Our method achieves the best performance compared to other state-of-the-ae approaches on the FDE with minimal 4 seconds of future information during testing; see Tab Due to the instability of GAN, we remove the diverging training cases and average the numbers over 3 stable runs. Although other methods do not report values 2s and 4s, can that the performance of HalentNet increases and HalentNet outperforms existing approaches as we predict more time steps in the future. The complementary learning mechanism and hallucinated intents show noticeable improvement in vehicle trajectory prediction.
We run more experiments to examine further our method's performance and Trajectron++ Salzz- mann et al. 2020) as Trajectron++ outperforms other baseline approaches. We used various metrics with the prediction horizon from 1 second seconds for all tracked objects with at least 6-second available future data. The evaluation results are demonstrated in Tab Our method outperforms Trajectron++ in almost all metries with a significant margin. Besides, the methods also generalize well when we extend the prediction horizon. We obtain about 26% on average FDE over the output distribution (FDE Full) and 52% for the road boundary violation improvement over the baseline model in the 6-second prediction case. Superiority is when the prediction horizon is more extended. HalentNet trajectories show more respect to road boundaries and output plausible trajectories pro- duced from hallucinated intents that are changed over the prediction horizon and are encouraged to deviate from the classified intents. The evaluation on the pedestrian nuScenes benchmark is listed We obtain 8% improvement in the 4s prediction horizon case.
in Tab 2
7

---

Published as a conference paper at ICLR 2021
Table 4: With Map. We evaluate our model and Trajec- tron++ (both trained from scratch) on the combination of ETH and Hotel datasets with maps. Maps help to learn a better discriminator, hence increase the performance of our method on ETH and Hotel sets. Note that Trajectron+++ also takes maps as imput in this experiment for a fair comparison. Our method is significantly better.
Methods
ADE ML
FDE ML
minADE-20
min/-DE-20
Trajectron++
0,48
1.15
0.37
0.89
Ours
0,46
1.08
0.31
0.70
Table 5: Ablation study nuScenes dataset.
Balancing classified la- tent intent behavior and hallucina- tive learning by selecting proper = 0.5 in Algo [2 helps to get a best FDE averaged over 2000 ran- dom samples)
Figure 2:
Components be
FDE Full
FDE ML.
B. Viclations
Dis
Ln
3s
4s
35
3s
4s
56
3s
4s
5s
1.21
2.17
3.41
1.00
1.88
3.06
1.30%
3.21%
6.00%
1,23
2.32
3.81
1.01
1.90
3.09
1.48%
4.91%
10.84%
1,29
2.54
4.27
1.02
1.99
3.31
1.75%
6.09%
12.44%
1.28
2.35
3.75
1.08
2.01
3.23
1.64%
4.62%
8.66%
1.06
7.04%
12.95%
Pedestrian Datasets To further demonstrate our performance, we train our model on two widely used pedestrian datasets; ETH(Pellegrini et al. (2099 and UCY(Leal-Taixx Taixé et al., 2014). UCY (no map). UCY does not provide map information that is important for our method. We still test our method in this case in Tab. i0 Appendix E This can be viewed as variant of our model since the map is not provided. We observe slight improvement in the FDE results with about 7% over Trajectron+++. As we show later, the improvement is more significant when map information is used that we think is available in most cases.
ETH (with map). We split the data by 70%, 15%, and 15% as training set, validation set, and test set, separately. Then, we combine these two sets as one big dataset and train both our method and Trajectron++ from scratch with map information. The assessment uses observation period of 8 timesteps (3.2s) and a projected horizon of 12 timesteps (4.8s). The results are shown in Tab Our method is significantly better than Trajectron++ with an improvement of about 20% on minFDE-20.
Ablation Study To better demonstrate and understand each component's effect in our model, we create model variants by removing the evaluated components step by step and showing their perfor- mance. The evaluation on the nuScenes dataset with the vehicle prediction for all tracked objects with at least 6-second available future data. The results are listed in Tab Dis, Le, and L,, denote the discriminator, the classification learning, and the ballucination learning, respectively.
Compared to the variant without all the components we list, the model with the discriminator out- performs it by 15% on average FDE over the output distribution (FDE Full) and 30% for the road boundary violation. The FDE of most likely prediction is also better after 3 seconds. This indicates that the discriminator helps to improve the quality of output distribution. One of the possible reasons is the injection of the map info. Although the generator takes the local map m as input, we do not guarantee that the plain model will use it. As a trajectory that violates the road boundary can be easily recognized as fake data by the discriminator, the map info is injected into the GAN learning objective. Hence, optimizing this loss helps to push map information into output distribution. We observe that we can gain additional improvement when add the classification loss L. We think this is because Le only encourages the classified intents 2 to be distinguishable from each other. And this property doesn't have clear relationship the performance. Our method benefits from the implicit behavior augmentation by the hallucinated intent. When we implicitly augment the data by hallucinative intent loss T.,, mixing intents during training with àn by combining clas- sified intents, we observe further boost in the performance. The FDE is more than 5% better than the discriminator only variant Dis, and the road boundary violation about 20-30% better, show- ing the effectiveness of the hallucinative learning; see Table 5 Note that although L. alone does not improve, it is still important to encourage the hallucinative signal L,, to be more explorative. This is since the exploration of Ln, depends on the classified intents' diversity that Le increases. Mathematically, the classification loss L. encourages reducing the entropy of the categorical output distribution over 2, and the hallucinative loss promotes that mixing these intents can still be plausi-
8

---

Published as a conference paper at ICLR 2021
Table 6: Ablation study on different designs for hallucinative loss L, nuScenes.
Methods
FDE Full
FDE ML
5s
5s
La (delsullseentropy))
1.21
2.17
3.41
1,00
1,88
3,06
La (mixxp))
1.19
2.20
3.60
1.02
1.97
3.60
Lu
1.35
2.69
4.58
1,20
2.63
4.66
Table 7: Human Evaluation Results
Compured Methods
Votes for method
Votes for compared method
Our Advaniage
Variant without LA
427
323
2%
Trajectron++
437
313
39.6%
ble. In other words, L trains the discriminator to classify trajectories and LA trains the generator to output trajectories that are hard to be classified if we use hallucinated intent. If we remove The and keep La only, our classifier cannot be trained, which make L. meaningless. Hence, the two losses are complementary to one another. The complementary importance of Le to Ln, can be explained by drop performance when Le only is discarded (second row in Tab 5
Fig 2 shows our exploration of how to balance the classification learning and hallucinative learn- ing. represents the importance of classification learning. When À 1, method is reduced to the variant without hallucinative learning. We set the in Alg from 0.0 to 1.0 for training sep- arately and plot the corresponding average FDE over the output distribution. The results suggest that properly balancing classified latent intent behavior and hallucinative learning helps improve performance.
Hallucinative loss Ln defined 5 is used to encourage the classification difficulty of the hal- lucinated trajectory nn. Ln is defined the cross entropy between the uniform distribution and the classification results in our method. Here we denote our original design choise as Ln (de faull) In addition to this design choice, we also experiment with another 2 possibilities: LA (mirup) and La (N + 1). LA (mmirup)) is defined as the cross entropy between a discrete distribution that only has non-zero probabilities on the latent codes (probabilities equal 50% for both) used together as the hallucinated intent 2; and the classification results. For T,, + 1). we define new class label for all the hallucinated trajectories. (N + 1) is the cross entropy between this new class and the prediction results. Results are shown in Tab Our design choice achieves the best performance, but the TwoHot variant also shows comparable results. The performance of AdditionClass is much worse compared to our design and TwoHot.
Human Evaluation We use Amazon Mechanical Turk to evaluate the quality of our prediction. We randomly selected 150 paired scenes, each of which is evaluated by five human subjects on MTurk who are requested to judge which model predicts better trajectory given a scene. Each scene evaluated by times. Therefore, each comparison contains 750 votes in total. Our method generates better trajectories compared to our variant without hallucinative learning measured by 32.2% more votes and Trajectron++ measured by 39.6% more votes. Results shown Tab 7
CONCLUSION
In this paper, we propose HalentNet, a probabilistic latent variable framework that hallucinates novel trajectories via transformations in discrete latent agent behavior space. Our method contains two complementary learning mechanisms that encourage diverse and novel generation to regulate the neural network behavior and achieve more accurate predictions show that HalentNet can significantly improve generalization for multi-modal future predictions in multi-agent settings and reduces the boundary violation metric by more than 50%.
uncertain scenarios. We
ACKNOWLEDGEMENT
This work is funded by a KAUST BAS/I/1685-0144 The authors wish to thank Amazon Mechanical Turkers without who helped with our human studies.

---

Published as a conference paper at ICLR 2021
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 961-971, 2016.
Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks. arXiv preprint arxivv:1111 17110:4400 2017.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv: 1027, 2019a.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. arXiv preprini arXiv: 1903. 1027, 2019b.
Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urtasun. Spatially-aware graph neural networks for relational behavior forecasting from sensor data. arXiv preprint arXiv:1910.08233, 2019.
Yuning Chai, Benjamin Sapp. Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple prob- abilistic anchor trajectory hypotheses for behavior prediction. arXiv preprint arXiv:1910.05449, 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 113-123, 2019.
Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou, Tsung-Han Lin, Thi Nguyen, Tzu- Kuo Huang, Jeff Schneider, and Nemanja Djurie. Multimodal trajectory predictions for au- tonomous driving using deep convolutional networks. CoRR, abs/1809.10732,, 2018. URL
httt:://arxiv> org/ www.:/gg//......... 10732
Nachiket Deo and Mohan M Trivedi. Multi-modal trajectory prediction of surrounding vehicles with maneuver based lstms. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1179-1184, IEEE, 2018.
pp.
Ahmed Elgammal, Bingchen Liu, Mohamed Elhoseiny, and Marian Mazzone. Can: Creative adver- sarial networks, generating" art" by learning about styles and deviating from style norms. arXiv preprint arXiv: 2017.
Mohamed Elhoseiny and Mohamed Elfeki. Creativity inspired zero-shot learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5784-5793, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: So- cially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2255-225522 2018.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3018- 3027, 2017.
Daniel Ho, Eric Liang, Chen, Ion Stoica, and Pieter Abbeel. Population based augmentation: Efficient learning of augmentation policy schedules. In International Conference Machine Learning. pp. 2731-2741, 2019.
Xin Huang, Stephen G. McGill, Jonathan A. DeCastro, Brian Charles Williams, Luke Fletcher, John Leonard, and Guy Rosman. Diversity-aware vehicle motion prediction via latent semantic sampling. ArXiv, 00//1911.12236 2019.
10

---

Published as a conference paper at ICLR 2021
Boris Ivanovic and Marco Pavone. The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. Proceedings of the IEEE International Conference on Computer Vision, pp. 2375-2384, 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Vineet Kosaraju, Amir Sadeghian, Roberto Martin- Martin, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. arXiv preprint arXiv: 03399, 2019.
Laura Leal-Taixé, Michele Fenzi, Alina Kuznetsova, Bodo Rosenhahn, and Silvio Savarese. Learn- ing image-based motion context for multiple people tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3542-3549, 2014.
L. Leal-Taixé, M. Fenzi, A. Kuznetsova, B. Rosenhahn, and S. Savarese. Learning an image-based motion context for multiple people tracking. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B. Choy, Philip H. S. Torr, and Manmohan Chandraker. DESIRE: distant future prediction in dynamic scenes with interacting agents. In 2017 IEEE Conference Computer Vision and Puttern Recognition, CVPR 2017, Honolula, HI. USA, July 21-26, 2017, pp. 2174, IEEE Computer Society, 2017. doi: 1..110//CPR.2211..220
URL https: htg.///i..ooggg/// org 1109/ CVPR. 2017.231 233
Bo-Yi Li, Felix Wu, Ser-Nam Lim, Serge J. Belongie, and Kilian Q. Weinberger. On feature nor- malization and data augmentation. ArXiv, abs/2002.11102, 2020a.
Kai Li, Yulun Zhang, Kunpeng Li, and Yun Fu. Adversarial feature hallucination networks for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13470-13477 13479, 2020b.
Bo Liu, Xudong Wang, Mandar Dixit, Roland Kwitt, and Nuno Vasconcelos. Feature space transfer for data augmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9090-9098, 2018.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Confer- on Computer Vision, pp. 2794-2802 2017.
Colin Martindale. The clockwork muse: The predictability of artistic change. Basic Books, 1990.
Jean Mercat, Thomas Gilles, Nicole El Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Pita Gil. Multi-modal simultancous forecasting of vehicle position sequences using social attention. arXiv preprint 190.0399000033500 2019.
Agnieszka Mikolajczyk and Michat Grochowski. Style transfer-based image synthesis as efficient regularization technique deep learning. In 2019 24th International Conference on Methods and Models in Automation and Robotics (MMAR), pp. 42 47. IEEE, 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint
05957, 2018
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In 2009 IEEE 12th International Conference Computer Vision, pp. 261-268. IEEE, 2009.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv preprint arXiv:1511.064:5 2015.
Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learn ing to compose domain speciffc transformations for data augmentation. In Advances in neural information processing systems, pp. 3236-3246, 2017.

---

Published as a conference paper at ICLR 2021
Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: tioned on goals in visual multi-agent settings. The IEEE International Conference on Computer Vision (ICCV), October 2019.
Prediction condi-
Daniela Ridel, Nachiket Deo, Denis Wolf, and Mohan Trivedi. Scene compliant trajectory forecast with agent-centric spatio-temporal grids. IEEE Robotics and Automation Letters, 5(2):2816 2823, 2020.
Amir Sadeghian, Ferdinand Legros, Maxime Voisin, Ricky Vesel, Alexandre Alahi, and Silvio Savarese. Car-net: Clairvoyant attentive recurrent network. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 151-167, 2018.
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical con- straints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1349-1358, 2019.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Multi- agent generative trajectory forecasting with heterogeneous data for control. 2020.
Othman Sbai, Mohamed Elhoseiny, Antoine Bordes, Yann LeCun, and Camille Couprie. Design:
Design inspiration from generative networks. In ECCV workshop, 2018.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):60, 2019.
Charlie Tang and Russ R Salakhutdinov. Multiple futures prediction. In Advances in Neural Infor- 2019.
mation Processing Systems, pp.
Anirudh Vemula, Katharina Muelling, and Jean Oh. Social attention: Modeling attention in human crowds. In 2018 IEEE international Conference on Robotics and Automation (ICRA), pp. 1-7. IEEE, 2018.
Eason Wang, Henggang Cui, Sai Yalamanchi, Mohana Moorthy, Fang Fang-Chieh Chou, and Nemanja Djuric. Improving movement predictions of traffic actors in bird' s-eye view models using gans and differentiable trajectory rasterization. arXiv preprint arXiv:2004.06244 06247, 2020.
Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. Implicit semantic data augmentation for deep networks. Advances Neural Information Processing Systems, pp. 12614-12623, 2019.
Xuesu Xiao, Bo Liu, and Peter Stone. Agile robot navigation through hallucinated learning and sober deployment. arXiv preprint arXiv:2010.08099, 2020a.
Xuesu Xiao, Bo Liu, Garrett Warnell, and Peter Stone. Toward agile maneuvers in highly constrained spaces: Learning from hallucination. arXiv preprint 2:Xiv:2007.144777 2020b.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceed- ings of the IEEE International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongguang Zhang, Jing Zhang, and Piotr Koniusz. Few-shot learning via saliency-guided hallu- cination samples. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2770-2779, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17X1r:1710.09412, 2017.
12

---

Published as a conference paper at ICLR 2021
A APPENDIX
In this document, we present more explanations and details about the training and the testing results on the NuScenes dataset and the pedestrian datasets with more visualization for our method. It contains the following sections:
Training Details
Qualitative Results
Pedestrian Experiments
The code and pretrained models will be released in the future (soon).
B TRAINING DETAILS
Original Loss of Generator In this dataset, we initialize the generator of our Halent model with the pretrained Trajectron++ model (Salzmann et al. 2020). The original Trajectron++ training loss Luyj kept in our method.
Lmmje- -E Eg-g(n x( Togp(y m. 2)] k DR DRr m.
m)) (x2n((;)
Here, 12 is set to 1. Instead of directly learning the distribution of latent intents p(z|x, m). Tra- jectron++ learns ((z|x, m, yot) which additionally conditioned on ground truth trajectory during training. p(z|x, x, m) is learned by reducing the KL divergence between ((z[x, x, m, Var and p(z|x, m). k is gradually increase to enhance the information transfer. Note that only p(z]x. m) is used during testing.
Differentiable Rasterizer To combine the trajectory and the local map m into a acceptable format for the CNN-based discriminator, we use differentiable rasterizer (Wang al. 2020) to convert y, which can be represented by sequence of T positions {(11, (T2, (2) "2)....(2:) Ur into T 2D occupancy grids {G), 92. GT) Each grid G. a tensor with same weight and height of m. In detail, it creates bivariate Gaussian distribution N(pr, Σx) for every time step where
14 = (Z Vt), diag(o², )²). a a hyperparameter. The value for cell (i,j) of G. is the scaled probability density location (i,j) in the map coordinate system
G( N((i, E,)
N(ulu, E.)
Here, we normalize the occupancy grids so the maximal amplitude equal to k. By this way, we obtain 2D trajectory grids (91, Gr,-..rr, which be processed by CNN and are differentiable w.r.t the original trajectory. In our experiments, we set k = 9 and o 5 based on the hyperparameter search on the validation set.
Training We set À = 0.5 to balance the classified latent intent behavior and hallucinative learning. The model is trained by Adam optimizer (Kingma & Bal 2014). The pretrained model trained by 12 epochs. We continued the training for another 23 epochs with our method and kept the original learning rate for our generator. The learning rate of the discriminator is lower compared to the generator to avoid large gradient at the beginning of training.
C ADDITIONAL RESULTS ON NUSCENES
Here report the ADE scores of our method compared to Trajectron+++ and the variants of our method in Tab as a supplementary to Tab and Tab The evaluation on the nuScenes dataset with the vehicle prediction for all tracked objectswith at least 4-second available future data. Com- pared to Trajectron++ (the last row), we obtain 25cm improvement in the 48 case measured by ADE-Full, which is about 21 % better.
13

---

Published as a conference paper at ICLR 2021
Table 8: ADE scores on nuScenes dataset.
Components
ADE Full
ADE ML
Dis
La
Ln
3s
+
0,53
0.91
0.43
0.76
0.44
0.55
0.44
0.79
0.56
0.97
0.47
0.83
0.67
1.16
0.45
0.82
Table Results for the variant without discriminator map input on nuScenes.
Methods
FDE Full
FDE ML
4s
3s
4s
Ours
1.21
2.17
3.41
1,00
1.88
3.06
Ours without discriminator map input
2.38
1.07
1.99
Trajectron++
1.52
2.80
1.06
2.06
3.46
In Tab we show the importance of the map information the discriminator by removing the map input to the discriminator and keep all the rest parts the same (the map input to the generator is kept). Due to the lack of map information, the diseriminator cannot be well trained and the performance drops compared to our full model. Models are evaluated on nuScenes dataset.
14

---

Published as a conference paper at ICLR 2021
D QUALITATIVE RESULTS
Here, we demonstrate qualitative results of our method compared with Trajectron++ for seconds prediction, trained for 3 seconds prediction. We randomly sample 50 trajectories from model for each prediction, use kernel density estimation to approximate the total output distribution from the samples, and print it out in Fig The ground truth trajectories are represented by white points. Compared to Trajectron++, our method reduces the uncertainty of the future by a large margin and also increase the accuracy.
The classified latent intent behavior helps to widen the difference among trajectories from dif- ferent behavior intents. To demonstrate this, we plot out trajectories for every latent intents, totally 25 intents including the unlikely latent intents given the input data, for both our method and Trajec- tron++ in Fig 6 The white points are ground truth trajectories. The red trajectories are the behaviors 2 with at least 5% probability (p(2)x, m) 2 0.05). The gray trajectories are behaviors which are less possible to occur (p(2) x, m) < 0.05). From the visualization we can see that the latent behaviors in method are more diverse and distinguishable compared to Trajectron++.
15

---

Published as a conference paper at ICLR 2021
Ours
(b) Trajectron++
(c) Ours
dd) Trajectron++
(c) Ours
S Trajectron++
(g) Ours
(h) Trajectron++
Figure 3: Qualitative results of method and Trajectron++. Compared Trajectron++, method significantly reduces the uncertainty of the prediction in all scenes with improved accuracy. White points denotes the ground truth trajectories.
16

---

Published as a conference paper at ICLR 2021
(a) Ours
(b) Trajectron++
Figure 4: The trajectories from all behavior intents generated by our method and Trajectron++. We force the model to predict trajectory for all behaviors matter the behaviors are possible judged by the model or not. White points denote the ground truth trajectories. The other points denote pre- dicted trajectories with different behavior intents. With the help of classified latent intent behavior, we obtain more diverse behaviors compared to Trajectron++. Note that red points comes from the intents which likely under the judgement of models given the input data. The gray points comes from intents which are very unlikely to happen and we forcibly set it for demonstration. Note that Trajectory++ predicts unsafe trajectory with high likelihood. While Our method have a capability to predict diverse trajectories but unsafe modes have vert low likelihood
17

---

Published as a conference paper at ICLR 2021
E PEDESTRIAN DATASETS
Here, train model on the pedestrians dataset ETH (Pellegrini al. 2009) and UCY (Leal- Taixé et al. 2014) without map information. A mave-one-our technique is used for evaluation, similar previous work (Alahi et al., 2016 Gupta al. 2018 Ivanovic & Pavone 2019 Kosaraju et al. 2019 Sadeghian et al. 2019) Salzmann et al. 2020). where the model is trained in four datasets and tested the fifth dataset. The assessment projected horizon of 12 timesteps (4.8s). Note that different from experiments in nuScenes dataset, our model trained from scratch here.
observation period of 8 timesteps (3.2s) and
We show in table I0 our performance on the UCY datasets. In addition, the model's deterministic ML output scheme used, which produces the most likely single trajectory of the model. With only using the the notion of classified intents and hallucinated intents that can be captured by discrete latent vector 2, we see slight improvement in the FDE results with almost 7% over Trajectron++.
Table 10: Without Map. The performance of our method on UCY pedestrian datasets. We don't have map information in this experiments. Although, our method still achieve comparable perfor- mance compared other state-of-the-art methods and get the best performance in the FDE of most likely trajectories averaged over datasets. Lower is better. Bold indicates best. Our method is significantly better.
ADE/FDE ML
Dataset
S-L.TMM
S-ATTN
Trajectron++
Ours
Gupta el al. 2018
Vemula el al. 2018
Salzmann et al. 2020
Univ
0.67/1
0.33/3.92
0.41/1.00 0.30/0.7 77
0.42/1.07
Zara
0.47/1.00
0.20/0.52
0.27/0.67
Zara
0.56/1.17
0.30/2.13
0.23/0.59
0.20/0.52
Average
0.57/1.19
0.28/2.19
0.31/0.81
0.29/0.75
18

---

Published as a conference paper at ICLR 2021
F DETAILED ALGORITHM
Algorithm 2: Detailed Training Process
Predicted horizon T, Integration model Inte Initialize Encog Deco Diss:
Initialize learning rate a, 33;
while converge do
// discriminaton
Yot, x, m Dataset
// Compute the high level features and the distribution of latent code
P(x] m), e Encel (x, m)
// Sample the classified intent 2
2 ~ P(z[x, m)
for t in range((") do
P( 11 x, m) = Decop (e, 2, âc-1) à X( m)
end
// convert the action into trajectories by integration model
y = (.. Tnle(yo. ã1, (o,,...d0
// Discriminator judge whether the given trajectory real/fake and classified to which // ž represents the classification result P(z m). The i-th element ž is the probability y belongs to i-th judged by the discriminator
D(y), 2 = Dis (f. m)
D(yat),- = Dis (y m)
Le CrossEntropul (2 = -
E, &s log ž, // classification loss In = (1 DD )) (D()))2 The // GAN loss plus classification loss
Φ Φ a VLD.D
// generator
-x m -Dtaset
P(z|x, m).c. m c Enco (x, m)
1/ Sample 2, 2' and time step the Their combination is viewed as the hallucinated intent 2,.
2,2' ~ -P(z|x, m), z + 2
In Uniform(2, 11)
1/ Generate hallucinated trajectory yn. for in range(t;)) do ((an,l@as.te x, m) Decsp (e, 2, ân,z-1) // Conditioned on à ān.t P( aā, 00ht-1:: 1,,,,1, m) end for range(t,, + 1,T) do
Dece, (e, 2', ân,t-1) // Conditioned ž ān. P(a,,1 0,,1-1. x, m)
P(an, l0, (t.../pn..11111 1-1: m)
end
y 1h.1 - = Inte(Wo, 0n,1, ân,2; ..., Gn.T)
D(S)), 2n, Dis (ýn. m)
// Make ÿn hard to be classified by reducing the cross entropy between a uniform distribution and the classification results ūt. N number of latent code
La Cross Entropy(UUU formorssss,, 2n.) Σ, N log 5n.s
Lg.e (D($) 1)2 Le
LG = (D(5,) 1D(5))111 1)2 Ln
0 0=0- 0
3Vo Vo(Le + A)La,))
Ygr, x. m Dataset
8 = 0 - AV, (Luga))) // Original Trajectron++ loss
end
19

---

Published as a conference paper at ICLR 2021
G
CODE PATCH FOR TRAINING LOoP
|
TRAINING
pbar = tędm (data_ loader, ncols=80)
pbar_gen iter (pbar)
d_loss g loss train_loss terch. (1)
not not_empty True
10
while not empty: srajectron.set_cue iter (curr_ (curr_iter)
tra annealers medde_type)
13
14
discriminator 1. Compute the high level featurese
15
# and the distribution of latent codez.
Sample the classified intent
18
# 3. convert the action into trajectories by integration model
19
20
# 4. Discriminator judge whether the given
trajectory is real/fake and classified to which
5. represents the classification result (zl y m))
23
The i-th element
is the probability
24
belongs to i-th judged by the discriminator optimizer d. zero zero_ (model registrar, "discriminator")
27
28
batch, not_empty = (prc_baachhbbarr_ooo if not net_empty: break
29
32
d_los,, do_loss, d_real, fake trajectron.gan_ batch, node_type, grid_std-args..rid
33
34
loss loss args.elast Lambda de loss (args loss www.:oosssseeeeees weight weight_total total less) .backwadd() opt wiiiierrdd
36
37
38
Generator
and # a time step th to assemble hallucinsted intent zh
1. Sample additional latent code
41
42
# 2. Generate hallucinated trajectory yh
43
# 3. Make yh hard to be classified by
reducing the cross entropy between
45
uniform distribution and the classification results 2h opt potimizer.oo zero_grad(
46
47
part2train
registtar, "generator")
batch, net_empty fetch batch (pbar_gen) it not not_empty: break
50
51
52
g loss, c_loss seajecttron.gan__losss batch, node_type, aadd_stddaggggridda... grid_raxeargs.griddn
55
56
regs.real_ratio * factor loss args.class lambda
c coss))
58
) (.backward()
59
g loss, creative_loss trajacttron.an.ggllos. l
60
batch,
node_type,
grid_atd--aggs.riid_s
20

---

Published as a conference paper at ICLR 2021
grid max.aaxxxaaacc.mm grid max, creative-creative_moee
65
66
loss =
(1 args.real_ratio)
args.g_factor
69
lg_s
creative_loss
70
less) backkard
opt mi zer.
74
75
trajectron+*
26
optimizer. zero zero_grad(
(artttraainmmdee___eeisttttte "generator")
78
79
batch, not_empty = (ptch_batchpbaarr..gg
80
if not net_empty:
break
train_loss traia taannnttrannnnoonee loss (batch, node_type) brain_loss.backwarde
83
84
optimizer.step
87
Stepping forward the learning rate scheduler and annealers. 1r_scheduler.steep))
88
89
1r 1=_schedule__d_steppl
curr_iter iter
Listing 1: Training Loop
21


Reviews:
[{"review_id": "uDwmrU-tEM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting technique to generate multimodal trajectory GAN and a carefully designed latent intent space. This latent intent space allows an operation termed as hallucination, which switches agent intents to enrich the latent spaces' coverage. \n\nThe paper is overall clear and well-written. However, there is one point regarding the hallucinative learning probably can benefit from more elaboration: the time step \"t\" is only introduced in the Hallucinative Learning paragraph. How is the time step being used in the network? Is the network only predicting y_{gt} as a simple waypoint (t=1), or a sequence of waypoints (a trajectory, in that case, what is the horizon)?  If I understand correctly, the main hallucination idea is to extract as many intentions in the latent space as possible, from the training data, and then the hallucination is implemented as randomly switching between different learned intents to generate augmented data? This point needs to be carefully clarified. \n\nRegarding the idea of using \"hallucination\" to generate more training data, many recent works have used the idea of hallucination for data augmentation, or even generating training data from scratch. For example, https://arxiv.org/pdf/2010.08098.pdf and https://arxiv.org/pdf/2007.14479.pdf generate training data using hallucination based on geometric feasibility, instead of latent intent, https://openaccess.thecvf.com/content_CVPR_2019/papers/Zhang_Few-Shot_Learning_via_Saliency-Guided_Hallucination_of_Samples_CVPR_2019_paper.pdf uses saliency to guide the generation of hallucination, https://openaccess.thecvf.com/content_CVPR_2020/papers/Li_Adversarial_Feature_Hallucination_Networks_for_Few-Shot_Learning_CVPR_2020_paper.pdf also used a very similar adversarial approach to generate feature hallucination. These works can help to set stage in the related work section. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "mRdP5xo5wiL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors tackle the problem of trajectory prediction in autonomous vehicles.  They propose a data augmentation scheme where they enrich the raw trajectories with synthetically generated trajectories to reduce spurious modes in the predictions. This enriched dataset is hoped to improve  accuracy in trajectory prediction problems.\n\nStrengths:\n\n1. The paper belongs to an active area of research. Reliable and accurate Trajectory prediction is one of the central problems in autonomous driving.\n\nWeaknesses:\n\n1. The paper is very hard to read. For example, the contributions are confusing and not clearly defined. In the first paragraph of the Introduction, a data augmentation scheme is presented. But then on Pg.2, the authors present two notions of intents as their contributions. What is the relation? From my understanding, the authors use $\\hat z \\sim P(z)$ to generate new data  (I'm looking at  Eqn 4). And this eqn 4 is termed as the \"classified intent\". There are several things to unpack here: \nFirst, these contributions are not well-defined. What is a \"classified intent\" and \"hallucinative intent\". These are not standard terms in the trajectory prediction literature. So these terms need to be defined and explained. I would like to see a mathematical definition and references to relevant citations in the traffic psychology literature.\nOnce it is formally defined what a \"classified intent\" and \"hallucinative intent\" is, the next thing would be to formally derive equations 4 and 5 from those definitions. Because eqns 4/5 are not derived with supporting derivations and justifications, the main concern here is that I don't find the motivation or relevance of these equations convincing.\nMy suggestion here is that most of the current material on Pg.4 can be moved to an implementation section. Instead, use that space to include the motivations, definitions, justifications, and derivations for eqns 4/5 as explained above.\n\n2. The questionable nature of the proposed equations (4, 5) directly relates to my next point, that is, the lack of any useful improvement over SOTA. In Tab. 1, the max FDE improvement is 18cm. In Tab 2. it is 5cm. Furthermore, the authors report gains in terms of percentages, which is misleading and even dangerous, since in the real world, it is important to know absolute errors and not percentage errors relative to another benchmark.\n\n3. Additionally, the authors primarily compare using the FDE (a weaker and insufficient metric than the ADE) on most occasions, most critical of which are Tab.1 which contains the comparison with other methods and Tab.5 which contains the ablation experiments that highlight the benefits of the proposed equations 4 and 5. For fair evaluation, it is necessary to also present comparisons using ADE.\n\nIn summary, the paper suffers from lack of a clear justification of the proposed contributions, unfair evaluations, and questionable significance of the results", "extra_scores": {}, "extra_text": {}}, {"review_id": "J38X45DGR6F", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper designed a framework for motion forecasting (trajectory prediction), with emphasis on multimodal distribution modeling and generalization. Specifically, they use latent code to model agent's intents. This latent code combined with historical trajectories and map were used to generate future trajectories, which were further judged by a discriminator. Besides, they added latent code classification and hallucinative data augmentation for performance boosting. \n\nReasons for score:\nThe major reason for the voting of accept is that this paper is solid and generally easy to understand. The interplay of latent code classification and hallucinative is interesting. \n\nPros:\n1. Solid method development, fluent method introduction, and systematic ablation studies. \n2. The proposed method seems to be interesting, especially the adding of latent code z to generate trajectories and then classify them.\n\nCons:\n1. The proposed methods seems to be combination of existing components, which limits its theoretical contributions. \n2. The underlying reasons for the success of different components (classification of latent intent and hallucinative latent intent) are hard to explain. \n\nPlease address the following questions:\n1. It would be interesting to see how this method compare with \"Mercat, Jean, et al. \"Multi-head attention for multi-modal joint vehicle motion forecasting.\" 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.\" which seems to be a state of art.\n2. During inference, how do you generate and select multiple trajectory candidates?\n3. It seems like only when both classification latent and hallucinative latent are combined can they improve the performance. I notice the authors discussed about this, but I think this interplay needs more discussion. \n4. Did you try to see what is the contribution of map?\n5. For equation (2), it is better to display it as multi lines for easy understanding. Also, P(z,|x,m),e = Encθ(x,m), the \",\" after z seems to be a typo.", "extra_scores": {}, "extra_text": {}}, {"review_id": "G_Rg1DfhBP_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**SUMMARY**\n\nThe present work considers the problem of multi-agent trajectory prediction. Its main contribution is incorporating generative augmentation losses for improving the quality of a trajectory predictor. This is achieved by allowing trajetcory predictors to model intent as an unobserved latent variable in the model and using this to generate trajectories corresponding to different intentions. The work also proposes to use a descriminative loss encouraging diversity of the intents and an additional \"hallucination\" loss that allows for modelling mixed intents.\n\nThe idea is integrated with Trajectron++ and it is shown how the newly approach improves upon the Final Displacement Error, a comon trajectory prediction metric. In human experiments, it is also shown that the trajectories predicted by the new approach are considered more realistic by humans in comparison to the baseline. \n\n**STRENGTHS**\n- The idea of a structured generative loss for creating a richer diversity of predictions has a lot of potential to improve the trajectory prediction tasks.\n- Human studies seem to be a very interesting idea for studying trajectory prediction.\n\n**WEAKNESSES**\n- Although the authors claim somewhat broad applicability of their approach, the idea is ultimately only demonstrated on Trajectron++\n- The evaluations require some more data to be better interpretable.\n\n**CLARITY**\n\nI found this work mostly clear to read. It sometimes makes an assumption about the reader's familiarity with more specialized techniques such as LSGAN and spectral normalization. While there is probably no space to introduce everything used by the authors and I do not hold it against them, maybe it is worth briefly motivating these and other non-obvious design choices without going much into detail. Further smaller clarity questions/remarks are:\n- What is the motivation behind the discreteness of the latent variable?\n- What do the authors mean by \"unsupervised discrete random variables\"? As I understand it, a random variable is a mathematical object that typically does not entail properties such as supervised or unsupervised.  Maybe simply write \"discrete latent variable\"?\n\n**REPRODUCIBILITY**\n\nI believe the work to be mostly reproducible. It is not possible to judge the usefulness of the code without it being available during the review process. I still applaud the author's intent to release it.\n\n**EVALUATION**\n- Focusing the evaluation on one predictor (Trajectron++) seems insufficient given that the authors make a much broader claim about the general usefulness of their approach.\n- Similarly, the claim that the proposed approach outperforms multiple state-of-the-art predictors seems misleading. In Table 1, Trajectron would have been already the clearly best model even without the addition proposed by the authors.  In Table 6, the proposed approach does not always outperform the baseline and does not even always outperform Trajectron++. While I do not believe that every paper has to outperform all previous papers always on all metrics, it would be interesting to see a discussion/analysis on why this happens. \n- Also why does Table 6 only contain UCY? Is it because ETH is already shown in Table 4? The comparisons seem different to me, so maybe worth also integrating ETH in table 6?\n- Because the proposed method's performance is still somewhat close to Trajectron++'s, it would be interesting to see several runs of the same experiments also reporting the standard deviation of the obtained metrics. At least for the comparison with Trajectron++.\n\n**SUMMARY**\n\nIn summary, I think the authors propose an interesting idea and I believe using humans to judge the quality of trajectories is a cool evaluation methodology. The paper's weak point is a much broader claim while it is only evaluated with Trajectron++. And, in the comparison with this baseline, it is not clear to what extent the work will constantly outperform it. That being said, I want to encourage the authors in following this line of work and providing further evaluations.\n\n**POST-DISCUSSION UPDATE**\n\nI believe the authors to have addressed some of my concerns and I appreciate the additional experiments demonstrating that even the close results were more than a mere statistical artifact. Some other points remain still open such as the limited focus on Trajectron in evaluations.  In summary, I believe that the paper has now surpassed the acceptance threshold and am happy to recommend its publication.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for future trajectory generation. The main contribution is in proposing a technique for data augmentation in the latent space which encourages prediction of trajectories that are both plausible, but also different from the training set. The results clearly show superior performance on standard benchmarks. The evaluation is thorough and ablations show that the proposed innovation matters. 

R2, R3, R4 recommend that the paper be accepted with scores 6, 8, and 6 respectively. R1 recommends the paper be rejected with a score of 5. The main concern of reviewers are: 

R1: " In summary, the paper suffers from lack of a clear justification of the proposed contributions, unfair evaluations, and questionable significance of the results." The authors addressed this concern in their rebuttal.  

R2: "Some other points remain still open such as the limited focus on Trajectron in evaluations." Since trajectron is a recent SOTA, I think this is not a big concern. Authors compare against other baseline methods too. 

R4: Comparison to Mercat, Jean, et al., ICRA 2020 is missing. The authors mention that their code is unavailable and therefore cannot compare.  

R4: "underlying reasons for the success of different components (classification of latent intent and hallucinative latent intent) are hard to explain". I agree with this and this is also my major concern which I detail below. 

The paper proposes to find diverse trajectories by generating two latent vectors: z, z'. The first h time steps are generated by latent vector z and the remainder using z'. The generated trajectory is evaluated by a discriminator that ensures plausibility. The latent vectors are chosen to be discrete and a classifier is trained to recognize z from ground truth trajectories. To encourage diverse trajectories, authors use a loss that encourages mis-classification of the latent variable inferred from the generated trajectory. Since the generated trajectory cannot be classified well, it is assumed to be different from the training set. 

This formulation is rather adhoc. If the trajectory is indeed different from the training distribution, then it will also fool the discriminator. If it doesnot, then it's not very different. The mis-classification, is akin to encouraging high entropy in the z space inferred from predicted trajectories. With this view, it is possible that there is no need to generate two latent vectors z, z', but simply generate one and use the entropy penalty. I would love to see this experiment and see the authors demystify their method. It would also lead to significant changes in writing. Even now, writing needs improvement. Due to the proposed method being a adhoc trick, that is not well justified, I would normally not recommend acceptance. However, the empirical results are strong, tilting the recommendation to acceptance.
Decision: Accept (Poster)