HalentNet: Multimodal Trajectory Forecasting with Hallucinative Intents

1 INTRODUCTION . The ability to forecast trajectories of dynamic agents is essential for a variety of autonomous systems such as self-driving vehicles and social robots . It enables an autonomous system to foresee adverse situations and adjust motion planning accordingly to prefer better alternatives . Because agents can make different decisions at any given time , future motion distribution is inherently multi-modal . Due to incomplete coverage of different modes in real data and interacting agents ’ combinatorial nature , trajectory forecasting is challenging . Several existing works focus on formulating the multi-modal future prediction only from training data ( e.g. , ( Tang & Salakhutdinov , 2019 ; Alahi et al. , 2016 ; Casas et al. , 2019 ; Deo & Trivedi , 2018 ; Sadeghian et al. , 2018 ; Casas et al. , 2019 ; Salzmann et al. , 2020 ) ) . This severely limits the ability of these models to predict modes that are not covered beyond the training data distribution , and some of these learned modes could be spurious especially where the real predictive spaces are not or inadequately covered by the training data . To improve the multimodal prediction quality , our goal is to enrich the coverage of these less explored spaces , while encouraging plausible behavior . Properly designing this exploratory learning process for motion forecasting as an implicit data augmentation approach is at the heart of this paper . ∗Work done prior to Amazon . Most data augmentation methods are geometric and operate on raw data . They also have been mostly studied on discrete label spaces like classification tasks ( e.g. , ( Zhang et al. , 2017 ; Yun et al. , 2019 ; Wang et al. , 2019 ; Cubuk et al. , 2019 ; Ho et al. , 2019 ; Antoniou et al. , 2017 ; Elhoseiny & Elfeki , 2019 ; Mikołajczyk & Grochowski , 2019 ; Ratner et al. , 2017 ) ) . In contrast , we focus on a multi-agent future forecasting task where label space for each agent is spatial-temporal . To our knowledge , augmentation techniques are far less explored for this task . Our work builds on recent advances in trajectory prediction problem ( e.g. , Tang & Salakhutdinov ( 2019 ) ; Salzmann et al . ( 2020 ) ) , that leverage discrete latent variables to represent driving behavior/intents ( e.g . Turn left , speed up ) . Inspired by these advances , we propose HalentNet , a sequential probabilistic latent variable generative model that learns from both real and implicitly augmented multi-agent trajectory data . More concretely , we model driving intents with discrete latent variables z . Then , our method hallucinates new intents by mixing different discrete latent variables up in the temporal dimension to generate trajectories that are realistic-looking but different from training data judged by discriminator Dis to implicitly augment the behaviors/intents . The nature of our augmentation approach is different from existing methods since it operates on the latent space that represents the agent ’ s behavior . The training of these latent variables is guided by a collaboration between discriminative and hallucinative learning signals . The discriminative loss increases the separation between intent modes ; we impose this as a classification loss that recognizes the one-hot latent intents corresponding to the predicted trajectories . We call these discriminative latent intents as classified intents since they are easy to classify to an existing one-hot latent intent ( i.e. , low entropy ) . This discriminative loss expands the predictive intent space that we then encourage to explore by our hallucinated intents ’ loss . As we detail later , we define hallucinated intents as a mixture of the one-hot classified latent intents . We encourage the predictions of trajectories corresponding to hallcuinated intents to be hard to classify to the one-hot discrete latent intents by hallucinative loss but , in the meantime , be realistic with a real/fake loss that we impose . The classification , hallucinative , and real/fake losses are all defined on top of a Discriminator Dis , whose input is the predicted motion trajectories and the map information . We show that all these three components are necessary to achieve good performance , where we also ablate our design choices . Our contributions are summarized as follows . • We introduce a new framework that enables multi-modal trajectory forecasting to learn dynamically complementary augmented agent behaviors . • We introduce the notion of classified intents and hallucinated intents in motion forecasting that can be captured by discrete latent variables z . We introduce two complementary learning mechanism for each to better model latent behavior intentions and encourage the novelty of augmented agent behaviors and hence improve the generalization . The classified intents ẑ is defined not to change over time and are encouraged to be well separated from other classified intents with a classification loss . The hallucinated intents ẑh , on the other hand , changes over the prediction horizon and are encouraged to deviate from the classified intents as augmented agent behaviors . • Our experiments demonstrate at most 26 % better results measured by average FDE compared to other state-of-the-art methods on motion forecasting datasets , which verifies the effectiveness of our methods . We also conducted human evaluation experiments showing that our forecasted motion is considered 39 % safer than the runner-up approach . Codes , pretrained models and preprocessed datasets are available at https : //github.com/ Vision-CAIR/HalentNet 2 RELATED WORK . Trajectory Forecasting Trajectory forecasting of dynamic agents has received increasing attention recently because it is a core problem to a number of applications such as autonomous driving and social robots . Human motion is inherently multi-modal , recent work ( Lee et al. , 2017 ; Cui et al. , 2018 ; Chai et al. , 2019 ; Rhinehart et al. , 2019 ; Kosaraju et al. , 2019 ; Tang & Salakhutdinov , 2019 ; Ridel et al. , 2020 ; Salzmann et al. , 2020 ; Huang et al. , 2019 ; Mercat et al. , 2019 ) has focused on learning the distribution from multi-agent trajectory data . ( Cui et al. , 2018 ; Chai et al. , 2019 ; Ridel et al. , 2020 ; Mercat et al. , 2019 ) predicts multiple future trajectories without learning low dimensional latent agent behaviors . ( Lee et al. , 2017 ; Kosaraju et al. , 2019 ; Rhinehart et al. , 2019 ; Huang et al. , 2019 ) encodes agent behaviors in continuous low dimensional latent space while ( Tang & Salakhutdinov , 2019 ; Salzmann et al. , 2020 ) uses discrete latent variables . Discrete latent variables succinctly capture semantically meaningful modes such as turn left , turn right . ( Tang & Salakhutdinov , 2019 ; Salzmann et al. , 2020 ) learns discrete latent variables without explicit labels . Built on top of these recent work , we hallucinate possible future behaviors by changing agent intents . As the forecast horizon is a few seconds , these are highly plausible . We use a discriminator to encourage augmented trajectories to look real . Data augmentation Data augmentation is a popular technique to mitigate overfitting and improve generalization in training deep networks ( Shorten & Khoshgoftaar , 2019 ) . New data is typically generated by transforming real data samples in the original input space . These transformations range from simple techniques ( e.g . random flipping , mirroring for images , mixup ( Zhang et al. , 2017 ) and Cutmix ( Yun et al. , 2019 ) ) to automatic data augmentation techniques ( e.g . AutoAugment ( Cubuk et al. , 2019 ) ) and class-identity preserving semantic data augmentation techniques ( Ratner et al. , 2017 ) ( e.g . changing backgrounds of objects ) . Recently data augmentation via semantic transformation in deep feature space ( Liu et al. , 2018 ; Wang et al. , 2019 ; Li et al. , 2020a ) has also been proposed . ISDA ( Wang et al. , 2019 ) proposes a loss function to implicitly translate training samples along with semantic directions in the feature space . For example , a certain direction corresponds to the semantic translation of ” make-bespectacled. ” When a person ’ s feature without glasses is translated along this direction , the new feature may correspond to the same person but with glasses . MoEx ( Li et al. , 2020a ) proposes a new augmentation method that leverages the first and second moments extracted and re-injected by feature normalization . Specifically , it replaces the moments of the learned features of one training image by those of another and interpolates the target labels . Our data augmentation is also in the latent space , which represents agent behavior . Imaginative/Hallucinative models . GANs ( Goodfellow et al. , 2014 ; Radford et al. , 2015 ) are a powerful generative model , yet they are not explicitly trained to go beyond the training data to improve generalization . Inspired by the theory of human creativity ( Martindale , 1990 ) , recent approaches on generative models were proposed to encourage novel visual content generation in art and fashion designs . In ( Elgammal et al. , 2017 ) , the authors adapted GANs to generate unconditional creative content ( paintings ) by encouraging the model to deviate from existing painting styles . In the fashion domain , ( Sbai et al. , 2018 ) showed that their model is capable of producing a non-existing shape like “ pants to extended arm sleeves ” that some designers found interesting . The key mechanism in these methods is the addition of a deviation loss , which encourages the generator to produce novel content . More recently , ( Elhoseiny & Elfeki , 2019 ) proposed a method for understanding unseen classes , also known as zero-shot learning ( ZSL ) , by generating visual representations of synthesized unseen class descriptors . These visual representations are encouraged to deviate from seen classes , leading to better generalization compared to earlier generative ZSL methods . ( Zhang et al. , 2019 ) and ( Li et al. , 2020b ) introduced methods to generate additional data based on saliency maps and adversarial learning for few-shot learning task , respectively . In the field of navigation , ( Xiao et al. , 2020a ) and ( Xiao et al. , 2020b ) utilized geometric information to hallucinate new navigation training data . In contrast to these earlier methods , our work has two key differences . First , our work is a sequential probabilistic generative model focusing on motion forecasting requiring time-series prediction in continuous space . Second , the deviation signal in ( Elgammal et al. , 2017 ; Sbai et al. , 2018 ; Elhoseiny & Elfeki , 2019 ) is based on defining labeled discrete seen styles and seen classes , respectively . In contrast , we model the deviation from a discrete latent space guided by a deviation signal to help the model imagine driver intents without supervision signal . Similar to MoEx ( Li et al. , 2020a ) , augmented trajectories are dynamically generated during training . We believe we are the first to propose a data augmentation method in latent space for trajectory forecasting . 3 METHOD . Problem Formulation We are aiming at predicting the future trajectory ygt of a specified agent given the input states x , which contains the historical information like positions and heading angles of the agent itself and the surrounding agents , and a semantic map patch m , which offers context information like drivable region , by generating a distribution P ( y|x , m ) to model the distribution of real future trajectory ygt . Our Model Motion forecasting in the real world is a multi-modal task . There are usually multiple possible futures given the same state . To accurately model this diversity , we define a latent code z to represent different intents of the predicted agent inspired by literature ( e.g. , ( Tang & Salakhutdinov , 2019 ; Salzmann et al. , 2020 ) ) . We denote the input state as x , a local map as m , and the corresponding ground truth future as ygt . The possible behaviors are modeled by the distribution of latent code z conditioned on the input state and the map P ( z|x , m ) . Then , the predicted trajectory distribution is calculated by conditioning on both input state and the latent code P ( y|z , x , m ) . For motion forecasting tasks , we use maximum likelihood estimation ( MLE ) loss on the ground truth future as the learning objective L = − log P ( ygt|x , m ) . Note that we do not have the label for the latent code z in the dataset . Similar to ( Tang & Salakhutdinov , 2019 ; Salzmann et al. , 2020 ) , we represent latent code as a discrete random variable . The learning objective can be rewritten as follows . L = − log P ( ygt|x , m ) = − log ∑ i [ P ( zi|x , m ) P ( ygt|zi , x , m ) ] ( 1 ) Hence , we obtain an unsupervised latent code z that captures some uncertainty of the future without knowing its label . The distribution P ( z|x , m ) can be modeled by any model that outputs a categorical distribution . P ( y|z , x , m ) is usually modeled by models that output multivariate Gaussian distribution . An overview of our model can be found in Fig.1 . It consists of two sub-networks , a generator module and a discriminator module described in the following paragraphs . Generator The generator is the prediction model that produces the future trajectory distribution P ( y|x , m ) given agent states x and a local map m. As the possible future is multi-modal , the output distribution should model this uncertainty . As we discussed earlier , we model distribution P ( z|x , m ) and P ( y|z , x , m ) by neural networks . We use a discrete random variable to represent the latent code z . The uncertainty of the future trajectory can be factorized hierarchically into intent uncertainty and control uncertainty ( Chai et al. , 2019 ) . The intent uncertainty reflects different intents or behavior modes of the agent . Furthermore , the control uncertainty covers other minor noise . As the simple Gaussian distribution P ( y|z , x , m ) is not expressive enough to model the complex uncertainty of multi-modal behaviors , this framework encourages the latent code distribution P ( z|x , m ) to cover more the intent uncertainty . We denote the modules that generate the latent code distribution and the trajectory distribution as encoder Encθ and decoder Decφ with parameter θ and φ , respectively . In addition , Encθ also encodes agent states x and the local map m into a feature vector e , which is part of the decoder ’ s input . Note that our method does not introduce further restrictions for the model structure . Any model that fits this framework can be used as our generator like MFP ( Tang & Salakhutdinov , 2019 ) and Trajectron++ ( Salzmann et al. , 2020 ) . In our experiments , we select Trajectron++ as our generator . Its original learning objective Ltraj++ is shown in Appx.B . In summary , the process of the generator can be represented by the following equations . P ( z|x , m ) , e = Encθ ( x , m ) ẑ ∼ P ( z|x , m ) ( 2 ) P ( y|ẑ , x , m ) = Decφ ( ẑ , e ) Discriminator The discriminator Disψ with parameters ψ takes either a real trajectory ygt or a generated one ŷ sampled from our predicted distribution together with the local map m as input to judge whether the trajectory is real or generated following GAN framework ( Goodfellow et al. , 2014 ) . This helps the decoder inject map information into the learning signal and alleviate the violation of road boundaries in prediction . Besides , we add a classification head to the discriminator . When the input data is generated , this head needs to recognize the latent code ẑ the generator used for creating the input trajectory . In this way , the generator is forced to increase the difference among latent codes and give us more distinct and semantically meaningful driving strategies . This is further discussed in the following paragraphs . The following equation describes the function of our discriminator . D ( y ) , P ( z|y , m ) = Disψ ( y , m ) ( 3 ) Trajectory y is either the ground truth future ygt or the sample from predicted trajectory distribution ŷ . D ( y ) is the score to indicate whether y is real or synthetic . P ( z|y , m ) is the classified distribution . Our discriminator is modified from the one in DCGAN ( Radford et al. , 2015 ) by adding a fully-connected classification head at the end to classify the latent code . Trajectories y are transformed into the format that convolutional layers can handle via differentiable rasterizer trick ( Wang et al. , 2020 ) and stacked together with the local map m as the input for the discriminator as described in Appx.B . Learning Methods Our architecture can be trained by a GAN learning objective , together with the original learning loss of the generator module depends on the model we choose to combine with . To learn a better behavior representation and improve the quality of predicted trajectory distribution , we introduce two new methods Classified latent Intent Behavior and Hallucinative Latent Intent for training . Algorithm 1 : Training Process Initialize EncθE , DecθD , Disφ ; Initialize learning rate α , β ; while not converge do // discriminator ygt , x ∼ Dataset Sample normal prediction ŷ LD = ( 1−D ( ygt ) ) 2+ ( D ( ŷ ) ) 2+Lc φ = φ− α∇φLD // generator , x ∼ Dataset Generate hallucinated trajectory ŷh ∼ P ( y|ẑh , x , m ) LG , c = ( D ( ŷ ) − 1 ) 2 + Lc LG , h = ( D ( ŷh ) − 1 ) 2 + Lh θ = θ− β∇θ ( λLG , c + ( 1− λ ) LG , h ) ygt , x ∼ Dataset θ = θ − β∇θ ( Ltraj++ ) // Trajectron++ loss end Classified latent Intent Behavior In the real world , humans can recognize different behavior intents by looking at the trajectories . To encourage the latent code to contain more information about the intent uncertainty and less about the control uncertainty , we mimic this phenomenon and let the discriminator classify the latent code behind the generated trajectories . The classification function can be trained by a cross-entropy loss . Lc = − ∑ i ẑi log z̃i , where ẑ ∼ P ( z|x , m ) ( 4 ) zi denotes the i-th dimension of the vector z. ẑ is the latent code under the input trajectory , which is a one-hot vector sampled from the multinoulli distribution P ( z|x , m ) . z̃ denotes the classified categorical distribution generated by our discriminator . Minimizing this loss encourages the decoder to widen the difference among predictions from different z to reduce the classification difficulty for the discriminator . Therefore , we reduce the overlap among output distributions from different latent codes and sharpen them to increase accuracy . Since our model is trained to classify trajectory into ẑ , we name ẑ classified intent . Note that this loss is only applied for generated trajectories since we do not have the latent code for ground truth trajectory . Hallucinative Learning Latent codes z are trained to model intents in the training data . Each predicted trajectory ŷ is calculated from single latent code ẑ for all the prediction steps . Assume the predicted horizon is T , ŷ = [ ŷ1 , ŷ2 , ... , ŷT ] , ŷ∗ at each step is generated conditioned on the same ẑ . And our discriminator is trained to recognize ẑ given the synthetic trajectory ŷ by the classification loss . Besides , the MLE loss encourages synthetic trajectories to be similar to the training data . Therefore , the discriminator implicitly classifies the training data into one of the latent code z . We propose a novel way to utilize this property and learn beyond the training data by encouraging the model to generate trajectories from unfamiliar driving behaviors . This is done by first sampling a second different latent code ẑ′ in addition to the original one ẑ and randomly selecting a time step th . The prediction until time step th in this case ( [ ŷ1 , ... , ŷth ] ) is conditioned on the first latent code ẑ and we switch to ẑ′ for the remaining steps ( [ ŷth+1 , ... , ŷT ] ) . By this way , we hallucinate a new intent by stacking 2 learned intents in the temporal dimension . We denote this mixed hallucinated intent as ẑh and name it hallucinated intent . The predicted distribution from such a intent is denoted as P ( y|ẑh , x , m ) . We aim to encourage the hallucinative trajectories ŷh to be plausible but different from the training data . To achieve this , we minimize the cross entropy between the uniform distribution and our intent class distribution . Lh = − ∑ i 1 N log z̃i ( 5 ) N indicates the number of latent codes . z̃i is the i-th dimention of the classified distribution z̃ . It encourages the hallucinative trajectory to be hard to be classified into any latent code z , and therefore , to be different from the training data . The plausibility of the hallucinative trajectory is encouraged by the additional GAN loss . In this way , we implicitly apply data augmentation in the latent space to train a more powerful discriminator and improve the generator prediction quality . We call this method hallucinative learning inspired from literature ( e.g. , Hariharan & Girshick ( 2017 ) ) . Training We use LSGAN ( Mao et al. , 2017 ) loss with spectral normalization ( Miyato et al. , 2018 ) as our GAN learning objective . We also keep the original Trajectron++ learning loss Ltraj++ to maintain the performance in case Trajectron++ is our generator . The combination of GAN learning , training of the original generator , classified latent intent behavior , and hallucinative learning is demonstrated in Alg.2 ( Detailed version in Appx.F ) . We use a hyperparameter λ to balance the training between classification learning and hallucinative learning for the generator by adjusting the weighting of the learning loss .

Summary: The paper designed a framework for motion forecasting (trajectory prediction), with emphasis on multimodal distribution modeling and generalization. Specifically, they use latent code to model agent's intents. This latent code combined with historical trajectories and map were used to generate future trajectories, which were further judged by a discriminator. Besides, they added latent code classification and hallucinative data augmentation for performance boosting.