Motion Forecasting with Unlikelihood Training

1 INTRODUCTION . For robotic applications deployed in the real world , the ability to foresee the future motions of agents in the surrounding environment plays an essential role for safe and intelligent decision making . This is a very challenging task . For example , in the autonomous driving domain , to predict nearby agents ’ future trajectories , an agent needs to consider contextual information such as their past trajectories , potential interactions , and maps . State of the art prediction models ( Salzmann et al. , 2020 ; Tang & Salakhutdinov , 2019 ; Rhinehart et al. , 2019 ) directly take contextual information as part of their input and use techniques such as graph neural networks to extract high-level features for prediction . They are typically trained with a maximum likelihood estimation ( MLE ) objective that maximizes the likelihood of ground truth trajectories in the predicted distribution . Although MLE loss encourages the prediction to be close to the ground truth geometrically , it does not focus on learning a good distribution that is plausible with respect to the contextual information . These models predict trajectories that violate the contextual information ( e.g. , go to opposite driving direction or out of the driving area ) but still closes to ground truth . In contrast , humans can easily notice that these trajectories are unlikely in a specific context . This phenomenon suggests that simply applying MLE loss can not fully exploit contextual information . To address the problem , we propose a novel and simple method , unlikelihood training , that injects contextual information into the learning signal . Our loss penalizes the trajectories that violate the contextual information , called negative trajectories , by minimizing their likelihood in the predicted distribution . To generate negative trajectories , we first draw a number of candidate trajectories from our model ’ s predicted distribution . Then , a context checker is used to cut out the trajectories that violate contextual information as negative trajectories . This context checker does not need to be differentiable . By minimizing the likelihood of negative trajectories , the model is forced to use the contextual information to avoid predictions that violate context . Therefore , the prediction quality is improved . Existing methods ( Casas et al. , 2020 ; Park et al. , 2020 ) using contextual information as learning signals either introduce new learning parameters or using high-variance learning methods such as the REINFORCE algorithm ( Casas et al. , 2020 ) . In contrast , our method injects rich contextual information into the training objective and keeps the training process simple . Unlikelihood training ( Welleck et al. , 2019 ) has been applied to neural text generation . We are the first to propose unlikelihood training for continuous space of trajectories . For the discrete space of token sequences , repeating tokens or n-grams in the generated sequence are chosen as negative tokens . In contrast , we design a context checker to select negative trajectories sampled from the continuous distribution of model predictions . Our method can be viewed as a simple add-on to any models that estimate the distribution of future trajectories . It improves their performance by encouraging models to focus more on contextual information without increasing the complexity of its original training process . Our contributions are summarized as follows : • We propose a novel and simple method , unlikelihood training for motion forecasting in autonomous driving that encourages models to use contextual information by minimizing the likelihood of trajectories that violate contextual information . Our method can be easily incorporated into state-of-the-art models . • Our experimental results on challenging real-world trajectory forecasting datasets , nuScenes and Argoverse , shows that unlikelihood training can improve prediction performance by 8 % and reduce the standard deviation by up to 50 % . 2 RELATED WORK . In this section , we briefly review the two most related topics . Trajectory Forecasting Trajectory forecasting of dynamic agents , a core problem for robotic applications such as autonomous driving and social robots , has been well studied in the literature . State-of-the-art models solves it as a sequence-to-sequence multi-modal prediction problem ( Lee et al. , 2017 ; Cui et al. , 2018 ; Chai et al. , 2019 ; Rhinehart et al. , 2019 ; Kosaraju et al. , 2019 ; Tang & Salakhutdinov , 2019 ; Ridel et al. , 2020 ; Salzmann et al. , 2020 ; Huang et al. , 2019 ) . ( Cui et al. , 2018 ; Chai et al. , 2019 ; Ridel et al. , 2020 ) predicts multiple future trajectories without learning low dimensional latent agent behaviors . ( Lee et al. , 2017 ; Kosaraju et al. , 2019 ; Rhinehart et al. , 2019 ; Huang et al. , 2019 ) encodes agent behaviors in continuous low dimensional latent space while ( Tang & Salakhutdinov , 2019 ; Salzmann et al. , 2020 ) uses discrete latent variables . Discrete latent variables succinctly capture semantically meaningful modes such as turn left , turn right . ( Tang & Salakhutdinov , 2019 ; Salzmann et al. , 2020 ) learns discrete latent variables without explicit labels . All of them use a maximum likelihood estimation ( MLE ) objective or its approximations ( e.g. , VAE ) . In this paper , we show that MLE loss can ignore contextual information such as maps and states of surrounding agents . As a result , models with such a loss can assign too much probability to unlikely trajectories . We propose an unlikelihood training objective to avoid such cases . All models with the maximum likelihood estimation objective can potentially benefit from our methods . Contrastive learning and unlikelihood training To date , several studies have investigated the possibilities to benefit from negative data . One of the popular direction is contrastive learning . Contrastive learning has achieved significant success in many fields ( Oord et al. , 2018 ; Kipf et al. , 2019 ; Ma & Collins , 2018 ; Abid & Zou , 2019 ; Welleck et al. , 2019 ) . NCE ( Ma & Collins , 2018 ) CPC ( Oord et al. , 2018 ) maximizes the mutual information between data and latent representation by a novel contrastive loss to extract useful representation from data . C-SWMs ( Kipf et al. , 2019 ) utilizes contrastive learning to learn a better world model for reinforcement learning tasks . Recently , unlikelihood training ( Welleck et al. , 2019 ) proposes a new method to utilize negative data . In addition , to maximize the likelihood of the ground truth token , it minimizes the likelihood of negative tokens for better text generation . Their method is on the discrete space of token sequences . Repeating tokens or n-grams in the generated sequence is chosen as negative tokens . In contrast , our proposed method works in the continuous space of trajectories . We design a novel method , context checker , to select negative trajectories . 3 METHOD . 3.1 PROBLEM FORMULATION . We are targeting at better predicting the future trajectory Yi of a vehicle i given input Xi . Xi can include any related information like rasterized maps or past positions of vehicle i and surrounding agents , depends on the design of the method . Here we skip the detailed choice of input and denote it as Xi for conciseness . Due to different driving strategies , driving intents , and the complex traffic environment , there are usually multiple possible future trajectories given an input xi ( although there is only one ground truth future trajectory yi , gt in a dataset recorded in the real world ) . To handle this situation , most state of the art methods ( Salzmann et al. , 2020 ) model a distribution of possible future trajectories pθ ( Yi | Xi ) to cover all the possibilities given the input Xi instead of predicting one trajectory . θ denotes the learning parameters of the model . To train such methods , most stateof-the-art models usually use maximum likelihood estimation ( MLE ) to maximize the likelihood of ground truth trajectory Yi , gt in the predicted distribution . For example , the loss of CVAE-based model Trajectron++ ( Salzmann et al. , 2020 ) is Eq.1 . This loss is used to maximize the lower bound of ground truth ’ s likelihood when the coefficient k = 1 . Ltraj++ = −Eẑ∼qθ3 ( z|Xi , Yi , gt ) [ log pθ2 ( Yi , gt |Xi , ẑ ) ] + kDKL ( qθ3 ( z |Xi , Yi , gt ) ‖pθ1 ( z |Xi ) ) − Iq ( Xi ; z ) ≥ − log p ( Yi , gt |Xi , Yi , gt ) − Iq ( Xi ; z ) , when k = 1 ( 1 ) Limitation of MLE on Motion Forecasting MLE encourages the model to predict a distribution that allocates reasonable probability mass to the region where Yi is located by minimizing the KL-divergence of predicted distribution and ground truth distribution . Because the domain of trajectory distribution is over the geometric locations , MLE makes these two distributions ” close ” to each other geometrically . However , we argue that maintaining the geometrical nearness only is not good enough for motion forecasting task in autonomous driving . In complex traffic scenarios , there can be many potential trajectories close enough to the ground truth geometrically but are very unlikely to happen . For example , if the ground truth trajectory Yi , gt is on the outermost lane , a trajectory that is close to Yi , gt but outside the drivable region is unlike to happen in the real world . However , MLE loss will not impose a significant enough penalty on such a case to avoid such a prediction . Fig.1 demonstrates a prediction example from Trajectron++ ( Salzmann et al. , 2020 ) where part of the distribution is outside of the derivable region or on the lane with the wrong direction . The MLE-based loss only offers learning signals that contain the geometric location information of the ground truth trajectories . All the other contextual information , like the drivable region and the lane direction , are missing in the learning signals . While the inputs to a model contain rich contextual information , the model can not use it to avoid the prediction that is geometrically close to ground truth but violates context . In contrast , this is quite a simple task for humans . 3.2 UNLIKELIHOOD LOSS . To mitigate this problem , we design a new loss term that encourages the model to consider the contextual information . Inspired by contrastive learning and unlikelihood training , we additionally train our model to minimize the likelihood of trajectories that violate the contextual information given input Xi . We denote them as negative trajectories Yi , neg . Let ’ s first assume that we already have a distribution of negative trajectories pneg ( Yi | Xi ) . One intuitive way is to directly minimize the log likelihood of Yi , neg in our predicted distribution , similar to MLE but in an opposite manner Lunlike = EXi , ∼D , Yi , neg∼pneg ( Yi|Xi ) [ log pθ ( Yi , neg |Xi ) ] ( 2 ) However , the gradient of log function tends to infinity when the input tends to 0 , which leads to unstable training since the model are optimized to minimize pθ ( Yi , neg | Xi ) and pθ ( Yi , neg | Xi ) ≥ 0 . To avoid the infinity gradient region of log function , we add a small constant to the likelihood . The final loss term we propose is Lunlike = EXi , ∼D , Yi , neg∼pneg ( Yi|X ) [ log ( pθ ( Yi , neg |Xi ) + ) ) ] ( 3 ) We call it unlikelihood loss . We use a coefficient γ to balance Lunlike . The final training objective in case we combine our method with Trajectron++ is L = Ltraj++ + γLunlike ( 4 ) Eq.4 is also easily adapted to combine with any other models that predict trajectory distribution as output . With the help of Lunlike , we inject the contextual information into the learning signal , force the model to better extract and use contextual information in Xi , and generate more reasonable predicted distribution to avoid high Lunlike .

Summary: The present paper considers the problem of context integration in probabilistic agent trajectory predictors, particularly Trajectron++. It starts with the observation that these predictors often do a bad job at considering non-drivable areas in their predictions even if context information is injected as part of the input. The paper proposes adjusting the loss function by adding an unlikelihood loss term. This term modifies existing MLE-type losses in a way that discourages context violations (in the present case, it discourages prediction of trajectories that are on non-drivable space).