Diverse Trajectory Forecasting with Determinantal Point Processes

1 INTRODUCTION . Forecasting future trajectories of vehicles and human has many useful applications in autonomous driving , virtual reality and assistive living . What makes trajectory forecasting challenging is that the future is uncertain and multi-modal – vehicles can choose different routes and people can perform different future actions . In many safety-critical applications , it is important to consider a diverse set of possible future trajectories , even those that are less likely , so that necessary preemptive actions can be taken . For example , an autonomous vehicle should understand that a neighboring car can merge into its lane even though the car is most likely to keep driving straight . To address this requirement , we need to take a generative approach to trajectory forecasting that can fully characterize the multimodal distribution of future trajectories . To capture all modes of a data distribution , variational autoencoders ( VAEs ) are well-suited generative models . However , random samples from a learned VAE model with Gaussian latent codes are not guaranteed to be diverse for two reasons . First , the sampling procedure is stochastic and the VAE samples can fail to cover some minor modes even with a large number of samples . Second , since VAE sampling is based on the implicit likelihood function encoded in the training data , if most of the training data is centered around a specific mode while other modes have less data ( Fig . 1 ( a ) ) , the VAE samples will reflect this bias and concentrate around the major mode ( Fig . 1 ( b ) ) . To tackle this problem , we propose to learn a diversity sampling function ( DSF ) that can reliably generate a diverse set of trajectory samples ( Fig . 1 ( c ) ) . 1https : //youtu.be/5i71SU IdS4 The proposed DSF is a deterministic parameterized function that maps forecasting context features ( e.g. , past trajectories ) to a set of latent codes . The latent codes are decoded by the VAE docoder into a set of future trajectory samples , denoted as the DSF samples . In order to optimize the DSF , we formulate a diversity loss based on a determinantal point process ( DPP ) ( Macchi , 1975 ) to evaluate the diversity of the DSF samples . The DPP defines the probability of choosing a random subset from the set of trajectory samples . It models the negative correlations between samples : the inclusion of a sample reduces the probability of including a similar sample . This makes the DPP an ideal tool for modeling the diversity within a set . In particular , we use the expected cardinality of the DPP as the diversity measure , which is defined as the expected size of a random subset drawn from the set of trajectory samples according to the DPP . Intuitively , since the DPP inhibits selection of similar samples , if the set of trajectory samples is more diverse , the random subset is more likely to select more samples from the set . The expected cardinality of the DPP is easy to compute and differentiable , which allows us to use it as the objective to optimize the DSF to enable diverse trajectory sampling . Our contributions are as follows : ( 1 ) We propose a new forecasting approach that learns a diversity sampling function to produce a diverse set of future trajectories ; ( 2 ) We propose a novel application of DPPs to optimize a set of items ( trajectories ) in continuous space with a DPP-based diversity measure ; ( 3 ) Experiments on synthetic data and human motion validate that our method can reliably generate a more diverse set of future trajectories compared to state-of-the-art generative models . 2 RELATED WORK . Trajectory Forecasting has recently received significant attention from the vision community . A large portion of previous work focuses on forecasting 2D future trajectories for pedestrians ( Kitani et al. , 2012 ; Ma et al. , 2017 ; Ballan et al. , 2016 ; Xie et al. , 2013 ) or vehicles ( Jain et al. , 2016a ) . Some approaches use deterministic trajectory modeling and only forecast one future trajectory ( Alahi et al. , 2016 ; Yagi et al. , 2018 ; Robicquet et al. , 2016 ) . As there are often multiple plausible future trajectories , several approaches have tried to forecast distributions over trajectories ( Lee et al. , 2017 ; Galceran et al. , 2015 ; Gupta et al. , 2018 ) . Recently , Rhinehart et al . ( 2018 ; 2019 ) propose a generative model that can accurately forecast multi-modal trajectories for vehicles . Soo Park et al . ( 2016 ) also use egocentric videos to predict the future trajectories of the camera wearer . Some work has investigated forecasting higher dimensional trajectories such as the 3D fullbody pose sequence of human motions . Most existing work takes a deterministic approach and forecasts only one possible future motion from past 3D poses ( Fragkiadaki et al. , 2015 ; Butepage et al. , 2017 ; Li et al. , 2017 ; Jain et al. , 2016b ) , static images ( Chao et al. , 2017 ; Kanazawa et al. , 2018 ) or egocentric videos ( Yuan and Kitani , 2019 ) . Differently , some probabilistic approaches ( Habibie et al. , 2017 ; Yan et al. , 2018 ) use conditional variational autoencoders ( cVAEs ) to generate multiple future motions . In constrast to previous work , our approach can generate a diverse set of future motions with a limited number of samples . Diverse Solutions have been sought after in a number of problems in computer vision and machine learning . A branch of these methods aiming for diversity stems from the M-Best MAP problem ( Nilsson , 1998 ; Seroussi and Golmard , 1994 ) , including diverse M-Best solutions ( Batra et al. , 2012 ) and multiple choice learning ( Guzman-Rivera et al. , 2012 ; Lee et al. , 2016 ) . Alternatively , previous work has used submodular function maximization to select a diverse subset of garments from fashion images ( Hsiao and Grauman , 2018 ) . Determinantal point processes ( DPPs ) ( Macchi , 1975 ; Kulesza et al. , 2012 ) are efficient probabilistic models that can measure both the diversity and quality of items in a subset , which makes it a natural choice for the diverse subset selection problem . DPPs have been applied for document and video summarization ( Kulesza and Taskar , 2011 ; Gong et al. , 2014 ) , recommendation systems ( Gillenwater et al. , 2014 ) , object detection ( Azadi et al. , 2017 ) , and grasp clustering ( Huang et al. , 2015 ) . Elfeki et al . ( 2018 ) have also used DPPs to mitigate mode collapse in generative adversarial networks ( GANs ) . The work most related ours is ( Gillenwater et al. , 2014 ) , which also uses the cardinality of DPPs as a proxy for user engagement . However , there are two important differences between our approach and theirs . First , the context is different as they use the cardinality for a subset selection problem while we apply the cardinality as an objective of a continuous optimization problem in the setting of generative models . Second , their main motivation behind using the cardinality is that it aligns better with the user engagement semantics , while our motivation is that using cardinality as a diversity loss for deep neural networks is more stable due to its tolerance of similar trajectories , which are often produced by deep neural networks during stochastic gradient descent . 3 BACKGROUND . 3.1 VARIATIONAL AUTOENCODERS . The aim of multi-modal trajectory forecasting is to learn a generative model over future trajectories . Variational autoencoders ( VAEs ) are a popular choice of generative models for trajectory forecasting ( Lee et al. , 2017 ; Walker et al. , 2016 ) because it can effectively capture all possible future trajectories by explicitly mapping each data point to a latent code . VAEs model the joint distribution pθ ( x , z ) = p ( z ) pθ ( x|z ) of each data sample x ( e.g. , a future trajectory ) and its corresponding latent code z , where p ( z ) denotes some prior distribution ( e.g. , Gaussians ) and pθ ( x|z ) denotes the conditional likelihood model . To calculate the marginal likelihood pθ ( x ) = pθ ( x , z ) /pθ ( z|x ) , one needs to compute the posterior distribution pθ ( z|x ) which is typically intractable . To tackle this , VAEs use variational inference ( Jordan et al. , 1999 ) which introduces an approximate posterior qφ ( z|x ) and decomposes the marginal log-likelihood as log pθ ( x ) = KL ( qφ ( z|x ) ‖pθ ( z|x ) ) + L ( x ; θ , φ ) , ( 1 ) where L ( x ; θ , φ ) is the evidence lower bound ( ELBO ) defined as L ( x ; θ , φ ) = Eqφ ( z|x ) [ log pθ ( x|z ) ] −KL ( qφ ( z|x ) ‖p ( z ) ) . ( 2 ) During training , VAEs jointly optimize the recognition model ( encoder ) qφ ( z|x ) and the likelihood model ( decoder ) pθ ( x|z ) to maximize the ELBO . In the context of multi-modal trajectory forecasting , one can generate future trajectories from p ( x ) by drawing a latent code z from the prior p ( z ) and decoding z with the decoder pθ ( x|z ) to produce a corresponding future trajectory x . 3.2 DETERMINANTAL POINT PROCESSES . Our core technical innovation is a method to learn a diversity sampling function ( DSF ) that can generate a diverse set of future trajectories . To achieve this , we must equip ourselves with a tool to evaluate the diversity of a set of trajectories . To this end , we make use of determinantal point processes ( DPPs ) to model the diversity within a set . DPPs promote diversity within a set because the inclusion of one item makes the inclusion of a similar item less likely if the set is sampled according to a DPP . Formally , given a set of items ( e.g. , data points ) Y = { x1 , . . . , xN } , a point process P on the ground set Y is a probability measure on 2Y , i.e. , the set of all subsets of Y . P is called a determinantal point process if a random subset Y drawn according to P has PL ( Y = Y ) = det ( LY ) ∑ Y⊆Y det ( LY ) = det ( LY ) det ( L + I ) , ( 3 ) where Y ⊆ Y , I is the identity matrix , L ∈ RN×N is the DPP kernel , a symmetric positive semidefinite matrix , and LY ∈ R|Y |×|Y | is a submatrix of L indexed by elements of Y . The DPP kernel L is typically constructed by a similarity matrix S , where Sij defines the similarity between two items xi and xj . If we use the inner product as the similarity measure , L can be written in the form of a Gram matrix L = S = XTX where X is the stacked feature matrix of Y . As a property of the Gram matrix , det ( LY ) equals the squared volume spanned by vectors xi ∈ Y . With this geometric interpretation in mind , one can observe that diverse sets are more probable because their features are more orthogonal , thus spanning a larger volume . In addition to set diversity encoded in the similarity matrix S , it is also convenient to introduce a quality vector r = [ r1 , . . . , rN ] to weigh each item according to some unary metric . For example , the quality weight might be derived from the likelihood of an item . To capture both diversity and quality of a subset , the DPP kernel L is often decomposed in the more general form : L = Diag ( r ) · S · Diag ( r ) . ( 4 ) With this decomposition , we can see that both the quality vector r and similarity matrix S contribute to the DPP probability of a subset Y : PL ( Y = Y ) ∝ det ( LY ) = ( ∏ xi∈Y r2i ) det ( SY ) . ( 5 ) Due to its ability to capture the global diversity and quality of a set of items , we choose DPPs as the probabilistic approach to evaluate and optimize the diversity of the future trajectories drawn by our proposed diversity sampling function .

Summary: This paper presents a novel approach for forecasting object trajectories (e.g., predicted paths of vehicles) forcing diversity of outputs. The authors adopt determinantal point processes (DPPs) to capture the diversity and propose a diversity sampling function (DSF) which consists of a neural network. Its trainable parameter (i.e., \gamma) maps the past trajectory into a set of latent codes and they are decoded to feasible trajectories by a pre-trained conditional variational autoencoder (cVAE). The DSF is trained by maximizing the diversity of the output trajectory. But, since the standard log-likelihood function can be singular, they additionally present an objective function for diversity by maximizing the expected cardinality, which admits replicated outputs. In experiments, the proposed method finds more diverse paths than other competitors under both for synthetic 2D objects and human motions.