Diverse Trajectory Forecasting with Determinantal Point Processes
Authors: Ye Yuan
Kris M. Kitani
Venue:  (2020)
Keywords: Diverse Inference
Generative Models
Trajectory Forecasting

Abstract: The ability to forecast a set of likely yet diverse possible future behaviors of an agent (e.g., future trajectories of a pedestrian) is essential for safety-critical perception systems (e.g., autonomous vehicles). In particular, a set of possible future behaviors generated by the system must be diverse to account for all possible outcomes in order to take necessary safety precautions. It is not sufficient to maintain a set of the most likely future outcomes because the set may only contain perturbations of a dominating single outcome (major mode). While generative models such as variational autoencoders (VAEs) have been shown to be a powerful tool for learning a distribution over future trajectories, randomly drawn samples from the learned implicit likelihood model may not be diverse -- the likelihood model is derived from the training data distribution and the samples will concentrate around the major mode of the data. In this work, we propose to learn a diversity sampling function (DSF) that generates a diverse yet likely set of future trajectories. The DSF maps forecasting context features to a set of latent codes which can be decoded by a generative model (e.g., VAE) into a set of diverse trajectory samples. Concretely, the process of identifying the diverse set of samples is posed as DSF parameter estimation. To learn the parameters of the DSF, the diversity of the trajectory samples is evaluated by a diversity loss based on a determinantal point process (DPP). Gradient descent is performed over the DSF parameters, which in turn moves the latent codes of the sample set to find an optimal set of diverse yet likely trajectories. Our method is a novel application of DPPs to optimize a set of items (forecasted trajectories) in continuous space. We demonstrate the diversity of the trajectories produced by our approach on both low-dimensional 2D trajectory data and high-dimensional human motion data.

Published as a conference paper at ICLR 2020
DIVERSE TRAJECTORY FORECASTING WITH DETERMINANTAL POINT PROCESSES
Ye Yuan, Kris M. Kitani Robotics Institute Camegie Mellon University
{yyuan2, , kkitani (écc.cco cmu. edu
ABSTRACT
The ability forecast set of likely yet diverse possible future behaviors of agent (e.g., future trajectories of a pedestrian) is essential for safety-critica per- ception systems (e.g., autonomous vehicles). In particular, set of possible future behaviors generated by the system must be diverse to account for all possible out- comes in order to take necessary safety precautions. It is not sufficient maintain a set of the most likely future outcomes because the set may only contain pertur- bations of dominating single outcome (major mode). While generative models such as variational autoencoders (VAEs) have been shown to be powerful tool for learning a distribution over future trajectories, randomly drawn samples from the learned implicit likelihood model may not be diverse the likelihood model is derived from the training data distribution and the samples will concentrate around the major mode of the data. In this work, we propose to learn diversity sampling function (DSF) that generates a diverse yet likely set of future trajectories. The DSF maps forecasting context features to of latent codes which be de- coded by generative model (e.g., VAF) into a set of diverse trajectory samples. Concretely, the process of identifying the diverse parameter estimation. To learn the parameters of the DSF, the diversity of the trajectory samples evaluated by diversity loss based on a determinantal point process (DPP). Gradient descent is performed over the DSF parameters, which turn moves the latent codes of the sample set find an optimal set of diverse yet likely trajectories. Our method is novel application of DPPs to optimize a set of items (forecasted trajectories) continuous space. We demonstrate the di- versity of the trajectories produced by our approach both low-dimensional 2D trajectory data and high-dimensional human motion data. (Vided)
of samples is posed DSF
INTRODUCTION
Forecasting future trajectories of vehicles and human has many useful applications in autonomous driving, virtual reality and assistive living. What makes trajectory forecasting challenging is that the future is uncertain and multi-modal vehicles can choose different routes and people can perform different future actions. In many safety-critical applications, is important to consider a diverse set of possible future trajectories, even those that are less likely, that necessary preemptive actions can be taken. For example, an autonomous vehicle should understand that a neighboring car can merge into its lane even though the is most likely keep driving straight. To address this requirement, we need to take a generative approach to trajectory forecasting that can fully characterize the multi- modal distribution of future trajectories. To capture all modes of data distribution, variational autoencoders (VAEs) are well-suited generative models. However, random samples from a learned VAE model with Gaussian latent codes are not guaranteed be diverse for two reasons. First, the sampling procedure is stochastic and the VAE samples can fail to cover some minor modes even with a large number of samples. Second, since VAE sampling based on the implicit likelihood function encoded in the training data, if most of the training data is centered around a specific mode while other modes have less data (Fig. [[ (a)), the VAE samples will reflect this bias and concentrate around the major mode (Fig. (b)). To tackle this problem, we propose to learn diversity sampling function (DSF) that can reliably generate a diverse set of trajectory samples (Fig. (c)).
httts::/ycoueee be/ 5171SJ 1684

---

Published as a conference paper at ICLR 2020
Data
VAE
DSF (Ours)
Modes
Latent Space
XXX %..
Past Thai
X
Tri
The Sampless
Latent Samplee
Traj
Space
(a)
(b)
(c)
Figure 1:
A toy trajectory forecasting example. (a) The three modes (pink, blue, purple) of the future trajectory distribution are shown in both the trajectory space and the latent space of learned VAE model. The data distribution is imbalanced, where the blue mode has most data and covers most of the latent space. (b) Random samples from the VAE only cover the major (blue) mode. (c) Our proposed DSF generates a diverse set of future trajectories covering all three modes.
The proposed DSF is deterministic parameterized function that maps forecasting context features (e.g., past trajectories) to a set latent codes. The latent codes are decoded by the VAE docoder into set of future trajectory samples, denoted the DSF samples. In order to optimize the DSF, formulate a diversity loss based on a determinantal point process (DPP) (Macchi 1975) to evaluate the diversity of the DSF samples. The DPP defines the probability choosing random subset from the set of trajectory samples. It models the negative correlations between samples: the inclusion of sample reduces the probability of including similar sample. This makes the DPP ideal tool for modeling the diversity within set. In particular, we use the expected cardinality of the DPP as the diversity measure, which is defined as the expected size of random subset drawn from the set of trajectory samples according to the DPP. Intuitively, since the DPP inhibits selection of similar samples, if the set of trajectory samples more diverse, random subset more likely to select more samples from the set. The expected cardinality of the DPP is easy to compute and differentiable, which allows us to use it as the objective to optimize the DSF to enable diverse trajectory sampling.
Our contributions are as follows: (1) We propose a new forecasting approach that learns a diversity sampling function to produce diverse set of future trajectories; (2) We propose novel application continuous space with a DPP-based diversity
of DPPs to optimize a set of items (trajectories) measure; (3) Experiments synthetic data and human motion validate that our method reliably generate a more diverse set of future trajectories compared to state-of-the-art generative models.
2 RELATED WORK
Trajectory Forecasting has recently received significant attention from the vision community. A large portion of previous work focuses forecasting 2D future trajectories for pedestrians (Ki- tani et al. 2012: Ma et al. 2017 Ballan et al. 2016 Xie et al.] 2013) or vehicles (Jain et al. 201 Some approaches use deterministic trajectory modeling and only forecast future tra- jectory Alahi et al.) 2016 Yagi et al. 2018 Robicquet et al.) 2016). As there are often multiple plausible future trajectories, several approaches have tried forecast distributions over trajecto- ries (Lee et al. 2017] Galceran et al., 2015 Gupta et al., 2018). Recently, RRiinhart et al. (2018) trajectories for vehicles. Soo Park et al. (016) also use egocentric videos to predict the future trajectories of the camera wearer. Some work has investigated forecasting higher dimensional trajectories such as the 3D full- body pose sequence of human motions. Most existing work takes a deterministic approach and forecasts only one possible future motion from past 3D poses (Fragkiadaki et al. 2015 Butepage et al. 2017 Li et al.) 2017 Jain et al.) (016bb), static images (Chao et al.) 2017 Kanazawa et al.) 2018 or egocentric videos Yuan and Kitani 2019). Differently, some probabilistic approaches (Habibie et al. 2017; Yan et al.] 2018) use conditional variational autoencoders (cVAEs) generate multiple future motions. In constrast to previous work, our approach can generate a diverse set of future motions with limited number of samples.
2019 propose generative model that can accurately forecast
2

---

Published as a conference paper at ICLR 2020
Diverse Solutions have been sought after in number of problems in computer vision and ma- chine learning. A branch of these methods aiming for diversity stems from the M- M-Best MAP prob- lem (Nilsson 1998; (Srouussi and Golmard| 1994), including diverse M-Best solutions (Batra al.) 2012) and multiple choice learning (Guzman- Rivera et al., 2012 Lee et al. 2016). Alternatively, previous work has used submodular function maximization select diverse subset of garments from fashion images (Hsiao and Grauman, 2018). Determinantal point processes (DPPs) (Macchi)
1975 Kulesza et al. 2012) are efficient probabilistic models that can measure both the diversity and quality of items in a subset, which makes it natural choice for the diverse subset selection prob-
lem. DPPs have been applied for document and video summarization (Kulesza and Taskar 2011
Gong et al.) 2014). recommendation systems (Gillenwater et al., 2014), object detection (Azadi jet al. 2017). and grasp clustering (Huang et al. 2015). Elfeki et al. (2018) have also used DPPs to mitigate mode collapse in generative adversarial networks (GANs). The work most related ours is Gillenwater et al. 2014), which also uses the cardinality of DPPs as a proxy for user engagement. However, there are two important differences between our approach and theirs. First, the context is different as they use the cardinality for a subset selection problem while we apply the cardinality as objective of a continuous optimization problem in the setting of generative models. Second, their main motivation behind using the cardinality is that it aligns better with the user engagement semantics, while our motivation is that using cardinality a diversity loss for deep neural networks is more stable due to its tolerance of similar trajectories, which are often produced by deep neural networks during stochastic gradient descent.
BACKGROUND
3.1
VARIATIONAL AUTOENCODERS
The aim of multi-modal trajectory forecasting is to learn a generative model over future trajectories. Variational autoencoders (VAEs) are popular choice of generative models for trajectory forecast- ing (Lee et al.] 2017, Walker et al.] (016) because it can effectively capture all possible future trajectories by explicitly mapping each data point latent code. VAžs model the joint distribution pe(X, z) = po (x]z) of each data sample x (e.g., a future trajectory) and its corresponding latent code z. where p(z) denotes some prior distribution (e.g., Gaussians) and pe po (x denotes the condi- tional likelihood model. To calculate the marginal likelihood po Po Po(x, (z)po(z)xx one needs to compute the posterior distribution po (z)x) which is typically intractable. To tackle this, VAEs use variational inference (Jordan et al. 1999) which introduces an approximate posterior (z and
decomposes the marginal log-likelihood as
(x) KL (go(2)x)]lpo(2)
C(x;0,0)
where L(x; 0, () is the evidence lower bound (ELBO) defined as
0(x;0, (x)x)) [x 2)] - KL (ao(z)x)]l(z))
(2)
During training, VAEs jointly optimize the recognition model (encoder) (z(z)) and the likelihood model (decoder) po (x)) to maximize the ELBO. the context of multi-modal trajectory forecast- ing, one can generate future trajectories from p(x) by drawing latent code from the prior p(z) and decoding z with the decoder pe (x|z) to produce a corresponding future trajectory x.
3.2 DETERMINANTAL POINT PROCESSES
Our technical innovation is a method to learn diversity sampling function (DSF) that can generate a diverse set of future trajectories. To achieve this, we must equip ourselves with a tool to evaluate the diversity of set of trajectories. To this end, we make use of determinantal point processes (DPPs) to model the diversity within a set. DPPs promote diversity within a set because the inclusion of one item makes the inclusion of similar item less likely if the set is sampled according to a DPP.
Formally, given a set of items (e.g., g. data points) y = (xi,..., XN J. point process P on the ground set )' is probability measure on 2 i.e., the set of all subsets of y. P is called a determinantal point process if a random subset Y drawn according to has
det (Lv) Pt(Y xY= Ersy (Lv) det + I)
det (Ly)
(3)
3

---

Published as a conference paper at ICLR 2020
where Y V is the identity matrix, L RNN is the DPP kernel, symmetric positive semidef- inite matrix, and Ly E |R| is submatrix of L indexed by elements of Y.
The DPP kernel L is typically constructed by a similarity matrix S, where Siy defines the similarity between two items x; and x.. If we use the inner product as the similarity measure, L can be written in the form of Gram matrix L = S = X"x where X is the stacked feature matrix of V. As property the Gram matrix, det (Ly) equals the squared volume spanned by vectors x; E Y. With this geometric interpretation in mind, one can observe that diverse sets are more probable because their features are more orthogonal, thus spanning a larger volume.
In addition to set diversity encoded in the similarity matrix S, it is also convenient introduce quality vector r = [r,,..., to weigh each item according to some unary metric. For example, the quality weight might be derived from the likelihood of item. To capture both diversity and quality of subset, the DPP kernel L is often decomposed in the more general form:
L Diag(r) S Diag(r).
With this decomposition, we can see that both the quality vector and similarity matrix S contribute the DPP probability of subset Y:
PL = Y) x det (Ly)
II
det (Sy).
(5)
X.EV
Due to its ability to capture the global diversity and quality of a set of items, we choose DPPs as the probabilistic approach to evaluate and optimize the diversity of the future trajectories drawn by our proposed diversity sampling function.
4 APPROACH
Safety-critical applications often require that the system can maintain a diverse set of outcomes covering all modes of predictive distribution and not just the most likely one. To address this re- quirement, we propose to learn diversity sampling function (DSF) to draw deterministic trajectory samples by generating a set of latent codes in the latent space of a conditional variational autoen- coder (cV)E) and decoding them into trajectories using the eVAE decoder. The DSF trajectory samples are evaluated with a DPP-based diversity loss, which in turn optimizes the parameters of the DSF for more diverse trajectory samples.
Formally, the future trajectory x E TR'x is a random variable denoting a D dimensional feature future time horizon T (e.g., vehicle trajectory or sequence of humanoid poses). The context & = {h, f provides the information to infer the future trajectory x, and it contains the past trajectory h E lg of last H/ time steps and optionally other side information f, such as an obstacle map. In the following, we first describe how we learn the future trajectory model po (x(1b) with a eVAE. Then, we introduce the DSF and the DPP-based diversity loss used to optimize the DSF.
4.1 LEARNING A CVAE FOR FUTURE TRAJECTORIES
In order to generate a diverse set of future trajectory samples, we need to learn a generative trajectory forecasting model pe (x/(b} that can cover all modes of the data distribution. Here (other proper generative models can also be used), which explicitly map data x with the encoder 4s(2) [z|x, 2)) to its corresponding latent code and reconstruct the data from the latent code using the decoder Po [x|2 4b). By maintaining this one-on-one mapping between the data and the latent code, cVAEs attempt to capture all modes of the data. As discussed in Sec. 3.1 cVAEs jointly optimize the encoder and decoder to maximize the variational lower bound:
cVAEs
L(x, ub; 0, Φ) (ys(z(x. [logre (x|2,, bb) KL (1)(2z)x,)))lp(( ())l)()))
(6)
We use multivariate Gaussians for the prior, encoder and decoder: p(z) Ni 0) (z)x,, +b (( μr, IDI), and po (x|z, x/) xxxx;, al). Both the encoder and decoder implemented as neural networks. The encoder network fo outputs the parameters of the posterior distribu- tion: (), = fo(x,{b) The decoder network go outputs the reconstructed future trajectory &:
4

---

Published as a conference paper at ICLR 2020
x 90 (b). Detailed network architectures given in Appendix B.1 parameterization of the eVAE, the objective in Eq. 6 can be rewritten as
Based on the Gaussian
D2 Σ D= 5=1
Levae (x, = V
X x +
(1+2ggg 21og - -1 -0)))
==1
where we take V samples from the posterior qo(z|x, b). Dg is the number of latent dimensions, and B 1/a is weighting factor. The training procedure for the cVAE is detailed in Alg. 2 (Appendix A). Once the cVAE model is trained, sampling from the learned future trajectory model po po(x up) is efficient:
sample latent code according to the prior p(z) and use the decoder (2(222 2)) to decode into a future trajectory x.
Algorithm Training the diversity sampling function (DSF) (1))
1: Input: Training data (x(*), 2 N=1 cVAE decoder network ga (z, a/)
2: Output: DSF S., (zb)
3: Initialize y randomly
4: while not converged do
5: for each z6 do
6: Generate latent codes 2 = [ Zn with the DSF S., (1b)
7:
Generate the trajectory ground set 3 = {xI....,XX, with the decoder go (z, x/)
8:
Compute the similarity matrix S and quality vector r with Eq. and
9:
Compute the DPP kernel = = S Diag(r)
10:
Calculate the diversity loss Ldiverse
11: Update with the gradient V Ldinerse
12:
end for
13:
end while
4.2 DIVERSITY SAMPLING FUNCTION (DSF)
As mentioned before, randomly sampling from the learned cVAE model according to the implicit likelihood function po 2²), i.e. sampling latent codes from the prior p(z), does not guarantee that the trajectory samples are diverse: major modes (those having more data) with higher likelihood will produce most of the samples while minor modes with lower likelihood will have almost no sample. This prompts us to devise new sampling strategy that can reliably generate a diverse set of samples covering both major and minor modes. We propose to learn diversity sampling function (DSF) (1b) that maps context &p to a set latent codes = {Zi..... .,. JNJ. The DSF implemented as 7-parameterized neural network which takes &/ input and outputs a vector of length N Dz (see Appendix B. for for network details). The latent codes Z are decoded into a diverse set future trajectories y {x,...... ......... XN which are denoted the DSF trajectory samples. We note that N is the sampling budget. To solve for the parameters of the DSF, we propose a diversity loss based on DPP defined over y. In this section, first describe how the DPP kernel L is defined, which involves the construction of the similarity matrix S and quality vector r. We then discuss how we use the DPP kernel L to formulate diversity loss optimize the parameters of the DSF.
Recall that the DPP kernel is defined as L = Diag(r) S Diag(r), where r defines the quality of trajectories. The DPP kernel L(>) is
each trajectory and S measures the similarity between function of as it is defined over the ground set y output by the DSF (zb).
Similarity. We measure the similarity Sty between two trajectories x; and x, as
S.j exp ( (-k: k: de (x;,xx))
(8)
where dx is the Euclidean distance and is a scaling factor. This similarity design ensures that 0 S.j < 1 and S 1. It also makes S positive definite since the Gaussian kernel we use is positive definite kernel.
Quality. It may be tempting use p(x|{b to define the quality of each trajectory sample. However, this likelihood based measure will clearly favor major modes that have higher probabilities, making it less likely generate samples from minor modes. This motivates to design quality metric that
5

---

Published as a conference paper at ICLR 2020
treats all modes equally. To this end, unlike the similarity metric which is defined in the trajectory space, the quality of each sample is measured in the latent space and is defined as
W w exp (-2, 2: + R²).
if 2 S R otherwise
T2
Geometrically, let R be the radius of a sphere & containing most samples from the Gaussian prior p(z). We treat samples inside & equally and only penalize samples outside +. In this way, samples from major modes are not preferred over those from minor modes as long as they are inside o while samples far away from the data manifold are heavily penalized as they are outside 4. The radius R is determined by where P percent of the Gaussian samples lie within, and we set p 90. To compute R. we use the percentage point function of the chi-squared distribution which models the distribution over the sum of squares of independent standard normal variables. The base quality is a hyperparameter which we set to I during training in our experiments. At test time, we can use larger w' to encourage the DPP to select more items from the ground set V The hyperparameter p (or R) allows for the trade-off between diversity and quality. When R is small, the quality metric is reduced to a pure likelihood-basee samples with high likelihood and result in a less diverse sample set. When At is large, most samples will have the same quality, and the resulting samples will be highly diverse but less likely. In practice, the choice of R should be application dependent, as one could imagine autonomous vehicles would need to consider more diverse scenarios including those less likely ones to ensure robustness. We note that after the diverse samples are obtained, it is possible to reassign the quality score for each sample based on its likelihood allow users to prioritize more likely samples.
metric (proportional to the latent likelihood), which will prefer
Diversity Loss. To optimize the DSF (1b), we need to define a diversity loss that measures the diversity of the trajectory ground set y generated by S. (v²). An obvious choice for the diversity loss log det((Ln)) + log (et(L(>)) + I). However, there is a problem with directly using the DPP log likelihood. The log likelihood heavily penalizes repeated items: if two trajectories inside y' are very similar, their corresponding rows in L will be almost identical, making det det() = A2..... tice, if the number of modes in the trajectory distribution p(x({b) is smaller than (3%, y will always have similar trajectories, thus making det (1))) always close to zero. In such cases, optimizing the negative log likelihood causes numerical issues, which is observed in our early experiments.
would be the negative log likelihood - log PL(y) (Y ))
zo O (An, is the n-th eigenvalue). In prac-
Instead, the expected cardinality of the DPP is a better measure for the diversity of V, which is defined as -F [Y]] Intuitively, since the DPP discourages selection of similar items, if y is more diverse. a random subset Y drawn according to the DPP is more likely to select more items from V/ thus having larger cardinality. The expected cardinality can be computed as (Eq. 15 and 34 in Kulesza et al. 22112):
An
E CI Σ
tr (I- (L(+)+1)-1)
(10)
The main advantage of the expected cardinality is that it is well defined even when the ground set ) has duplicated items, since it does not require all cigenvalues of L to be non-zero the log likelihood does. Thus, our diversity loss is defined as Ldiverse (7) = tr (I (L(x) + +1)-)). The
training procedure for S., (v)) is outlined in Alg. I
Inference. At test time, given current context &b plwwe use the learned DSF (xb) to generate the fu- ture trajectory ground set y In some cases, ) may still contain some trajectories that are similar to others. In order to obtain a diverse set of trajectories without repetition, we aim to perform MAP in- ference for the DPP to find the most diverse subset Y = arg maxy PL (Y). ). A useful property of DPPs is that the bog-rrbbbbility function is submodular (Gillenwater et al.) 2012). Even though submodular maximization is Na-hard, we use a greedy algorithm (Nemhauser et al. 1978) which is popular optimization procedure that works well in practice. As outlined in Alg. 3 the output set Yy is initialized as 6. and at each iteration, the trajectory which maximizes the log probability
arg max log det (Lyyutx)) xE>rYr
(11)
is added to Yy, until the marginal gain becomes negative or Y, y.

---

Published as a conference paper at ICLR 2020
5 EXPERIMENTS
The primary focus of our experiments is to answer the following questions: (1) Are trajectory sam- ples generated with our diversity sampling function more diverse than samples from the cVÆE and other baselines? (2) How does method perform on both balanced and imbalanced data? (3) Is
our method general enough to perform well on both low-dimensional and high-dimensional tasks?
Metrics. A problem with trajectory forecasting eval- uation is that in real data each context u5(9) usually only has one future trajectory x()), which means we only have one sample from a multi-modal distribu- tion. Let us consider a scenario of three data exam- ples {x(*) (5(")) ) as shown in Fig. (red, purple, blue). The contexts (past trajectories) of the three examples are instances of the same trajectory but they are slightly different due to noise. As these three contexts have the same semantic meaning, they should share the future trajectories, e.g., the purple and blue future trajectories are also valid for the red context. If we evaluate each example (x(1), ()) only with its own future trajectory x(),, a method can achieve high scores by only forecasting the mode corresponding x(6) and dropping other modes. This is undesirable because we want a good method to capture all modes of the fu- ture trajectory distribution, not just a single mode. To allow for multi-modal evaluation, we pro- pose collecting multiple future trajectories for each example by clustering examples with simi- lar contexts. Specifically, we augment each data example (x(+), (b()) with future trajectory set (())) = ((x(011l<bb) - 1 I <E = I,...,,... MM and metrics are calculated based on x)) instead of x (1) i.e., we compute metrics for each x E 200 and average the results.
Context r/
Future X
Figure 2: In real data, contexts (past trajecto-
ries) seldom the same due to noise.
We use the following metries for evaluation: (1) Average Displacement Error (ADE); average mean square error (MSE) over all time steps between the ground truth future trajectory x and the closest sample
the forecasted set of trajectories Yy. (2) Final Displacement Error (FDE): MSE between the final ground truth position x" and the closest sample's 's final position x (3) Average Self Distance (ASD): average L2 distance over all time steps between a forecasted sample x; and its closest neighbor &, in Yz. (4) Final Self Distance (FSD): I.2 distance between the final position of sample x, and its closest neighbor's final position x". The ADE and FDE are common metrics used in prior work on trajectory forecasting (Alahi et al.) 2016 Lee et al. 2017 Rhinehart et al.) 2018 Gupta et al., 2). However, these two metrics do not penalize repeated samples. To address this, we introduce two new metrics ASD and FSD to evaluate the similarity between samples in the set of forecasted trajectories. Larger ASD and FSD means the forecasted trajectories are more non-repetitive and diverse.
Baselines. We compare our Diversity Sampler Function (DSF) with the following baselines: (1) eVAE: method that follows the original sampling scheme of cVAE by sampling latent codes from a Gaussian prior p(z). (2) MCL: an approach that uses multiple choice learning (Lee et al., 2016) to optimize the sampler S,, (xb) with the following loss: Lmat mingey x where the ground truth future trajectory. (3) R2P2: a method proposed in (Rhinehart et al. 2018) that uses reparametrized pushforward policy improve modeling of multi-modal distributions for vehicle trajectories. (4) cGAN: generative adversarial networks (Goodfellow et al., 2014) conditioned on the forecasting context. We implement all baselines using similar networks and perform hyperpa- rameter search for each method for fair comparisons. For methods whose samples are stochastic, we use 10 random seeds and report the average results for all metrics.
5.1 SYNTHETIC 2D TRAJECTORY DATA
We first use synthetic data to evaluate our method's performance for oow-dimensional tasks. We design virtual 2D traffic scene where vehicle comes to crossroad and choose three different future routes: forward, left, and right. We consider two types of synthetic data: (1) Balanced data, which means the probabilities of the vehicle choosing one of the three routes are the same; (2)
7

---

Published as a conference paper at ICLR 2020
DSF (Ours)
cVAE
MCL
R2P2
cGAN
Balanced
Data
Imbalanced Data
Figure 3: Qualitative results on synthetic data for both balanced and imbalanced data distribution
when N 10. Blue represents the past trajectory and red represents forecasted future trajectories.
Balanced Data
Imbalanced Data
Method
ADE
FDE 4
ASD f
FSD f
ADE +
FDE
ASD
FSD 1
DSF (Ours)
0.182
0.344
0.147
0.340
0.198
0.371
0.207
0.470
<VAE
0.262
0.518
0.022
0.050
0.332
0.662
0.021
0.050
MCL
0.276
0.548
0.012
0.030
0.457
0.938
0.005
0.010
R2P2
0.211
0.361
0.047
0.080
0.393
0.776
0.019
0.030
cGAN
0.808
1.619
0.018
0.010
1.784
3.744
0.006
0.001
Table I: Quantitative results on synthetic data (numbers scaled by 10) when = 10.
Imbalanced data, where the probabilities of the vehicle going forward. left and right are 0.8, 0.1, 0.1, respectively. We synthesize trajectory data by simulating the vehicle's behavior and adding Gaussian noise vehicle velocities. Each data example (x(*), (5(+) contains future trajectories of steps and past trajectories of 2 steps. We also add an obstacle map around the current position to the context (b)) In total, we have around 1 100 training examples and 1000 test examples. Please refer Appendix B for more implementation details.
Table summarizes the quantitative results for both balanced and imbalanced data when the sam- pling budget N is 10. We
that method DSF outperforms the baselines in all metrics in both test settings. As shown in Fig. 3, our method generates more diverse trajectories and is less affected by the imbalanced data distribution. The trajectory samples of method are also less repetitive, feature afforded by our DPP formulation. Fig. shows how ADE changes as a function of the sampling budget N.
5.2 DIVERSE HUMAN MOTION FORECASTING
To further evaluate method's performance for more complex and high-dimensional tasks, apply method to forecast future hu- man motions (pose sequences). We use motion capture obtain 10 motion sequences includ- ing different types of motions such as walking, turning, jogging, bending, and crouching. Each sequence is about I minute long and each pose consists of joint angles. We use past poses s0. to forecast next 30 poses (1s). There are around 9400 training examples and 2000 test exam- ples where we use different sequences training and testing. More implementation details can be
Method
ADE
FDE 4
ASD
FSD 1
DSF (Ours)
0.259
0.421
0.115
0.282
eVAE
0.332
0.642
0.034
0.098
MCL
0.344
0.674
0.036
0.122
cGAN
0.652
1.296
0.001
0.003
Table Quantitative results for human motion
forecasting when N =10.
found in Appendix B
present quantitative results Table and we can see that our approach outperforms other meth- ods in all metries. As the dynamics model used in R2P2 (Rhinehart et al. 2018) does not generalize well for high-dimensional human motion, we find the model fails to converge and we do not com- pare with it in this experiment. Fig. showws that our method achieves large improvement when the sampling budget is big (N = 50). We also present qualitative results Fig, 5 where we show the starting pose and the final pose of all 10 forecasted motion samples for each method. We can clearly
8

---

Published as a conference paper at ICLR 2020
Synthetic - Balanced
Synthetic - Imbalanced
Human
0.7
0.7
DSF
DSF
0.45
DSF
0.6
VAE
VAE
VAE
0.40
0.5
MCL
0.5
MCL
MCL
ADE
ADE
ADE
0.4
R2P2
R2P2
0.35
0.30
0.2
0.2
0.25
0.1
20
40
20
Number of Samples N
Number of Samples N
Number of Samples N
Figure 4: ADE vs. N for synthetic data and human motion forecasting. cGAN is not shown in this plot much worse than other methods due to mode collapse.
DSF (Ours)
CVAE
MCL
cGAN
Start Pose
Figure 5: Qualitative results for human motion forecasting when N = 10. The left shows the starting pose, and the right shows for each method the final pose of all 10 forecasted motion samples.
see that our method generates more diverse future human motions than the baselines. Please refer to Appendix C Cld our video for additional qualitative results.
5.3 ADDITIONAL EXPERIMENTS WITH DIVERSITY-EE EAVERSIT-BBASED BASELINES
In this section, we perform additional experiments on a large human motion dataset (3.6 million frames), Human3.6., (lonescu al. 2013), to evaluate the generalization ability of our approach. We predict future motion of 2 seconds based on observed motion of 0.5 seconds. Please refer Appendix B.3 for implementation details. We also several variants of our method (DSF) and the cAE to validate several design choices of our method, including the choice of the expected cardinality over the negative log likelihood (NLL) of the DPP as the diversity loss. Specifically, we use the following new baselines: (1) DSF-NLL: a variant of DSF that uses NLL as the diversity loss instead of the expected cardinality. (2) DSF-COS: DSF variant that uses cosine similarity to build the similarity matrix S for the DPP kernel L. (3) DSF-NLL: variant the cVAE that samples 100 latent codes and performs DPP MAP inference on the latent codes to obtain diverse set of latent codes, which are then decoded into trajectory samples.
new selection of baselines including
We present quantitative results Table when the number of samples N is and 50. The baseline DSF-COS is able to achieve very high diversity (ASD and FSD) but its samples are overly diverse and have poor quality which is indicated by the large ADE and FDE. Compared with DSF-NLL,
=
10
N = 50
Method
ADE L
FDE
ASD
FSD 1
ADE 4
FDE J
ASD 1
FSD
DSF (Ours)
0.340
0.521
0.381
0.621
0.236
0.306
0.313
0.415
DSF-NLL
0.335
0.514
0.343
0.496
X
X
X
DSF-COS
2.588
1.584
5.093
5.718
0.978
0.891
2.007
1.968
eVAE
0.363
0.549
0.235
0.360
0.276
0.369
0.160
0.220
eVAE-LDPP 0.373
0.554
0.280
0.426
0.277
0.365
0.176
0.240
Table 3: Quantitative results on Human3.66 3.6M means the method is unable learn model due numerical instability.
(Tonescu et al., 2013) for N = 10 and N = 50. X

---

Published as a conference paper at ICLR 2020
method achieves better diversity (ASD and FSD) and similar ADE and FDE when the number of samples is small (N = 10). For a larger number of samples (N = 50), NLL becomes unstable even with a large (Ie-3) added to the diagonal. This behavior of NLL, i.e., stable for small N but unstable for large N, matches our intuition that NLL. becomes unstable when samples become similar (as discussed in Sec. 4.2), because when there are more samples, it is easier have similar samples during the SGD updates of the DSF network. The baseline eVAE-LDPP also performs worse than DSF in all metrics even though is able to outperfom the cVAE. We believe the reason is that diversity in sample space may not be well reflected in the latent space due to the non-linear mapping from latent codes to samples induced by deep neural networks.
6 CONCLUSION
We proposed novel forecasting approach using DSF optimize over the sample space of generative model. Our method learns the DSF with a DPP-based diversity measure to generate a diverse set of trajectories. The diversity measure is novel application of DPPs to optimize a set of items in continuous space. Experiments have shown that our approach can generate more diverse vehicle trajectories and human motions compared state-of-the-ae of-the-artt baseline forecasting approaches.
Acknowledgment. This project was sponsored in part by JST CREST (EMJCRI4EI)) NSF NRI (1637927) and TARPA (D17PC00340).
REFERENCES
A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction crowded spaces. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-971, 2016.
S. Azadi, J. Feng, and T. Darrell. Learning detection with diverse proposals. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 7149-7157, 2017.
L. Ballan, F. Castaldo, A. Alahi, F. Palmieri, and S. Savarese. Knowledge transfer for scene-specific motion prediction. In European Conference on Computer Vision, pages 697-713. Springer, 2016.
D. Batra, P. Yadollahpour, A. Guzman-Riverta and G. Shakhnarovich. Diverse m-best solutions in markov random fields. In European Conference on Computer Vision, pages 1-16. Springer, 2012.
J. Butepage, M. J. Black, D. Kragic, and H. Kjellstrom, Deep representation learning for human motion predic- tion and classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6158-6166 2017.
Y.-W Chao, Yang. Price, S. Cohen, and J. Deng. Forecasting human dynamics from static images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 548-556, 2017. M. Elfeki, C. Couprie, M. Riviere, and M. Elhoseiny. Gdpp: Learning diverse generations using determinantal point process, arXiv preprint arXiv:1812.0000 1200000 2018.
K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik. Recurrent network models for human dynamics. In Pro- ceedings of the IELE International Conference on Computer Vision, pages 4346-4354, 2015.
E. Galceran, A. G. Cunningham, R. M. Eustice, and E. Olson. Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction. In Robotics: Science and Systems, volume 1, 2015.
J. Gillenwater, A. Kulesza, and B. Taskar. Near-optimal map inference for determinantal point processes. In Advances in Neural Information Processing Systems, pages 2735-2743,201
J. A. Gillenwater, A. Kulesza, E. Fox, and B. Taskar. aopectationmmaximizatte point processes. In Advances in Neural Information Processing Systems, pages 3149-3157, 2014. B. Gong, W.-L. Chao, K. Grauman, and F. Sha. Diverse sequential subset selection for supervised video summarization. In Advances in Neural Information Processing Systems, pages 2069-2077, 2014.
for learning determinantal
I. Goodfellow, J. Pouget-Ahadic, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014. A. Gupta, J. Johnson, I., Fei-Fei, S. Savarese, and A. Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pages 2255-2264 2018.
A. Mirrman-Rivera, D. Batra, and P. Kohli. Multiple choice learning: Learning to produce multiple structured
outputs. In Advances in Neural Information Processing Systems, pages 1799-1807, 2012.
I. Habihie, D. Holden, J. Schwarz, J. Yearsley, and T. Komura. recurrent variational autoencoder for human motion synthesis. BMVC17, 2017.
W.-.. Hsian and K. Grauman. Creating capsule wardrohes from fashion images. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 7161-7170, 2018.
10

---

Published as a conference paper at ICLR 2020
D.-A. Huang, M. Ma, W.-C. Ma, and K. M. Kitani, How do hands? discovering diverse set of common grasps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 666-675, 2015.
C. lonescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m:
Large scale datascts and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-13) 1339, 2013.
A. Jain, A. Singh, H. S. Koppula, S. Soh, and A. Saxena. Recurrent neural networks for driver activity ticipation via sensory-fusion architecture. In Robotics and Automation (ICRA), 2016 IEEE International Conference on, pages 3118-3125. IEEE, 2016a.
A. Jain, A. R. Zamir, S. Savarese, and A. Saxena. Structural-rnn: Deep learning spatio-tempont graphs.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2016b.
M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul. An introduction to variational methods for graphical models. Machine learning. 37(2):183-333, 1999.
A. Kanazawa, J. Zhang, P. Felsen, and J. Malik. Learning 3d human dynamics from video. arXiv preprint 1112v:112.00044
2018.
D. P. Kingma and Ba. Adam: A method for stochastic optimization. arXiv preprint 11Xiv:1412.6900 2014.
K. M. Kitami, B. D. Zicbart, J. A. Bagnell, and M. Hebert. Activity forecasting. In European Conference on Computer Vision, pages 201-214. Springer, 2012.
A. Kulesza and B. Taskar. k-dpps: Fixed-size determinantal point processes. In Proceedings of the 28th International Conference on Machine Learning (ICML-II), pages 1193-1200, 2011.
A. Kulesza, B. Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning, 3(2-3):123-288) 2012.
N. Lee, W. Choi, P. Vernza,, B. Choy. H. Torr, and M. Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 336-345, 2017.
Lee, S. S. Prakash, M. Cogswell, V. Ranjan, D. Crandall, and D. Batra. Stochastic multiple choice learning for training diverse deep ensembles. In Advances in Neural Information Processing Systems, pages 2119-2122 2016.
Li, Y. Zhou, S. Xiao, C. Hc, Z. Huang, and H. Li. Auto-conditioned recurrent networks for extended complex human motion synthesis. arXiv preprint
2017.
W.C. Ma, D.-. Huang, N. Lee, and K. M. Kitani. Forecasting interactive dynamies of pedestrians with fictitious play. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
774-782, 2017.
O. Macchi. The coincidence approach to stochastic point processes. Advances in Applied Probability, 7(1):
83- 1975.
J. Martinez, R. Hossain, J. Romero, and J. J. Little. A simple yet effective bascline for 3d human pose estima- tion. In Proceedings of the IEEE International Conference on Computer Vision, pages 2640-2649, 2017. G. L. Nemhauser, I.. A. Wolsey, and M. L.. Fisher. An analysis of approximations for maximizing submodular set functionsi. Mathematical programming, 14(1):265-294 1978.
D. Nilsson. An efficient algorithm for finding the m most probable configurationsin probabilistic expert sys- tems. Statistics and computing, 1(2):159-173 1998.
G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to-hie volumetric prediction for single-image 3d human pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7025-7034, 2017.
D. Pavilo, C. Feichtenhofer, D. Grangier, and M. Auli. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7753-7762, 2019.
N. Rhinchart, K. M. Kitani, and P. Vernaza R2p2: A reparameterized pushforward policy for diverse, precise generative path forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), pages
772-788, 2018
N. Rhinehart, R. McAllister, K. Kitani, and S. Levine. Precog: Prediction conditioned goals in visual
multi-agent settings. arXiv preprint arXiv: 10tp:0/250.06 2019.
A. Robiequet, A. Sadeghian, A. Alahi, and S. Savarese. Learning social etiquette: Human trajectory under- standing in crowded scenes. In Eumpean conference on computer vision, pages 549-565. Springer, 2016. B. Seroussi and J.-L. Golmard. An algorithm directly finding the most probable configurations in hayesian networks. International Journal of Approximate Reasoning, 1 1133)225-2333 1994.
H. Soo Park, J.JJ Hwang, Y. Niu, and J. Shi. Egocentric future localization. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pages 4697-4705, 2016.
J. Walker, C. Doersch, A. Gupta, and M. Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In European Conference on Computer Vision, pages 835-851. Springer, 2016.
11

---

Published as a conference paper at ICLR 2020
D. Xie, Todorovic, and S.-C. Zhu. Inferring "dark matter" and "dark energy" from videos. 2013 IEEE International Conference on Computer Vision, pages 2224-2231, 2013.
T. Yagi. K. Mangalam, R. Yonetani, and Y. Sato. Future person localization in first-person videos. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
X. Yan, A. Rastogi. R. Villegas, K. Sunkavalli, E. Shechtman, S. Hadap, E. Yumer, and H. Lee. Mt-vae: Learning motion transformations to generate multimodal human dynamics. In Proceedings of the European Conference on Computer Vision (ECCV), pages 265-28! 2018.
Y. Yuan and K. Kitani. Ego-pose estimation and forecasting as real-time pd control. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 10082-10092, 2019.
A ALGORITHMS
Algorithm 2 Training the eVAE
I: Input: Training data (x(*), 2b (b)) M
[==1
2: Output: cVAE encoder network [x 4/) and decoder network go(2, s/)
3: Initialize & and 0 randomly
4: while not converged
5: for each (x(*), ab())) do
6:
Compute parameters (μ, or) of the posterior distribution (z 2b) using fø ab)
7: Sample Gaussian noises (El, from A/ I)
8:
Transform noises to latent samples from Qa x,], +/):
p 2 Decode latent samples into reconstructed trajectories {xI..... .a.....,yy. Xy using go v))
9:
10:
Calculate the e VAE loss Levae according Eq.
11: Update & and with V VoLerae and VgLevee
12:
end for
13: end while
Algorithm 3 Inference with the DSF (yb))
1: Input: Context vb, DSF S,, (vb), cVE decoder network go(z, t/))
2: Output: Forecasted trajectory set Y,
3: Generate latent codes 2 {zz,.... .... 2N } with the DSF (zb)
4: Generate the trajectory ground set y = { XN3 with the decoder go (z, v)
5: Compute the DPP kernel L Diag(r) S Diag(r)
6: Y5 < 0, U < y
7: while U not empty do
8: x" +- arg maxxeu log det (Ly,)(x))
9:
if log det (Lyju(x))) log det (Ly,) < <0 then
10:
break
11:
end if
12:
Y + Y, U fx*)
U U U \x*)
14:
end while
12

---

Published as a conference paper at ICLR 2020
B IMPLEMENTATION DETAILS
Networks for Synthetic Data
Encoder
Decoder
DSF
(2)
2
h
LSTM Cell
Networks for Human Motion Decoder
MLP Layer
Encoder
DSF
CNN
(
xx
h
h
for synthetic data, we CNN to process the obstacle map f and directly flatten trajectories x and h into vectors. The reconstructed trajectory X is decoded with an MLP. Bottom: for human motion, we use Bi-LSTMs extract temporal features for x and h and decode the reconstructed trajectory & with forward LSTM.
Figure 6: Network architectures for synthetic data and human motion. Top:
B. NETWORK ARCHITECTURRS
Synthetic data. Fig. 6 (Top) shows the network architecture for synthetic data. The number of latent dimensions is By default, we use ReLU activation for all networks. The future trajectory E R3x2 consists of 3 future positions of the vehicle. The context Ab contains past trajectories E 2222 of 2 time steps and obstacle map E 1J,5525x25 spanning a x area around the current position of the vehicle (the road width is 2). For the encoder, neural network (CNN) with three 32-channel convolutional layers to process f. The first two layers have kernel size 4 and stride while the last layer has kernel size 6 and stride 1. The obtained CNN features are concatenated with flattened x and h into a unified feature, which is feed into a multilayer perceptron (MLP). The MLP has one 128-dim hidden layer and two heads outputing the mean and variance a of the latent distribution. For the decoder, we concatenate the CNN feature from with the latent code through an MLP with one 128-dim hidden layer which outputs the reconstructed future trajectory R For the diversity sampler function (DSF). we concatenate the CNN feature from f with the flattened h and pass it through an MIP with one 128-dim hidden layer to obtain set of latent codes (.,,..., 2N) which are represented by a vector length 2N.
convolutional
IR2 and flattened h into a unified feature. The feature passed
Human motion. Fig. 6 (Bottom) shows the network architecture for synthetic data. The number of latent dimensions is The future trajectory x E x0 consists future poses of 30 time steps (Is). The context contains past poses h E R3x59 of 3 time steps (0.1s). Each pose consists of 59 joint angles. For the encoder, we use two 128-dim bidirectional LSTMs (Bi-LSTMs)) and mean pooling obtain the temporal features for and h. We then concatenate the temporal features into a unified feature and feed it into an MLP with two hidden layers (300, 200) and two heads to obtain the mean i and variance of the latent distribution. For the decoder,
the Bi-LSTM of
13

---

Published as a conference paper at ICLR 2020
the encoder for the context h and 128-dim forward LSTM decode the future trajectory X. At each time step i, the forward LSTM takes as input the previous pose x (h 8 for = 0), the latent code R and the temporal features from h, and outputs 128-dim feature. The feature is then passed through an MLP with two hidden layers 200) to generate the reconstructed pose x For the DSF, feed into an MLP with a 128-dim hidden layer to produce set of latent codes ( ZN3 which represented by vector of length S.V.
different 128-dim Bi-LSTM to obtain the temporal feature for h, which is
B.2 TRAINING AND EVALUATION
When training the cVEE model using Eq. 7, take V sample from the posterior 4s (z) x, .)). The weighting factor for the KL term is set to 0.1 for synthetic data and le-4 for human motion. We use Adam (Kingma and Ba 2014) to jointly optimize the encoder and decoder. The learning rate is set to le-4 and we use mini batch size of 32 for synthetic data. We optimize the model for 500 epochs for synthetic data and 100 epochs for human motion.
When training the DSF, the scale factor for the similarity matrix is set to for synthetic data and 1e-2 for human motions. For both synthetic data and human motions, rate le-4 to optimize the DSF for 20 epochs.
Adam with learning
Recall that in the metrics section (Sec. 5.1), need the grouping threshold build the ground truth future trajectory set x {I((1)|[2b)) b S = 1, I...., MM. For synthetic data, is
set to 0.1 and we only use past trajectories h to compute the distance between contexts. For human motion, set to 0.5.
B.3 IMPLEMENTATION DETAILS FOR EXPERIMENTS HUMAN3.6N
Following previous work (Martinez et al. 2017 Pavlakos et al. 2017 Pavllo et al. 2019), we con- vert the motion sequences the dataset into sequences of 3D joint positions, and adopt 17-joint skeleton. We train on five subjects (S1, $5, $6, S7, S8), and test on two subjects (S9 and S11).
We use the same network architectures (Fig 6 (Bottom)) this experiment as the one used in the human motion forecasting experiment above. The number of latent dimensions is 128. When train- ing the cVAE model, the weighting factor set to 0.1. We sample 5000 training examples every epoch and optimize the cVEE for 500 epochs using Adam and a learning rate of le-4. We set the batch size to 64 for the optimization.
The scale factor k for the similarity matrix S of the DPP kernel is set to 5. When learning the DSF, we use a batch size of 64 and sample 1000 training examples every epoch and optimize the DSF for 20 epochs using Adam and learning rate of le-3.
When computing the metrics, we set the grouping threshold e to 0.1.
14

---

Published as a conference paper at ICLR 2020
C ADDITIONAL VISUALIZATION
We also show additional qualitative results for human motion forecasting Fig. 7 diversity of the forecasted motions are best seen in our video
The quality and
The final pose of 10 forecasted motions
DSF (Ours)
cVAE
MCL
cGAN
Start Pose
DSF (Ours)
CVAE
MCL
cGAN
Start
Pose
DSF (Ours)
CVAE
MCL
cGAN
Start
Pose
DSF (Ours)
CVAE
MCL
cGAN
Start Pose
Figure 7: Additional visualization for human motion forecasting. The left shows the starting pose, and the right show for each method the final pose of 10 forecasted motion samples.
https://ycutu..e/5555 b/5/715 1d84
15


Reviews:
[{"review_id": "rklqqtBhYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel approach for forecasting object trajectories (e.g., predicted paths of vehicles) forcing diversity of outputs. The authors adopt determinantal point processes (DPPs) to capture the diversity and propose a diversity sampling function (DSF) which consists of a neural network. Its trainable parameter (i.e., \\gamma) maps the past trajectory into a set of latent codes and they are decoded to feasible trajectories by a pre-trained conditional variational autoencoder (cVAE). The DSF is trained by maximizing the diversity of the output trajectory. But, since the standard log-likelihood function can be singular, they additionally present an objective function for diversity by maximizing the expected cardinality, which admits replicated outputs. In experiments, the proposed method finds more diverse paths than other competitors under both for synthetic 2D objects and human motions.\n\nThis paper is well-written and easy to understand. The contribution can be important as it performs better than other generative networks that are not forcing diversity. Unlike them, the proposed method can capture the diversity trajectories, which requires for safety-critical applications. \n\nMain concerns:\n\n1. The DPP kernel consists of a similarity (equation (8)) and a quality score (equation (9)). However, it is unnatural that the similarity is defined in the data space (x) and the quality is defined in latent space (z). A more naive approach is to define the DPP kernel as a function of x or z. Is there any specific reason to define those scores are defined in different space?\n\n2. To maximize the expected cardinality, it is enough that the eigenvalues of L(\\gamma) become large. Does the network find the trivial solution? E.g., all eigenvalues are the same as a very large value. In addition, the proposed diverse loss is not a new approach. The maximum induced cardinality of a DPP was proposed by [1] and its various properties were also studied therein. \n\n3. In a work of [1], a method to optimize the induced cardinality (a similar to diverse loss in this paper) is proposed. And it would be great to compare the maximum induced cardinality (also can be approximated by the greedy algorithm) to the MAP for inference of diverse trajectory.\n\n4. It is also possible to apply DPP MAP inference to a set of latent codes generated from the encoder of cVAE. Then, the decoder can map the diverse latent variables to feasible data trajectories. Are these outputs also diverse? or does the diversity of latent space reflect the diversity in the data trajectory space? \n\n5. Computing the gradient of the proposed diverse loss is expensive since it is the trace of an inverse of a parameterized matrix. How long does it take to learn the proposed DSF compared to other methods? \n\n6. To avoid that the determinants become zero, a popular choice is to shift all eigenvalues with a small amount (this can be done by adding eps * identity matrix to the kernel matrix). Did the authors investigate other practical diverse losses?\n\n7. It seems to be possible to train the parameters of the kernel matrix, i.e., k in equation (8) and \\omega in equation (9). Did the author try to learn those parameters?\n\nOverall, this work proposes an approach combined cVAE with a DPP for forecasting diverse trajectories and the empirical results are promising as it outperforms other methods. But, its novelty is incremental and competitors are not actually the models capturing diversity. I vote for a weak acceptance but depending on clarifications on the above concerns in an author response, I would be willing to increase the score.\n\nMinor comments:\n\n1 . Please specify the network architecture of DSF and details on the parameter \\gamma.\n\n\n[1] Gillenwater, Jennifer A., et al. \"Maximizing induced cardinality under a determinantal point process.\" Advances in Neural Information Processing Systems. 2018.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "rkgSq-KstS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Caveat: I am very familiar with DPPs, but unfamiliar with the literature of trajectory forecasting, autonomous driving, etc.\n\nSummary: This work introduces a generative model for diverse sequences based on a DPP, with the goal of providing likely yet non-overlapping possible future trajectories to models that require such information for safety concerns (eg, autonomous driving). Rather than using the DPPs negative log-likelihood as a measure of the trajectory set's diversity, the authors use the DPP's expected sample size as a proxy for a diversity metric. Experimentally, the authors show on two tasks that this approach generates more diverse, relevant trajectories than several competing baselines.\n\nRecommendation: I recommend this paper be accepted, with the caveat that I am not well equipped to evaluate the experimental contribution of this paper, which in my opinion is the most important part of this work. I would have liked to see more extensive experiments, in particular (a) other baselines (DPP NLL as a diversity loss, different DPP kernels) and (b) experiments on larger datasets.\n\nHigh level comments and questions:\n\n- When generating the trajectory set (Alg. 3), your algorithm may generate sets of variable size (as the DPP NLL is log-submodular but not necessarily increasing). Did you also consider generating sets of fixed size, or of a minimal size? \n\n- Previous work has also looked at using DPPs to alleviate mode collapse in GANs (Elfeki et al., ICML 2019). Could you comment on how you expect such a method would perform on your chosen tasks? \n\n- The intuition that the DPP negative log-likelihood will overwhelmingly penalize subsets that contain a few similar trajectories seems very reasonable. However, I would have liked to see that intuition verified through an experiment that used the NLL as a diversity loss.\n\n- The training sets for both experiments seem fairly small (1100 and 9400 training examples). Could you comment on this choice, and on whether you expect your experimental results to generalize to much larger datasets?\n\n- Your motivating example is autonomous vehicles and safety concerns; with that example in mind, could you comment on how to select the radius R (or, equivalently, \\rho) in Eq. 9? It seems like this choice would have significant downstream implications on the trade-off between accuracy and robustness to unpredictability.\n\n- The idea of using the expected subset size as a metric for diversity is compelling. The authors may want to take a look at (Gillenwater et al., NeurIPS'18), which uses a similar metric as a proxy for user engagement. \n\nMinor comments: \n- after Eq. 2: please consider defining the acronym ELBO before using it.\n- Eq. 8: if you are using an exponentiated quadratic, I believe the distance should be squared.\n- Eq. 10: consider citing the relevant proposition from (Kulesza & Taskar).\n ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HJlvetnpFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method to diversify samples generated from a VAE. The method is based on determinantal point processes in the latent space and relies on specifying a kernel in the sample space and a quality metric in the latent space. \n\nIncreasing diversity of samples from VAE-like models is an important and common problem and the authors present a reasonably generic method for solving it. The experiments are sensible and show clear improvement over a reasonable selection of baselines. One improvement I would suggest is adding a more explicit discussion of how the proposed method avoids generating overly diverse, very improbable trajectories. It is also somewhat suspicious that the trajectories shown in Figure 3 are terminated so early.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach for forecasting diverse object trajectories using determinantal point processes (DPP). Past trajectory is mapped to a latent code and a conditional VAE is used to generate the future trajectories. Instead of using log-likelihood of DPP, the propose method optimizes expected cardinality as a measure for diversity. While there are some concerns about the core method being incremental in novelty over some existing DPP based methods, the context of the paper is different from these papers (ie, diverse trajectories in continuous space) and reviewers have appreciated the empirical improvements over the baselines, in particular over DPP-NLL and DPP-MAP in latent space. 
Decision: Accept (Poster)