MGF: Mixed Gaussian Flow for Diverse Trajectory Prediction

Authors: jiahe  chen
jinkun  cao
dahua  lin
kris  kitani
jiangmiao  pang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:29:34

Abstract: To predict future trajectories, the normalizing flow with a standard Gaussian prior suffers from weak diversity. 
The ineffectiveness comes from the conflict between the fact of asymmetric and multi-modal distribution of likely outcomes and symmetric and single-modal original distribution and supervision losses.
Instead, we propose constructing a mixed Gaussian prior for a normalizing flow model for trajectory prediction.
The prior is constructed by analyzing the trajectory patterns in the training samples without requiring extra annotations while showing better expressiveness and being multi-modal and asymmetric.
Besides diversity, it also provides better controllability for probabilistic trajectory generation.
We name our method Mixed Gaussian Flow (MGF). It achieves state-of-the-art performance in the evaluation of both trajectory alignment and diversity on the popular UCY/ETH and SDD datasets. Code is available at https://github.com/mulplue/MGF.

Main paper: 
 introduction: 
 In this work, we aim to improve the diversity for probabilistic trajectory prediction. In trajectory prediction, diversity describes the fact that agents (pedestrians) can move in different directions, speeds, and interact with other agents. Because the motion intentions of agents can not be determined by their historical positions, there is typically no global-optimal strategy to predict a single outcome of future trajectories. Therefore, recent works have focused on probabilistic methods to generate multiple likely outcomes. However, existing solutions are argued to lack good diversity and they often fail to generate the under-represented future trajectory patterns in the training data.Different motion patterns are usually imbalanced in a dataset. For example, agents are more likely to move straight than turn around in most datasets. Thus, many motion patterns are highly underrepresented though discoverable. Therefore, intuitively, an ideal distribution to represent the possible future trajectories should be asymmetric, multi-modal, and expressive to represent long-tailed patterns.However, most existing generative models solve the problem of trajectory prediction by modeling it as a single-modal and symmetric distribution, i.e., standard Gaussian. This is because the standard Gaussian is tractable and there is a belief that it can be transformed into any desired distribution of the same or a lower dimension. However, deriving a desired complex distribution from a simple and symmetric prior distribution is challenging, especially with limited and imbalanced data. Moreover, when we derive the target distribution by transforming from the tractable original distribution as Normalizing Flows, GANs, and VAEs do, a dilemma arises: an over-smoothing transformation model can neglect under-represented samples while an over-decorated transformation model will overfit. Especially for normalizing flow, some studies [22, 3] discussed the difficulty of training normalizing flow in practice to represent a complex target distribution.To solve this dilemma, we propose a prior distribution with more expressiveness and data-driven statistics. It is asymmetric, multi-modal, and adaptive to the training data in the form of a mixed set of Gaussians. Compared to the standard Gaussian, the mixture of Gaussians can better summarize the under-represented samples scattered far away in the representation space. This relieves the sparsity issue of rare cases and thus enhances the diversity of the generated outcomes. Besides diversity, as Figure 1 : Non-invertible generative models (a), e.g., CVAE, GAN, and diffusions, lack the invertibility for probability density estimation. Flow-based methods (b) are invertible while, sampling from the symmetric standard Gaussian, undermines the diversity and controllability of generation. Our proposed Mixed Gaussian flow (c) maps from a mixed Gaussian prior instead. Summarizing distributions from data and controllable edits, it achieves better diversity and controllability for trajectory prediction.the mixed Gaussian prior is parametric and transparent during construction, we could control the generation by manipulating this prior, such as adjusting the weights of different sub-Gaussians or manipulating the mean value of them. All these manipulations change generation results statistically without requiring fine-tuning or other re-training. Upon the prior distribution, we choose to construct the generative model by normalizing flow with the unique advantage of being invertible. We thus could estimate the likelihood of each single generated outcome. By combining the designs, we propose a normalizing flow-based trajectory prediction model named Mixed Gaussian Flow (MGF). It enjoys better diversity, controllability, interruptibility, and invertibility for trajectory prediction. During our study, we find that though several evaluation tools have been proposed for measuring diversity [49, 62, 33, 37] , they employ varying calculation method and have not gained widespread adoption within the research community. The most popular evaluation metrics (APD/FDE scores) focus on how similar a generated trajectory is to the single ground truth. It is calculated in a "best-of-M " fashion where only one candidate in a batch of M predictions is taken into the measurement. This protocol encourages the methods to generate outcomes approaching the mean (most likelihood) of the learned distribution and provides no sense of the diversity of generation outcomes. Therefore, building upon previous research in the field of human motion prediction [62] , we formulate a metric set of Average Pairwise Displacement (APD) and Final Pairwise Displacement (FPD), which measure the diversity of a batch of M generated samples. This helps us to have a concrete study about generation diversity and avoid bias from the "best-of-M " evaluation protocol. With the proposed metrics, we demonstrate that the proposed architecture design improves the diversity of generated trajectories. Still, we estimate the "best-of-M " candidate's alignment with the ground truth under widely adopted APD/FDE metrics. Surprisingly, MGF also achieves state-of-the-art performance.To conclude, In this work, we focus on enhancing the diversity of trajectory prediction. We propose Mixed Gaussian Flow (MGF) by reforming the prior distribution of normalizing flows as a novel design of mixed Gaussians. It achieves state-of-the-art performance with respect to both the "bestof-M " alignment metrics and diversity metrics. We demonstrate that the proposed MGF model is capable of diverse and controllable trajectory predictions.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We focus on improving the diversity and the controllability of the probabilistic trajectory generation.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related works: 
 Generative Models for Trajectory Prediction. Trajectory prediction aims to predict the positions in a future horizon given historical position observations of multiple participants (agents). Early studies solve the problem by deterministic trajectory prediction [20] where Social forces [14] , RNNs [1, 35, 52] , and the Gaussian Process [53] are proposed to model the agent motion intentions. Recent works mostly seek multi-modal and probabilistic solutions for trajectory prediction instead, which is a more challenging but faithful setting. Though some of them leverage reinforcement learning [25, 4] , the mainstream uses generative models to solve the problem as generating likely future trajectories. Auto-encoder [18] and its variants, such as CVAE [23, 63] , are widely adopted. GANs make another line of work [13] . More recently, the diffusion [15] model is also used in this area [31] . However, they are typically not capable of estimating outcome probability as the generation process is not invertible. Normalizing flow [21] is preferred in many cases for being invertible.Normalizing Flow for Trajectory Prediction. In this work, we would like the predicted trajectories diverse and controllable. We prefer the generation process invertible to allow tractable generation likelihood. We thus choose normalizing flow [21] generative models. Normalizing flow [36] constructs complex distributions by transforming a probability density through invertible mappings from tractable distribution. Normalizing flow has been studied for trajectory prediction in some previous works [40, 41, 12] . In the commonly adopted evaluation protocol of "best-of-M " trajectory candidates, normalizing flow-based methods are once considered not capable of achieving state-ofthe-art performance. However, we will show in this paper that with proper design of architecture, normalizing flow can be state-of-the-art. And much more importantly, its invertibility allows more controllable and explainable trajectory prediction.Gaussian Mixture models as prior. Though the standard Gaussian is chosen by mainstream generative models as the original sampling distribution, some previous works explored how Gaussian mixture models (GMM) can be an alternative to help with generation or classification tasks. [8] uses a GMM prior in VAE models to enhance the clustering and classification performance. [2] adopts GMM to enhance the conditional generation of GAN networks. FlowGMM [17] uses GMM as the prior for flow-based models to deal with the classification task in a semi-supervised way. A recent work PluGen [54] proposes to mix two Gaussians to model the conditional presence of two binary labels to control generation tasks. Existing methods mostly use GMM to describe the presence of multiple auxiliary labels and they typically require additional annotations to construct the GMM. In this work, we use GMM as the distribution prior for normalizing flows without requiring any label annotations. It is designed to enhance the diversity of the generation and relieve the difficulty of learning transforming the tractable prior distribution to the desired complex and multi-modal target distribution for future trajectory generation.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limited by the page limit, we make the discussion about the limitations in the appendix instead of in the main paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Our proposed method is based on the normalizing flow paradigm for invertible trajectory generation while we construct a mixed Gaussian prior as the original distribution instead of the naive standard Gaussian to allow more diverse and controllable outcomes. In this section, we first provide the formal problem formulation in Section 3.1. Then we introduce normalizing flow in Section 3.2 and the proposed Mixed Gaussian Flow (MGF) model in Section 3.3. We detail the training/inference implementations in Section 3.4. At last, we introduce the proposed metrics set to measure the diversity of generated trajectories in Section 3.5. The overall illustration of MGF is shown in Figure 2 .We focus on 2D agent trajectory forecasting and represent the agent positions by 2D coordinates. Given a set of multiple agents, i.e., pedestrians in our case, we denote the 2D position of an agent a at time t as x a t and a trajectory from t i to t j (t i < t j ) as x a ti:tj . Given a fixed scene with map M and a period T: t 0 , t 1 , t 2 , ..., t c , ..., t T , there are N agents that have appeared during the period T, denoted as A t0:t T = {a 0 , a 1 , ..., a N -1 }. Without loss of generality, given a current time step t c ∈ (t 0 , t T ), the task of trajectory prediction aims to obtain a set of likely trajectories x a tc:t T with the past trajectories of all observed agents X At 0 :tc t0:tc = {x a t0:tc , a ∈ A t0:tc } as input, where a is an arbitrary agent that has shown up during t : t 0 -→ t c . For each agent a ∈ A t0:tc we seek to sample plausible and likely trajectories of it over the remaining time steps t c -→ t T by a generative model Φ, i.e., xa tc:t T = Φ(XEQUATIONat the same time, when there are other variables such as the observations of the maps are provided, we can use them as additional input information. By denoting the observations until t as O t0:tc we haveEQUATIONIf the generation process is probabilistic instead of deterministic, the outcome of the solution is a set of trajectories instead of a single one. The formulation thus turns toEQUATIONwhere i is the index of one candidate in the predicted batch.For some generative models relying on transforming from a sample point in a known distribution D 0 to the target distribution, e.g., GANs and normalizing flows, the set is generated by mapping from different sample points, i.e., p ∈ D 0 . Therefore, the full formulation becomesEQUATIONwhere P = {p 0 , ..., p K } is a set of sampled points from D 0 . Implicitly, the model Φ is required to construct a transformation (Jacobians) between the two distributions. Usually, D 0 is chosen as a symmetric and tractable distribution, such as a standard Gaussian. However, the distribution of the target distribution can be shaped by many data-biased asymmetries thus posing a challenge to learning the transformation effectively and inclusively. This often causes failure of generating under-represented trajectory patterns for trajectory forecasting and hurts the diversity of the outcomes. This observation motivates us to propose a probabilistic generative model for more diverse outcomes by representing the original distribution with more expressiveness.Normalizing flow [21] is a genre of generative model that constructs complex distributions by transforming a simple distribution through a series of invertible mappings. We choose normalizing flow over other probabilistic generative models as it can provide per-sample likelihood estimates thanks to being invertible. This property is critical to more comprehensively understand the distribution of different future trajectory patterns, especially when typically only sampling dozens of outcomes and considering the existence of long-tailed trajectory patterns. We denote a normalizing flow as a bijective mapping f which transforms a simple distribution p(z) to a complex distribution p(x). The transformation is often conditioned on context information c. With the change-of-variables formula, we can derive the transformation connecting two smooth distributions as follows:EQUATIONGiven the formulations, with a known distribution z ∼ D 0 , we can calculate the density of p(x) following the transformations and vice versa. However, the equations require the Jacobian determinant of the function f to obtain the distribution density p(x). The calculation of it in the high-dimensional space is not trivial. Recent works propose to use deep neural networks to approximate the Jacobians.To maintain the inevitability of the normalizing flows, some carefully designed layers are inserted into the deep models and the coupling layers [9] are one of the most widely adopted ones.More recently, FlowChain [28] is proposed to enhance the standard normalizing flow models by using a series of Conditional Continuously-indexed Flows (CIFs) [5] to estimate the density of outcomes. CIFs are obtained by replacing the single bijection f in normalizing flows with an indexed family F (•; u) u∈U , where U ⊆ R is the index set and each F (•; u) : z -→ x is a bijection. Then, the transformation is changed toEQUATIONPlease refer to [5] for more details about CIFs and their connection with variational inference. In this work, we follow the idea of using a stack of CIFs from [28] to achieve fast inference and the updates of trajectory density estimates.Normalizing flow based model samples from a standard Gaussian, z ∼ N (0, 1), usually results in overfitting to the most-likelihood for trajectory prediction. This is because each data sample from the training sample is considered extracted as the mode of a standard Gaussian. Only the mode value (the ground truth) is directly supervised and the underlying target distribution is assumed to be perfectly symmetric, which is not aligned with the usual real-world facts. Related discussion can be found in many previous literatures [43, 19] . This typically results in degraded expressiveness of the model to fail to capture under-represented motion patterns from the data and thus hurts the outcome diversity.We propose Mixed Gaussian Flow (MGF) to enhance the diversity and controllability in trajectory prediction. MGF consists of two stages as summarized in Figure 2 . First, we construct the mixed Gaussian prior by fitting the parametric model of a combination of K Gaussians,{N (µ k , σ 2 k )}, (1 ≤ k ≤ K).The parametric model is obtained with the data samples from training sets. Then, during inference, we sample points from the mixture of Gaussian and map them into a trajectory latent in the target distribution by a stack of CIF layers with the historical trajectories of all involved agents as the condition. We will introduce the two stages in detail below.MGF maps from a mixture of Gaussians instead of a single Gaussian to the target distribution. To maintain the inevitability of the model, the mixed Gaussian prior can not be arbitrary. We obtain the parametric construction of the mixed Gaussian by fitting it with training data. In this fashion, we can derive multiple Gaussians to represent different motion patterns in the dataset, such as going straight or turning left and right. In a simplified perspective, we regard the mixture as combining multiple clusters, each of which represents a certain sub-distribution. By sampling from the mixture of Gaussians instead of a standard Gaussian, our constructed model has more powerful expressiveness than the standard normalizing flow model. This results in more diverse trajectory predictions. Also, by manipulating the mixed Gaussian prior, we can achieve controllable trajectory prediction.Mixed Gaussian Prior Construction. For the data pre-processing, we transfer motion directions into relative directions with respect to a zero-degree direction. All position footage is represented in meters. Given the trajectory between t 0 -→ t c to predict the trajectory between t c -→ t T , we would put the position pivot at t c , i.e., x tc , as the origin and convert the position on all other time steps to be the offset from x tc . Then, we cluster the preprocessed future trajectories into K clusters, which is a hyper-parameter. We note the mean of the clusters as µ = {µ i } i=1,...,K .These cluster centers reveal the mean value of K representative patterns of pedestrians' motion, e.g. go straight, turn left. They will be the means of the Gaussians. The variances of the Gaussian, i.e., σ 2 k , can be pre-determined or learned. The final mixture of Gaussians is denoted asEQUATIONwhere β k are the weights assigned to each cluster following the k-means clustering of the training data. By default, we perform clustering by K-means with K = 8.Flow Prediction. Once the mixed Gaussian prior is built, we can do trajectory prediction by mapping samples from the distribution to future trajectories conditioned on historical information(e.g. social interaction features extracted by a Trajectron++ [46] encoder). Here, we ignore the intermediate transformation by CIFs as Equation ( 6) shows while following the original formulations of normalizing flows as Equation ( 5) for simplicity. We distribute the samples from different Gaussians by their weights. Given the i-th sample from N (µ k , σ 2 k ), we can transform it to the i-th predicted trajectoriesEQUATIONFor a sample ziEQUATIONk ), we have the probability estimateEQUATIONand the transformation is converted toEQUATIONwhich can be also invested back for the density estimate by the normalizing flow lawEQUATIONThe training loss of MGF comes from two directions: the forward process to get mixed flow loss and the inverse process to get minimum ℓ 2 loss.Forward process. Given a ground truth trajectory sample x a tc:t T , we need to assign it to a cluster in the mixed Gaussian prior by measuring its distance to the centroidsEQUATIONwith a tractable probability density function p k(•). Through the inverse process f -1 of flow model, we transform x a tc:t T into its corresponding latent representation, here denoted asEQUATION)Then we can compute the forward mixed flow loss: Instead of computing negative-loglikelihood(NLL) loss of ẑ in the mixed distributionEQUATIONEQUATIONk ), we compute NLL loss in the sub-Gaussian with the nearest centroid β kN (µ k, σ k2 ) because each centroid is independent to others in the mixed distribution and we encourage the model to learn specified motion patterns to avoid overwhelming by the major data patterns. Calculating NLL loss over the mixed distribution may fuse other centroids and damage the diversity of model outputs. By our design, the mixed Gaussian prior can maintain more capacity for expressing complicated multi-modal distribution than the traditional single Gaussian prior, which typically constrains the target distribution to be single-modal and symmetric.Inverse Process. This process repeats the flow prediction process to get generated trajectories. To predict M candidates, we sampleEQUATIONWe compute the minimum ℓ 2 loss between M predictions and ground truth trajectory as [13] does:EQUATIONWe sample z i from sub Gaussians by their weight. This is approximately equal to sampling from the original mixed Gaussians but makes the reparameterization trick doable.Although approximated differential backpropagation techniques, such as the Gumbel-Softmax trick, can be employed to make the sampling process of mixed Gaussians differentiable, computing the Negative Log-Likelihood (NLL) loss between a sample point and the mixed Gaussian distribution remains challenging becauseEQUATIONcontains exponential operations on matrices, which can be simplified through logarithmic operations in single Gaussian condition. Computing this term requires iterative optimization methods, such as the Expectation-Maximization algorithm [6] for approximation [60, 47] , which makes the computing process much more complex. Therefore, in practice, sampling from individual Gaussian components is preferred for computing efficiency. Furthermore, applying the Gumble-softmax to learn a mixture of Gaussians in generative models has been reported difficult in practice in some cases [39] due to gradient vanishing problem.The forward and inverse losses encourage the model to predict a well-aligned sample in a sub-space from the prior without hurting the flexibility and expressiveness of other sub-spaces. We combine the forward and inverse losses by a ratio γ to be a Symmetric Cross-Entropy loss [40] , which was proved beneficial for better balancing the "diversity" and "precision" of predicted trajectories:EQUATIONThe widely adopted average/final displacement error (ADE/FDE) scores measure the alignment (precision) between the ground truth future trajectory and one predicted trajectory. Under the common "best-of-M " evaluation protocol, ADE/FDE scores encourage nothing but finding a single "aligned" trajectory with the ground truth. ADE encourages the position on all time steps to be aligned with the single ground truth and FDE chooses the trajectory with the closest endpoint while all other trajectories are neglected in score calculating. Such an evaluation protocol overwhelmingly encourages the methods to fit the most likelihood from a certain distribution and all generated candidates race to be the most similar one as the distribution mean. Under the single-mode and symmetric assumption, this usually tends to fit into a Gaussian with a smaller variance. However, this tendency hurts the diversity of predicted trajectory hypotheses.To provide a tool for quantitative trajectory diversity evaluation, we formulate a set of metrics. Following the idea of average displacement error (ADE) and final displacement error (FDE), we measure the diversity of trajectories by their pairwise displacement along the whole generated trajectories and the final step. Follow Dlow [62] , we name that average pairwise displacement (APD) and final pairwise displacement (FPD). We note that the diversity metrics are measured in the complete set of generated trajectory candidates instead of between a single candidate and the ground truth. The formulation of APD and FPD are as belowEQUATIONwhere APD measures the average displacement along the whole predicted trajectories and FPD measures the displacement of trajectory endpoints. We would mainly follow the widely adopted ADE/FDE for benchmarking purposes while using APD/FPD as a secondary metric set to better understand the diversity of the generated future trajectories.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experiments: 
 In this section, we provide experiments to demonstrate the effectiveness of our method. We first introduce experiment setup in Section 4.1 and benchmark with related works to evaluate the trajectory prediction alignment and diversity in Section 4.2. Then, we showcase the diversity and controllability of MGF in Section 4.3 and Section 4.4. Finally, we ablate key implementation components in Section 4.5.Datasets. We evaluate on two major benchmarks, i.e., ETH/UCY [24, 38] and SDD [42] . ETH/UCY consists of five subsets. We follow the widely used Social-GAN [13] benchmark. SDD dataset consists of 20 scenes captured in bird's eye view. We follow the TrajNet [44] benchmark. We note that in the community of trajectory prediction, previous works have inconsistent evaluation protocol details and thus have made unfair comparisons. Please refer to the appendix in supplementary materials for details.Metrics. We use the widely used average displacement error (ADE) and final displacement error (FDE) to measure the alignment of the predicted trajectories and the ground truth. ADE is the average L2 distance between the ground truth and the predicted trajectory. FDE is the L2 distance between the ground truth endpoints and predictions. Most previous works choose the "Best-of-M " evaluation protocol and we follow it to choose M = 20 as default.Here, we note that, under different assumptions of distribution spreading and variance, the evaluation is ideally done with different values of M . However, most existing methods only provide results with M = 20 and many of them do not open-source the code of the models so we can not rebenchmark with other value choices of M . Besides the metrics for trajectory alignment, we also use the proposed metrics set APD and FPD to measure the diversity of the predicted trajectory candidates. Implementation Details. We enhance our model using a similar technique as "intension clustering" [58] and we name it "prediction clustering". The key difference is that we directly cluster the entire trajectory instead of the endpoints. To make a fair comparison, we followed the data processing from FlowChain [28] and Trajectron++ [46] . We also follow FlowChain's implementations of CIFs that each layer consists of a RealNVP [10] with a 3-layer MLP and 128 hidden units. We use a Trajectron++ [46] encoder to encode historical trajectories. All models were trained on a single NVIDIA V100 GPU for 100 epochs(approximately 4 to 8 hours).We benchmark MGF with a line of recent related works on ETH/UCY dataset in Table 1 . The results of Trajectron++ and MID are updated according to a reported implementation issue 1 . MGF achieves on-par state-of-the-art performance with Eqmotion [59] . Specifically, Our method achieves the best ADE and FDE in 3 out of 5 subsets and the best ADE and FDE score by averaging all splits. Here we note that we build MGF as a normalizing flow-based method as its invertibility is key property we desire, though normalizing flow is usually considered inferior regarding the alignment evaluation. Therefore, such a good performance on the alignment is surprising to us. To compare with other normalizing flow-based methods, our method significantly improves the performance compared to FlowChain, achieving 27.6% improvement by ADE and 34.6% improvement by FDE.On the SDD dataset, where the motion pattern is considered more diverse than UCY/ETH, the benchmark results are shown in Table 2 . Our method outperforms all baselines measured by ADE/FDE for trajectory alignment. Specifically, Our method reduces ADE from 8.56 to 7.74 compared to the current state-of-the-art method MemoNet, achieving 9.6% improvement. Our method also significantly improves the performance of FlowChain for 22.1% by ADE and 29.7% by FDE. According to the benchmarking on the two popular datasets, we demonstrate the state-of-the-art alignment (precision) of our proposed method. Here we note again that the alignment with the deterministic ground truth is not the highest priority when we design our method, we will discuss the main advantages of MGF, diversity, and controllability, in the next paragraphs. By leveraging the mixed Gaussian prior, our model can generate trajectories from the corresponding clusters, resulting in a more diverse set of trajectories than sampling from a Gaussian. This is intuitively due to less difficulty in learning the Jacobians for distribution transformation. We present examples in Figure 4 . Given a past trajectory, there is a single ground truth future trajectory possibility from the dataset. We select four samples with different ground truth intentions, i.e., going straight, U-turn, left-turn, and right-turn. By sampling noise from the clustered distributions, we could generate future trajectories with diverse intentions. From the visualizations, we could notice, of course, that we generate outcomes that are very similar to the ground truth with close intentions while we also generate outcomes that have very diverse intentions. The well-aligned single trajectory accounts for the high ADE and FDE score our method achieves. And the impressive diversity demonstrates the effectiveness of our design, especially considering they are well controlled by the clusters where they are sampled from.Quantitatively, we evaluate the generation diversity according to our proposed metrics on ETH/UCY dataset since most existing methods did not either make experiments on SDD or open-source training code/checkpoint on SDD. The results are presented in Table 3 . We can observe that MGF achieves the best or second-best APD and FPD score on all splits among sota methods. Besides, our method significantly improves the performance compared to FlowChain, achieving 103.7% improvement by APD and 140.6% improvement by FPD. The only method that can achieve close diversity with our method is Agentformer [63] , which designs sampling from a set of conditional VAE to improve the diversity. However, compared to MGF, Agentformer is more computation-intensive and shows significantly lower alignment according to ADE/FDE scores in Table 1 . Also, Agentformer is not fully invertible, which is considered a key property we desire for trajectory forecasting. The superior quantitative performance according to the alignment (precision) and diversity metrics suggests the effectiveness of our method by balancing these two adversarial features.We also find that APD/FPD metrics are not sensitive to M , which is a natural result, see appendix B.The generated sample from MGF is highly correlated with the original sample drawn from the mixed Gaussian prior. If the prior distribution is a standard Gaussian as in the canonical normalizing flow method, we can have almost no control over the generated sample. The only controllability is to sample near the mode to generate a sample similar to the learned most-likelihood outcome or far from the mode to make them more different. However, as we discussed, after sufficient training and supervision by the forward loss, the variance of the latent Gaussian distribution of the outcome is usually very small, which further hurts the controllability. However, as we chose a transparent mixed Gaussian prior for the sampling, we can control the generation flexibility. First, by adjusting sub-Gaussians in the mixture prior, we can manipulate the generation process statistically. Figure 5 shows that by editing cluster compositions, we can control the predictions of MGF with good interpretability. By editing the weights of sub-Gaussians, we can control the ratio of splatting into directions. By editing the directions of the cluster means, we can control the intentions of samples statistically.Besides cluster centers, we can also edit the variance of Gaussian to control the density of generated trajectories or combine a set of operations to get expected predictions. We provide more discussions and examples in the appendix in the supplement. We ablate some key components of our implementation for both ADE/FDE and APD/FPD metrics, see Table 4 and Table 5 . ( 1)Prediction clustering is a common post-processing method, which improves the ADE/FDE as expected. However, it hurts the diversity for nomalizing flow model with single Gaussian prior. This is reasonable as the single Gaussian prior tends to generate trajectories densely close to the most likelihood and prediction clustering can't cluster them into well-separated clusters for different motion intentions. (2)Mixed Gaussian prior help the model generates more diverse outputs and achieves higher APD/FPD scores and this improvement can be further enhanced by prediction clustering. It also increases ADE/FDE scores a lot, we believe this is because mixed Gaussian prior relieves the difficulty of learning the Jacobians for distribution transformation. Thus more under-explored patterns, which may be selected as the "best-of-M " samples in rare but plausible scenarios, have the chance to be expressed. (3)Learnable variance improve ADE/FDE while bring down APD/FPD a bit. We find that the learnable variance usually converges to a smaller value than the fixed situation. This is encouraged by the supervision from the ground truth (most likelihood) to a desired steeper Gaussian, thus hurting the diversity. However, its substantial improvement in ADE/FDE indicates that it remains a valuable component of the model architecture. ( 4)Inverse loss provides a straightforward supervision of the trajectory in the coordinate space, which is also proved beneficial for ADE/FDE and APD/FPD scores. 
conclusion: 
 We focus on improving the diversity while keeping the estimated probability tractable for trajectory forecasting in this work. We noticed the poor expressiveness of Gaussian distribution as the original sampling distribution for normalizing flow-based methods to generate complicated and clustered outcome patterns. We thus propose to construct a mixed Gaussian prior to help learn Jacobians for distribution transformation with less difficulty and higher flexibility. Based on this main innovation, we propose Mixed Gaussian Flow (MGF) model for the diverse and controllable trajectory generation. The cooperating strategy of constructing the prior distribution and training the model is also designed.According to the evaluation of popular benchmarks, we demonstrate that MGF achieves state-of-theart prediction alignment and diversity. It also has other good properties such as controllability and being invertible for probability estimates.misalignments, wishing future research aware of this to avoid potential continuity of the mis-practice and provide a fair comparison.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Yes, we already included the source code and the models in the supplementary materials for the reproduction. Also, we will open source the code and models.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
limitations: 
 Limited by computing resources, we did not utilize the map information in our model. Some generated trajectories may overlap with obstacles, thus decreasing the upper bound of MGF's ability. Also, we found that agents can occasionally collide with each other due to the limited ability of the history encoder. Future works may take more consideration to the collision among agents or between agents and the environment.NeurIPS Paper ChecklistQuestion: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide a detailed introduction about the training details. Limited the page length limit, we may skip some implementation details if we inherit from previous methods. All details are available in our provided source code in the supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: To make the consistent benchmarking practice, we do not report the variance of the evaluation results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: we have read the code of ethics and conform with it during the work of this paper.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: there is no societal impact of the work performed, we use the human trajectory data but no human face or other data that might involve private information is used. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Our code is partly based on FlowChain codebase, which is unlicensed on github. We test our mehthod on ETH/UCY [24, 38] and SDD dataset. ETH/UCY dataset are licensed under CC BY-SA 4.0, SDD [42] dataset are licensed under CC BY-NC-SA 3.0. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: When comparing baseline results, many previous studies fail to meticulously verify whether they adhere to the same convention, thereby leading to unfair comparisons. To compare the performance of various models fairly, we follow the practice that the community adheres to the most: Social-GAN's convention for ETH/UCY and TrajNet's convention for SDD.More specifically, for ETH/UCY dataset, we recommend to use the preprocessed data and dataloader from SocialGAN [13] /Trajectron++ [46] /AgentFormer [63] . For the SDD dataset, we recommend using the preprocessed data and dataloader from Y-Net [29] . Although their data processing methods may differ, they share the same data source and data splitting approach, facilitating fair comparisons.We also note that, in the early version of Trajectron++, a misuse of the np.gradient function during computation resulted in the model accessing future information. Rectifying this bug typically leads to a significant decrease in scores. Consequently, several Trajectron++-based studies have achieved improved scores.As we all know, the ADE/FDE employ a "best-of-M " computation approach, where M significantly influences the results: larger M values yield lower ADE/FDE scores. We wonder whether APD/FPD metrics demonstrate similar sensitivity to M . Since APD/FPD calculations consider all trajectories collectively, intuitively, their values would remain stable with varing M . Our experimental results confirmed this, see Table 6 . The APD/FPD metrics for FlowChain and MGF(w/o Pred. Clustering) remain stable as M increases, whereas MGF exhibits a slight decrease. This decrease can be attributed to MGF's prediction clustering mechanism, which initially generates fixed J samples (e.g., here J=500) and then clustering them into M output trajectories. As M increases, the impact of prediction clustering gradually diminishes (when M = J, prediction clustering is deprecated). Given that prediction clustering contributes to improving diversity in our ablation study, its weakening effect leads to a monotonic decrease in APD/FPD Besides the fashion of manipulating generation results given the good property of our constructed mixed Gaussian prior in the main paper, we also use data augmentation to alter the data patterns in our training set, thereby obtaining different priors. This enables our model to fix corner cases that are difficult to handle with traditional flow-based models like FlowChain. Taking Figure 6 as an example, to generate the clusters representing green "U-turn", purple "left-turn", and cyan "right-turn" clusters in the right-middle figure, we duplicate and rotate the original future trajectory data by 180°, 90°, and -90°, respectively. Subsequently, we mix these rotated data with the original data in a fixed ratio to produce the augmented dataset (in this particular case, a 2:2:1:1 ratio is employed for the original:180°:90°:-90°). Then we apply k-means to the augmented dataset, thereby obtaining the new augmented prior distribution (depicted in the right-middle figure). Finally, we train the model using this augmented dataset. Compared to the generated results from FlowChain, after using the augmented data to construct the mixed Gaussian prior, our method can generate the Figure 6 : By adding augmented data along with their corresponding clusters into the construction of the mixed Gaussian prior, we could manipulate the generation patterns as we desire. For example, we could inject some under-represented trajectory patterns. Then the model can generate corner cases that existing models fail to generate in a reasonable probability. We selected three examples from the UNIV dataset, namely sharp left/right turns and U-turns. The left column shows the predictions from FlowChain, while the middle column shows the predictions of MGF with augmented priors under-explored trajectory patterns with a higher chance. After this manipulation, we could change the mixed Gaussian prior as we desire, such as amplifying the chance of generating corner cases in this example.On the other hand, we quantitatively evaluate the ability to generate under-represented trajectory patterns. Table 7 compares the ADE/FDE scores of their worst-N samples on the UNIV dataset. Typically, the samples from the test set with the worst ADE/FDE relate to the under-represented corner cases of future trajectories. The results demonstrate quantitatively that MGF can better generate the under-represented motion patterns after injecting the desired corner cases on purpose by manipulating the mixed Gaussian prior as mentioned above. We note that all the provided examples of manipulating the mixed Gaussian prior to controlling the generation statistics do not require fine-tuning or any operation to the normalizing flow itself. As manipulating the mixed Gaussian prior is purely a parameter updating processing without any training and gradient backpropagation, all the manipulation is very fast in practice. This suggests the good efficiency and flexibility of our proposed method to achieve controllable generations.https://github.com/StanfordASL/Trajectron-plus-plus/issues/53 


Decision: This paper introduces a normalizing flow (NF)-based approach for diverse trajectory prediction. In contrast to standard Gaussian priors used in NFs, the authors propose using a data-driven Gaussian mixture model (GMM), built from the analysis of trajectory patterns in the training dataset. The resulting Mixed Gaussian Flow (MGF) model is trained with both forward and inverse processes. The authors claim to include new diversity metrics, and experiments were conducted on the ETH/UCY and SDD datasets.

The paper initially received mixed reviews, with one weak acceptance recommendation (6), one weak rejection recommendation (4), and one rejection recommendation (3). The main concerns raised by the reviewers related to the justification of the method, the diversity metrics used in the submission, and several concerns regarding the experiments (e.g., relevance of the task, results with HD maps, comparison to other state-of-the-art methods). There was extensive discussion during the rebuttal period. The authors successfully addressed both theoretical and experimental concerns from reviewer 1BoH and reviewer qRNj, who respectively raised their ratings to 7 and 6, both recommending acceptance. However, reviewer pmqe maintained their initial recommendation, but did not respond to the authors' feedback.

The AC has thoroughly reviewed the submission and the discussions. The AC considers the problem of probabilistic forecasting to be highly important and appreciates the authors' efforts to fairly evaluate different methods. The proposed GMM data-driven prior is a simple yet relevant idea in the context of trajectory forecasting, and the forward and inverse processes are sound. In my opinion, a clearer positioning with respect to determinantal point processes (DPPs) [1] for representing diversity would have been beneficial. Additionally, the AC notes that the proposed diversity metrics have previously been introduced in the literature and urges the authors to properly credit related work—other diversity metrics, such as rF [2], have also been used. Therefore, the AC recommends acceptance but strongly encourages the authors to include the elements discussed during the rebuttal in the final paper.

[1] Diverse Trajectory Forecasting with Determinantal Point Processes. Ye Yuan, Kris Kitani. ICLR 2020.\
[2] Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding. S. H. Park, G. Lee, J. Seo, M. Bhat, M. Kang, J. Francis, A. Jadhav, P. P. Liang, and L. P. Morency. ECCV 2020.

Reviews: review: paper_summary: paper_summary: The authors proposed a new normalizing flow-based human trajectory prediction method called Mixed Gaussian Flow (MGF), which promotes diversity and controllability of prediction. The model uses a mixture of Gaussian model as the initial distribution to transform, rather than single-modal standard Gaussian distribution. The paper proposed using clustering of training data to mine modes of movements and use these modes to fit the initial sets of Gaussians for the mixture. The authors also claimed they proposed new metric measure diversity (APD, FPD). The authors have shown MGF can exceed SOTA performance on ETH-UCY and SDD datasets during their quantitative analysis, and showcase by controlling the mixture of Gaussian, the types of trajectory to generate can be controlled.
strength_weakness: strength_weakness: - Line 112, 113 using $t_0, t_1, t_2$ might not be the best choice, the reviewer suggest maybe consider $t_0,t_1, \dots ,t_{c}, \dots, t_{T}$
- Suggest tidy up the variable representation of this section, for example using $X$ for observation sequence and $Y$ for future sequence
- Line 159 160, about normalizing flow overfitting, it would be great if the author could provide a reference to this claim or if this claim is by the authors themselves (also 129-130)
- Line 218-219, NLL loss is in sub-Gaussian, but how to decide the mixing coefficient (the reviewer is aware line 199 says it is decided from training data but specifically how?)
- Line 332, typo what is AED for APD or ADE?
- The ablation shows the ADE and FDE performance had a big increase with the prediction clustering, without prediction clustering MGF will have performance similar to AgentFormer and GroupNet and will not achieve SOTA, so does this means prediciton clustering/interntion clustering is decisive in accuracy performance of the model?
- The authors claimed they proposed APD metric, but it seems it is already in used in previous literatures under the same name and same formulation

@inproceedings{yuan2020dlow,
  title={Dlow: Diversifying latent flows for diverse human motion prediction},
  author={Yuan, Ye and Kitani, Kris},
  booktitle={Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part IX 16},
  pages={346--364},
  year={2020},
  organization={Springer}
}
questions: questions: - Abalation study was on ADE FDE only, but what is the effect on the diveristy? It would be great if the author could showcase this for the abalation, as diversity is the main contribution of the paper.
- If taking out mixture of gaussian and only use predictive clustering would the model achieve same level of ADE/FDE performance?
- Besides, this quesiton it would be great if the author could address the concerns in weakness section.
- The reviewer will consider increase the score if the authors could address and respond to the concerns mentioned above.
limitations: limitations: The reviewer appreciate the author mention in the supplementary about the limitation, where the model does not utilzed map information and predicitons will contain collision among agent. Although per guideline, the reviewer shall not punish this limitation, however, it is worth pointing out diversity without context might not be a sound logic. As solely promoting diversity might making the prediction output actually unrealistic considering the environmental and social context of the prediciton output. Although the author mentioned potential way to incorporating map information in the main paper, such proposal is more on the implicit side (as latent embeddings), which might not be able to address environmental collision issue.
paper_summary: paper_summary: Due to the asymmetric and multi-modal nature of future trajectories, the authors point out that the standard Gaussian prior with a neural network-based transformation is insufficient for probabilistic trajectory prediction. They propose Mixed Gaussian Flow (MGF), a method that uses the mixed Gaussian prior in the normalizing flow model. MGF allows diverse and controllable generation by leveraging the prior distribution. A minimum-of-M loss is designed for the inverse process to increase the predicting precision. Substantial experimental results on well-known datasets like ETH/UCY and SDD verify the effectiveness of MGF.
strength_weakness: strength_weakness: W1. This paper challenges the standard normal prior assumption in the normalizing flow model, but lacks in theoretical support. Also see Q6.
questions: questions: Q1. The prior distribution is obtained by clustering trajectories in the training set. Will it suffer from a distribution shift in the testing set? Specifically, in ETH/UCY dataset, the testing set contains scenes that do not appear in the training set. Can MGF deal with this issue? If not, can you figure out any possible solution?

Q2. The asymptotic behavior of ADE/FDE is well-studied [Ref.1]. Is it possible to theoretically or empirically analyze the asymptotic behavior of APD/FPD for a sufficiently large $M$? Is the value of APD/FPD sensitive to $M$?

[Ref.1] Analyzing the Variety Loss in the Context of Probabilistic Trajectory Prediction. In ICCV, 2019.

Q3. Some other papers have proposed metrics for measuring the diversity of trajectories, like the average maximum eigenvalue [Ref.2] and the energy score [Ref.3]. Can you discuss the connection and difference of APD/FPD with these metrics? Is it possible to evaluate the trajectory diversity of different methods using these metrics?

[Ref.2] Social-implicit: Rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation. In ECCV, 2022.

[Ref.3] Evaluation of Trajectory Distribution Predictions with Energy Score. In ICML, 2024.

Q4. In Section 4.4, how do you edit the directions of the cluster means?

Q5. Can you provide some empirical evidence for the claim of "limited and imbalanced data" in Line 30 of Section 1, e.g., the weights of each Gaussian component of the prior distribution?

Q6. Is it possible to find some theoretical support for the claim that "deriving a desired complex distribution from a simple and symmetric prior distribution is challenging, especially with limited and imbalanced data" in Line 29-30 of Section 1?

Q7. In Line 230-232, the authors say "Instead, it is hard to back-propagate the gradient through the sampling process of a Gaussian mixture with the reparametrization applied." I think the gumbel-softmax trick can be used for selecting components while allowing back-propagation. What do you mean by saying "it is hard to back-propagate"?

Comment.

C1. The authors clearly describe the subtle differences in evaluation in existing methods. It really saves my time and helps researchers in trajectory prediction.

C2. Appendix B contains some important facts. Please summarize these discoveries in the main text.

**Update:**

I have read the authors' response. They have addressed most of my concerns. I decide to raise my rating to 7. I hope that the authors can better organize the paper so that the presentation is clearer and it is easier for the readers to grasp the paper's significance.
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes to solve the trajectory prediction task by normalizing flow based model with mixture Gaussian assumption. The trajectories are clustered to multiple Gaussian distributions in the pre-processing stage over training data. During inference, priors are sampled from the mixture Gaussian distribution as the guidance of trajectory prediction. Moreover, new metrics are introduced to evaluate the diversity of multiple trajectory generation.
strength_weakness: strength_weakness: 1. The mixture Gaussian assumption is not accurate.

    Human trajectories are always affected by the interaction with other pedestrians and the environment. I agree that the possibility of trajectory conditional on i) historical trajectory ii) trajectory of other pedestrians iii) environment (e.g. maps) could be a mixture Gaussian distribution. However, all trajectories follow mixture Gaussian distribution without any conditions may not be an good assumption. Because the scenes in training and testing set are different. The learned distribution only fit on the training set with specific scenes, which cannot be generealized to different scenes.


2. The proposed diversity metrics is not reasonable.

    The diversity in multiple trajectory generation is necessary only on the premise that the generated trajectories make sense. The definition of APD and FPD only consider how different amont the generated trajectories but ignore whether the trajectories are reasonable or not. For example, 20 future trajectories are manually set to equally cover all directions from time step t_1, which will have very good score on APD and FPD but we cannot say these future trajectories are good predictions.


3. This paper ignores lots of recent works in SOTA comparison. E.g.
    Mangalam, Karttikeya, et al. "From goals, waypoints & paths to long term human trajectory forecasting." ICCV. 2021.

    Weibo Mao, et al.. Leapfrog diffusion model for stochastic trajectory prediction. CVPR, 2023.

    Jiangbei Yue, et al. Human trajectory prediction via neural social physics. ECCV, 2022

    Yonghao Dong,  et al. Sparse instance conditioned multimodal trajectory prediction. ICCV, 2023.

    ...

    Although authors mentioned that for some of the recent works, the processed data is slightly different. It would be more convincible to comapre to them with same training/testing data instead of ignoring these works as these works represent the most recent progress compared to original benchmarks such as socialGAN.
questions: questions: Please see the questions in weaknesses.

Besides, here are some suggestions.

1. The readability of notations and equations can be improved. For example, the paper does not include the scene map modeling, then map M (line 111) is useless. If O_{t0:t1} (line 119) only contains map information, then O can also be excluded from equations.

2. This task may not be the appropriate task for the proposed method.

    Although the task of trajectory prediction contains multiple trajectory output which usually involves generative models, this task is not a pure generation problem. In this task, the accuracy is still with highest priority. For example, a self-driving vehicle need to predict a pedestrian's (or another vehicle's) future trajectory to avoid collision, where only the correct trajectory is required. Since motion intentions of agents cannot be determined by their historical positions (as described in introduction), multiple predictions are necessary to cover the most possible futures instead of multiple predictions as diverse as possible (or controllable predictions). 

    I believe it would be more appropriate to evaluate the proposed method in the tasks such as motion simulation in synthetic data, where diversity and controllability motion trajectory is useful to make the agents look real.
limitations: limitations: Authors discussed the limitations in the appendix.

Comments: title: More discussion about the task setting
Follow-up meesage
Follow-up reply to Reviewer qRNj [2/2]
Follow-up reply to Reviewer qRNj [1/2]
Reply to Reviewer 1BoH
Reply to Reviewer 1BoH
comment: Dear Reviewer pmqe,

We understand your concern that our task setting can't be fit to certain user scenarios directly, such as auto-driving trajectory planning. We would like to elaborate more about it to clarify potential misunderstandings. 

Our problem setting is not to directly solve the auto-driving problem and we didn't claim it. We follow a long-standing computer vision problem setting for trajectory prediction/forecasting. It has been widely recognized as a main task for studying probabilistic generation diversity. A line of works published on top-tier venues, such as [1,2,3,4,5,6,7,8] (there are many more unlisted), follow this task setting. They all study on the same dataset as we use in this submission (ETH/UCY or SDD). They are proposed to study the balance between (1) more aligned/accurate max-likelihood prediction and (2) more diverse probabilistic generation. Both goals are expected to be achieved from a single model in a balanced fashion. Focusing on a single goal only causes defective solutions, either overfit deterministic methods (if only caring about accuracy/alignment) or meaningless random generation (if only caring about diversity). 

The related works mentioned by Reviewer pmqe also emphasize that accurate and diverse trajectory generation is a main contribution of their method. For example:
1. [6] claims its main contribution as `achieves precise and diverse predictions with fast inference speed`. 
2. [7] claims its contribution as ` exploits the proposed epistemic & aleatoric structure for diverse trajectory predictions across long prediction horizons`. 
3. [8] claims its main contributions including `a balanced solution, i.e., a novel sparse instance, ...to guide the multimodal trajectory prediction...for the multimodal trajectory
prediction`. 

If you have any concerns not addressed, please let use know. The authors will be pleased for follow-up discussions. 

Reference:
> [1] "AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting", ICCV'2021

> [2] "Remember Intentions: Retrospective-Memory-based Trajectory Prediction", CVPR'2022

> [3] "Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion", CVPR'2022

> [4] "Progressive Pretext Task Learning for Human Trajectory Prediction", ECCV'2024

> [5] "Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction", CVPR'2024

> [6] "Leapfrog Diffusion Model for Stochastic Trajectory Prediction", CVPR'2023

> [7] "From goals, waypoints & paths to long term human trajectory forecasting", ICCV'2021

> [8] "Sparse instance conditioned multimodal trajectory prediction", ICCV'2023
We appreciate Reviewer qRNj for the reply and raising the review score. We thank Reviewer qRNj for recognizing that the added ablation studies have addressed the concern. We will add them to the revised version of the paper. 

We agree that environmental information is important for trajectory forecasting. We will discuss it in the revised paper. 

Our proposed method does model the social constraint between agents though we didn't put the `social interaction` or similar terms explicitly in the paper. We would like to elaborate more about the social interaction/constraint referred to in the related works in this area for clarification. As stated in the `Implementation Details` in the paper, we follow the historical encoder as in Trajectron++[1]. Encoding and fusing historical trajectories of all or multiple agents is a typical way to model social interaction in this area. Some related works[1,2,3] explicitly claim social interaction modeling by using the same multi-agent historical trajectory encoder as ours. Moreover, there are some other genres of historical encoders used in this area to capture social interaction, such as LSTM encoder[4,5,6] and LSTM+GAT encoder[7]. Limited by the page limit and considering that social interaction is not a focus and contribution in this work (though MGF indeed models it), we didn't discuss it in the original paper. We will add a corresponding discussion about it in the revised paper and explicitly mention that our model handles social constraints between agents.

We will definitely acknowledge the previously proposed APD metric and improve the paper according to previous suggestions in the paper revision.


> [1] "Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data"

> [2] "Fast Inference and Update of Probabilistic Density Estimation on Trajectory Prediction"

> [3] "Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion"

> [4] "Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks"

> [5] "Remember Intentions: Retrospective-Memory-based Trajectory Prediction"

> [6] "Collaborative Motion Predication via Neural Motion Message Passing"

> [7] "STGAT: Modeling Spatial-Temporal Interactions for Human Trajectory Prediction"
We thank the author for the detailed response. the two new tables summarizing the ablation, especially the comparisons between first two rows are good demonstration of the  effectiveness of Mixture of Gaussian on the diversity aspect. Please add them to the final paper along with the analysis you provided in the second rebuttal. the reviewer believe that this concern is largely addressed.

The reviewer still believe it is important to consider environment and social constraint, especially when discussing diversity, (as most of the works benchmarked in the main paper consider one or both factors), however, the proposal for mixture of Gaussian prior and the demonstration of effectiveness on controllability and diversity of trajectory prediction is certainly an important contribution and the reviewer recognize the merit of the paper. However, how to combine social and environmental factor into the current method is still an important future research direction. Therefore, the reviewer will raise the score and would strongly recommend the author discuss such limitation of the current work in the final paper. Please also make sure to address the APD issue and consider other previous suggestions.
Dear Reviewer pmqe,

Please let us know if our responses have addressed your concerns or if you have any questions or concerns regarding our submission. We are glad to further discuss them. Thanks!
**Experiments on SDD:** In the original paper, we raised SDD in the paper to encourage further more intensive research on the challenging dataset. However, we had to put the dataset as secondary because of some practical difficulties:
1. SDD is a more challenging dataset than ETH/UCY with more complicated motion patterns and intentions. Also, SDD is a more recent benchmark. Many existing methods do not study SDD and just report results on ETH/UCY. 
2. Among the baselines we included in the benchmark on SDD, only PECNet[1] provides the checkpoint to reproduce their reported results. 
3. Otherwise, only MID[2] and MemoNet[3] open-source their training code (no pre-trained checkpoint is provided). Moreover, we unfortunately could not reproduce MID's result as reported in its paper by the released code. We could reproduce MemoNet's results on SDD close to its reported result. 

We encourage open-sourced research and transparent experiment settings on all datasets in this area. We would expect a more established benchmark on SDD with baseline methods. We wish to provide the diversity comparison for methods on SDD in the future version.

> [1] "It is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction"

> [2] "Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion"

> [3] "Remember Intentions: Retrospective-Memory-based Trajectory Prediction"

**Problem setups:** The task of trajectory prediction/forecasting has been studied for a long time in the community of computer vision. We followed the common problem setting of this long-standing task. According to the additional information on map/surroundings and social interaction:
1. Most works in this area do not take the map information into consideration. Moreover, adding extra map/environment information makes the task a totally different task, asking for a different line of benchmarking. Under such problem setups, more noise is included, especially the map information encoder, which is designed in different fashions in related works. This will confuse and blur our main focus in this work when seeking transparent and fair experimental evidence. 
2. In our design, our method can actually capture the social relations of different pedestrians (by feeding the historical trajectory of all agents into History Encoder). Also, we did compare the methods modeling social interaction explicitly. Our method's performance is much better than many works that model social interaction explicitly, such as Social-GAN.

**more qualitative results:** We did provide the visualization of prediction results in Figure 4 (in the paper) and Figure 6 (in the supplementary). Figure 6 is a good example to help understand the behavior and diversity difference between naive Gaussian prior and mixed Gaussian prior. We also provide illustration videos in the supplementary to understand the behaviors of different sub-Gaussians in our built model. We understand that Reviewer qRNj desires more quantitative visualization to assist in understanding the ablation study results. However, the qualitative difference is not significant for most pairs of comparison in the ablation study per our check. Moreover, we can't provide images or videos here per the rebuttal policy. We will consider the suggestion for the next version. Thanks.
We appreciate the detailed feedback about our comments from Reviewer qRNj. To address Reviewer qRNj's concerns, we provide follow-up replies here.

**Mixing coefficient**: The weight of a cluster is defined as the (number of trajectories belonging to that cluster)/(number of all trajectories).

**APD originality issue:** Yes, though the two tasks are different, they are related. We promise that we will carefully review the related metrics and give full credit to the earlier proposal of the metric and the naming of APD.

**Significance of prediction clustering and mixed Gaussian prior**: 
First of all, we claimed the main contribution of this work as the proposal of mixed Gaussian prior to normalizing flow to enhance the generation diversity and not to hurt or even boost the alignment/accuracy. 

Second, to enhance the generation diversity, prediction clustering itself makes no significant contribution as observed in the ablation study. The mixed Gaussian prior is necessary for our method to achieve outstanding generation diversity (adding extra prediction clustering further boosts the diversity). Therefore, we are afraid that the reviewer's argument `"although there is sign the mixture of Gaussian as prior show certain improvement of accuracy and diversity under certain circumstances, such improvement is less significant comparing with the improvement brought by the prediction clustering."` is not accurate. 

In the previous ablations, the inverse loss is used together with mixed Gaussian priors and prediction clustering which can make potential noise to the comparison. We originally put prediction clustering as an optional component in the ablation study to provide more transparency though it is actually used in many existing works as a default module for post-processing.

Now, we provide more ablation settings as in Table 5 and Table 6 below. We could have conclusions:
1. Prediction clustering can't independently boost the generation diversity significantly.
2. Using the Mixed Gaussian prior can improve both the diversity and the accuracy. Its improvement of diversity is much more significant than prediction clustering. Its improvement of accuracy/alignment is comparable to prediction clustering.
3. Mixed Gaussian prior's improvements to diversity and accuracy/alignment can be stacked with prediction clustering.
4. Learnable variance improves the accuracy but slightly hurts the diversity as we discussed in previous replies.

*Table 5: More ablations about the generation diversity (APD/FPD, the higher the better)*
| Inv. Loss | Mixed Gauss. | Learnable Var. | Pred. Clustering | ETH/UCY   | SDD         |
|----------:|------------:|---------------:|-----------------:|-----------|-------------|
|           |             |                |                  | 0.39/0.76 | 14.82/27.22 |
|           |           √ |                |                  | 0.78/1.70 | 23.18/44.90 |
|           |             |                |                √ | 0.41/0.80 | 15.52/28.50 |
|           |           √ |                |                √ | 1.09/2.33 | 32.42/65.43 |
|           |           √ |              √ |                √ | 0.96/2.12 |  30.1/60.20 |

*Table 6: More ablations about the prediction accuracy/alignment (ADE/FDE, the lower the better)*
| Inv. Loss | Mixed Gauss. | Learnable Var. | Pred. Clustering | ETH/UCY   | SDD         |
|----------:|------------:|---------------:|-----------------:|-----------|-------------|
|           |             |                |                  | 0.33/0.61 |  11.9/21.33 |
|           |           √ |                |                  | 0.29/0.48 |  11.38/19.28 |
|           |             |                |                √ | 0.29/0.54 | 10.63/18.80 |
|           |           √ |                |                √ | 0.27/0.48 |  9.19/15.86 |
|           |           √ |              √ |                √ | 0.23/0.39 |  8.71/14.86 |

*Note: here we retrained the model to remove the inverse loss and make sure the same randomness for all entries in the tables.*

The research of many computer vision tasks is always advanced by progressive improvement by continuous innovations. We believe as long as a method is effective in boosting performance and can be stacked with previous arts, it makes its own value to advance this area. Moreover, considering that our focus is to enhance the diversity of generation and prediction clustering can not actually boost the diversity, we believe our proposed mixed Gaussian priors have proven their unique value per the ablation studies.

We note that inverse loss and prediction clustering are both techniques widely used in existing works in the area. We did not claim them as our contribution. To conclude, our innovation of Mixed Gaussian prior has been proven effective in improving both the diversity and the accuracy/alignment of the generation. The improvement can be stacked to the effect of existing tricks of inverse loss and prediction clustering.
We appreciate Reviewer 1BoH for raising the Gumble-softmax trick to us which can be an interesting add-on component to compare with in the paper. We thank Reviewer 1BoH for increasing the review score. 

We just got some preliminary experimental observations that we would need careful parameter study to avoid the gradient vanishing. We promise to conduct a discussion and empirical study about it in the revised paper.
The reviewer acknowledges the contribution of mixture of Gaussians to boost the controllability of trajectory generation; its effectiveness of mining out different motion patterns out of the training data. The paper is well written and many intuitions and formulations are clearly explained. However, considering improving diversity as one of the main contribution of the paper, the experiments are showing mix signals and the diversity metric is only evaluated on one dataset for extensive benchmark against other SOTA models. The diversity metric is also not as original as the authors claimed. Plus, current submission lacks qualitative analysis showcasing the behavior of the different model component and model behavior under different type of inputs (for example, observation with slow speed vs. fast speed). Furthermore, the improvement of diversity alone might not be meaningful without considering the environmental and social factors in the prediction model, as social interaction and environmental constraint also dictates the behavior of human movement. Solely promote diversity without considering these constraints might not be a sound argument. Hence, the reviewer will keep the current score in terms of assessment.
The reviewer thanks the author for the response and the follow-up ablation studies. The author is able to address some of the original concerns and offers more insights on some of the questions.

**Writing Issue**: The reviewer thanks the author for considering updating the manuscript and clarifying the typo.

**Normalizing Flow Overfitting**: The reviewer thanks the authors' explanation and provide reference. 

**Mixing coefficient**: Thank you for the clarification, however, could you elaborate a bit more on the weight and how is it defined in the k-mean clustering? Is it based the number instance belongs to that cluster? Or other definition? Because general k-mean clustering does not usually involve weights.

**APD originality issue**: The reviewer appreciates the author acknowledging this metric has been published before, however, these two tasks, although different in nature, are quite similar technically in their settings of the output. Please do make sure to revise the paper and give credit to the original literature, despite original work was on human motion prediction, motion trajectory and human trajectory could be both evaluated by APD metric, as the authors also chose to do so here, therefore it is not reasonable for the author to claim that metric as novel contribution here without mentioning the original work that it was first proposed.

**Prediction clustering + Ablation:**
- Indeed, using mixture of gaussian boosts ADE/FDE, compared to using stardard gaussian. Yet, the boost to ADE/FDE value with the addition of prediction clustering is significantly larger, the reviewer acknowledge ADE/FDE only reflect the best-of-m mode of the prediction, and does not reflect diversity or other quality of all model output. However, compare with switching to mixture gaussian as prior, ADE/FDE improvement by adding intention/prediction clustering is much more significant.
- The reviewer really appreciates the author providing new abalation study on the effect of model component on diversity.  Indeed, the prediction clustering offers significant boost to APD/FPD to the model with mixture of gaussion prior (row 4 and row 5 in table 1) and less or no boost to even negative effect on APD/FPD (row1/row6 or row2/row7). However, if comparing row2 and row3, where the different is only between standard gaussian and mixture of gaussian as prior, the improvement on APD/FPD is less significant, especially on SDD dataset. Furthermore, in the main paper, the authors only offer APD/FPD benchmark against other model on ETH-UCY not SDD.
- The learnable var. indeed slightly hurts diversity, however the difference is less significant compare with the point above and the reviewer thanks the author for the analysis.
- Overall, although there is sign the mixture of gaussian as prior shows certain improvement of accuracy and diversity under specific circumstances, such improvement is less significant compared with the improvement brought by the prediction clustering. As a modeling choice, prediction clustering as a post-processing step is reasonable, however, since the author’s claim on mixture of gaussian improving sampling diversity is one of the main foci of the paper, more experiments to support the diversity claim on mixture of Gaussian are needed. One of such experiments would be to benchmark against other SOTA model on the SDD dataset, since the ablation on SDD especially shows mix signal and certain SOTA model such as AgentFormer is quite close in the diversity metric on the provided ETH-UCY diversity benchmark. The other suggested experiment would be showing more qualitative analysis to showcase through visualization of prediction outputs the effect of mixture Gaussian prior and prediction clustering. The reviewer understand such addition would be a significant amount of work, and might not be able to be added to the current submission.
Thanks for the recommending the blog and clarifying the details in metric evaluation. You can consider reporting both the results of Table 3 and Table 4 in th revised paper. I would consider rasing the rating to 7.
**AMD & AMV:** Yes, by carefully check the [code implementation](https://github.com/abduallahmohamed/Social-Implicit/blob/main/amd_amv_kde_metrics.py) of AMD & AMV, we found the 2 metrics can be solely calculated based on sampled trajectories. The GMM can be fit by the internal computing process without confidence estimation. We will carefully review the recently proposed related metrics and add the them in the revised version of paper.

**Gumble-softmax trick:** Really a good suggestion! Gumbel-softmax trick provides a solution[1,2,3] with approximation to derive a differentialble gradient when sampling from categorical distributions (in our case, the mixture of multiple sub-Gaussians). But applying the Gumble-softmax to learn a mixture of Gaussians in generative models has been reported difficult in practice in many cases[2,4] due to graident vanish. Still, adding Gumble-Softmax as a re-parameterization trick for graident BP would be an interesting component in the revised paper though that makes no impact to the main contribution of this work. We will leave it for future works.

> [1] "Categorical Reparameterization with Gumbel-Softmax"
> [2] "Invertible Gaussian Reparameterization: Revisiting
the Gumbel-Softmax"
> [3] "Gradient Estimation with Stochastic Softmax Tricks"
> [4] https://gfchen01.cc/post/gmm_vae/

**concern about APD/FPD:** Very good insight! Intuitively, the APD/FPD are expected to be stable for a sufficiently large M, as what FlowChain performs in the added ablation study (Table 3). 
To add more detailed observations, the decrease of APD/FPD by MGF when M increases can be attributed to the use of prediction clustering in enhancing diversity.
In Table 3 in the general response pdf, predictions for M=10, 20, ..., 80 are obtained by first sampling 500 trajectories and then clustering them into M clusters. As M increases, the impact of prediction clustering gradually diminishes (when M=500, prediction clustering is deprecated). Given that prediction clustering contributes to improving diversity, its weakening effect leads to a monotonic decrease in APD/FPD until prediction clustering becomes entirely ineffective at M=500.
By removing the prediction clustering, we provide new results as shown in Table 4 below:

*Table4: APD/FPD of MGF(w/o Prediction Clustering)*
| M   | ETH/UCY   | SDD         |
|-----|-----------|-------------|
| 10  | 0.75/1.66 | 24.11/47.60 |
| 20  | 0.81/1.80 | 24.84/48.78 |
| 30  | 0.83/1.83 | 26.56/52.26 |
| 40  | 0.84/1.87 | 26.33/51.72 |
| 50  | 0.84/1.86 | 26.16/51.43 |
| 60  | 0.84/1.87 | 26.16/51.33 |
| 70  | 0.84/1.86 | 26.20/51.53 |
| 80  | 0.84/1.87 | 26.48/52.20 |
| 100 | 0.85/1.88 | 26.51/52.07 |
| 200 | 0.85/1.89 | 26.74/52.49 |
| 300 | 0.85/1.89 | 26.65/52.36 |
| 400 | 0.85/1.89 | 26.72/52.47 |
| 500 | 0.85/1.89 | 26.72/52.50 |

Without prediction clustering, APD/FPD of MGF remains stable as M increases, converging to approximately 0.85/1.89 on ETH/UCY and 26.7/52.5 on SDD.  
Now, we observe that the when M increases, APD/FPD scores remain much more stable. This observation is well aligned with the intuition provided by you.
Thanks for your response. The literature on normalizing flow is also helpful for me. You may consider adding descriptions of editing the cluster directions and the empirical evidence of data imbalance to the revised paper. For distribution shift, according to your description, I tend to think that clustering based on relative positions other than absolute positions will help learn transferable moving patterns. I would like to point out that:

1. From my understanding, AMD and AMV are solely based on sampled trajectories. Confidence outputs from probabilistic methods are generally not a prerequisite. EI and ED are recently proposed metrics. You do not have to compare your metrics with them according to the submission guidelines of NeurIPS. Still, I believe properly evaluating the performance of all methods on more metrics can help better understand their differences, which can be added to the revision or left for future work.

2. As I understand, selecting a component from a Gaussian mixture model is a process of sampling from a categorical distribution, where the gumbel-softmax trick can play a role in allowing back-propagation. I am not sure if it will be practical and effective and can be left for future work.

Still, I have one minor concern.

**The influence of the value of M to APD/FPD:** The results in Table 3 of the attached pdf show that APD/FPD is stable for FlowChain, yet a noticeable decrease can be observed for MGF on both ETH/UCY and SDD datasets as $M$ increases. Since APD/FDP are computed by taking the average over sample distances, it is expected that they will be stable for a sufficiently large $M$ (I may be wrong). This expectation and the results for FlowChain do not match the results for MGF even for $M=80$. Can you provide more explanation?