MGF: Mixed Gaussian Flow for Diverse Trajectory Prediction
Authors: Jiahe Chen
Jinkun Cao
Dahua Lin
Kris M. Kitani
Jiangmiao Pang
Venue: NeurIPS 2024 poster (2024)
Keywords: trajectory prediction
trajectory forecasting
Primary area: machine_vision

Abstract: To predict future trajectories, the normalizing flow with a standard Gaussian prior suffers from weak diversity. 
The ineffectiveness comes from the conflict between the fact of asymmetric and multi-modal distribution of likely outcomes and symmetric and single-modal original distribution and supervision losses.
Instead, we propose constructing a mixed Gaussian prior for a normalizing flow model for trajectory prediction.
The prior is constructed by analyzing the trajectory patterns in the training samples without requiring extra annotations while showing better expressiveness and being multi-modal and asymmetric.
Besides diversity, it also provides better controllability for probabilistic trajectory generation.
We name our method Mixed Gaussian Flow (MGF). It achieves state-of-the-art performance in the evaluation of both trajectory alignment and diversity on the popular UCY/ETH and SDD datasets. Code is available at https://github.com/mulplue/MGF.

MGF: Mixed Gaussian Flow for Diverse Trajectory Prediction
Jiahe Chen 1.2+ Jinkun Cao 3x1
Dahua Lin 1,2,5
Kris Kitani Jiangmiao Pang31
1 Zhejiang University
Shanghai AI Laboratory
Carnegie Mellon University
"The Chinese University of Hong Kong
CPII under InnoHK
*: co-first authors
t: co-corresponding authors
Abstract
To predict future trajectories, the normalizing flow with standard Gaussian prior suffers from weak diversity. The ineffectiveness comes from the conflict between the fact asymmetric and multi-modal distribution of likely outcomes and symmetric and single-mone modal original distribution and supervision losses. Instead, we propose constructing a mixed Gaussian prior for a normalizing flow model for trajectory prediction. The prior is constructed by analyzing the trajectory patterns in the training samples without requiring extra annotations while showing better expressiveness and being multi-modal and asymmetric. Besides diversity, it also provides better controllability for probabilistic trajectory generation. We name our method Mixed Gaussian Flow (MGF). It achieves state-of-the- of-tth--rttttttt-- performance in the evaluation of both trajectory alignment and diversity on the popular UCY/ETH and SDD datasets. Code is available at
1 Introduction
In this work, we aim to improve the diversity for probabilistic trajectory prediction. trajectory prediction, diversity describes the fact that agents (pedestrians) can move in different directions, speeds, and interact with other agents. Because the motion intentions agents can not be determined by their historical positions, there is typically no global-optimal strategy to predict single outcome of future trajectories. Therefore, recent works have focused on probabilistic methods to generate multiple likely outcomes. However, existing solutions are argued to lack good diversity and they often fail to generate the under-represented future trajectory patterns the training data.
Different motion patterns are usually imbalanced in a dataset. For example, agents are more likely to move straight than turn around most datasets. Thus, many motion patterns are highly under- represented though discoverable. Therefore, intuitively, an ideal distribution to represent the possible future trajectories should be asymmetric, multi-modal, and expressive to represent long-tailed patterns.
However, most existing generative models solve the problem of trajectory prediction by modeling it as single-modal and symmetric distribution, i.e., standard Gaussian. This because the standard Gaussian is tractable and there is belief that it can be transformed into any desired distribution of the same or lower dimension. However, deriving desired complex distribution from a simple and symmetric prior distribution is challenging. especially with limited and imbalanced data. Moreover, when we derive the target distribution by transforming from the tractable original distribution as Normalizing Flows, GANs, and VAFs do, dilemma arises: over-smoothing transformation model can neglect under-represented samples while an over-decorated transformation model will overfit. Especially for normalizing flow, some studies 22 3 discussed the difficulty of training normalizing flow in practice represent a complex target distribution.
To solve this dilemma, we propose a prior distribution with more expressiveness and data-driven statisties. It is asymmetric, multi-modal, and adaptive the training data in the form of mixed set of Gaussians. Compared to the standard Gaussian, the mixture of Gaussians can better summarize the redresrepreeented samples scattered far away in the representation space. This relieves the sparsity issue of rare cases and thus enhances the diversity of the generated outcomes. Besides diversity, as
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Generate
Mixed
(o) Non-invertible Generative Model
Generate
Right Straight
Left
Inverse
Generate
10%
20%
206
30%
(c) Mixed Gaussian Flow(Ours)
Inverse
(b) Flow- based Model
Figure 1: Non-invertible generative models (a), c-g., CVAE, GAN, and diffusions, lack the invertibility for probability density estimation. Flow-hased methods (h) are invertible while, sampling from the symmetric standard Gaussian, undermines the diversity and controllability of generation. Our proposed Mixed Gaussian flow (c) maps from mixed Gaussian prior instead. Summarizing distributions from data and controllable edits, achieves better diversity and controllability for trajectory prediction.
the mixed Gaussian prior is parametric and transparent during construction, we could control the generation by manipulating this prior, such as adjusting the weights of different sub Gaussians or manipulating the mean value of them. All these manipulations change generation results statistically without requiring fine-tuning or other re-training. Upon the prior distribution, we choose to construct the generative model by normalizing flow with the unique advantage of being invertible. We thus could estimate the likelihood of each single generated outcome. By combining the designs, we propose normalizing flow-based trajectory prediction model named Mixed Gaussian Flow (MGF). enjoys better diversity, controllability, interruptibility, and invertibility for trajectory prediction. During our study, we find that though several evaluation tools have been proposed for measuring diversity [49 62 33 37,, they employ varying calculation method and have not gained widespread adoption within the research community. The most popular evaluation metrics (APD/FDE scores) focus on how similar a generated trajectory is to the single ground truth. It is calculated a "best-of- M" fashion where only one candidate in batch of M predictions is taken into the measurement. This protocol encourages the methods to generate outcomes approaching the mean (most likelihood) of the learned distribution and provides sense of the diversity of generation outcomes. Therefore, building upon previous research in the field of human motion prediction (22) we formulate a metric set of Average Pairwise Displacement (APD) and Final Pairwise Displacement (FPD), which measure the diversity of batch of M generated samples. This helps us to have a concrete study about generation diversity and avoid bias from the "besttof-M"" evaluation protocol. With the proposed metrics, we demonstrate that the proposed architecture design improves the diversity of generated trajectories. Still, we estimate the "best-of-M"" candidate's 's alignment with the ground truth under widely adopted APD/FDE metrics. Surprisingly, MGF also achieves state-of-the-ar performance.
To conclude, In this work, we focus enhancing the diversity of trajectory prediction. We propose Mixed Gaussian Flow (MGF) by reforming the prior distribution of normalizing flows as a novel design of mixed Gaussians. It achieves state-of-the-art performance with respect to both the "best- of-M" alignment metrics and diversity metrics. We demonstrate that the proposed MGF model is capable of diverse and controllable trajectory predictions.
2 Related Works
Generative Models for Trajectory Prediction. Trajectory prediction aims to predict the positions in future horizon given historical position observations of multiple participants (agents). Early studies solve the problem by deterministic trajectory prediction 220 where Social forces [14], RNNs 0 355 (5], and the Gaussian Process (3 are proposed to model the agent motion intentions. Recent works mostly seek multi-modal and probabilistic solutions for trajectory prediction instead, which more challenging but faithful setting. Though some of them leverage reinforcement learning (25 4 the mainstream uses generative models to solve the problem as generating likely future trajectories. Auto-encoder [18] and its variants, such as CVAE [23 63 are widely adopted. GANs make another line of work [13 More recently. the diffusion [15 model is also used in this
area 31. However, they are typically not capable of estimating outcome probability as the generation process is not invertible. Normalizing flow (21] is preferred in many cases for being invertible.
Normalizing Flow for Trajectory Prediction. this work, we would like the predicted trajectories diverse and controllable. We prefer the generation process invertible to allow tractable generation likelihood. We thus choose normalizing flow 21 generative models. Normalizing flow (36) constructs complex distributions by transforming probability density through invertible mappings

---

from tractable distribution. Normalizing flow has been studied for trajectory prediction in some previous works 40 [2]. In the commonly adopted evaluation protocol of "best-of-AV of- trajectory candidates, normalizing flow-based methods are once considered not capable of achieving state-of- the-art performance. However, we will show this paper that with proper design of architecture, normalizing flow can be state-of-the-art. And much more importantly, its invertibility allows more controllable and explainable trajectory prediction.
Gaussian Mixture models as prior. Though the standard Gaussian is chosen by mainstream generative models as the original sampling distribution, some previous works explored how Gaussian mixture models (GMM) can be an alternative to help with generation or classification tasks. g uses
GMM prior in VAE models to enhance the clustering and classification performance. [2] adopts
GMM to enhance the conditional generation of GAN networks. Flow FlowGMM [17] uses GMM as the prior for flow-based models to deal with the classification task in semi-supervised way. recent work PluGen
54 proposes to mix two Gaussians to model the conditional presence of two binary labels to control generation tasks. Existing methods mostly use GMM to describe the presence of multiple auxiliary labels and they typically require additional annotations to construct the GMM. In this work, we use GMM as the distribution prior for normalizing flows without requiring any label annotations. It is designed to enhance the diversity of the generation and relieve the difficulty of learning transforming the tractable prior distribution to the desired complex and multi-moda target distribution for future trajectory generation.
3 Method
Our proposed method is based on the normalizing flow paradigm for invertible trajectory generation while we construct mixed Gaussian prior as the original distribution instead of the naive standard Gaussian to allow more diverse and controllable outcomes. In this section, we first provide the formal problem formulation in Section 3.1 Then we introduce normalizing flow in Section 3.2 and the proposed Mixed Gaussian Flow (MGF) model in Section [..3 We detail the training/inference implementations in Section At last, we introduce the proposed metries to measure the diversity
of generated trajectories in Section 3.5 The overall illustration of MGF is shown in Figure
3.1 Problem Formulation
We focus on 2D agent trajectory forecasting and represent the agent positions by 2D coordinates. Given a set of multiple agents, i.e., pedestrians in our case, we denote the 2D position of an agent time / as X and trajectory from 4 15(44 < 12) as i Given fixed scene with map M and a period T: to, 11, 12. l...,,,,.. -.. br, there are N agents that have appeared during the period T, denoted as Aott = {oo, a1...., -N- Without loss of generality, given a current time step te E (fo, tr), the task of trajectory prediction aims to obtain set of likely trajectories x2,tr with
Aigie (xg,;,, a E Atacta) as input, where is an the past trajectories of all observed agents tytt, arbitrary agent that has shown up during t to - te. For each agent a E Atott, we seek to sample plausible and likely trajectories of it over the remaining time steps te - tr by generative model Φ i.e.
Ga + X
Atg:le fotte
(1)
at the same time, when there are other variables such as the observations of the maps are provided, we can use them as additional input information. By denoting the observations until t as O tgpte we have
O Aigite, O
(x
(2)
If the generation process is probabilistic instead of deterministic, the outcome of the solution is set of trajectories instead of a single one. The formulation thus turns to
( 2 D(xx Atgite Orgita)
where i is the index of one candidate in the predicted batch.
For some generative models relying on transforming from sample point a known distribution Do to the target distribution, e.g., GANs and normalizing flows, the set is generated by mapping from different sample points, i.c., p E Do. Therefore, the full formulation becomes
(1)g = (
Atgite, Orgiter P);
where P = {ppo..., ..., } is set of sampled points from Do.
3

---

Clustering
History Encoder
Motion Clusters
Preprocess
History Observation
Flow Model
Misod
Sample
Gff Layer 2
OF Layee
GFF Layer
Straight
Training Set
Mixed Gaussian Prior
Future Prediction
Mixed Gaussian Prior Construction
Flow Prediction
The illustration our proposed Mixed Gaussian Flow (MGF). During training, we construct a mixed Gaussian prior by statistics from the training set. During sampling, the initial noise samples are from the constructed mixed Gaussian prior. MGF keeps tractable prior distribution and invertible inference process while the novel mixed Gaussian prior provides more diversity and controllability the generation outcomes.
Figure 2:
Implicitly, the model is required to construct a transformation (Jacobians) between the two distributions. Usually, Do is chosen as symmetric and tractable distribution, such standard Gaussian. However, the distribution of the target distribution can be shaped by many data-biased asymmetries thus posing challenge learning the transformation effectively and inclusively. This often causes failure of generating under-represented trajectory patterns for trajectory forecasting and hurts the diversity of the outcomes. This observation motivates to propose probabilistic generative model for more diverse outcomes by representing the original distribution with more expressiveness.
3.2 Normaling Flow
Normalizing flow (1 is genre of generative model that constructs complex distributions by trans- forming simple distribution through a series of invertible mappings. We choose normalizing flow over other probabilistic generative models as it can provide per-sample likelihood estimates thanks to being invertible. This property is critical to more comprehensively understand the distribution of different future trajectory patterns, especially when typically only sampling dozens of outcomes and considering the existence of long-tailed trajectory patterns. We denote normalizing flow as bijective mapping f which transforms a simple distribution p(z) to a complex distribution p(x). The transformation is often conditioned context information With the change-off soangg--o-vriiaan variables formula, we can derive the transformation connecting two smooth distributions as follows:
(z;
p(x) = p(z) del (dee c))l:
(5)
log(p(x)) = - log(p(z)) - log(1 det (xt(Vxx.=(xxx)))]
Given the formulations, with known distribution z Do. we can calculate the density of p(x) following the transformations and vice versa. However, the equations require the Jacobian determinant of the function / to obtain the distribution density p(x). The calculation of it in the high-dimensional space is not trivial. Recent works propose To maintain the inevitability of the normalizing flows, some carefully designed layers are inserted into the deep models and the coupling layers are one of the most widely adopted ones.
deep neural networks approximate the Jacobians.
More recently, FlowChain 2 is proposed to enhance the standard normalizing flow models by using Flows (CIFs) [5] to estimate the density of outcomes. CIFs are obtained by replacing the single bijection normalizing flows with an indexed family F(::u) where U C R is the index set and each F(:; u)
series of Conditional Continuously-indexee
x is a bijection. Then, the
transformation is changed to
(6) Please refer (5] for more details about CIFs and their connection with variational inference. In this work, we follow the idea of using a stack of CIFs from (28) to achieve fast inference and the updates of trajectory density estimates.
z p(z),
U U~ (-)))) x:= x== F(z:)).
Normalizing flow based model samples from a standard Gaussian, z ~ NO 1), usually results in overfitting the most-likelise likelihood for trajectory prediction. This is because each data sample from the

---

training sample is considered extracted as the mode of a standard Gaussian. Only the mode value (the ground truth) directly supervised and the underlying target distribution is assumed to be perfectly symmetric, which is not aligned with the usual real-world facts. Related discussion can be found in many previous literatures [33 19]. This typically results in degraded expressiveness of the model to fail to capture under-represented motion patterns from the data and thus hurts the outcome diversity.
3.3 Mixed Gaussian Flow (MGF)
We propose Mixed Gaussian Flow (MGF) enhance the diversity and controllability in trajectory
prediction. MGF consists of two stages as summarized in Figure 2 First, we construct the mixed Gaussian prior by fitting the parametric model of a combination of K Gaussians, {.V o (1 < k S K). The parametric model is obtained with the data samples from training sets. Then, during inference, we sample points from the mixture of Gaussian and map them into a trajectory latent in the target distribution by a stack of CIF layers with the historical trajectories of all involved agents as the condition. We will introduce the two stages in detail below.
MGF maps from a mixture of Gaussians instead of a single Gaussian to the target distribution. To maintain the inevitability of the model, the mixed Gaussian prior can not be arbitrary. We obtain the parametric construction of the mixed Gaussian by fitting it with training data. In this fashion, we can derive multiple Gaussians to represent different motion patterns in the dataset, such as going straight or turning left and right. In a simplified perspective, we regard the mixture as combining multiple clusters, each of which represents a certain sub-distribution. By sampling from the mixture of Gaussians instead of a standard Gaussian, our constructed model has more powerful expressiveness than the standard normalizing flow model. This results in more diverse trajectory predictions. Also, by mani pulating the mixed Gaussian prior, we can achieve controllable trajectory prediction.
Mixed Gaussian Prior Construction. For the data pre-processing, we transfer motion directions into relative directions with respect to a zero-degree direction. All position footage is represented be to predict the trajectory between te - Ir. we
in meters. Given the trajectory between to would put the position pivot at fe. i.e., xt., as the origin and convert the position on all other time steps to be the offset from Xt. Then, we cluster the preprocessed future trajectories into K clusters, which is a hyper- parameter. We note the mean of the clusters as μ |
These cluster centers reveal the mean value of K representative patterns of pedestrians' motion, e.g. go straight, turn left. They will be the means of the Gaussians. The variances of the Gaussian, i.e., of can be pre-determined leamee. The final mixture of Gaussians is denoted
D = Σ (HK, o
where 3g are the weights assigned to each cluster following the k-means clustering of the training data. By default, perform clustering by K-means with K = 8.
Flow Prediction. Once the mixed Gaussian prior built, we can do trajectory prediction by mapping samples from the distribution to future trajectories conditioned on historical information(e.g. social interaction features extracted by Trajectron++ [46 encoder). Here, we ignore the intermediate trans- formation by CIFs as Equation (6) shows while following the original formulations of normalizing flows as Equation (5) for simplicity. We distribute the samples from different Gaussians by their weights. Given the i-th sample from M(p,, o.),, we can transform it to the i-th predicted trajectories
2i D
20 R = $ (x
Algile
O Dotta Zi).
(8)
For sample Zi De N( )), we have the probability estimate
Cop m2
p(z;) = 3x
20
(9)
Tb V 2π
5
and the transformation is converted to
exp( (z+ )2
8k
+ log
det fla 22)1,
(10)
20
Th V
which can be also invested back for the density estimate by the normalizing flow law
[f- ((0) 2g. ft. Otmte) pk] )
((2,) = Bk V4
exp(
(11)
20 k
5

---

3.4 Training and Inference
The training loss of MGF comes from two directions: the forward process to get mixed flow loss and the inverse process to get minimum lg loss.
Forward process. Given ground truth trajectory sample xi we need to assign it to a cluster in the mixed Gaussian prior by measuring its distance the centroids
k = arg (iin(x ua)² D BLN(
(12)
with tractable probability density function Pk (-). Through the inverse process f of flow model, we transform teitr into its corresponding latent representation, here denoted as
2 f (xf Owuste).
(13)
Then we can compute the forward mixed flow loss:
forward = log log(p(xf.arr) = log(p; (2)) - log( det V Vx xi 2)
(14)
Instead of computing negative-log- (NL))))ooddNLL) loss of in mixed distribution Σx= JuN(uue of). we compute NLL loss in the sub-Gaussian with the nearest centroid BgNV(ug, k because each centroid is independent others in the mixed distribution and we encourage the model to learn specified motion patterns to avoid overwhelming by the major data patterns. Calculating NLL loss over the mixed distribution may fuse other centroids and damage the diversity model outputs. By our design, the mixed Gaussian prior can maintain more capacity for expressing complicated multi-modal distribution than the traditional single Gaussian prior, which typically constrains the target distribution be single-modal and symmetric.
Forward Loss
Det3 Loss
Forward
Flow
t
Left
Kmeans Predictor
History +
Inverse
Flow
Right
mini2 Loss = Inverse Loss
During training, the model is trained at both forward and inverse process of the normalizing flow.
Figure 3:
Inverse Process.
This process repeats the flow prediction process to get generated trajectories. To predict M candidates, we sample z; Σ (uN o.), 1, ..., M and transform them into M trajectories
( ( = {f(z, Ot::t.) }}+ = 2.., .... ......M........ M.
(15)
We compute the minimum & loss between M predictions and ground truth trajectory as B does:
M
x 1t:
Linverse min
(16)
to to
We sample z; from sub Gaussians by their weight. This is approximately equal to sampling from the original mixed Gaussians but makes the reparameterization trick doable.
Although approximated differential backpropagation techniques, such as the Gumbel-Softmax trick, can be employed to make the sampling process of mixed Gaussians differentiable, computing the Negative Log- Likeelihooood (NLL) loss between a sample point and the mixed Gaussian distribution remains challenging because
K
(8-14)² 20
log(po=(()) (2)) = log
CC
(17)
Ok
k=
contains exponential operations on matrices, which can be simplified through logarithmic operations in single Gaussian condition. Computing this term requires iterative optimization methods, such as
the Expectation Expectation-Maximizaati
algorithm 6 for approximation (60) 47). which makes the computing process much more complex. Therefore, in practice, sampling from individual Gaussian components is preferred for computing efficiency. Furthermore, applying the Gumble-softmax to learn a mixture
of Gaussians generative models been reported difficult practice in some cases (39] due to gradient vanishing problem.

---

The forward and inverse losses encourage the model to predict a well-aligned sample in sub-space from the prior without hurting the flexibility and expressiveness of other sub-spaces. We combine the forward and inverse losses by ratio y to be Symmetric Cross-Entropy loss [40]. which was proved beneficial for better balancing the "diversity" and "precision" " of predicted trajectories:
L = Lorward + Linverse-
(18)
3.5 Diversity Metrics
The widely adopted average/final displacement error (ADE/FDE) scores measure the alignment (precision) between the ground truth future trajectory and predicted trajectory. Under the common "best-of-M"" evaluation protocol, ADE/EDE scores encourage nothing but finding a single "aligned"" trajectory with the ground truth. ADE encourages the position on all time steps to be aligned with the single ground truth and FDE chooses the trajectory with the closest endpoint while all other trajectories are neglected in score calculating. Such an evaluation protocol overwhelmingly encourages the methods to fit the most likelihood from a certain distribution and all generated candidates race to be the most similar one as the distribution mean. Under the single-mode and symmetric assumption, this usually tends to fit into a Gaussian with a smaller variance. However, this tendency hurts the diversity of predicted trajectory hypotheses.
To provide a tool for quantitative trajectory diversity evaluation, we formulate a set of metrics. Following the idea of average displacement error (ADE) and final displacement error (FDE), measure the diversity of trajectories by their pairwise displacement along the whole generated trajectories and the final step. Follow Dlow 622, name that average pairwise displacement (APD) and final pairwise displacement (FPD). We note that the diversity metrics are measured in the complete set of generated trajectory candidates instead of between single candidate and the ground truth. The formulation of APD and FPD are as below
Σ EM
M
M j=1 Σ
EI. (0) ()) x2)²
Σ E
(08g -0) 80, )²
APD
FPD
M (lT 12)
M
(19)
where APD measures the average displacement along the whole predicted trajectories and FPD measures the displacement of trajectory endpoints. We would mainly follow the widely adopted ADE/FDE for benchmarking purposes while using APD/FPD secondary metric set to better understand the diversity of the generated future trajectories.
Experiments
In this section, we provide experiments to demonstrate the effectiveness of our method. We first introduce experiment setup in Section [.] benchmark with related works to evaluate the trajectory prediction alignment and diversity in Section 4.2 Then, we showcase the diversity and controlla- bility of MGF in Section 4.3 and Section 4.4 Finally, we ablate key implementation components
in Section
4.1 Setup
Datasets. We evaluate on two major henchmarks, i.e., ETH/UCY [24
38 and SDD
42 ETH/UCY
consists of five subsets. We follow the widely used Social-GAN [3] consists of 20 scenes captured in bird's eye view. We follow the TrajNet 4 that in the community of trajectory prediction, previous works have inconsistent evaluation protocol details and thus have made unfair comparisons, Please refer to the appendix in supplementary materials for details.
benchmark. SDD dataset benchmark. We note
Metrics. We use the widely used average displacement error (ADE) and final displacement error (FDE) to measure the alignment of the predicted trajectories and the ground truth. ADE is the average L2 distance between the ground truth and the predicted trajectory. FDE is the L2 distance between the ground truth endpoints and predictions. Most previous works choose the "Best-of-M" evaluation protocol and we follow it to choose M = 20 as default.
Here, we note that, under diffferntt assumptions of distribution spreading and variance, the evaluation is ideally done with different values of M. However, most existing methods only provide results with M = 20 and many of them do not open-source the code of the models so not rebenchmark with other value choices of M. Besides the metrics for trajectory alignment, we also use the proposed metrics set APD and FPD to measure the diversity of the predicted trajectory candidates.
7

---

Table 1: Results ETH/UCY datasct with Best-of-20 metrics. Scores are in meters, lower better. bold and
underlined scores denote the best and the second-best scores.
CTH
HOTEL
UNIV
ZARAI
ZARA2
Meun
Method
ADE
FDE
ADE
FDE
ADE
FDE
ADE
FDE
ADE
FDE
ADE
FDE
Social-GAN 13
0.87
1.62
0.67
1.37
0.76
1.52
0.35
0.68
0.42
0.84
0.61
1.21
STGAT 16
0.65
1.12
0.35
0.66
0.52
1.10
0.34
0.69
0.29
0.60
0.43
0.83
Social-STGGCNN
32
0.64
1.11
0.49
0.85
0.44
0.79
0.34
0.53
0.30
0.48
0.44
0.75
Trajectron++ 46
0.61
1.03
0.20
0.28
0.30
0.55
0.24
0.41
0.18
0.32
0.31
0.52
MID [1]
0.55
0.88
0.20
0.35
0.30
0.55
0.29
0.51
0.20
0.38
0.31
0.53
PECNet
0.54
0.18
0.24
0.35
0.60
0.22
0.39
0.17
0.30
0.29
0.48
GroupNet [57
0,46
0.73
0.15
0.25
0.26
0.49
0.21
0.39
0.17
0.33
0.25
0.44
AgentFormer 63
0.45
0.75
0.14
0.22
0.25
0.45
0.18
0.30
0.14
0.24
0.23
0.39
ExMottoo 59
0.40
0.61
0.12
0.18
0.23
0.43
0.18
0.32
0.13
0.23
0.21
0.35
FlowChain
0.55
0.20
0.35
0.29
0.54
0.22
0.40
0.20
0.34
0.52
MGF(Ours)
0.39
0.59
0.13
0.20
0.21
0.39
0.17
0.29
0.14
0.24
0.21
0.34
Table 3: Results ETH/UCY dataset with diversity metries. Scores are in meters, higher means more diverse prediction. bold and underlined scores denote the best and the second-hest scores.
ETH
HOTEL
UNIV
ZARAI
ZARA2
Mean
Method
APD
FPD
APD
FPD
APD
FPD
APD
FPD
APD
FPD
APD
FPD
Social GANi-GGNN
13
0.680
1.331
0.566
1.259
0.657
1.502
0.617
1.360
0.515
1.119
0.607
1.314
Social-STGCNN
32
0.404
0.633
0.591
0.923
0.333
0.497
0.490
0.762
0.417
0.657
0.447
0.694
Trajectron++
0.704
1.532
0.568
1.240
0.648
1.404
0.697
1.528
0.532
1.161
0.630
1.373
AgentFormer
61
1,998
4.560
0.995
2.333
1.049
2,445
0.774
1.772
0.849
1.982
1,133
2.618
MemoNet
5X
1.232
2,870
0.950
2.030
0.847
1.822
0.844
1.919
0.880
2.120
0.951
2.152
FlowChain
28
0.814
1,481
0.484
0.833
0.636
1.094
0.505
0.890
0.492
0.859
0.586
1.031
MGF(Ours)
1.624
3,555
1.138
2.387
1.115
2.163
1.029
2.119
1.065
2.182
1.194
2
Table Evaluation results on SDD (in pixels).
Implementation Details. We enhance our model us- ing a similar technique as "intension clustering" [58] and name it "prediction clustering". The key
Method
ADE
FDE
Social-GAN 13
27.25
41.44 1,44
difference is that we directly cluster the entire tra- jectory instead of the endpoints. To make fair
STGAT 16
14.85
28.17
Social-STGCNN 32
20.76
33.18
comparison, we followed the data processing from FlowChain (28] and Trajectron++ [46). We also fol- low FlowChain's 's implementations of CIFs that each layer consists of RealNVP [T0] with a 3-layer MLP and 128 hidden units. We use Trajectron++ [46] encoder to encode historical trajectories. All models were trained on single NVIDIA V100 GPU for 100 epochs(aprroximately
Trajectron++ 46
19.30
32.70
MID [I]
10.31
17.37
PECNet 330
9.97
15.89
GroupNet [7
9.31
16.11
EqMotion 59
8.80
14.35
McmoNet 58
8.56
12.66
FlowChain 28
9.93
17.17
hours).
MGF (Ours)
7.74
12.07
4.2 Benchmark Results
We benchmark MGF with a line of recent related works on ETH/UCY dataset in Table The results of Trajectron++ and MID are updated according reported implementation issue MGF achieves on-par state-of-the-art performance with Eqmotion [59]. Specifically, Our method achieves the best ADE and FDE in out of 5 subsets and the best ADE and FDE score by averaging all splits. Here we note that we build MGF as a normalizing flow-based method as its invertibility is key property we desire, though normalizing flow is usually considered inferior regarding the alignment evaluation. Therefore, such a good performance on the alignment is surprising to us. To compare with other normalizing flow- methods, our method significantly improves the performance compared to FlowChain, achieving 27.6% improvement by ADE and 34.6% improvement by FDE.
On the SDD dataset, where the motion pattern is considered more diverse than UCY/ETH, the bench- mark results are shown in Table 2 Our method outperforms all baselines measured by ADE/FDE for trajectory alignment. Specifically, Our method reduces ADE from 8.56 to 7.74 compared to the cur- rent state-of-the-att method MemoNet, achieving 9.6% % improvement. Our method also significantly improves the performance of FlowChain for 22.1 % by ADE and 29.7% % by FDE. According the benchmarking on the two popular datasets, we demonstrate the ofte-of--the-ar alignment (precision) of our proposed method. Here we note again that the alignment with the deterministic ground truth is not the highest priority when we design our method, we will discuss the main advantages of MGF, diversity, and controllability, in the next paragraphs.
8

---

Clustering
Diverse Prediction
Original
Weights
Edit Directions
Samples
Intertion:
Claster Centara
Internian: Tight-Tum
CT Intertian: Lafi-Turn
... Port Tispezorry
--t Clsttr Center
Grand Truth Trojectory
Predizzed Trajectory
Trajectory
Center
Prodicted Trajectory
Figure 4: MGF predictions on ETH dataset. The color of trajectories corresponds to the cluster in the mixed Gaussian prior, from which the sample belongs to.
Figure Controllable generation on ETH datasct. By editing cluster centers, we can control the predictions.
4.3 Diverse Generation
By leveraging the mixed Gaussian prior, our model can generate trajectories from the corresponding clusters, resulting in a more diverse set of trajectories than sampling from Gaussian. This is intuitively due to less difficulty in learning the Jacobians for distribution transformation. We present examples in Figure Given past trajectory, there is single ground truth future trajectory possibility from the dataset. We select four samples with different ground truth intentions, i.c., going straight, U-turn, left-turn, and right-turn. By sampling noise from the clustered distributions, we could generate future trajectories with diverse intentions. From the visualizations, we could notice, of course, that we generate outcomes that are very similar to the ground truth with close intentions while we also generate outcomes that have very diverse intentions. The well-al well-aligned single trajectory accounts for the high ADE and FDE score our method achieves. And the impressive diversity demonstrates the effectiveness of our design, especially considering they are well controlled by the clusters where they are sampled from.
Quantitatively, we evaluate the generation diversity according to our proposed metries on ETH/UCY dataset since most existing methods did not either make experiments on SDD or open-source training code/checkpoint on SDD. The results are presented in Table We can observe that MGF achieves the best or second-best APD and FPD score on all splits among sota methods. Besides, our method significantly improves the performance compared to FlowChain, achieving 103.7 improvement by APD and 140.6 % improvement by FPD. The only method that can achieve close diversity with our method is Agentformer (63] which designs sampling from a set of conditional VAE to improve the diversity. However, compared to MGF, Agentformer is more computation-intensii and shows significantly lower alignment according to ADE/FDE scores in Table Also, Agentformer not fully invertible, which is considered key property we desire for trajectory forecasting. The superior quantitative performance according to the alignment (precision) and diversity metrics suggests the effectiveness of method by balancing these two adversarial features.
We also find that APD/FPD metries are not sensitive M, which is a natural result, see appendix B.
4.4 Controllable Generation
The generated sample from MGF is highly correlated with the original sample drawn from the mixed Gaussian prior. If the prior distribution is standard Gaussian in the canonical normalizing flow method, we can have almost no control over the generated sample. The only controllability is to sample the mode generate sample similar the learned most-likelihood outcome far from the mode to make them more different. However, as we discussed, after sufficient training and supervision by the forward loss, the variance of the latent Gaussian distribution of the outcome is usually very small, which further hurts the controllability. However, as we chose a transparent mixed Gaussian prior for the sampling, we can control the generation flexibility. First, by adjusting sub- Gaussians in the mixture prior, we can manipulate the generation process statistically. Figure 3 shows that by editing cluster compositions, we can control the predictions of MGF with good interpretability. By editing the weights of sub- Gaussians, we can control the ratio of splatting into directions. By editing the directions of the cluster means, we can control the intentions of samples statistically. Besides cluster centers, we can also edit the variance of Gaussian to control the density of generated trajectories or combine set of operations to get expected predictions. We provide more discussions and examples in the appendix in the supplement.

---

Table 4: Ablation study of ADE/DD on the ETHUCCY and SDD dataset.
ETH/UCY
SDD
Pred.
Mixed Gaus.
Learnable Var.
Inv. Loss
Clustering
ADE
FDE
ADE
FDE
0.33
0,61
11.90
21.33
0.29(J0.04)
0,48(10.13)
11.38((0.52)
19.28((2.05)
0.29
0.54
10.63
18.80
0.27()0.02)
0.48(10.06)
9.19(11.44)
15.86([2.994
V
0.23([0.04)
0.39(10.09)
8.71((0.48)
14.86(L1.00)
V
0.21(10.02)
0.34(10.05)
7.74(10.97)
12.07((2.79)
Table 5: Ablation study of APD/FPD on the ETH/UCY and SDD datasct.
ETH/CCY
SDD
Pred. Clustering
Mixed Gaus.
Learnable Var.
Inv. Loss
APD
FPD
APD
FPD
0.39
0.76
14.82
27.22
0.78(10.39)
1.70(10.94)
23.18(18.36)
44.90 17.668
0.41
0.80
15.52
28.50
V
1.09(00.68)
2.33(11.53)
32.42(+1169)
65.43((336.33)
V
0.96((0.13)
2.12((0.21)
30. 30.10((2.32)
60.20(L5.23)
/
1.19(10.77)
2.48(10.36)
31.56(11.46)
64.52(14.32)
4.5 Ablation Study
We ablate some key components of our implementation for both ADE/EDE and APD/FPD metrics, Table 4 and Table 5 P1)drediction clustering is common post-processing method, which improves the ADE/DDE as expected. However, hurts the diversity for nomalizing flow model with single Gaussian prior. This reasonable the single Gaussian prior tends to generate trajectories densely close to the most likelihood and prediction clustering can't cluster them into well-separated clusters for different motion intentions. (2)Mixed Gaussian prior help the model generates more diverse outputs and achieves higher APD/FPD scores and this improvement can be further enhanced by prediction clustering. It also increases ADE/FDE scores lot, we believe this is because mixed Gaussian prior relieves the difficulty of learning the Jacobians for distribution transformation. Thus more under-explored patterns, which may be selected the "best-of-M"" samples in rare but plausible scenarios, have the chance to be expressed. Learnabee variance improve ADE/FDE while bring down APD/FPD a bit. We find that the learnable variance usually converges smaller value than the fixed situation. This is encouraged by the supervision from the ground truth (most likelihood) to desired steeper Gaussian, thus hurting the diversity. However, its substantial improvement in ADE/FDE indicates that it remains a valuable component of the model architecture. (4) loss provides a straightforward supervision the trajectory the coordinate space, which is also proved beneficial for ADE/FDE and APD/FPD scores.
5 Conclusion
We focus on improving the diversity while keeping the estimated probability tractable for trajectory forecasting in this work. We noticed the poor expressiveness of Gaussian distribution the original sampling distribution for normalizing flow-based methods to generate complicated and clustered outcome patterns. We thus propose construct mixed Gaussian prior help learn Jacobians for distribution transformation with less difficulty and higher flexibility. Based on this main innovation, we propose Mixed Gaussian Flow (MGF) model for the diverse and controllable trajectory generation. The cooperating strategy of constructing the prior distribution and training the model is also designed. demonstrate that MGF achieves state-of-the- art prediction aligmment and diversity. It also has other good properties such as controllability and being invertible for probability estimates.
According the evaluation of popular benchmarks,
Acknowledgments
This project is funded in part by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK, Dahua Lin is of CPI under the InnoHK. This project is also supported by Shanghai Artificial Intelligence Laboratory.
10

---

References
[1] Alexandre Alahi. Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-977 2016.
[2] Matan Ben- Yosef and Daphna Weinshall. Gaussian mixture generative adversarial networks for diverse datasets, and the unsupervised clustering of images, 2018.
[3] Sam Bond-Taylor, Adam Leach, Yang Long, and Chris G Willcocks. Deep generative modelling: A comparative review of vaes, gans, normalizing flows, energy-based and autoregressive models. IEEE transactions on pattern analysis and machine intelligence, 44(1 1):7377-73377 2021 2021.
[4] Jinkun Cao, Xin Wang, Trevor Darrell, and Fisher Yu. Instance- aware predictive navigation in multi-agent environments. In 2021 I IEEE International Conference on Robotics and Automation (ICRA), pages 5096-5102. IEEE, 2021.
[5] Rob Cornish, Anthony Caterini, George Deligiannidis, and Arnaud Doucet. Relaxing bijec- tivity constraints with continuously indexed normalising flows. In International conference machine learning, pages 2133-2143. PMLR, 2020.
[6] Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological), 39(1):1-22, 1977.
[7] Patrick Dendorfer, Sven Elflein, and Laura Leal-Taixé. Mg-gan: A multi-generator model preventing out-of-distribution samples in pedestrian trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3158-13161 2021.
[8] Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, Matthew C. H. Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders, 2017.
[9] Laurent Dinh, Jascha Sohl- Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint XrX:v: 088003 2016.
[10] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp.
arXiv preprint arXiv: 10XKvv:600.0000 08800, 2016.
[11] Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17113-17122,
2022.
[12] Jiaqi Guan, Ye Yuan, Kris Kitani, and Nicholas Rhinehart. Generative hybrid representations for activity forecasting with no-regret learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 173-182, 2020.
[13] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2255-2264, 2018.
[14] Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical review
E, 4252222, 1995.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
[16] Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial-temporal interactions for human trajectory prediction. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6272-6281. 2019.
[17] Pavel Izmailov, Polina Kirichenko, Mare Finzi, and Andrew Gordon Wilson, Semi-supervised learning with normalizing flows, 2019.

---

[18] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[19] Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of-distribution data. Advances in neural information processing systems, 33:20578-20588 2020.
[20] Kris M Kitani, Brian D Ziebart, James Andrew Bagnell, and Martial Hebert. Activity forecasting. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part IV 12, pages 201-214. Springer, 2012.
[21] Ivan Kobyzev, Simon JD Prince, and Marcus A Brubaker. Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence. 43(11))3944-3979, 2020.
[22] Frederic Koehler, Viraj Mehta, and Andrej Risteski. Representational aspects of depth and conditioning in normalizing flows. In International Conference on Machine Learning, pages 0628-5636. PMLR, 2021.
[23] Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Tomr, and Man- mohan Chandraker. Desire: Distant future prediction dynamic scenes with interacting agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 336-345, 2017.
[24] Alon Lemner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. In Computer graphics forum, volume 26, pages 655-664. Wiley Online Library, 2007.
[25] Jiachen Li, Fan Yang, Hengbo Ma, Srikanth Malla, Masayoshi Tomizuka, and Chiho Choi Rain: Reinforced hybrid attention inference network for motion forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages (6096-16106) 2021.
[26] Junwei Liang, Lu Jiang, and Alexander Hauptmann. Simaug: Learning robust representations from simulation for trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings. Part XIII 16, pages Springer, 2020.
[27] Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander G Hauptmann, and Li Fei-Fei. Peeking into the future: Predicting future person activities and locations in videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[28] Takahiro Maeda and Norimichi Ukita. Fast inference and update of probabilistic density estimation on trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9795-9805, 2023.
[29] Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From goals, waypoints & paths to long term human trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15233-15242, 2021.
[30] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 759-776. Springer, 2020.
[31] Weibo Mao, Chenxin Xu, Qi Zhu, Siheng Chen, and Yanfeng Wang. Leapfrog diffusion model for stochastic trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5517-5526, 2023.
[32] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgenn: social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14424-14432, 2020.
12

---

[33] Abduallah Mohamed, Deyao Zhu, Warren Vu, Mohamed Elhoseiny, and Christian Claudel. Social-implicitt Rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation. In European Conference on Computer Vision, pages 463 Springer, 2022.
[34] Alessio Monti, Alessia Bertugli, Simone Calderara, and Rita Cucchiara. Dag-net: Double attentive graph neural network for trajectory forecasting. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 2551-2558. IEFF, 2021.
[35] Jeremy Morton, Tim A Wheeler, and Mykel J Kochenderfer. Analysis of recurrent neural networks for probabilistic modeling of driver behavior. IEEE Transactions on Intelligent Transportation Systems, (8(5):1289-12) 1298, 2016.
[36] George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. The Journal of Machine Learning Research, 22(1):2617-2668 2021.
[37] Scong Hyeon Park, Gyubok Lee, Jimin Seo, Manoj Bhat, Minseok Kang, Jonathan Francis, Ashwin Jadhav, Paul Pu Liang, and Louis-Philippe Morency. Diverse and admissible trajectory forecasting through multimodal context understanding. In Computer Vision ECCV 2020: 16th European Conference, Glasgow. UK. August 23 28, 2020, Proceedings, Part XI 16, pages 282-298. Springer, 2020.
[38] Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In 2009 IEEE 12th international conference on computer vision, pages 261-268. IEEE, 2009.
[39] Andres Potapczynski, Gabriel Loaiza-Ganem, and John P Cunningham. Invertible gaussian repa- rameterization: Revisiting the gumbel-softmax. Advances in Neural Information Processing Systems, 33:1231 13:11211-23221, 2020.
[40] Nicholas Rhinehart, Kris M Kitani, and Paul Vernaza. R2p2: A reparameterized pushfor- ward policy for diverse, precise generative path forecasting. In Proceedings of the European Conference on Computer Vision (ECCV), pages 772-788, 2018
[41] Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and Sergey Levine. Precog: Prediction con- ditioned on goals in visual multi-agent settings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2821-2830, 2019.
[42] Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In Computer Aasion-ECCC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part VIII 14, pages 549 149-565. Springer, 2016.
[43] Jonas Rothfuss, Fabio Ferreira, Simon Bochm, Simon Walther, Maxim Urrich, Tamim Asfour, and Andreas Krause. Noise regularization for conditional density estimation. arXiv preprint arXiv:1907.08983 2019.
[44] Amir Sadeghian, Vineet Kosaraju, Agrim Gupta, Silvio Savarese, and Alexandre Alahi. Trajnet: Towards a benchmark for human trajectory prediction. arXiv preprint, 2018
[45] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophic: An attentive gan for predicting paths compliant to social and phys- ical constraints. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1349-1358, 2019.
[46] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajec- tron++: Dynamically-feasibbc trajectory forecasting with heterogeneous data. In Computer Visi 2020: 16th European Conference. Glasgow, UK, August 23-28, 2020. Proceedings, Part XVIII 16, pages 683-700. Springer, 2020.
[47] Matthias Schwab, Agnes Mayr, and Markus Haltmeier. Deep gaussian mixture model for unsupervised image segmentation. arXiv preprint arXiv:2404. 12252, 2024.
13

---

[48] Nasim Shafiee, Taskin Padir, and Ehsan Elhamifar. Introvert: Human trajectory prediction via conditional 3d attention. In Proceedings of the IEEE/evf Conference on Computer Vision and Pattern recognition, pages 16825, 2021.
[49] Novin Shahroudi, Mihkel Lepson, and Meelis Kull. Evaluation of trajectory distribution predictions with energy score. In Forty-first International Conference on Machine Learning.
[50] Jianhua Sun, Yuxuan Li, Liang Chai, and Cewu Lu. Stimulus verification is universal and effective sampler in multi-modal human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22014 220023, 2023.
[51] Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu. Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. In Proceedings of the IEER/CVF International Conference on Computer Vision, pages 13250-13259, 2021.
[52] Anirudh Vemula, Katharina Muelling, and Jean Oh. Social attention: Modeling attention in human crowds. In 2018 IEEE international Conference on Robotics and Automation (ICRA), pages 4601-4607. IEEE, 2018
[53] Jack M Wang, David J Fleet, and Aaron Hertzmann. Gaussian process dynamical models for human motion. IEEE transactions on pattern analysis and machine intelligence, 30(2):283-298, 2007.
[54] Maciej Wolezyk, Magdalena Proszewska, łukasz Maziarka, Maciej Zieba, Patryk Wielopolski, Rafat Kurezab, and Marek Smieja. Plugen: Multi-label conditional generation from pre-trained models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 8647-8656, 2022.
[55] Conghao Wong, Beihao Xia, Qinmu Peng, and Xinge You. Another vertical view: A hi- erarchical network for heterogeneous trajectory prediction via spectrums. arXiv preprint arXiv:2304.05106, 2023.
[56] Conghao Wong, Beihao Xia, and Xinge You. Socialcircle: Learning the angle-based social interaction representation for pedestrian trajectory prediction. arXiv preprint arXiv:2310.05370, 2023.
[57] Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6498 6507, 2022.
[58] Chenxin Xu, Weibo Mao, Wenjun Zhang, and Siheng Chen. Remember intentions: Retrospective- memooy-maoedd based trajectory prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1488-6497, 2022.
[59] Chenxin Xu, Robby Tan, Yuhong Tan, Siheng Chen, Yu Guang Wang, Xinchao Wang, and Yanfeng Wang. Eqmotion: Equivariant multi-agent motion prediction with invariant interaction reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4410-1420, 2023.
[60] Guorong Xuan, Wei Zhang, and Peiqi Chai. Em algorithms of gaussian mixture model and hidden markov model. In Proceedings 2001 international conference on image processing (Cat. No. 01CH37205), volume 1, pages 145-148. IEEE, 2001.
[61] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. Computer Vision-ECCY 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XII 16, pages 507-523. Springer, 2020.
[62] Ye Yuan and Kris Kitani. Dlow: Diversifying latent flows for diverse human motion prediction. In Computer VCCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part IX 16, pages 364. Springer, 2020.
14

---

[63] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transform- ers for socio-temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9813 9813-9823, 2021.
[64] Pu Zhang. Wanli Ouyang, Pengfei Zhang, Jianru Xue, and Nanning Zheng. Sr-Istm: State refinement for Istm towards pedestrian trajectory prediction. Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12085- 12094, 2019.
15

---

A Inconsistent Practice of the Evaluation for Trajectory Prediction
Previous research in the area of trajectory forecasting. i.c. trajectory prediction, has focused on multiple datasets for quantitative evaluation. However, we notice that the evaluation settings of previous works are inconsistent thus making noisy and unfair comparisons in the benchmarks we usually refer to.
A.1 Inconsistent evaluation convention on ETH/UCY
The ETH/UCY [24] 38
dataset, comprising five subsets of data, serves the primary benchmark for human trajectory prediction. It is not proposed as single dataset in the first place but merges many different resources. Therefore, there are official guidelines for data splitting and model evaluation metrics. Consequently, previous studies employ different evaluation conventions and falsely confuse results from different conventions together for benchmarking.
The benchmark widely adopted by the research community was initially proposed by Social- GAN [13]. It adheres to the following principles:
1. Utilizing data with a sampling rate of 10 FPS in all subsets.
2. Employing leave-one-out approach for splitting, where the model is trained on four subsets and tested on the remaining one subset.
3. Dividing the raw trajectory samples into specific train, eval, and test sets.
Subsequent works like Trajectron++ 46
AgentFormer
0S and EqMotion 59 have widely
embraced this benchmarking and evaluation setting.
However, not all studies adhere to this benchmark. To identify examples, conducted review of recent open-access conference papers, which reveals the following divergence.
1. Sampling Rate on ETH dataset. There are two widely used sampling rates for the evaluation of
the ETH dataset. While Social-GAN [T3] utilized the data with a sampling rate of 10FPS (SR=10),
other works, such SR-LSTM (64), V2Net (55), SocialCircle (56), STAR [6]], PCCSNet [5]]. Stimulus Verification [50), and MG-GAN (7), used the version with a sampling rate of 6FPS (SR=6). The SR=6 version contains more data, with total of 8,908 frames, whereas the SR=10 version consists of only 5,492 frames. Based on our experience, the same model tends to yield higher evaluation scores (ADE/FDE) the SR=6 version compared to the SR=10 version.
Data Splitting. Social-GAN follows a specific scheme and ratio for splitting the data into train/val/test sets, while some works have adopted different conventions. For instance, Sophie [45 selected fewer training scenes, MG-GAN 7 used the complete training scene data for training while separating portion of the test set for evaluation. Also, works that choose the SR=6 version data in the ETH dataset adopt different splitting conventions, because they do not share the same raw trajectory data samples with Social-GAN, which uses the SR=10 version data.
Inconsistent Data Pre-processing. Some other studies, such Y-Net (2), Introvert 48 and Next [27], provide processed data without the raw data and the processing scripts. The provided processed data for val/test sets does not align with the Social-GAN benchmark.
A.2 Inconsistent evaluation convention on SDD
Compared to ETH/UCY, SDD [42) is more recent dataset consisting of 20 scenes captured in bird's eye view. SDD contains various moving agents such as pedestrians, bicycles, and cars.
Most works follow the setting of TrajNet 44 which comes from a public challenge. However, some works adopt different evaluation way compared to TrajNet. SimAug (26) reprocesses the raw videos and gets a set of data files different from TrajNet's. 's. Besides, it uses a different data splitting convention. Subsequent works such as V2Net (55] and SocialCirele [56] follow the same setting that SimAug starts. DAG-Net (4 shared the same data file with TrajNet, but used a different data splitting approach. Social- ImplictI
33 followed its setting.
Therefore, there are multiple different evaluation protocol conventions on the ETH/UCY and SDD datasets. Because the data splitting for training/test and evaluation details are different, putting the evaluation numbers from them together provides misleading quantitative observations, which the community has been using for while. We point it out and make complete summary of these
16

---

misalignments, wishing future research aware of this to avoid potential continuity of the mis practice and provide a fair comparison.
A.3 Summary and Our Practice
When comparing baseline results, many previous studies fail to meticulously verify whether they adhere to the same convention, thereby leading unfair comparisons. To compare the performance of various models fairly, we follow the practice that the community adheres to the most: Social-GAN's convention for ETH/UCY and TrajNet's convention for SDD.
More specifically, for ETH/UCY dataset, we recommend to use the preprocessed data and dataloader from SocialGAN Trajecctrott++ [46//AgenfFormer [63]. For the SDD dataset, recommend using the preprocessed data and dataloader from Y-Net [29]. Although their data processing methods may differ, they share the same data source and data splitting approach, facilitating fair comparisons.
We also note that, the early version of Trajectron++ a misuse of the up.gradient function during computation resulted in the model accessing future information. Rectifying this bug typically leads to significant decrease in scores. Consequently, several Trajectron+++basce studies have achieved improved scores.
B Sensitivity of diversity metrics
As we all know, the ADE/FDE employ a "best-of-MM" computation approach, where M significantly influences the results: larger M values yield lower ADE/FDE scores. We wonder whether APD/FPD metries demonstrate similar sensitivity to M. Since APD/FPD calculations consider all trajectories collectively, intuitively, their values would remain stable with varing M. Our experimental results confirmed this, see Table 6 The APD/FPD metrics for FlowChain and MGF(w/o Pred. Clustering) remain stable M increases, whereas MGF exhibits slight decrease. This decrease can be attributed to MGF's prediction clustering mechanism. which initially generates fixed J samples (c.g.. here J=500) and then clustering them into M output trajectories. As M increases, the impact of prediction clustering gradually diminishes (when M = J, prediction clustering is deprecated). Given that prediction clustering contributes to improving diversity in our ablation study, its weakening effect leads to a monotonic decrease in APD/FPD
Table 6: APD/FPD performance of Flow Chain and MGF on ETH/UCY and SDD dataset when M varies.
FlowC
MGF
MGF(w/o Pred. Clustering)
M
ETH/UCY
SDD
ETH/UCY
SDD
ETH/UCY
SDD
10
0.37/0.72
14.05/25.83
0.97/2.18
31.96/65.54 54
0.75/1.66
24.11/47.66 .00
20
0.39/0.76
14.81/27.23
1.02/2.28
31.50/64.39
0.81/1.80
24.84/48.78
30
0.40/0.77
15.05/27.68
0.99/2.23
30.62/62.29
0.83/1.83
26.56/52.26
40
0.40/0.78
15.18/27.91
0.97/2.16
29.85/60.47
0.84/1.87
26.33/51.72
50
0.40/0.78
15.25/28.03
0.95/2.11
29.46/59.45
0.84/1.86
26.16/51.44
60
0.40/0.78
15.29/28.10
0.93/2.07
29.00/58.34
0.84/1.87
26.16/51.33
70
0,40/0.79
15.31/28.12
0.91/2.03
28.78/57.75
0,84/1.86
26.20/51.53
80
0.40/0.79
15.34/28.19
0.90/2.01
28.54/57.10
0.84/1.87
26.48/52.20
C Controllable generation by data augmentation
Besides the fashion of manipulating generation results given the good property of constructed mixed Gaussian prior in the main paper, we also use data augmentation to alter the data patterns in our training set, thereby obtaining different priors. This enables our model fix corner cases that are difficult to handle with traditional flow-based models like FlowChain. Taking Figure 6 as an example, generate the clusters representing green "U-turn", purple "left-turn", and cyan "right-turn' clusters in the right-middle figure, we duplicate and rotate the original future trajectory data by 180%, 90", and -90%. respectively. Subsequently, we mix these rotated data with the original data in fixed ratio to produce the augmented dataset (in this particular case, a 2:2:1:1 ratio is employed for the original:1807 1000:900:-00")" Then we apply k-means to the augmented dataset, thereby obtaining the new augmented prior distribution (depicted in the right-middle figure). Finally, we train the model using this augmented dataset. Compared to the generated results from FlowChain, after using the augmented data to construct the mixed Gaussian prior, our method can generate the
17

---

Table 7: minADD/mmnfFDE of worst-N predictions UNIV dataset. By adding augmented data along with their corresponding cluster centers, our method significantly improves the performance on corner cases.
FlowChain
Augment MGF
MGgFent-MGF
N
ADE
FDE
ADE
FDE
10
3.13
6.54
0.75
1.30
50
2.40
4.90
0,90
1.57
100
2.06
4.29
1.07
1.86
FlowChain
Augment-MGF
Augmenting
Adding Clusters
4.17/8.13
Past Trajectory
Trajectory
Gluster Center
2.905.300
0.631/0.588
Trajectory
Figure 6: adding augmented data along with their corresponding clusters into the construction of the mixed Gaussian prior, could manipulate the generation patterns under-represented trajectory patterns. Then the model can generate corner cases that existing models fail to generate in reasonable probability. We selected three examples from the UNIV dataset, namely sharp left/right turns and U-turns. The left column shows the predictions from FlowChain, while the middle column shows the predictions of MGF with augmented priors
desire. For example, we could inject some
under-explored trajectory patterns with a higher chance. After this manipulation, we could change the mixed Gaussian prior desire, such amplifying the chance of generating corner in this example.
On the other hand, quantitatively evaluate the ability to generate under-represented trajectory patterns. Table Table'ccome compares the ADE/FDE scores their worst-IV samples on the UNIV dataset. Typically, the samples from the test with the worst ADE/FDE relate the under-represented corner cases future trajectories. The results demonstrate quantitatively that MGF can better generate the under-represented motion patterns after injecting the desired corner cases on purpose by manipulating the mixed Gaussian prior as mentioned above. We note that all the provided examples of manipulating the mixed Gaussian prior to controlling the generation statistics do not require fine-tuning or any operation to the normalizing flow itself. As manipulating the mixed Gaussian prior purely a parameter updating processing without any training and gradient backpropagation, all the manipulation is very fast in practice. This suggests the good efficiency and flexibility of our proposed method to achieve controllable generations.
Limitations
Limited by computing resources, we did not utilize the map information in our model. Some generated trajectories may overlap with obstacles, thus decreasing the upper bound of MGF's ability. Also, we found that agents can occasionally collide with each other due the limited ability of the history encoder. Future works may take more consideration to the collision among agents or between agents and the environment.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We focus on improving the diversity and the controllability of the probabilistic trajectory generation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limited by the page limit, we make the discussion about the limitations the appendix instead of in the main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [NA]
This paper is mostly an application paper without essential theoretical contri- butions. We borrow the established theoretical framework about normalizing flow, mixed Gaussian model, and CIFs in this paper without theoretical extension. Therefore, we make no new assumptions or proofs.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided the details to reproduce the method and results we present in the paper. We also include source code and the checkpoints required to reproduce the results in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
20

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, we already included the source code and the models in the supplementary materials for the reproduction. Also, we will open source the code and models.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.cc.
Sttii///giides//oeeSuboee itt.eesouuuiisiiooooooooy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. //nips.cc/publll wwtpccpwwwpcpcudeeseespppppppppe wtt.gudddee///ccmmiiiaaa sondesuumisssioPoll1ee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetalls
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide a detailed introduction about the training details. Limited the page length limit, may skip implementation details if we inherit from previous methods. All details are available in our provided source code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: To make the consistent benchmarking practice, we do not report the variance of the evaluation results.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details of experiments compute resources in the Implementation Details part of Experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: we have read the code of ethics and conform with it during the work of this paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
22

---

Justification: there is no societal impact of the work performed, we use the human trajectory data but no human face or other data that might involve private information used.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our code is partly based FlowChain codebase, which is unlicensed on github. We test our mehthod on ETH/UCY [24 and SDD dataset. ETH/UCY dataset
are licensed under CC BY-SA 4.0, SDD 42 dataset are licensed under CC BY-NC-SA 3.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
23

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We adopt MIT License our code, and we also provide a detailed instruction of how to run the code for reproducing experimental results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "YepR059hP1", "reviewer": "Reviewer_qRNj", "summary": "The authors proposed a new normalizing flow-based human trajectory prediction method called Mixed Gaussian Flow (MGF), which promotes diversity and controllability of prediction. The model uses a mixture of Gaussian model as the initial distribution to transform, rather than single-modal standard Gaussian distribution. The paper proposed using clustering of training data to mine modes of movements and use these modes to fit the initial sets of Gaussians for the mixture. The authors also claimed they proposed new metric measure diversity (APD, FPD). The authors have shown MGF can exceed SOTA performance on ETH-UCY and SDD datasets during their quantitative analysis, and showcase by controlling the mixture of Gaussian, the types of trajectory to generate can be controlled.", "questions": "- Abalation study was on ADE FDE only, but what is the effect on the diveristy? It would be great if the author could showcase this for the abalation, as diversity is the main contribution of the paper.\n- If taking out mixture of gaussian and only use predictive clustering would the model achieve same level of ADE/FDE performance?\n- Besides, this quesiton it would be great if the author could address the concerns in weakness section.\n- The reviewer will consider increase the score if the authors could address and respond to the concerns mentioned above.", "limitations": "The reviewer appreciate the author mention in the supplementary about the limitation, where the model does not utilzed map information and predicitons will contain collision among agent. Although per guideline, the reviewer shall not punish this limitation, however, it is worth pointing out diversity without context might not be a sound logic. As solely promoting diversity might making the prediction output actually unrealistic considering the environmental and social context of the prediciton output. Although the author mentioned potential way to incorporating map information in the main paper, such proposal is more on the implicit side (as latent embeddings), which might not be able to address environmental collision issue.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well motivated, and by adopting Mixture of Gaussian, the model was able to achieve diversity and controllability\n- The paper has shown normalizing flow-based HTP model can achieve SOTA performance with good design\n- The model has good analysis (4.3, 4.4) showcasing the diversity and controllability of the model output.", "weaknesses": "- Line 112, 113 using $t_0, t_1, t_2$ might not be the best choice, the reviewer suggest maybe consider $t_0,t_1, \\dots ,t_{c}, \\dots, t_{T}$\n- Suggest tidy up the variable representation of this section, for example using $X$ for observation sequence and $Y$ for future sequence\n- Line 159 160, about normalizing flow overfitting, it would be great if the author could provide a reference to this claim or if this claim is by the authors themselves (also 129-130)\n- Line 218-219, NLL loss is in sub-Gaussian, but how to decide the mixing coefficient (the reviewer is aware line 199 says it is decided from training data but specifically how?)\n- Line 332, typo what is AED for APD or ADE?\n- The ablation shows the ADE and FDE performance had a big increase with the prediction clustering, without prediction clustering MGF will have performance similar to AgentFormer and GroupNet and will not achieve SOTA, so does this means prediciton clustering/interntion clustering is decisive in accuracy performance of the model?\n- The authors claimed they proposed APD metric, but it seems it is already in used in previous literatures under the same name and same formulation\n\n@inproceedings{yuan2020dlow,\n  title={Dlow: Diversifying latent flows for diverse human motion prediction},\n  author={Yuan, Ye and Kitani, Kris},\n  booktitle={Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part IX 16},\n  pages={346--364},\n  year={2020},\n  organization={Springer}\n}", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Writing issues:** We appreciate the detailed suggestions about writing, we could consider them and improve the writing details. L332 should be ADE and FDE. \n\n**Normalizing flow overfitting:** We observe that normalizing flow tends to overfit to the single mode value (ground truth annotation) when using the naive Gaussian prior. This is because of the internal conflict between a probabilistic model and a deterministic and single ground truth supervision under the unimodal simple Gaussian prior. On the other hand, normalizing flow directly transfers a noise to a single outcome point, further causing the model to collapse to the max-likelihood value. As a comparison, VAE predicts a distribution where we can sample outcomes instead of a single outcome, relieving the overfitting issue with a reasonable value of distribution variance. Related discussion can be found in many previous literature, such as [1,2].\n\n> [1] \"Noise Regularization for Conditional Density Estimation\"\n\n> [2] \"Why Normalizing Flows Fail to Detect Out-of-Distribution Data\"\n\n\n**mixing coefficient:** In our implementation, we decide the coefficients by steps. We use k-means clustering to partition the trajectory data into clusters. The mixing coefficient of each sub-Gaussian corresponds to the weight of the corresponding cluster. \n\n\n**Prediction clustering to improve accuracy:** We add ablation experiments in Table 2 in the pdf file. The prediction clustering improves ADE/FDE with both single Gaussian and mixed Gaussian prior. Adding mixed Gaussian prior further boosts the performance. \n\nMoreover, we provide the results of ground truth alignment (ADE/FDE) to demonstrate that our method can output a single \"likely\" trajectory in a batch at a similar level as the best arts. Its flexibility and advantage of enhancing generation diversity is our main goal. \n\t\t\n**APD in Dlow:** Yes, we agree that a similar APD has been defined by Dlow[3]. We missed it as it was proposed on a different task and benchmarks. We will revise this paper and acknowledge where the metric was first proposed.\n\n> [3] \"DLow: Diversifying Latent Flows for Diverse Human Motion Prediction\"\n\n**Ablation studies on diversity:** Great suggestion! We add the ablation study to measure the impact of modules on diversity in Table 1 in the pdf file. There are two interesting observations: \n1. The prediction clustering can significantly improve the diversity only with mixed Gaussian prior. With the naive Gaussian prior, it makes no contribution to the diversity. This is reasonable as the single Gaussian prior tends to generate trajectories densely close to the most likelihood and prediction clustering can't cluster them into well-separated clusters for different motion intentions. \n2. The learnable var. slightly hurts the diversity. This is because the learnable var. usually converges to a smaller value than the fixed situation ($\\sigma=1$). For instance, the average value is ~0.4-0.6 on ETH/UCY. This is encouraged by the supervision from the ground truth (most likelihood) to a desired steeper Gaussian, thus hurting the diversity. It is the trade-off between (1) aligning with the single likelihood provided by ground truth and (2) generating diverse outcomes that hide behind this observation.\n\n**Adding map information:** We agree that adding map information can be potentially more useful in realistic use cases. However it would lead to a different convention of benchmarking and the different processing of map information would introduce significant noise when comparing methods, especially considering many baselines have no open-sourced code or checkpoints. To focus on the main assumption and innovation in this work (a mixed Gaussian prior can improve the diversity of the normalizing flow-based trajectory forecasting), we decided not to use map information and follow the most widely adopted problem set up in this area. Adding map information would be an interesting future work but significant efforts are required to fairly compare different arts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yurFNx52DA", "reviewer": "Reviewer_1BoH", "summary": "Due to the asymmetric and multi-modal nature of future trajectories, the authors point out that the standard Gaussian prior with a neural network-based transformation is insufficient for probabilistic trajectory prediction. They propose Mixed Gaussian Flow (MGF), a method that uses the mixed Gaussian prior in the normalizing flow model. MGF allows diverse and controllable generation by leveraging the prior distribution. A minimum-of-M loss is designed for the inverse process to increase the predicting precision. Substantial experimental results on well-known datasets like ETH/UCY and SDD verify the effectiveness of MGF.", "questions": "Q1. The prior distribution is obtained by clustering trajectories in the training set. Will it suffer from a distribution shift in the testing set? Specifically, in ETH/UCY dataset, the testing set contains scenes that do not appear in the training set. Can MGF deal with this issue? If not, can you figure out any possible solution?\n\nQ2. The asymptotic behavior of ADE/FDE is well-studied [Ref.1]. Is it possible to theoretically or empirically analyze the asymptotic behavior of APD/FPD for a sufficiently large $M$? Is the value of APD/FPD sensitive to $M$?\n\n[Ref.1] Analyzing the Variety Loss in the Context of Probabilistic Trajectory Prediction. In ICCV, 2019.\n\nQ3. Some other papers have proposed metrics for measuring the diversity of trajectories, like the average maximum eigenvalue [Ref.2] and the energy score [Ref.3]. Can you discuss the connection and difference of APD/FPD with these metrics? Is it possible to evaluate the trajectory diversity of different methods using these metrics?\n\n[Ref.2] Social-implicit: Rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation. In ECCV, 2022.\n\n[Ref.3] Evaluation of Trajectory Distribution Predictions with Energy Score. In ICML, 2024.\n\nQ4. In Section 4.4, how do you edit the directions of the cluster means?\n\nQ5. Can you provide some empirical evidence for the claim of \"limited and imbalanced data\" in Line 30 of Section 1, e.g., the weights of each Gaussian component of the prior distribution?\n\nQ6. Is it possible to find some theoretical support for the claim that \"deriving a desired complex distribution from a simple and symmetric prior distribution is challenging, especially with limited and imbalanced data\" in Line 29-30 of Section 1?\n\nQ7. In Line 230-232, the authors say \"Instead, it is hard to back-propagate the gradient through the sampling process of a Gaussian mixture with the reparametrization applied.\" I think the gumbel-softmax trick can be used for selecting components while allowing back-propagation. What do you mean by saying \"it is hard to back-propagate\"?\n\nComment.\n\nC1. The authors clearly describe the subtle differences in evaluation in existing methods. It really saves my time and helps researchers in trajectory prediction.\n\nC2. Appendix B contains some important facts. Please summarize these discoveries in the main text.\n\n**Update:**\n\nI have read the authors' response. They have addressed most of my concerns. I decide to raise my rating to 7. I hope that the authors can better organize the paper so that the presentation is clearer and it is easier for the readers to grasp the paper's significance.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. The motivation is strong and the method is intuitive.\n\nS2. MGF allows controllable generation by manipulating the prior distribution. It also secures under-represented future trajectory patterns.\n\nS3. The experimental results are solid and convincing.", "weaknesses": "W1. This paper challenges the standard normal prior assumption in the normalizing flow model, but lacks in theoretical support. Also see Q6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Theoretical support behind our claim:** Great suggestion! Our claim about the difficulty of training normalizing flow to transferring a naive and symmetric distribution, e.g., simple Gaussian, to a complex distribution, is mainly from our empirical observations. But there are some insights behind this observation:\n1. VAE or Diffusion learns a conditional distribution to sample outcomes, whose representation is low-dimensional. For example, only $\\mu$ and $\\sigma$ as the output of VAE. On the other hand, normalizing flow is trained to learn a Jacobian between distributions, requiring a high-dimensional generation capacity.\n2. Given the same input noise, VAE or Diffusion will generate different outcomes in multiple rollouts, only required to be constrained in an underlying distribution. However, normalizing flow has to do a deterministic transformation from the input noise to a certain sample in the target distribution.\n3. Normalizing flow is typically trained with a single ground truth. With the naive Gaussian prior, if the target distribution is complex and multi-modal, we need to transform the single mode in the original distribution to more than one target mode. This causes a contradiction between training gradients.\n4. Normalizing flow is designed to be inversible. To learn a Jacobian between a single-modal original distribution and a complex/multi-modal target distribution, the one-to-many mapping also causes difficulty when training inversely.\n\nThere are many discussions about the difficulty of training normalizing flow in practice to represent a complex target distribution, such as[1,2,3]. In [3], normalizing flow is considered inefficient and difficult to train due to invertibility constraints on transformations and as a consequence input and output spaces with the same dimension. This matches our discussion in the insight #1 above. \n\nWe have to admit that normalizing flows is usually not as good as GAN or diffusion for high-dimensional data generation, such as images. However, its unique property of invertibility is irreplaceable for controllable and explicable generation as we desire in this work.\n\n> [1] \"Representational aspects of depth and conditioning in normalizing flows\"\n\n> [2] https://distill.pub/2019/gan-open-problems/#tradeoff\n\n> [3] \"Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models\"\n\n**Distribution shift:** The problem of distribution shift exists in all supervised learning arts. We believe the common and shared intentions across data are important to adapt to test scenes. And the history encoder allows MGF to adjust generated trajectories based on the input historical trajectories. For example, if there is a group of people located in front of the agent, trajectories of “go straight” sub-Gaussian will be transferred into left/right/other trajectories to avoid collisions. As long as the shared intentions such as \"move to avoid collision\" hold, our work can generate reasonable trajectories in new situations. \n\n**The influence of the value of M to APD/FPD:** We add the ablation study about the value of $M$ in Table 3 in the submitted pdf file with $M$ increasing from 10 to 80. According to the results, APD/FPD is not sensitive to the value of $M$.\n\n\n**Comparison to other diversity metrics:** Here we compare the mentioned diversity metrics with ours: \n1. AMD (Average Mahalanobis Distance) and AMV(Average Maximum Eigenvalue) are proposed in [4]. AMD computes the Mahalanobis distance to measure how close the generated samples are to the ground truth, similar to ADE. AMV quantifies the overall spread of the predictions for diversity measuring. It requires the confidence of predictions, which limits its application in many cases where no confidence or probability is available for predictions.\n2. The metrics of ED and EI are proposed in [5]. ED measures accuracy and EI shares the same formulation with APD.  We note that [5] is accepted to ICML'2024 and was not publicly available yet at the due date of NeurIPS submission.\n\n> [4] \"Social-implicit: Rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation\"\n\n> [5] \"Evaluation of Trajectory Distribution Predictions with Energy Score.\"\n\n**Edit the direction of the cluster:** The cluster center is represented as a vector in shape $12 \\times 2$, which is the mean value of the corresponding sub-Gaussian distribution, sharing the same dimensionality as the output trajectory. We can directly manipulate the value of this vector to represent different motion intentions.\n\n\n**Empirical evidence of the data imbalance:** Existing trajectory forecasting datasets contain imbalanced data where most trajectories are going straightforward. In Figure 4 in the paper, the weights of each cluster (represented by colors) are: Blue: 0.28, Brown: 0.09, Red: 0.12, Green: 0.1, Yellow: 0.16, Cyan:0.19, Purple: 0.05. This suggests very imbalanced data distributions for different motion intentions. Moreover, across all five subsets in the ETH/UCY datasets and one set in the SDD dataset, similar data patterns are observed: approximately 30% of the trajectories exhibit a 'stay' behavior, while 50-60% correspond to 'go straight' trajectories with varying speeds. Conversely, only 10-20% of the trajectories involve 'turn left/right' maneuvers, with 'U-turn' and other corner cases being extremely rare occurrences.\n\n**Difficulty of back-propagation:** After sampling a noise point **x** from a mixture of Gaussian **D**, it is non-trivial to back-propagate the gradient from applied to **x** to the learnable parameters of **D**. \n[6] has addressed this issue using approximations. It requires substantially more computation. Gumbel-Softmax trick deals with the problem of “sampling discrete data from a categorical distribution”, it is not a viable solution for our case.\n    \n> [6]: https://github.com/vsimkus/torch-reparametrised-mixture-distribution", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iHybvaGev9", "reviewer": "Reviewer_pmqe", "summary": "This paper proposes to solve the trajectory prediction task by normalizing flow based model with mixture Gaussian assumption. The trajectories are clustered to multiple Gaussian distributions in the pre-processing stage over training data. During inference, priors are sampled from the mixture Gaussian distribution as the guidance of trajectory prediction. Moreover, new metrics are introduced to evaluate the diversity of multiple trajectory generation.", "questions": "Please see the questions in weaknesses.\n\nBesides, here are some suggestions.\n\n1. The readability of notations and equations can be improved. For example, the paper does not include the scene map modeling, then map M (line 111) is useless. If O_{t0:t1} (line 119) only contains map information, then O can also be excluded from equations.\n\n2. This task may not be the appropriate task for the proposed method.\n\n    Although the task of trajectory prediction contains multiple trajectory output which usually involves generative models, this task is not a pure generation problem. In this task, the accuracy is still with highest priority. For example, a self-driving vehicle need to predict a pedestrian's (or another vehicle's) future trajectory to avoid collision, where only the correct trajectory is required. Since motion intentions of agents cannot be determined by their historical positions (as described in introduction), multiple predictions are necessary to cover the most possible futures instead of multiple predictions as diverse as possible (or controllable predictions). \n\n    I believe it would be more appropriate to evaluate the proposed method in the tasks such as motion simulation in synthetic data, where diversity and controllability motion trajectory is useful to make the agents look real.", "limitations": "Authors discussed the limitations in the appendix.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "1. The authors analyze the limitations of current metrics ADE and FDE, which only focus on trajectory level prediction accuracy.\n\n2. The idea of controllability for trajectory prediction based on learned mixture Gaussian distribution is interesting, although I do not think it is an appropriate task to demonstrate the controllability (see details in Questions)", "weaknesses": "1. The mixture Gaussian assumption is not accurate.\n\n    Human trajectories are always affected by the interaction with other pedestrians and the environment. I agree that the possibility of trajectory conditional on i) historical trajectory ii) trajectory of other pedestrians iii) environment (e.g. maps) could be a mixture Gaussian distribution. However, all trajectories follow mixture Gaussian distribution without any conditions may not be an good assumption. Because the scenes in training and testing set are different. The learned distribution only fit on the training set with specific scenes, which cannot be generealized to different scenes.\n\n\n2. The proposed diversity metrics is not reasonable.\n\n    The diversity in multiple trajectory generation is necessary only on the premise that the generated trajectories make sense. The definition of APD and FPD only consider how different amont the generated trajectories but ignore whether the trajectories are reasonable or not. For example, 20 future trajectories are manually set to equally cover all directions from time step t_1, which will have very good score on APD and FPD but we cannot say these future trajectories are good predictions.\n\n\n3. This paper ignores lots of recent works in SOTA comparison. E.g.\n    Mangalam, Karttikeya, et al. \"From goals, waypoints & paths to long term human trajectory forecasting.\" ICCV. 2021.\n\n    Weibo Mao, et al.. Leapfrog diffusion model for stochastic trajectory prediction. CVPR, 2023.\n\n    Jiangbei Yue, et al. Human trajectory prediction via neural social physics. ECCV, 2022\n\n    Yonghao Dong,  et al. Sparse instance conditioned multimodal trajectory prediction. ICCV, 2023.\n\n    ...\n\n    Although authors mentioned that for some of the recent works, the processed data is slightly different. It would be more convincible to comapre to them with same training/testing data instead of ignoring these works as these works represent the most recent progress compared to original benchmarks such as socialGAN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Mixed Gaussian assumption:** Our motivation is the difficulty of transforming a single-modal and symmetric original distribution to a complex target distribution by normalizing flow. We seek a solution to relieve the difficulty. We choose the problem setting to better provide a fair quantitative evaluation protocol. In the area of trajectory forecasting, the problem setups are different. Most works do not consider the map information (at least on the benchmarks of ETH/UCY and SDD) while some others consider the map information. We agree that map information and other factors can be important in practice, but we focus on the simple convention without considering them because (1) we follow the convention with the most baseline works to compare with; (2) our innovation is mainly about enhancing the expressiveness of complicated target distribution of normalizing flow. We would like to choose the setting with the minimum necessary factors to avoid irrelevant inductive bias, such as map information encoder design in different baseline works. Adding other conditional information will steer our focus to a different problem setting, though worth studying for practical purposes, which may confuse the contribution and intention of our work.\n\n\n**APD/FPD not reasonable:** Without other information, such as the obstacle in the scene, we can't judge if a trajectory is `reasonable` or not. Our focus in this work is to design a method with a better diversity of the generation results by normalizing flow. The fundamental is to enhance the expressiveness when transforming to a complicated target distribution. I believe that can't say that the long-standing task of motion generation/forecasting is meaningless if no scene information is considered as it is a good platform to study the generative models' capacity as a probabilistic time series prediction task. Further adding other information can make the solutions from this task setting better fit with practical situations. \n\nWe agree that a single APD/FPD is not a perfect metric, it should be used along with the ADE/FDE. For example, ``equally cover all directions`` as a solution can reach high APD/FPD, but would probably cause low ADE/FDE which are the main metrics in the area of trajectory forecasting for many years.\n\nThe APD/FPD metrics are more inspiring through the new ablation provided in the pdf file (Table 1&2). It highlights that mixed Gaussian prior can significantly increase the diversity compared to the single Gaussian prior, while also boosting the accuracy measured by ADE/FDE even though the two sets of metrics are usually considered a hard trade-off.\n\n\n**Missing recent works:** We appreciate the mentioned recent related works. As we emphasize in the appendix, it is chaotic for benchmarking in the area of trajectory forecasting that different methods use different data pre-processing and evaluation conventions but compare together. We made intensive efforts to present the different conventions and choose the most widely adopted one for our benchmarking. Please refer to Appendix A in the supplementary for details. We did not include some mentioned related works in the main paper due to this reason:\n1. Ynet[1] and Yue, et al[2] use a different data convention from the widely adopted Social-Gan data convention. We explain this in Appendix A (Lines 575-577). So we can't include them in the benchmark.\n2. [3] only provides NBA dataset processing details. In their GitHub code, they only provide the training and evaluation on the NBA dataset. It is not able to know the details of evaluation/training on the ETH/UCY dataset. \n\nThere are some recent works that we should have included, such as [4,5]. We will add these works to the comparison.\n\nWe agree that it would be more convincible to compare more related works following their separate data pre-processing and evaluation conventions. However, this requires us to train and evaluate our model in at least seven different evaluation protocols. We choose the benchmarking convention following Social-GAN which includes most previous arts as we believe it has the best representativeness. Moreover, many works such as [3] provide no details, code, or processed data for us to know their data processing and evaluation protocol. Therefore, this is impossible for us to compare with these works fairly. We may benchmark under other data conventions if enough baselines can be gathered to provide a more complete sense of our method's performance.\n\n> [1] \"From goals, waypoints & paths to long term human trajectory forecasting\"\n\n> [2] \"Human trajectory prediction via neural social physics` share the same data with Ynet\"\n\n> [3] \"Leapfrog diffusion model for stochastic trajectory prediction\"\n\n> [4] \"Sparse instance conditioned multimodal trajectory prediction\"\n\n> [5] \"LG-Traj: LLM Guided Pedestrian Trajectory Prediction\"\n\n\n**Not an appropriate task:** Our goal is to design a generative model with good controllability, diversity, and invertibility and it is capable of modeling a complicated distribution with imbalanced training data. We understand the concern that diversity is not a high priority for practical trajectory forecasting cases (again, this goes back to the different opinions about whether the problem formulation is proper or worth studying). But we believe that it is a proper task for us to demonstrate our method because (1) the `best-of-M` evaluation protocol in this area provides the focus on both accuracy and diversity therefore capable of estimating whether our method can generate diverse results with faithful probability; (2) the task of trajectory prediction is a long-standing task in the computer vision community, thus provides a rich line of baselines to compare with. However, we agree that some other tasks, such as motion simulation and animation in a virtual environment, can also be good. They would be interesting future works to migrate our method into the graphics and animation communities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response (GR)
We thank all the reviewers for their valuable suggestions and comments. We add new experiments in the separate pdf file to assist our responses to the reviewers' questions. We would also resolve the writing issues mentioned by reviewers in the paper revision.

Meta-review: This paper introduces a normalizing flow (NF)-based approach for diverse trajectory prediction. In contrast to standard Gaussian priors used in NFs, the authors propose using a data-driven Gaussian mixture model (GMM), built from the analysis of trajectory patterns in the training dataset. The resulting Mixed Gaussian Flow (MGF) model is trained with both forward and inverse processes. The authors claim to include new diversity metrics, and experiments were conducted on the ETH/UCY and SDD datasets.

The paper initially received mixed reviews, with one weak acceptance recommendation (6), one weak rejection recommendation (4), and one rejection recommendation (3). The main concerns raised by the reviewers related to the justification of the method, the diversity metrics used in the submission, and several concerns regarding the experiments (e.g., relevance of the task, results with HD maps, comparison to other state-of-the-art methods). There was extensive discussion during the rebuttal period. The authors successfully addressed both theoretical and experimental concerns from reviewer 1BoH and reviewer qRNj, who respectively raised their ratings to 7 and 6, both recommending acceptance. However, reviewer pmqe maintained their initial recommendation, but did not respond to the authors' feedback.

The AC has thoroughly reviewed the submission and the discussions. The AC considers the problem of probabilistic forecasting to be highly important and appreciates the authors' efforts to fairly evaluate different methods. The proposed GMM data-driven prior is a simple yet relevant idea in the context of trajectory forecasting, and the forward and inverse processes are sound. In my opinion, a clearer positioning with respect to determinantal point processes (DPPs) [1] for representing diversity would have been beneficial. Additionally, the AC notes that the proposed diversity metrics have previously been introduced in the literature and urges the authors to properly credit related work—other diversity metrics, such as rF [2], have also been used. Therefore, the AC recommends acceptance but strongly encourages the authors to include the elements discussed during the rebuttal in the final paper.

[1] Diverse Trajectory Forecasting with Determinantal Point Processes. Ye Yuan, Kris Kitani. ICLR 2020.\
[2] Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding. S. H. Park, G. Lee, J. Seo, M. Bhat, M. Kang, J. Francis, A. Jadhav, P. P. Liang, and L. P. Morency. ECCV 2020.
Decision: Accept (poster)