BCDiff: Bidirectional Consistent Diffusion for Instantaneous Trajectory Prediction
Authors: Rongqing Li
Changsheng Li
Dongchun Ren
Guangyi Chen
Ye Yuan
Guoren Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Trajectory prediction
instantaneous observation

Abstract: The objective of pedestrian trajectory prediction is to estimate the future paths of pedestrians by leveraging historical observations, which plays a vital role in ensuring the safety of self-driving vehicles and navigation robots. Previous works usually rely on a sufficient amount of observation time to accurately predict future trajectories. However, there are many real-world situations where the model lacks sufficient time to observe, such as when pedestrians abruptly emerge from blind spots, resulting in inaccurate predictions and even safety risks. Therefore, it is necessary to perform trajectory prediction based on instantaneous observations, which has rarely been studied before. In this paper, we propose a Bi-directional Consistent Diffusion framework tailored for instantaneous trajectory prediction, named BCDiff. At its heart, we develop two coupled diffusion models by designing a mutual guidance mechanism which can bidirectionally and consistently generate unobserved historical trajectories and future trajectories step-by-step,  to utilize the complementary information between them. Specifically, at each step, the predicted unobserved historical trajectories and limited observed trajectories guide one diffusion model to generate future trajectories, while the predicted future trajectories and observed trajectories guide the other diffusion model to predict unobserved historical trajectories. Given the presence of relatively high noise in the generated trajectories during the initial steps, we introduce a gating mechanism to learn the weights between the predicted trajectories and the limited observed trajectories for automatically balancing their contributions. By means of this iterative and mutually guided generation process, both the future and unobserved historical trajectories undergo continuous refinement, ultimately leading to accurate predictions. Essentially, BCDiff is an encoder-free framework that can be compatible with existing trajectory prediction models in principle. Experiments show that our proposed BCDiff significantly improves the accuracy of instantaneous trajectory prediction on the ETH/UCY and Stanford Drone datasets, compared to related approaches.

BCDiff: Bidirectional Consistent Diffusion for Instantaneous Trajectory Prediction
Rongqing Li Beijing Institute of Technology lirongging9 90 gma com
Changsheng Li
Beijing Institute of Technology lesêbit. edu. on
Dongchun Ren ALLRIDE.AI
Guangyi Chen
CMU & MBZUAI guangyichen1992 @eaellgmmall..... com com
Bangchun.ren@alllee..
Ye Yuan Beijing Institute of Technology yuan-ye@bit... wwwabiit.euu.... cn
Guoren Wang
Beijing Institute of Technology wanggrbit@ 22126 com
Abstract
The objective of pedestrian trajectory prediction is to estimate the future paths of pedestrians by leveraging historical observations, which plays vital role in ensuring the safety of self-driving vehicles and navigation robots. Previous works usually rely sufficient amount of observation time accurately predict future trajectories. However, there are many real-world situations where the model lacks sufficient time to observe, such as when pedestrians abruptly emerge from blind spots, resulting in inaccurate predictions and even safety risks. Therefore, it is necessary to perform trajectory prediction based instantaneous observations, which has rarely been studied before. In this paper, we propose a Bi-directional Consistent Diffusion framework tailored for instantaneous trajectory prediction, named BCfiff. At its heart, we develop two coupled diffusion models by designing a mutual guidance mechanism which can bidirectionally and consistently generate unobserved historical trajectories and future trajectories step-by-step. to utilize the complementary information between them. Specifically, at each step, the predicted unobserved historical trajectories and limited observed trajectories guide one diffu- sion model to generate future trajectories, while the predicted futuree trajectories and observed trajectories guide the other diffusion model predict unobserved historical trajectories. Given the presence relatively high noise the generated trajectories during the initial steps, introduce gating mechanism learn the weights between the predicted trajectories and the limited observed trajectories for automatically balancing their contributions. By means of this iterative and mutually guided generation process, both the future and unobserved historical trajectories undergo continuous refinement, ultimately leading to accurate predictions. Essen- tially, BCDiff is an encoder-free framework that can be compatible with existing trajectory prediction models in principle. Experiments show that proposed BCDiff significantly improves the accuracy of instantancous trajectory prediction the ETH/UCY and Stanford Drone datasets, compared related approaches.
"Changsheng Li (les bit.edu.cn) is the corresponding author
37th Conference Neural Information Processing Systems (NeurIPS 2023).

---

Introduction
Pedestrian trajectory prediction aims to predict future trajectories conditioned on their past movements, which is an important task for autonomous driving [25] 57 and navigation robot [ Previous pedestrian trajectory prediction approaches usually rely on long enough observation time (typically,
to 3 seconds) for a pedestrian to precisely predict the future trajectories [53] 48
[4 [3]. However,
in many real-world situations, e.g., when pedestrians suddenly emerge from blind spots and are in close proximity to autonomous vehicles, traditional trajectory prediction methods do not have ample time to collect a sufficient number of locations. This leads to sub-optimal prediction performance and potentially unsafe behaviors in the decision- making of autonomous vehicles and robots. Therefore, it is quite necessary to forecast future trajectories based on limited or instantancous observations.
The prediction of instantancous trajectories for pedestrians is a highly challenging task due to the limited observation time. In some cases, as extreme as it can be, merely two frames of locations can be observed. In the face of such challenging task, there have been only a few academic works to date. MOE 46 is the first to propose the problem of instantaneous trajectory prediction, and thus is the most relevant to ours. MOE incorporates scene context information into limited observations and introduces the masked trajectory complement and context restoration as seff-supervised tasks to pre- train the model. However, since MOE only utilizes instantaneous temporal information acquired from limited trajectories, it might he hard to accurately predict the future trajectories of a pedestrian with complex behavior such as turning and yielding. DTO (34] focuses on lowering the influence of noise introduced by incorrect detection and tracking. and attempts to employ limited observed trajectory to alleviate this problem. It utilizes the knowledge distillation technique to distill knowledge from teacher model trained with an ample amount of long observations, and transfer the knowledge to a student model receiving fewer observations as input. Although these approaches have shown some effectiveness in instantaneous trajectory prediction, the representation of a pedestrian is restricted to two frames of locations, which contains extremely limited temporal information.
In this paper, we propose BCDiff, a bidirectional consistent diffusion framework specifically designed for the instantaneous trajectory prediction task. As we know, the diffusion model is a generative model, which has been successfully applicd to various generation tasks, including image synthesis [39 (3), image denoising [23], etc. Different from them, we leverage the diffusion model for instantancous trajectory prediction. We devise two coupled diffusion models to bidirectionally generate previous unobserved trajectories and future trajectories from random noises, which can address the issue of temporal information scarcity in limited observations. The underlying intuition behind this idea is: Both previous unobserved historical trajectories and future trajectories contain information of the same pedestrian at different timesteps, and thereby they provide complementary information to each other. It will be beneficial for the prediction of future trajectories if we can design elegant method to simultaneously generate previous unobserved historical trajectories and future trajectories by fully leveraging the complementary information between them.
To accomplish this, we devise a step-by-step mutual guidance mechanism in two coupled diffusion models to simultancously generate previous unobserved historical trajectories and future trajectories. Specifically, at each step, the predicted unobserved historical trajectories, together with the limited observed trajectories serve as guidance for one diffusion model to predict denoising intensity, which is then used to generate future trajectories of the subsequent step. Likewise, the predicted future trajectories, together with the observed trajectories, guide the other diffusion model predict unobserved historical trajectories of the next step. Meanwhile, considering there exists relatively high noise in the generated trajectories during the initial steps, devise gating mechanism to learn the weights between the predicted trajectories and the limited observed trajectories for automatically controlling the proportion of the guidance information from two kinds of trajectories during each generation step. Through this iterative and mutually guided generation process, the future and unobserved historical trajectories are continuously refined, ultimately leading to precise predictions. Notably, our proposed BCDiff is encoder-free and is compatible with existing trajectory prediction encoders in principle, allowing them to gracefully handle cases with instantaneous observations.
Our contributions can be summarized as follows: 1) We propose BCDiff, a diffusion model based framework tailored for instantaneous trajectory prediction. BCDiff can simultaneously generate both future and unobserved historical trajectories in a consistent manner, which can effectively leverage complementary information between them. 2) We devise step-by-step mutual guidance mechanism to couple two diffusion models for trajectory generation, and present gating strategy to adaptively
2

---

adjust the contributions of the guidance information between two kinds of trajectories. 3) Experiments demonstrate our proposed BCDiff significantly improves the accuracy of instantaneous prediction and outperforms the state-of-the-ar methods on ETH/UCY and Stanford Drone datasets.
2 Related Works
2.1 Traditional Trajectory Prediction
Traditional trajectory prediction methods aim to predict future trajectories given sufficient observation time. To capture complex interactions between pedestrians, many methods have been proposed 40]55 These models utilize social mechanism to aggregate neighboring actors and broadcast
information to each actor. In addition, graph neural networks [49] [T] 19, [2] [2 (8) and transformer
introduced to encapsulate implicit interactions among pedestrians. To resolve the problem of high uncertainty in pedestrians, researchers propose stochastic generative
architectures [52, 36 35 [1]
models, such as GAN [I [2 40
45 55],
VAE (25 24
B1 53,, and Diffusion Models [4] 32].
to better capture the variability in future trajectories. Various sampling strategies are designed to 30) However, the aforementioned methods accurately predict trajectories only when a sufficient amount of long observation trajectories are available. The accuracy cannot be guaranteed given the limited observation time. In contrast to these methods, our goal is to address the trajectory prediction problem under instantaneous observation scenarios.
avoid purely random sampling 6
2.2 Instantancous Trajectory Prediction
Instantaneous trajectory prediction aims to predict future trajectories given limited number of observed trajectory points. In the most extreme scenarios, merely two frames of locations can be observed. This task poses significant challenges due to the exceedingly short observation period. 34 are two trajectory prediction methods based on limited observed trajectory MOE [46] and DTO
points. MOE proposes feature extractor incorporate image semantic information and develops self-supervised task to enhance the representational ability of instantaneous observations. Mcanwhile, DTO investigates the influence of noise introduced by detection or tracking trajectory prediction, and intends to use fewer trajectory points in a knowledge distillation framework to address the issue. Despite these advances, they fail address the inherent lack of temporal information in instantaneous trajectory prediction. In our study, we attempt to predict and leverage previous unobserved trajectories to capture more temporal information, thereby improving the prediction accuracy of future trajectories.
2.3 Diffusion Models
The diffusion model is a class of stochastic generation models, which exhibits amazing performance in a diverse range of fields such as image synthesis 9 39 33 audio synthesis 77 21 and text generation 22. 12 8. Among these works, a typical diffusion model is the Denoising diffusion probabilistic model (DDPM) 44 16.. DDPM is inspired by the non-equilibrium thermodynamics, in which a forward Markov process perturbs real data into noise, and a reverse Markov process converts noise back to real data. DDPM has been widely used in various tasks, including image super-resolution [4] 3D point cloud generation (2 56) etc. For instance, CDM [17 cascades
multiple DDPM models to generate images of increasing resolution. The work in (2) employs
heat bath mechanism on DDPM to facilitate the generation of 3d point clouds. Different from these works, we leverage DDPM to solve the problem of instantaneous trajectory prediction.
3 Methods
3.1 Problem Formulation
In this work, we aim to tackle the task of instantaneous trajectory prediction, where we assume only two frames observed, i.c., the most extreme case. We denote Xobe {21, 22 } as the observation locations. The ground-truth future trajectory is symbolized as X = (IT, I4 ..., +2,, where Thut is the prediction length, and x; E RR² is the 2D coordinate of a trajectory location. Moreover, we characterize the previous unobserved trajectories as Xunobs = { ... 1-,, 20 where T Tumats represents the length of the unobserved trajectories. Our objective is to develop a diffusion
3

---

Enc-Dec Architecture
(b) Decoding Process of Bi-Directional Consistent Diffusion (BCDiff)
X obs
X
yuc
Encoder
Vobs
BCDUnit
BCDUnit
BCDUnit
Vobs
Decoder
X
umons
fut
unohs
X-1
X unobs
X unabs
Prut
0 fuc
X unobs
Bi-LSTM
frut
Gate
m
Fusion
Transformer
W fut
Efut
Lyut
Vobs
Xful
Bi-LSTM
Fusion
unabs
uuobs
0 uunbs
L
Lunobs
Punobs
Yuoobs
unobs
(c) Structure of BCDUnit
Figure 1: An illustration of our BCDiff framework. (a) The overall architecture comprises encoder and a decoder. The encoder is utilized to generate features of trajectories by incorporating social and scene context. The decoder is depicted in Figure (b). The decoder, i.e., our proposed BCDiff, generates previous unobserved historical trajectories and future trajectories step-by-ster through two coupled diffusion models. (c) The BCDUnit describes the details of two diffusion models marked in red and yellow. We design a mutual guidance mechanism and predict denoise intensities that are used for generating unobserved historical trajectories and future trajectories at the next step.
models based method to generate both previous unobserved historical and future trajectories, so that temporal information can be captured better predict future trajectories, given only two observed frames. Since we utilize the diffusion model, we denote the maximum diffusion steps as M, Fut and X represent the future and unobserved trajectories following a diffusion of m steps or a denoising of M m steps, respectively. Note that Xunobs X unobs and X X ful-
and use X
3.2 Overall Architecture
The overall architecture consists of encoder and a decoder, as shown in Figure (a). The encoder encodes Xobs as Vobs. and captures social and scene context e. The decoder is our proposed BCDiff framework, shown in Figure [ (b), which simultaneously generates previous observed historical trajectories and future trajectories step-by-step through the Bidirectional Consistent Denoising Unit (BCDUnit). The BCDUnit comprises two coupled diffusion models, As shown Figure (c). the diffusion model {Ørue, fur} marked in red color, is responsible for generating future trajectories, while the diffusion model (Bunots: )unabs), depicted yellow, is used for generating unobserved historical trajectories. The two diffusion models are coupled through a mutual guidance mechanism. To be specific, as the th step, the network ó == [drats Punobs) leverages two bidirectional LSTMs to encode unobserved trajectories and future trajectories mutual guidance 97n,. and ganabs used for
generating each other in the m th step. Considering the guidance information contains relatively high noise during the initial steps, we introduce a gate mechanism to balance the contributions between the observed guidance Toba and future guidance goutt as well as the observed guidance Vobs and unobserved guidance Iuppha: ultimately producing the appropriate guidance Ca. and Co.unobs- Then the network 6 = [(0 Ounobs fuses the guidance with both predicted unobserved historical and future trajectories current steps generate the denoise intensities, which used simultaneously generate X xm-1 and X X of the next step. It is noteworthy that both previous unobserved historical trajectories and future trajectories contain the information of the same pedestrian at different timesteps, thus we adopt parameter-share transformer across f ut and 6 fnnobs-
fut

---

3.3 Bidirectional Consistent Diffusion
In this section, we introduce our proposed BCDiff framework, which contains two coupled diffusion models to consistently generate trajectories in two directions, i.e. simultaneously predicting previous unobserved trajectories and future trajectories. In this paper, we utilize DDPM [T6] as our basic diffusion model, because of its excellent performance in various tasks. For the generation of each direction, the diffusion model executes diffusion and conditional denoising processes. The diffusion process aims to intentionally add a series of noises to ground-truth trajectory, while the conditional denoising process recovers the trajectory from noise inputs conditioned on the guidance.
The diffusion process is defined as Markov chain, conditioned on the ground- truth trajectories X fuee X umokss To write conveniently, we omit the subscripts, allowing X to represent unobserved historical trajectories or future trajectories. The diffusion process generates the sequence { by accumulating noise M times, i.e.,
Diffusion Process.
P=
q(x 1:M (
"(X"Ixm-1) [X"''"==),, g(X (X" m=1
II
= N(X"; Va
II),,, (1)
where N denotes the Gaussian distribution, and " represents the noise intensity from X X Typically, at is equal to 1 - -8"h where 8rhh is pre-defined value belonging to the interval [0,1].
Due to the additivity of Gaussian distributions, we are able to directly obtain X from X
M
x(x xx) = N Va x², 11-1-- (1-a"))), a
II
a
(2)
where T.t I a.. Note that as M becomes sufficiently large, o approaches to zero, making q(X M ) converge to the standard Gaussian distribution. By employing the above process, the ground-truth trajectory is transformed into the Gaussian noise X.M I).
Conditional Denoising Process. In this process, we aim to generate X from the Gaussian noise XM (0. We can reverse the aforementioned diffusion process, to gradually denoise from the Gaussian noise XM and reconstruct X Based on the proof in [TO]: If (X"""IXX") follows Gaussian distribution and 8rt is sufficiently small, q(X" "X also satisfies a Gaussian dis- tribution. Therefore, we formulate ( as Gaussian Markov process. However, is
(X
intractable to directly obtain q( (²). Consequently, we employ denoising neural network to estimate its mean and variance:
M II m=1
po 0:M cT) = p(x (c
pe X -
(Xm=1
| c,)),
(3)
pe pe(X Xm c. = N(
me (Xm m, c,)), Ee(X", ,,, c)).)
where = {0, {) is the parameter of the neural network. He and Eg are the predicted mean and variance by O. o is produced by network é, serving conditions to guide the denoising (we will introduce it in detail later). Our ultimate goal in the denoising process is to ensure that the denoising step becomes the inverse process of the diffusion step. thus enabling Pe mm X )
and 9 ( to have the same distribution. For more details, please refer to Section 3.4
Bidirectional Consistent Denoising Unit. To utilize more temporal complementary information in trajectories, we propose the BCDUnit to couple two diffusion models. One called the backward model, used to generate previous unobserved historical trajectories. The other, named as the forward model, is employed to produce future trajectories. We design mutual guidance mechanism in the two diffusion models: At each step, the predicted unobserved historical trajectories and observed trajectories are jointly utilized to guide the forward model to generate future trajectories of the next step. Likewise, the predicted future trajectories together with observed trajectories are responsible for guiding the backward model to generate the unobserved historical trajectories. In this way, the BCDUnit continuously refines the future and unobserved historical trajectories, ultimately leading to accurate predictions.
We denote the network in BCDUnit as θ = fd Ofut), ). (dunnbs Ounohs }}. As illustrated in Figure I (c), the forward model (Øfue; Ofut). marked as red, is used to generate future trajecto- rics, while the backward model {ounods, P Punobe) ). depicted in yellow, is responsible for generating
5

---

unobserved historical trajectories. Here, the guidance information can be obtained by the network f Punobs At the m step, first generates future guidance Iout and unobserved guid-
ance Fumoha by sending the unobserved historical trajectories umobs and future trajectories X fus to
Bidirectional Long- Shortt Term Memory (Bi-LSTM),, respectively:
fut Bi-LSTM unobs)) Punobs Bi-LSTM(
(5)
We further incorporate observed trajectory guidance Vobs into future and unobserved guidance obtain more informative guidance eg and c², Cojunobe. c
Ce.unobs Lowross Vobs. ml, where [.-]] represents the concatenation operation. However, considering that contains relatively high noise in the initial steps due to the inherent property of the diffusion model, is not appropriate directly concatenate them the beginning. To this end, we adopt gating mechanism to automatically learn the weights for balancing the contributions between two kinds of guidance information. We first calculate the weights for the future guidance gout and observed trajectory guidance Voia produce appropriate guidance. Formally,
Lofuee Vobe, ml. t
Yftt Gate (( Voba: ml),
(6)
where Fut is the learnt weight. Gate is a two layers MLP with the Sigmoid activation in this paper.
The guidance em fut can be then obtained by:
m ffffut Jut + (1 - (fut)
> Vobe-
Coaunobs can be obtained the same way. After obtaining fur and cm Couuohs: the network 0 {0fut; Ounoha) utilizes them to perform mutual guidance. They fuse guidance with the predicted trajectories at the mh step. Then, a transformer is leveraged to capture temporal dependencies in the fused features. Finally, the outputs of the transformer are passed through two MLPs, i.e., fus and W Wanobs. to obtain denoising intensities euu and em anods respectively. These denoising intensities are employed to generate trajectories of the next steps, i.e., x and X
Similarly, cm
In this way, we can bidirectionally and consistently generate unobserved historical trajectories and future trajectories step-by-step. effectively utilizing the complementary information between them.
3.4 The Objective Function
We define the objective as the negative log- likeeiid of the model pe under X and X
unobs as
L E[-loope (X")).
(
Here, we also omit the subscript for convenient writing. By minimizing the objective C. the original trajectories Xu and X X can be recovered through the denoising process. However, it is difficult directly compute L. Therefore, we employ the variational methods to derive the Variational Lower Bound (VLB) (20 of the expectation, denoted as:
L < LLVLB Eg log x(x11MM ( = LL [x'))
pe (
(ogpee(X' cb) + Σ ( 1 [ cm))].
The first term of LVLB approximates to 0, as both (XX M (X and pel ( X 1c are approximate to (0, J). The second term can be formulated as special case of the third term when m = The third term computes the KL divergence between the estimated distribution pe pe(X"* (X"=I| ,c and the true posterior distribution distribution and the ground- -ruth posterior distribution. To determine the q X)),, apply the Bayes formula as follows:
"I""", aiming to lower the error between the estimated
q( {(" ((m=-1199) = g (X=IIX=-) q( 9(x
((x=11 1x x²) =
X
(9)
q(Xm) (0)
9(8m) ((X)XX0)

---

By applying the Bayes formula, we observe each term can be calculated with Equation 2 We then substitute the results of Equation 2 to Equation 9 to obtain the mean and variance of the posterior distribution q( q( X
1-" ) as:
mm 1 1 a 8m ("(XX"XX) Va am a
3m
X
a XO.
(10)
Note that 6thh is constant value related to 8th thus the KL in the third term can be further derived as:
(T(g(("='(X"") l 1xx"""1X"" "px"m )) x ||7" (Xm x²)
ue (X"
"x.",m ,m, mn,"")ll
(11)
By reparameterizing μ4 and substituting into Equation 11 we can further simplify the expression and finally obtain the diffusion loss.
Lunabs = Ex /.mlle e - E X X unober m. em (o.nobb)ll2
fut = Ex R.lll. re X suc m. f.r
La = Liut + Lunobs
(12)
where (0, T), X fut and Xm unobs are calculated by Equation and ée represents the aforemen-
tioned BCDUnit.
3.5 Mutual Guidance based Optimizing and Inference
When optimizing the model, take the following four steps: Firstly, sample future ground- truth trajectories fut- instantaneous observations Xoss, and unobserved historical ground-truth trajectories X ynobs from the training dataset. We also sample timestep
(nii and
a Gaussian noise ~ N NO, I). Secondly, we use the encoder to encode Xobs as the Vobs. and employ the Equation 2 to obtain the fue and X Moobs Thirdly,
utilize the Equation 5 and to obtain
mutual guidance c cpuunbs and c fu. Finally, we calculate the loss defined in Equation 12 and take the gradient descent to optimize the model until it converges.
For inference, first sample instantaneous observations Xobs from the testing data and two Gaussian noises X fut and X.M from NO I). Following the third step in the optimizing stage. obtain mutual guidance c.M fut and MM "ounoos' Finally, we execute the following two updates from m M to 1 so that the predicted trajectories can be continuously refined until X fut and Xu unobe are generated,
5T
X mm-1_ Fut
=
Ver
(& fut
VI
Eo
(Xm Futs m,
7h f.,ftt) + -
(13)
8m
wnobs
(Am
unohs
Eo unobes m, m c")unohs)) T
(14)
m
where is a random noise sampled from the standard Gaussian distribution. The details of training and inference procedures are provided in Appendix.
4 Experiments
4.1 Experiment Settings
Dataset. We verify the effectiveness of our proposed method on the widely used ETH/UCY [37 26 and Stanford Drone (38) Dataset (SDD). ETH/UCY is dataset group. It consists of 5 different scenes, among which scenes (ETH, HOTEL) are from the ETH dataset, and the other three scenes (UNIV, ZARAI, and ZARA2) come from the UCY dataset. The whole dataset includes more than 1500 pedestrians. We follow the widely used leave-one oonse--ne-ccee-out protocol, i.e., the models are trained on 4 scenes and tested on the remaining scenes. It contains various agents such as pedestrians, bicycles, and vehicles.
[1] 42
SDD is large scale dataset consisting of 20
we employ the Average Dis- placement Error (ADE) and Final Displacement Error (FDE) as metrics to evaluate the performance of future trajectory predictions. In the instantaneous trajectory prediction setting, the observations
Evaluation Metrics. Following previous works
43
19
14
53
48

---

Table 1: Comparisons of different methods on the ETH/UCY dataset. The metrics are presented as ADE/FDE (m).
Dataset
Model
Methods
ETH
HOTEL
UNIV
ZARAI
ZARA2
AVG
Instantaneous
0.76/1.43
0.30/0.56
0.36/0.74 0.22/0.42
0.18/0.34
0.36/0.70
MOE [46]
0.64/1.12
0.20/0.33
0.33/0.62
0.22/0.42
0.17/0.32
0.31/0.56
Trajectron++ MOE w/o Image
0.68/1.22
0.25/0.49
0.35/0.68
0.22/0.42
0.18/0.33
0.34/0.63
DTO 34
0.70/1.23
0.22/0.45
0.32/0.62
0.22/0.42
0.17/0.33
0.33/0.61
BCDiff
0.61/1.09
0.16/0.28
0.28/0.53
0.22/0.41
0.18/0.33
0.29/0.53
Instantaneous
0.34/0.65
0.14/0.25
0.31/0.63
0.23/0.46
0.16/0.37
0.24/0.47
MOE [46]
0.31/0.57
0.13/0.21
0.25/0.53
0.20/0.41
0.14/0.31
0.20/0.41
PCCSNet
MOE w/o Image
0.32/0.61
0.14/0.24
0,28/0.57
0.21/0.45
0.15/0.34
0.22/0.44
DTO 34
0.33/0.64
0.14/0.24
0.31/0.62
0.22/0.46
0.15/0.35
0.23/0.46
BCDiff
0.30/0.56
0.13/0.20
0.25/0.52
0.18/0.37
0.14/0.31
0.19/0.39
Instantancous
0.88/1.66
0.55/1.16
0.38/0.71
0.30/0.54
0.25/0.46
0.47/0.91
MOE 46
0.74/1.41
0.45/0.85
0.38/0.71
0.29/0.54
0.25/0.45
0.42/0.79
SGCN
MOE w/o Image
0.79/1.52
0.50/1.05
0.38/0.71
0.30/0.54
0.25/0.46
0.44/0.85
DTO 34]
0.80/1.56
0.49/1.02
0,38/0.71
0.30/0.54
0.25/0.46
0.44/0.86
BCDiff
0.66/1.18
0.34/0.62
0.38/0.70
0.30/0.54
0.25/0.44
0.39/0.72
Instantancous
0.64/1.10
0.21/0.34
0.27/0.51
0.22/0.39
0.18/0.34
0.30/0.54
MOE [46
0.57/1.01
0.17/0.29
0.26/0.44
0.22/0.36
0.17/0.32
0.28/0.48
SocialVAE
MOE w/o Image
0.59/1.06
0.20/0.33
0.27/0.49
0.22/0.38
0.18/0.33
0.29/0.52
DTO 34
0.61/1.06
0.18/0.31
0.25/0.43
0.22/0.38
0.17/0.33 0.29/0.50
BCDiff
0.53/0.91
0.17/0.27
0.24/0.40 0.21/0.37 0.16/0.26 0.26/0.44
reduced to 2 frames, and the length of future predictions is 12 frames. Following previous works 46 15 31,, we sample 20 future predicted trajectories, and report the final error by the minimum error over all predicted trajectories.
Backbone and Baselines. To demonstrate the compatible ability of our BCDiff, we apply it to three popular trajectory prediction models, Trajectron++ 42), PCCSNet 47). SGCN [3], and Social [50] by replacing their decoders with our BCDiff. Moreover, we compare BCDiff with the following baselines:
Instantaneous means directly predicting the trajectories of the next 12 frames conditions on 2 frames of observations using the above three backbones. Additionally, we take MOE (46 and
DTO 33 as two baselines for instantaneous trajectory prediction. Since the original MOE employs image semantic information, we also implement MOE without using image semantic information, denoted as MOE w/o Image, in order to fairly compare with other methods.
4.2 Experiment Results and Analysis
Performance Instantaneous Trajectory Prediction. The overall performance are listed in Table [ The results by applying our BCDiff to three different backbones, consistently outperform the baseline methods on all the datasets. This demonstrates the effectiveness of our proposed method for instantancous trajectory prediction. Mcanwhile, it also illustrates our method can be well compatible with different trajectory prediction models. Note that the performance of MOE declines when image information is not included, which shows that MOE heavily depends on image semantic information. In contrast, our proposed method does not rely on any additional image information.
A Ablation Study. We perform the ablation studies, as listed in Table 3 We first utilize two separate diffusion models to predict previous unobserved trajectories and future trajectories, respectively, denoting it as guide the generation of future trajectories, denoted BCDiff- Unii Guidance. BCDiff- UCDiff-Uni. Guidance is better than BCDiff-w/o, wu.dia-w/..uuidnneee, meaning that the guidance from previous unobserved historical trajectories helpful for improving the prediction performance of future trajectories. Moreover, by
httpt//wo....dcc..e.. Then, we only utilize unobserved and observed trajectories to
8

---

Table 2: Comparisons of different methods the Stanford Drone dataset. The metrics presented as ADE/FDE (m).
Methods
Trajectron++
PCCSNet
SGCN
SocialVAE
Instantaneous
13.07/22.88
9.19/17.71
15.40/25.69
9.56/16.10
MOE|46
11.71/19.54
8.40/16.08
14.45/24.88
9.12/14.98
MOE w/o Image
12.41/21.46
8.87/1680
15.02/25.13
9.39/15.87
DTO
12.32/20.79
8.93/16.92
14,99/25.07
9.28/15.58
BCDiff
11.56/19.32
8.32/15.87
13.67/23.92
9.05/14.86
Table Ablation Studies using Trajectron++ on the ETH/UCY and Stanford Drone datasets.
Method
ETH/UCY
SDD
BCDiff-wwo.Guidance
0.36/0.70
13.07/22.88
BCDiffi Guidance
0,33/0.60
12.57/21.26
BCDiff-Bi.Guidde Guidanee
0,32/0.57
11.96/19.88
CCDiff-Bi.Guidancc & Gate
0.29/0.53
11 11.56/19.32
incorporating bidirectional guidance, called BCDiff-Bi PeDiff-Bii.uun Guidance we observe the performance is further improved. This illustrates the guidance from future trajectories is beneficial for predicting previous unobserved trajectories, thereby improving the final prediction performance. Finally, we integrate the gating mechanism into model, named BCDiff-Bi. Gu.idancce & Gate. It achieves the best performance, showing the gating mechanism is effective for trajectory prediction.
Analysis on Length of Predicted Unobserved Points. We investigate the impact of the number of predicted unobserved points on trajectory prediction. use the Trajectron++ encoder [42] as listed in Table g As the number of predicted points increases, the accuracy of future predictions gradually improves, which can be attributed to the additional unobserved points providing more usefull information. When T = 5, the prediction performance starts to decline. This is because it becomes unprecise when predicting previous unobserved historical points with larger length, thus introducing noise into method.
Table 4: Analysis of different unobserved point lengths T Tunobs
Dataset
Direction
Tunobs
Tunoha 2
T Tanobs 3 Tunoba 4 Tunots
T Tunobe 6
Future
0.31/0.55
0.30/0.54
0.30/0.53
0.29/0.53
0.32/0.57
0.34/0.60
ETH/UCY
Unobserved
0.006/0.006
0.034/0.033
0.055/0.063
0.086/0.089
0. 108/0.133
0.149/0.212
Future
11 11.86/19.78
11.81/19.71
11.68/19.50
11.56/19.32
11.92/20.01
12.33/20.98
SDD
Unobserved 0.311/0.312 1.564/1.492
2.626/2.678
3.552/3.884
4.828/5.423
6.177/7.894
Qualitative Analysis. We visualize the predicted trajectories in four different scenarios: Walking side-by-side, walking along, turning and yielding, as shown in Figure? BCDiff can accurately predict future trajectories, compared to MOE and DTO in all four scenarios. This because BCDiff utilizes temporal information of unobserved trajectories to aid the prediction of future trajectories. Note that when pedestrians are walking side-by-side, as depicted in the first column of Figure 2 BCDiff utilizes the predicted un- observed historical trajectories to capture the walking patterns between pedestrians, thereby precisely forecasting future tra- jectories. However, DTO and MOE predict deviated trajectories, due only using two observed frames. Moreover, visualize the gating weights Tut and ym in the denoising process from m = 40 to m = 1. As shown in Figure 3 the weights are gradually increased. It indicates our gating mechanism can adaptively learn weights, assigning lower weights to the predicted guidance in the initial steps.
m
Fut
30
20
10
77
mn Tinado
40
30
20
10
Figure 3: weights fuur and Twnahs- Darker colors represent larger values.
Visualization of m

---

b)
Figure 2: Visualization of predicted trajectories on the ETH/UCY Dataset. Given the instantancous observed trajectories (red), we predict the future trajectories (green) by (a) MOE, (b) DTO and (c) Our BCDifT. The ground-truth future trajectories the predicted unobserved trajectories (blue) by our BCDiff and ground-truth (cyan). Our predicted future trajectories are closer the ground-trute compared to other methods.
shown in orange color. In addition, we also draw
5
Conclusion
We proposed a diffusion model based framework for the task of instantaneous trajectory prediction. The proposed framework simultaneously generated both unobserved historical and future trajectories by designing a mutual guidance mechanism to couple two diffusion models, such that more temporal information be leveraged for future trajectory prediction. We introduced gating strategy, auto- matically balancing the contributions different guidance information. Experiments demonstrated our proposed framework achieved superior performance the state-of-the-art methods.
Acknowledgments and Disclosure of Funding
This work was supported by the NSFC under Grants 62 1220113, U2001211. This work was also supported by the Innovative Development Joint Fund Key Projects of Shandong NSF under Grants ZR2022LZH007.
References
[1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robiequet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-971, 2016.
[2] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances Neural Information Processing Systems,
2021.
[3] Inhwan Bae, Jin-Hwi Park, and Hae-Gon Jeon. Non-probability sampling network for stochastic human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6477-6487, 2022.
[4] Changan Chen, Yuejiang Liu, Sven Kreiss, and Alexandre Alahi. Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning. In 2019 international conference robotics and automation (ICRA), pages 6015-6022, IERE, 2019.
10

---

[5] Guangyi Chen, Zhenhao Chen, Shunxing Fan, and Kun Zhang. Unsupervised sampling promot- ing for stochastic human trajectory prediction. arXiv preprint arXiv:2304.0422 04299,, 2023.
[6] Guangyi Chen, Junlong Li, Nuoxing Zhou, Liangliang Ren, and Jiwen Lu. Personalized trajec- tory prediction via distribution discrimination. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15800-15899 2021.
[7] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.0077 2020.
[8] Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208. 04200, 2022.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-877 0794,, 2021.
[10] William Feller. On the theory of stochastic processes, with particular reference to applications. In Selected Papers I, pages 769-798. Springer, 2015.
[11] Jiyang Gao, Chen Sun, Hang Zhao, Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11525-11533, 2020.
[12] Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq; Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210008993, 2022.
[13] Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15303-15312, 2021.
[14] Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao. Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17113-17122 2022.
[15] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2255-2266 2018.
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 118840-6551,, 2020.
[17] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23(47):1-333 2022.
[18) Xin Huang, Stephen G McGill, Jonathan A DeCastro, Luke Fletcher, John Leonard, Brian C Williams, and Guy Rosman. Diversitygan: Diversity-aware vehicle motion prediction via latent semantic sampling. IEEE Robotics and Automation Letters, 5(4):5089-5066 2020.
[19] Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial-temporal interactions for human trajectory prediction. In Proceedings of the IEFF/CVF international conference computer vision, pages 6272-6281, 2019.
[20] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13Xiv:1312.61 14, 2013.
[21] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.0970 097661,99.77, 2020.
[22] Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. Advances in Neural Information Processing Systems, 32, 2019.
11

---

[23] Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. Sinddm: A single image denoising diffusion model. arXiv preprint arXiv:2211.16582, 2022.
[24] Mihee Lee, Samuel S Sohn, Seonghyeon Moon, Sejong Yoon, Mubbasir Kapadia, and Vladimir Pavlovic. Muse-vae: multi-scale vae for environment-aware long term trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2022.
[25] Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philipp HS Torr, and Manmo- han Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 336-345, 2017.
[26] Alon Lemnrr, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. In Computer graphies forum, volume 26, pages 655-664 Wiley Online Library, 2007.
[27] Jiachen Li, Fan Yang. Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. Advances in neural information processing systems, 33:19783-19794, 2020.
[28] Xin Li, Xiaowen Ying. and Mooi Choo Chuah. Grip++: aware trajectory prediction for autonomous driving. arXiv preprint 1907iv/907..00 07779977, 2019.
Enhanced graph-based interaction-
[29] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[30] Yecheng Jason Ma, Jeevana Priya Inala, Dinesh Jayaraman, and Osbert Bastani. Likelihood- based diverse sampling for trajectory forecasting. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 13279-13288, 2021.
[31] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part 11 16, pages 759-776. Springer, 2020.
[32] Weibo Mao, Chenxin Xu, Qi Zhu, Siheng Chen, and Yanfeng Wang. Leapfrog diffusion model for stochastic trajectory prediction. arXiv preprint arXiv:2303. 1009955 2023.
[33] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
[34] Alessio Monti, Angelo Porrello, Simone Calderara, Pasquale Coscia, Lamberto Ballan, and Rita Cucchiara. How many observations are enough? knowledge distillation for trajectory forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6553-6562, 2022.
[35] Nigamaa Nayakanti, Rami AI-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Ben- jamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. arXiv preprint 2wxiv:2207.058444 2022.
[36] Jiguan Ngiam. Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao- Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In International Conference on Learning Representations, 2022.
[37] Stefano Pellegrini, Andreas Ess, and Lue Van Gool. Improving data association by joint modeling of pedestrian trajectories and groupings. In Computer Vision-ECCV 2010: 1ith European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part I 1I, pages 452-465. Springer, 2010.
12

---

[38] Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In Computer AsionnEECCS 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 11-14, 2016, Proceedings, Part VIII 14, pages 169-565..0 Springer, 2016.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[40] Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 1349-1358, 2019.
[41] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[42] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasiin trajectory forecasting with heterogeneous data. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16. pages 683-700 Springer, 2020.
[43] Liushuai Shi, Le Wang, Chengjiang Long, Sanping Zhou, Mo Zhou, Zhenxing Niu, and Gang Hua. Sgen: Sparse graph convolution network for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8994-9003, 2021.
[44] Jascha Sohl- Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages
PMLR, 2015.
[45] Hao Sun, Zhiqun Zhao, and Zhihai He. Reciprocal learning networks for human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7416-7425, 2020.
[46] Jianhua Sun, Yuxuan Li, Liang Chai, Hao-Shu Fang, Yong-Lu Li, and Cewu Lu. Human trajectory prediction with momentary observation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 6467-6476, 2022.
[47] Jianhua Sun, Yuxuan Li, Hao-Shu Fang. and Cewu Lu. Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13250-13259, 2021.
[48] Conghao Wong, Beihao Xia, Ziming Hong, Qinmu Peng, Wei Yuan, Qiong Cao, Yibo Yang, and Xinge You. View vertically: A hierarchical network for trajectory prediction via fourier spectrums. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 682-700. Springer, 2022.
[49] Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2022.
[50] Pei Xu, Jean-Bernard Hayet, and Ioannis Karamouzas. Socialvae: Human trajectory prediction using timewise latents. In European Conference on Computer Vision, pages 511-528. Springer, 2022.
[51] Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pages 507-523. Springer, 2020.
13

---

[52] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transform- ers for socio-temporal agent forecasting. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9813-9823, 2021.
[53] Jiangbei Yue, Dinesh Manocha, and He Wang. Human trajectory prediction via neural social physics. In Computer Vision-ECC ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV, pages 376-394. Springer, 2022.
[54] Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp. Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. In Conference Robot Learning, pages 895-904. PMLR, 2021.
[55] Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12126-1215 12134, 2019.
[56] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835, 2021.
[57] Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8823 8833, 2022.
14


Reviews:
[{"review_id": "CUABMQEeQq", "reviewer": "Reviewer_GWuJ", "summary": "The authors propose a bidirectional consistent diffusion framework for instantaneous trajectory prediction. Specifically, two coupled diffusion models are employed to predict unobserved historical and future trajectories jointly. In addition, a gating mechanism is presented to deal with the noisy situation at the initial steps. Experiments on ETH-UCY and SDD show that the proposed method is able to achieve the promising performance.", "questions": "My questions are raised in the above Weaknesses Section.", "limitations": "I didn't see any limitations discussed in the submission.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear-written and organized.\n2. Promising results on several datasets.", "weaknesses": "1. I am concerned about the consistency between the discussed motivation and the experiments. In the Abstract and Introduction Section, the motivation is that \"there are many real-world situations where the model lacks sufficient time to observe\". However, in the Experiments Section, the evaluation protocol is the commonly used leave-one-out setting, which is somewhat inconsistent with motivation. I assume the training and the inference phase both take only 2 frames as the input (this is another unclear point), which makes the  \"real-world situation\" that the training scenario lacks adequate observation and the testing scenario also lacks adequate observation.\n\n2. In addition to the above point, another concern is that only human trajectories are predicted. While such real-world situations often happen in the driving scenario.\n\n3. To better demonstrate the motivation, some motivated experiments could be included, i.e., existing methods fail to make accurate predictions when the observations are not adequate.\n\n4. In Lines 60-62, it claims that jointly predicting unobserved historical trajectories and future trajectories can be beneficial to each other. This claim is very similar to [1]. However, there is no experiment on this point, such as conducting these two jobs separately, or first predicting unobserved historical trajectories and then predicting future trajectories based on the predicted historical results.\n\n5. In the Method Section, LSTM is used as in Eq.5, why don't use the Bi-directional LSTM?\n\n6. Some details are missing. In my opinion, there is a gap between the training and the inference phase. During the training, the method has access to the future gound-truth trajectory, in other words, there could have a forward and backward process. While in the inference phase, the method mustn't have access to future information, and the details of how to evaluate the trained model are missing.\n\n\n\n[1] Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction. CVPR'23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: I am concerned about the consistency between the motivation and experimental setting. \n\nA1: Thanks for raising this concern which helps us to clarify the experimental settings. We guess this concern is caused by the misunderstanding on the \"commonly used leave-one-out (LOO) setting.\" Here, the LOO setting in our work is a protocol for dataset split, which has no influence on input length during inference. Instead of using 8 frames as observations input for the model, we reduce them to 2 frames in both the training and inference. Please refer to Lines 261-270 for more details. \n\nAdditionally, our motivation \"There are many real-world situations where the model lacks sufficient time to observe\" indicates that the observation is inadequate during inference but is not related to the training process. We can collect adequate trajectory points in an offline manner in training. But merely 2 points can be observed in inference in our setting.\n\nOverall, the experiment setting is consistent with the motivation. \n\nBesides, please kindly note that the setting is the same as that of MOE [2], the first work in instantaneous trajectory prediction.\n\n>Q2: Another concern is only human trajectories are predicted.\n\nA2: Thanks for your advice. Pedestrian trajectory prediction holds significance in autonomous driving [4], and the past decade has seen a surge in research efforts [5]. Our primary focus here is on pedestrian trajectory prediction. Diverging from prevailing approaches, we address a more challenging scenario, where only 2 frames are observed to predict future trajectories.\n\nWe're now actively extending our method to predict trajectories of other moving agents like vehicles.\n\n>Q3: To better demonstrate the motivation, motivated experiments could be included, i.e., existing methods fail due to insufficient observations.\n\nA3: Following your suggestion, we conducted experiments using Trajectron++ and SGCN for better motivation. The results are listed in Table 1 of the PDF in Global Response. The Standard employs 8 frames to predict the next 12 trajectory points, while Instantaneous employs just 2 frames. Results reveal major drops when observed points decrease (e.g., 0.65 to 0.91 for SGCN on ETH/UCY). This accentuates the need for instantaneous trajectory prediction methods in autonomous driving. Remarkably, with only 2 frames as input, our BCDiff significantly enhances performance (e.g., 0.91 to 0.72 for SGCN on ETH/UCY).\n\n>Q4: It suggests that predicting historical and future paths can mutually benefit. This resembles [1], but lacks experiments, like separate predictions or sequential forecasting.\n\nA4: Thanks for your comments. We've incorporated discussions and citations in our latest revision. Our work diverges from [1] in terms of tasks and technical contributions. Our focus lies in addressing instantaneous trajectory prediction (with only 2 observed frames), whereas [1] deals with predicting future trajectories amidst occlusion (observation points are missing). In terms of technical contributions, we design two coupled diffusion models and propose a stepwise mutual guidance mechanism for generating both unobserved and future trajectories. In contrast, [1] employs a unified GCN model for predicting both missing and future trajectories.\n\nActually, we have verified the efficacy of joint prediction in Table 3 of the submission. BCDiff-w/o.Guidance means two diffusion models for separate predictions. BCDiff-Bi.Guidance integrates the proposed bidirectional guidance mechanism for joint prediction. BCDiff-Bi.Guidance is better than BCDiff-w/o.Guidance, showing the efficacy of joint prediction.\n\nFollowing your suggestion, we add an experiment to sequentially forecast unobserved historical trajectories and future trajectories, denoted as BCDiff-two-step in Table 2 of the PDF in Global Response. BCDiff-Bi.Guidance is better than BCDiff-two-step, further indicating the superiority of joint prediction.\n\n> Q5: Why use LSTM in Eq.5, instead of the Bi-directional LSTM?\n\nA5: Sorry for confusing you. The LSTM in Eq.5 is the bi-directional LSTM. We will update it in the final version.\n\n>Q6: Details are lacking regarding the gap between training and inference. Training uses future ground-truth, but inference should be future-blind. Evaluation for the trained model is also unspecified.\n\nA6: Thank you for pointing out this concern which has improved the readability of our paper. Our method utilizes future ground-truth trajectories solely in training but doesn't use them in inference.\nWe guess the misunderstanding is caused by the diffusion model [3] we employed. In training, we engage in both forward and backward processes. The former ensures ground-truth supervision, while the latter focuses on trajectory prediction.\nIn inference, only backward is used for predicting future trajectories. We first initialize predicted future trajectories using Gaussian noises and exploit the trained network to gradually denoise $M$ times to obtain the final predicted trajectories. Please kindly refer to Algorithms 1 and 2 in [3] for the gaps between the training and inference in detail. We also include our training and inference details to clarify the gaps between them in Section C in the Appendix. \n\nTo evaluate it, we use the trained network to predict future trajectories based on 2 observed points. Then, we calculate ADE & FDE between predicted and ground-truth future trajectories as the performance of the model.\n\n>Q7: I didn't see limitations in the submission.\n\nA7: We put the limitations in Section B of Appendix.\n\nReference\n\n[1] Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction. CVPR'23\n\n[2] Human Trajectory Prediction with Momentary Observation. CVPR'22\n\n[3] Denoising Diffusion Probabilistic Models. NeurIPS'20\n\n[4] Codes for View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrums. ECCV'22\n\n[5] Human Motion Trajectory Prediction: A Survey. ICRR'20"}, {"review_id": "Wo00UQABDk", "reviewer": "Reviewer_ytbj", "summary": "The paper presents a bi-directional consistent diffusion approach for handling instantaneous trajectory prediction, i.e. trajectory prediction without requiring a large number of observation time. It relies on two coupled diffusion models to generate historical and future trajectories. The forward diffusion model predicts future trajectories conditioned on the predicted trajectories of the other model and the limited observed trajectories, while the backward model predicts the historical trajectories guided by the predicted future trajectories and the observed trajectories. Since the amount of noise in the first few diffusion steps is inherently relatively high, a gating mechanism is introduced to control the contribution of the two types of trajectories. In the evaluation, the approach achieves state-of-the-art performance on the TH/UCY and Stanford Drone datasets by replacing the decoder of three popular trajectory prediction models with the proposed one.\n\n\n\nHaving read the reviews and the rebuttal, I am going to maintain my positive assessment. The rebuttal addressed a lot of concerns, especially the experimental ones.", "questions": "- Clarify on the possibility to perform standard trajectory prediction and also discus the hyper-parameter selection.", "limitations": "Not applicable.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea and method of the paper are easy to follow.\n\n- The use of diffusion models for trajectory generation is novel and interesting.\n\n- The approach works even with very sparse observations based on the experimental evaluation.\n\n- There are extensive experiments with different backbone architectures and datasets. \n\n- The proposed approach beats the current state-of-the-art in almost every experiment.", "weaknesses": "- The task of instantaneous trajectory prediction is not standard. It would value for the approach to be evaluated for the standard trajectory prediction protocol. Then it would be allowed to be compared with a wide range of methods. This is a particularly interesting point since the diffusion models are heavy in terms of parameters. It would allow for a comparison with the current state-of-the-art methods. \n\n- It is not clear how many observations are being used during the test time, for example, if it is $M = 7$, then the approach does not seem to improve the current state-of-the-art in trajectory. prediction. Moreover, it would be interesting to have an ablation study on the influence of step M on performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: The task of instantaneous trajectory prediction is not standard. It would value for the approach to be evaluated for the standard trajectory prediction protocol. Then it would be allowed to be compared with a wide range of methods. This is a particularly interesting point since the diffusion models are heavy in terms of parameters. It would allow for a comparison with the current state-of-the-art methods.\n\nA1: Thanks for your suggestions. We would like to kindly remind you that there might be a misunderstanding regarding our paper caused by the problem formulation. The standard trajectory prediction takes 8 frames as observations to predict future trajectories. Instead, we focus on utilizing merely 2 frames of observations to predict future trajectories, which strictly follows the pioneering work MOE [1] for instantaneous trajectory prediction. Therefore, due to the different settings in terms of the number of observation points, it is unfair to compare instantaneous trajectory prediction methods with standard trajectory prediction methods. We will refine the problem formulation to be clearer in the final version.\n\n>Q2: It is not clear how many observations are being used during the test time, for example, if it is $M=7$, then the approach does not seem to improve the current state-of-the-art in trajectory prediction. Moreover, it would be interesting to have an ablation study on the influence of step $M$ on performance.\n\nA2: We apologize again for confusing you, and we'd like to kindly remind you that the notation $M$ you mentioned in the Question denotes the maximum diffusion steps in the diffusion mode, instead of the number of observation points. \n\nThen, as we mentioned in the A1 that we focus on the instantaneous trajectory prediction, where 2 frames of observations are input, studying trajectory prediction with the arbitrary number of observation points as input is beyond the research scope of this paper. We are instead committed to improving accuracy in the instantaneous setting with only two observation points. \n\nHowever, as you suggested, using the arbitrary lengths of observed trajectories to predict future trajectories is interesting to study. We are willing to explore the topic in future works. \n\n\nReference\n\n[1] Human Trajectory Prediction with Momentary Observation. CVPR'22"}, {"review_id": "hrvEnfhkaQ", "reviewer": "Reviewer_VvSx", "summary": "This paper focuses on the trajectory prediction in an instantaneous observation by generative diffusion models. Specifically, the authors propose a bidirectional consistent diffusion framework, named BCDiff, to address the issue of temporal information scarcity. BCDiff employs two coupled diffusion models are devised to predict unobserved historical trajectories and future trajectories with a mutual guidance mechanism, which ensures the consistency between predicted unobserved trajectories and predicted future trajectories.  The authors conduct extensive experiments to show the effectiveness of proposed method.", "questions": "As shown in WEAKNESS,\n1)\tI suggest the authors provide more comparison with the latest baselines in the setting of instantaneous trajectory prediction.\n2)\tStandard diffusion model suffer from expensive inference time, the author should provide detailed inference speed to evaluate the proposed model comprehensively.\n", "limitations": "Refer to the weakness and questions. My concerns mainly lies on the comparison with the latest baselines and the inference time.  Providing above information could be helpful to improve this paper comprehensively.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Instantaneous trajectory prediction is a common challenge in traffic scene. It is promising that employ the diffusion model in instantaneous trajectory prediction.  The conducted experimental results show proposed BCDiff is effective in the scenes of ETH-UCY and SDD. \nMoreover, the authors claim that the proposed method has friend compatibility with existing trajectory prediction models. Also, the organization of paper is well.", "weaknesses": "The experiments is insufficient.\n1: The compared baselines are not the newest methods.  The authors should add more baselines with high accuracy, such as SocialVAE[1], SIT[2], MemoNet[3]. These baselines could be trained under the setting of the instantaneous trajectory prediction, such as observing 2 steps and predicting the next 12 steps. \n2: The inference time is missing. Instantaneous trajectory prediction often occurs on the emergency scene. The inference time is an important metrics to evaluate model. \n\n[1] Pei Xu, Jean-Bernard Hayet, and Ioannis Karamouzas. Socialvae: Human trajectory prediction using timewise latents. In ECCV, pages 511–528. Springer, 2022\n[2] Liushuai Shi, Le Wang, Chengjiang Long, Sanping Zhou, Fang Zheng, Nanning Zheng, and Gang Hua. Social interpretable tree for pedestrian trajectory prediction. In AAAI, pages 2235–2243, 2022\n[3] Chenxin Xu, Weibo Mao, Wenjun Zhang, and Siheng Chen. Remember intentions: Retrospective-memory-based trajectory prediction. In CVPR, pages 6488–6497, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: The compared baselines are not the newest methods. The authors should add more baselines with high accuracy, such as SocialVAE, SIT, MemoNet. These baselines could be trained under the setting of the instantaneous trajectory prediction, such as observing 2 steps and predicting the next 12 steps. \n\nA1: Following your suggestion, we add experiments to compare with SocialVAE [3] that is proposed recently on ETH/UCY Dataset. \nThe results are listed in the below table. We observe that our BCDiff still performs better than baselines, which demonstrates the superiority of our proposed method for instantaneous trajectory prediction. We will include the results in the final version.\n\n| Model        | Methods       |           |           | Dataset   |           |           |           |\n|--------------|---------------|-----------|-----------|-----------|-----------|-----------|-----------|\n|              |               | ETH       | Hotel     | UNIV      | ZARA1     | ZARA2     | AVG       |\n|              | Instantaneous | 0.64/1.10 | 0.21/0.34 | 0.27/0.51 | 0.22/0.39 | 0.18/0.34 | 0.30/0.54 |\n|              | MOE [1]       | 0.57/1.01 | 0.17/0.29 | 0.26/0.44 | 0.22/0.36 | 0.17/0.32 | 0.28/0.48 |\n| SocialVAE[3] | MOE w/o Image | 0.59/1.06 | 0.20/0.33 | 0.27/0.49 | 0.22/0.38 | 0.18/0.33 | 0.29/0.52 |\n|              | DTO [2]       | 0.61/1.06 | 0.18/0.31 | 0.25/0.43 | 0.22/0.38 | 0.17/0.33 | 0.29/0.50 |\n|              | BCDiff        | 0.53/0.91 | 0.17/0.27 | 0.24/0.40 | 0.21/0.37 | 0.16/0.26 | 0.26/0.44 |\n\n>Q2: The inference time is missing. Instantaneous trajectory prediction often occurs on the emergency scene. The inference time is an important metric to evaluate the model.\n\nA2: Thanks for your comments. We admit that inference time is one of our limitations due to the nature of the diffusion model, and we have put it as a limitation in the Section B of Supplementary Material. \nCurrently, we primarily focus on the accuracy of instantaneous trajectory predictions. We will further consider how to accelerate it in our future work. The following are some potentially feasible acceleration strategies:\n\n1. From the perspective of diffusion algorithms: Our current framework is built based on a typical diffusion model, DDPM. DDPM models the trajectory generation process as a Markov process that requires continuous M-steps to generate trajectories. There are many cutting-edge methods for accelerating inference [4,5,6,7], by relaxing the constraints of the Markov process and generating trajectories using skipping steps. In this way, the model can use fewer steps to generate a trajectory, and thereby shorten the reference time.\n\n2. From the perspective of model structure: We can combine our method with Half-Precision/Model Quantization techniques [8,9] to lower the computational cost, Model Pruning [10,11] to simplify the model or Knowledge Distillation [12,13] to utilize a small but effective model for inference. \n\nReference\n\n[1] Human Trajectory Prediction with Momentary Observation. CVPR'22\n\n[2] How Many Observations Are Enough? Knowledge Distillation for Trajectory Forecasting. CVPR'22\n\n[3] SocialVAE: Human Trajectory Prediction using Timewise Latents. ECCV'22\n\n[4] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 steps. NeurIPS'22\n\n[5] Leapfrog Diffusion Model for Stochastic Trajectory Prediction. CVPR'23\n\n[6] Progressive Distillation for Fast Sampling of Diffusion Models. ICLR'22\n\n[7] Denoising Diffusion Implicit Models. ICLR'21\n\n[8] Structural Pruning for Diffusion Models. arXiv'23\n\n[9] Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients. ICLR'22\n\n[10] Q-Diffusion: Quantizing Diffusion Models. arXiv'23\n\n[11] Post-Training Quantization on Diffusion Models. CVPR'23\n\n[12] On Distillation of Guided Diffusion Models. CVPR'23\n\n[13] Progressive Distillation for Fast Sampling of Diffusion Models. ICLR'22\n"}, {"review_id": "871rIdOzpP", "reviewer": "Reviewer_WQWs", "summary": "In this paper, the authors propose a diffusion framework for instantaneous trajectory prediction (only observing two frames in history). Specifically, they develop two coupled diffusion models by designing a mutual guidance mechanism that can bidirectionally and consistently generate unobserved historical trajectories and future trajectories step-by-step, to utilize the complementary information between them. The experimental results demonstrate the proposed solution improves the accuracy of instantaneous trajectory prediction. ", "questions": "1. Providing more information about failure cases will help readers better understand the performance of the proposed solution. \n2. Discussing some potential ways to speed up the proposed solution will be better. \n", "limitations": "The long computation time should be the biggest limitation of the proposed solution. The authors discussed the issues in the supplementary material, but no potential doable solution. It may need more exploration. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The intuition of the proposed solution is compelling. As mentioned in line 60, \"Both previous unobserved historical trajectories and future trajectories contain information of the same pedestrian at different timesteps, and thereby they provide complementary information to each other. It will be beneficial for the prediction of future trajectories if we can design an elegant method to simultaneously generate previous unobserved historical trajectories and future trajectories by fully leveraging the complementary information between them.\"\n2. The proposed solution (that simultaneously generates both future and unobserved historical trajectories in a consistent manner) sounds solid and should work in theory. \n3. The experimental results on multiple datasets and models prove the validity of the proposed solution. \n4. This paper is well-organized and easy to understand. \n5. The supplementary material provides more details about the model and experiments, which makes it easier for the reader to understand their work.\n\n\n", "weaknesses": "1. The proposed solution is time-consuming (4 seconds for predicting trajectories once). This problem appears to be particularly serious, especially in the area of autonomous driving. The long computation time should be caused by the model design (diffusion model and using an iterative way to generate predicted trajectories). \n2. Some details are missing. For example, what are the social and scene context e, and how to generate them for the Encoder is not clear. \n3. It will be better to show some failure cases. Considering that the trajectory prediction task is usually used in autonomous driving, failure cases are more valuable to study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: The proposed solution is time-consuming (4 seconds for predicting trajectories once). This problem appears to be particularly serious, especially in the area of autonomous driving. The long computation time should be caused by the model design (diffusion model and using an iterative way to generate predicted trajectories). Discussing some potential ways to speed up the proposed solution will be better.\n\nA1: Thanks for your comments. We admit the inference time is one of our limitations due to the nature of the diffusion model, and have put it as a limitation in Appendix B of Supplementary Material. \nCurrently, we primarily focus on the accuracy of instantaneous trajectory predictions. We will further consider how to accelerate it in our future work. The following are some potentially feasible acceleration strategies:\n\n1. From the perspective of diffusion algorithms: Our current framework is built based on a typical diffusion model, DDPM. DDPM models the trajectory generation process as a Markov process that requires continuous M-steps to generate trajectories. There are many cutting-edged methods for accelerating inference [1,2,3,4], by relaxing the constraints of the Markov process and generating trajectories using skipping steps. In this way, the model can use less steps to generate a trajectory, and thereby shorten the reference time.\n\n2. From the perspective of model structure:} We can combine our method with Half-Precision/Model Quantization techniques [7,8] to lower the computational cost, or Model Pruning [5,6] to simplify the model, or Knowledge Distillation [9, 10] to utilize a small but effective model for inference. \n\nWe will add them in the final version.\n\n>Q2: Some details are missing. For example, what are the social and scene context e, and how to generate them for the Encoder is not clear.\n\nA2: Sorry for confusing you. The social context is the information of other moving pedestrians around the target pedestrian whose future trajectories are to be predicted. The scene context represents the map information around the target agent. Generally speaking, these two kinds of information will be integrated into the encoder of the trajectory prediction model to boost the representation ability of historical trajectories. Since our proposed BCDiff is actually a decoder, in other words, it is an encoder-agnostic framework, our attention is not paid to the contextual information. We will include the above details in the final version.\n\n>Q3: It will be better to show some failure cases. Considering that the trajectory prediction task is usually used in autonomous driving, failure cases are more valuable to study.\n\nA3: Thanks for your suggestions. We provide some failure cases in the attached PDF file in the Global Response. As shown in Figure 2, our model sometimes fails when pedestrians start to walk or abruptly change the intention. We will further optimize our method according to these failure cases in our future work. \n\nReference\n\n[1] DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps. NeurIPS'22\n\n[2] Leapfrog Diffusion Model for Stochastic Trajectory Prediction. CVPR'23\n\n[3] Progressive Distillation for Fast Sampling of Diffusion Models. ICLR'22\n\n[4] Denoising Diffusion Implicit Models. ICLR'21\n\n[5] Structural Pruning for Diffusion Models. arXiv'23\n\n[6] Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradients. ICLR'22\n\n[7] Q-Diffusion: Quantizing Diffusion Models. arXiv'23\n\n[8] Post-Training Quantization on Diffusion Models. CVPR'23\n\n[9] On Distillation of Guided Diffusion Models. CVPR'23\n\n[10] Progressive Distillation for Fast Sampling of Diffusion Models. ICLR'22"}, {"review_id": "OvlbfjTqrv", "reviewer": "Reviewer_JXan", "summary": "This paper proposed a  diffusion-based model specifically for instantaneous trajectory prediction. It leverages the consistency of the prediction for unobserved historical trajectory and the future trajectory to improve the accuracy of the future trajectory. Regarding the noise in the prediction wh, especially in the early steps, the paper introduced a gating mechanism to guide the fusion. ", "questions": "1. Given the first point in [Weakness], for a fair comparison, I was curious to know the evaluation results if the existing methods also utilize unobserved trajectories as a prediction target for data augmentation. For instance, in Table 1, during the training of Trajectron++, Trajectron++ with MOE, and Trajectron++ with DTO, in addition to supervising future trajectories, we reverse the observed historical trajectories, use them as input to predict unobserved trajectories and evaluate the models. Another approach to demonstrate this could be seen in Table 3, BCDiff-w/o.Guidance, where instead of employing two diffusion models, only one model is used to predict both reversed unobserved and future trajectories.\n\n2. Regarding the gating mechanism, it is intuitive that the early steps should have lower weights given the noise level. Have authors tried simpler ways to handle the issue? Such as manually assigning lower weights to the early steps, and gradually increasing the weight till the last steps. \n\n3. Since the behavior of pedestrians can be very spontaneous, can you provide the evaluation comparison with other methods on some hard cases, especially for instantaneous trajectory prediction? Such as the cases where there are large contrasts between the unobserved trajectories and the future trajectories. \n\n4. Table 4 is very interesting. Can you provide some more quantitative results related to it? For instance, the numeric results of the accuracy on unobserved trajecotires and the relationship between which and the performance of the future predictions. ", "limitations": "The authors have not discussed the limitation. From my point of view, the limitation of the work is that the input information is so little that it can hardly provide valid predictions for samples in the wild or more complex situations. Furthermore, the dynamic or interaction types of the tested datasets are also very limited, the results can be biased without testing the methods on more diverse datasets, such as some autonomous driving datasets. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is written-well and the method section is generally straightforward and coherent. \n2. The idea is intuitive. Since longer trajectories would provide more dynamic information and motion hints to help the system make better predictions, even those longer trajecotires are also predicted and contain a certain amount of noise. \n3. The experiments and comparison are thorough and have covered two popular datasets in pedestrian trajectory forecasting and three SOTA methods. \n4. The improvements are significant compared to the existing instantaneous trajectory forecasting methods. ", "weaknesses": "Maybe there is some misunderstanding but my major concern is that the motivation of the method may not align with the motivation of the task very well. \n\nThe task of instantaneous trajectory prediction wants the system to extract valuable information when the objects' dynamic information is extremely limited.  As the authors mentioned in the related works L110-L111, the lack of temporal information is an inherent challenge of the task. Therefore, MOE[1] tries to use map information as an informative reference to help the system conduct more accurate predictions, and the motivation of DTO[2] is to do a thorough analysis of how limited observed information can lead to comparatively accurate predictions. \n\nBut in this paper, the authors let the system access the unobserved historical trajectories during the training time as one of the supervision signals. This data-augmentation-like design may make the comparison with other methods not fair enough. It would be necessary to rule out the influence of data augmentation when evaluating the boost brought by the system and prove the performance boost all comes from that the system has fully dug the potential of the limited observation. \n\n[1] Jianhua Sun, Yuxuan Li, Liang Chai, Hao-Shu Fang, Yong-Lu Li, and Cewu Lu. Human trajectory prediction with momentary observation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6467–6476, 2022.\n\n[2] Alessio Monti, Angelo Porrello, Simone Calderara, Pasquale Coscia, Lamberto Ballan, and Rita Cucchiara. How many observations are enough? knowledge distillation for trajectory forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6553–6562, 2022.\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Q1: The authors let the system access the unobserved historical trajectories during the training time as one of the supervision signals. It would be necessary to rule out the influence of unobserved historical trajectories when evaluating the boost brought by the system and prove the performance boost all comes from that the system has fully dug the potential of the limited observation.\n\nA1: Thanks for your suggestions for improving the quality of our method. Following your suggestions, we modify the baseline Trajectron++, Trajectron++ (MOE), and Trajectron++ (DTO) with the supervision of both future trajectories and unobserved trajectory ground-truth. The results are listed below. We observe our BCDiff still outperforms existing methods, which rules out the influence of unobserved trajectories and demonstrates the superiority of the proposed approach.\n\n|Model|Methods|||Dataset||||\n|-|-|-|-|-|-|-|-|\n|||ETH|Hotel|UNIV|ZARA1|ZARA2|AVG|\n||Instantaneous|0.71/1.36|0.25/0.50|0.34/0.74|0.22/0.42|0.18/0.34|0.34/0.67|\n||MOE|0.63/1.10|0.18/0.31|0.32/0.59|0.22/0.41|0.17/0.32|0.30/0.55|\n|Trajectron++|MOE w/o Image|0.66/1.16|0.22/0.39|0.33/0.62|0.22/0.42|0.18/0.33|0.32/0.58|\n||DTO|0.70/1.21|0.22/0.45|0.31/0.60|0.22/0.41|0.17/0.33|0.32/0.60|\n||BCDiff|0.61/1.09|0.16/0.28|0.28/0.53|0.22/0.41|0.18/0.33|0.29/0.53|\n\n>Q2: Regarding the gating mechanism, Have authors tried simpler ways to assign weight?\n\nA2: Thanks for your suggestions. We add an experiment to adjust the weights manually. We linearly increase the weights from 0 to 1 with a step size of $1/M$, where $M$ is the number of steps in the diffusion model. The results of using heuristic weight assigning (BCDiff-manually) are indeed better than that of using equal weights (BCDiff-w/oGate). Importantly, our gate mechanism achieves the best performance, since our method is expected to automatically learn the optimal weights through gradient descent.\n\n|Method|ETH/UCY|\n|-|-|\n|BCDiff-w/o Gate|0.32/0.57|\n|BCDiff-manually|0.31/0.55|\n|BCDiff-Gate|0.29/0.53|\n\n\n>Q3: Can you provide comparisons with other methods on some hard cases for instantaneous trajectory prediction? \n\nA3: Thanks for your suggestions. We provide some hard cases in the attached PDF file in the Global Response. As shown in Figure 1, we observe our method can better handle the hard cases and predict more accurate trajectories than other methods.\n\n>Q4: Can you provide more quantitative results related to Table 4?\n\nA4: Thanks for your suggestions. We provide the ADE/FDE of unobserved trajectories, as shown in the below table. \n\nAs the number of unobserved points ($T_{unobs}$) increases, the accuracy of unobserved trajectory prediction decreases. This is reasonable since it becomes more difficult when more points need to be predicted. In addition, as $T_{unobs}$ increases, the accuracy of future trajectory prediction gradually improves. This can be attributed to leveraging the additional unobserved points to provide useful information.  When $T = 5$, the prediction performance starts to decline. This is because it becomes imprecise when predicting previous unobserved historical points with a larger length, thus introducing noise into our method.\n\n|Dataset|Type|$T_{unobs}=1$|$T_{unobs}=2$|$T_{unobs}=3$|$T_{unobs}=4$|$T_{unobs}=5$|$T_{unobs}=6$|\n|-|-|-|-|-|-|-|-|\n|ETH/UCY|Future|0.31/0.55|0.30/0.54|0.30/0.53|**0.29/0.53**|0.32/0.57|0.34/0.60|\n||Unobserved|0.006/0.006|0.034/0.033|0.055/0.063|0.086/0.089|0.108/0.133|0.149/0.212|\n|SDD|Future|11.86/19.78|11.81/19.71|11.68/19.50|**11.56/19.32**|11.92/20.01|12.33/20.98|\n||Unobserved|0.311/0.312|1.564/1.492|2.626/2.678|3.552/3.884|4.828/5.423|6.177/7.894|\n\n>Q5: Limitation of the work: Not enough input information for accurate predictions in complex situations. Tested datasets have limited types of movement and interaction, which could make results biased. Need to test on diverse datasets.\n\nA5: Thanks for your comments.\nWe put the limitations in Section B of the Appendix due to space limitations.\nPedestrian trajectory prediction is an important research problem for autonomous driving [1,2], and many works have been proposed for studying it in the past decade [3,4]. Thus, we mainly focus on the pedestrian trajectory prediction problem in this work.\nDifferent from most of the existing pedestrian trajectory prediction methods, we consider a more challenging scenario, where merely two frames of trajectories can be observed to predict future trajectories.\n\nIt is indeed an interesting idea to apply our method to predict trajectories on autonomous driving datasets [5,6] for other moving agents, e.g., vehicles. We are currently considering such a problem setting and look forward to experimenting with our method. It is anticipated for our method to work well on vehicles. The reasons are two-fold:\n\n1. Pedestrian trajectory prediction is a more challenging problem, due to the moving directions of pedestrians being more random than those of vehicles driving on the road.\n\n2. Our proposed BCDiff is an encoder-agnostic framework that operates independently of specific encoders. It can be integrated with diverse existing trajectory prediction methods. By leveraging a strong encoder, intricate scenarios, and interactions can be fully captured and represented. Moreover, we will be also actively committed to further explore for effectively addressing complex environmental dynamics in our approach, thereby enhancing the accuracy of predictive capabilities.\n\nReference\n\n[1] Codes for View Vertically: A Hierarchical Network for Trajectory Prediction via Fourier Spectrums. ECCV'22\n\n[2] SocialVAE: Human Trajectory Prediction using Timewise Latents. ECCV'22\n\n[3] Human Trajectory Prediction via Neural Social Physics. ECCV'22\n\n[4] Human Trajectory Prediction with Momentary Observation. CVPR'22\n\n[5] Argoverse: 3D Tracking and Forecasting with Rich Maps. CVPR'19\n\n[6] Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset. ICCV'21"}]

Author rebuttal:
We really thank the reviewers for their valuable feedback. We are encouraged as reviewers found that our idea is novel and interesting (ytbj), promising (VvSx), solid (WQWs), and intuitive (JXan), our experiments show good results (GWuJ, ytbj, JXan) and are thorough (JXan), our paper is easy to follow (ytbj), easy to understand (WQWs), well-organized (GWuJ, VvSx, WQWs) and well-written (GWuJ, JXan). We have made the point-to-point response to each reviewer, and we include additional experimental results in the attached \textbf{PDF file} in the Global Response to assist reviewers in better understanding our responses.

Here, we list the additional experiments according to each reviewer's comments.

**[GWuJ, Q3]**. We include an experiment to better motivate our method. Results are shown in Table 1 in the attached PDF.

**[GWuJ, Q4]**. We include an experiment to further demonstrate the superiority of joint prediction. Results are shown in Table 2 in the attached PDF.

**[VvSx, Q1]**. We conduct an experiment to compare with the methods using the recently proposed SocialVAE as the backbone. Results are shown in the corresponding rebuttal thread.

**[WQWs, Q3]**. We provide some failure cases for instantaneous trajectory prediction, as shown in  Figure 2 in the attached PDF. Results are shown in the corresponding rebuttal thread.

**[JXan, Q1]**. We provide the performance of baseline that also utilized unobserved trajectories as supervision. Results are shown in the corresponding rebuttal thread.

**[JXan, Q3]**. We provide some hard cases for instantaneous trajectory prediction, as shown in Figure 1 in the attached PDF.

**[JXan, Q4]**. We provide the performance of unobserved trajectories when $T_{unobs}$ increases. Results are shown in the corresponding rebuttal thread.


Finally, we once again thank all reviewers for their insightful comments which are very helpful for improving the quality of our paper.

Meta-review: The paper attempts to address a relevant problem space, but there are mixed feelings about the approach and its relative novelty. The proposed method revolves around completing the incomplete data and forecasting the future using diffusion learning. While the technique has been explored previously with marginal distinctions, the following points merit consideration:

Lack of Comprehensive Comparisons: The paper falls short in comparing its diffusion model with other contemporaneous models, particularly those presented at CVPR in recent years. This omission could detract from the paper's value, as the readers are left without a full understanding of how this model stacks up against the latest in the field.

Outdated Benchmarking: The benchmark models (Trajectron++, PCCSNet, and SGCN) selected for comparison in Tab1 are from 2020 and 2021. For a more robust assessment, the paper would have benefited from incorporating more recent encoder models and demonstrating that their method could achieve state-of-the-art results.

Positive Rebuttal Feedback: Notwithstanding the initial concerns, it's evident that the authors were responsive in their rebuttal. Their clarifications and additional inputs during the rebuttal phase have improved the perception of their submission, leading to an increase in reviewer ratings.

Given the constructive engagement of the authors during the rebuttal phase and the potential merit of their work, it is suggested that the paper be accepted . However, the authors are encouraged to take into account the feedback for future works and possibly provide more exhaustive comparisons with recent models to solidify their contributions.
Decision: Accept (poster)