Title: BCDiff: Bidirectional Consistent Diffusion for Instantaneous Trajectory Prediction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Trajectory prediction

instantaneous observation
Abstract: The objective of pedestrian trajectory prediction is to estimate the future paths of pedestrians by leveraging historical observations, which plays a vital role in ensuring the safety of self-driving vehicles and navigation robots. Previous works usually rely on a sufficient amount of observation time to accurately predict future trajectories. However, there are many real-world situations where the model lacks sufficient time to observe, such as when pedestrians abruptly emerge from blind spots, resulting in inaccurate predictions and even safety risks. Therefore, it is necessary to perform trajectory prediction based on instantaneous observations, which has rarely been studied before. In this paper, we propose a Bi-directional Consistent Diffusion framework tailored for instantaneous trajectory prediction, named BCDiff. At its heart, we develop two coupled diffusion models by designing a mutual guidance mechanism which can bidirectionally and consistently generate unobserved historical trajectories and future trajectories step-by-step,  to utilize the complementary information between them. Specifically, at each step, the predicted unobserved historical trajectories and limited observed trajectories guide one diffusion model to generate future trajectories, while the predicted future trajectories and observed trajectories guide the other diffusion model to predict unobserved historical trajectories. Given the presence of relatively high noise in the generated trajectories during the initial steps, we introduce a gating mechanism to learn the weights between the predicted trajectories and the limited observed trajectories for automatically balancing their contributions. By means of this iterative and mutually guided generation process, both the future and unobserved historical trajectories undergo continuous refinement, ultimately leading to accurate predictions. Essentially, BCDiff is an encoder-free framework that can be compatible with existing trajectory prediction models in principle. Experiments show that our proposed BCDiff significantly improves the accuracy of instantaneous trajectory prediction on the ETH/UCY and Stanford Drone datasets, compared to related approaches.

BCDiff: Bidirectional Consistent Diffusion for Instantaneous Trajectory Prediction

Abstract

The objective of pedestrian trajectory prediction is to estimate the future paths of pedestrians by leveraging historical observations, which plays a vital role in ensuring the safety of self-driving vehicles and navigation robots. Previous works usually rely on a sufficient amount of observation time to accurately predict future trajectories. However, there are many real-world situations where the model lacks sufficient time to observe, such as when pedestrians abruptly emerge from blind spots, resulting in inaccurate predictions and even safety risks. Therefore, it is necessary to perform trajectory prediction based on instantaneous observations, which has rarely been studied before. In this paper, we propose a B i-directional C onsistent Diff usion framework tailored for instantaneous trajectory prediction, named BCDiff . At its heart, we develop two coupled diffusion models by designing a mutual guidance mechanism which can bidirectionally and consistently generate unobserved historical trajectories and future trajectories step-by-step, to utilize the complementary information between them. Specifically, at each step, the predicted unobserved historical trajectories and limited observed trajectories guide one diffusion model to generate future trajectories, while the predicted future trajectories and observed trajectories guide the other diffusion model to predict unobserved historical trajectories. Given the presence of relatively high noise in the generated trajectories during the initial steps, we introduce a gating mechanism to learn the weights between the predicted trajectories and the limited observed trajectories for automatically balancing their contributions. By means of this iterative and mutually guided generation process, both the future and unobserved historical trajectories undergo continuous refinement, ultimately leading to accurate predictions. Essentially, BCDiff is an encoder-free framework that can be compatible with existing trajectory prediction models in principle. Experiments show that our proposed BCDiff significantly improves the accuracy of instantaneous trajectory prediction on the ETH/UCY and Stanford Drone datasets, compared to related approaches.

1 Introduction

Pedestrian trajectory prediction aims to predict future trajectories conditioned on their past movements, which is an important task for autonomous driving [25, 57] and navigation robot [4]. Previous pedestrian trajectory prediction approaches usually rely on long enough observation time (typically, 2 to 3 seconds) for a pedestrian to precisely predict the future trajectories [53, 48, 54, 13]. However, in many real-world situations, e.g., when pedestrians suddenly emerge from blind spots and are in close proximity to autonomous vehicles, traditional trajectory prediction methods do not have ample time to collect a sufficient number of locations. This leads to sub-optimal prediction performance and potentially unsafe behaviors in the decision-making of autonomous vehicles and robots. Therefore, it is quite necessary to forecast future trajectories based on limited or instantaneous observations.

The prediction of instantaneous trajectories for pedestrians is a highly challenging task due to the limited observation time. In some cases, as extreme as it can be, merely two frames of locations can be observed. In the face of such a challenging task, there have been only a few academic works to date. MOE [46] is the first to propose the problem of instantaneous trajectory prediction, and thus is the most relevant to ours. MOE incorporates scene context information into limited observations and introduces the masked trajectory complement and context restoration as self-supervised tasks to pretrain the model. However, since MOE only utilizes instantaneous temporal information acquired from limited trajectories, it might be hard to accurately predict the future trajectories of a pedestrian with complex behavior such as turning and yielding. DTO [34] focuses on lowering the influence of noise introduced by incorrect detection and tracking, and attempts to employ limited observed trajectory to alleviate this problem. It utilizes the knowledge distillation technique to distill knowledge from a teacher model trained with an ample amount of long observations, and transfer the knowledge to a student model receiving fewer observations as input. Although these approaches have shown some effectiveness in instantaneous trajectory prediction, the representation of a pedestrian is restricted to two frames of locations, which contains extremely limited temporal information.

In this paper, we propose BCDiff, a bidirectional consistent diffusion framework specifically designed for the instantaneous trajectory prediction task. As we know, the diffusion model is a generative model, which has been successfully applied to various generation tasks, including image synthesis [39, 33], image denoising [23], etc. Different from them, we leverage the diffusion model for instantaneous trajectory prediction. We devise two coupled diffusion models to bidirectionally generate previous unobserved trajectories and future trajectories from random noises, which can address the issue of temporal information scarcity in limited observations. The underlying intuition behind this idea is: Both previous unobserved historical trajectories and future trajectories contain information of the same pedestrian at different timesteps, and thereby they provide complementary information to each other. It will be beneficial for the prediction of future trajectories if we can design an elegant method to simultaneously generate previous unobserved historical trajectories and future trajectories by fully leveraging the complementary information between them.

To accomplish this, we devise a step-by-step mutual guidance mechanism in two coupled diffusion models to simultaneously generate previous unobserved historical trajectories and future trajectories. Specifically, at each step, the predicted unobserved historical trajectories, together with the limited observed trajectories serve as a guidance for one diffusion model to predict a denoising intensity, which is then used to generate future trajectories of the subsequent step. Likewise, the predicted future trajectories, together with the observed trajectories, guide the other diffusion model to predict unobserved historical trajectories of the next step. Meanwhile, considering there exists relatively high noise in the generated trajectories during the initial steps, we devise a gating mechanism to learn the weights between the predicted trajectories and the limited observed trajectories for automatically controlling the proportion of the guidance information from two kinds of trajectories during each generation step. Through this iterative and mutually guided generation process, the future and unobserved historical trajectories are continuously refined, ultimately leading to precise predictions. Notably, our proposed BCDiff is encoder-free and is compatible with existing trajectory prediction encoders in principle, allowing them to gracefully handle cases with instantaneous observations.

Our contributions can be summarized as follows: 1) We propose BCDiff, a diffusion model based framework tailored for instantaneous trajectory prediction. BCDiff can simultaneously generate both future and unobserved historical trajectories in a consistent manner, which can effectively leverage complementary information between them. 2) We devise a step-by-step mutual guidance mechanism to couple two diffusion models for trajectory generation, and present a gating strategy to adaptively

adjust the contributions of the guidance information between two kinds of trajectories. 3) Experiments demonstrate our proposed BCDiff significantly improves the accuracy of instantaneous prediction and outperforms the state-of-the-art methods on ETH/UCY and Stanford Drone datasets.

2 Related Works

2.1 Traditional Trajectory Prediction

Traditional trajectory prediction methods aim to predict future trajectories given sufficient observation time. To capture complex interactions between pedestrians, many methods have been proposed [1, 15, 40]. These models utilize a social mechanism to aggregate neighboring actors and broadcast information to each actor. In addition, graph neural networks [49, 11, 19, 22, 27, 28] and transformer architectures [52, 36, 35, 51] are introduced to encapsulate implicit interactions among pedestrians. To resolve the problem of high uncertainty in pedestrians, researchers propose stochastic generative models, such as GAN [15, 22, 40, 45, 55], VAE [25, 24, 31, 53], and Diffusion Models [14, 32], to better capture the variability in future trajectories. Various sampling strategies are designed to avoid purely random sampling [6, 3, 18, 5, 30]. However, the aforementioned methods can accurately predict trajectories only when a sufficient amount of long observation trajectories are available. The accuracy cannot be guaranteed given the limited observation time. In contrast to these methods, our goal is to address the trajectory prediction problem under instantaneous observation scenarios.

2.2 Instantaneous Trajectory Prediction

Instantaneous trajectory prediction aims to predict future trajectories given a limited number of observed trajectory points. In the most extreme scenarios, merely two frames of locations can be observed. This task poses significant challenges due to the exceedingly short observation period. MOE [46] and DTO [34] are two trajectory prediction methods based on limited observed trajectory points. MOE proposes a feature extractor to incorporate image semantic information and develops a self-supervised task to enhance the representational ability of instantaneous observations. Meanwhile, DTO investigates the influence of noise introduced by detection or tracking to trajectory prediction, and intends to use fewer trajectory points in a knowledge distillation framework to address the issue. Despite these advances, they fail to address the inherent lack of temporal information in instantaneous trajectory prediction. In our study, we attempt to predict and leverage previous unobserved trajectories to capture more temporal information, thereby improving the prediction accuracy of future trajectories.

2.3 Diffusion Models

The diffusion model is a class of stochastic generation models, which exhibits amazing performance in a diverse range of fields such as image synthesis [9, 39, 33], audio synthesis [7, 21] and text generation [2, 12, 8]. Among these works, a typical diffusion model is the Denoising diffusion probabilistic model (DDPM) [44, 16]. DDPM is inspired by the non-equilibrium thermodynamics, in which a forward Markov process perturbs real data into noise, and a reverse Markov process converts noise back to real data. DDPM has been widely used in various tasks, including image super-resolution [41, 17], 3D point cloud generation [29, 56] etc. For instance, CDM [17] cascades multiple DDPM models to generate images of increasing resolution. The work in [29] employs a heat bath mechanism on DDPM to facilitate the generation of 3d point clouds. Different from these works, we leverage DDPM to solve the problem of instantaneous trajectory prediction.

3 Methods

3.1 Problem Formulation

In this work, we aim to tackle the task of instantaneous trajectory prediction, where we assume only two frames are observed, i.e., the most extreme case. We denote X obs = { x 1 , x 2 } as the observation locations. The ground-truth future trajectory is symbolized as X fut = { x 3 , x 4 , ..., x T fut +2 } , where T fut is the prediction length, and x i ∈ R 2 is the 2D coordinate of a trajectory location. Moreover, we characterize the previous unobserved trajectories as X unobs = { x 1 -T unobs , ..., x -1 , x 0 } , where T unobs represents the length of the unobserved trajectories. Our objective is to develop a diffusion

𝒗

!"#

𝒈

$%&

'

Encoder

𝑿

!"#

Decoder

𝑿

%(!"#

)

𝑿

$%&

)

x

M

𝒗

𝒐𝒃𝒔

𝑿

$%&

-

𝑿

$%&

-./

𝑿

$%&

/

𝑿

$%&

)

𝑿

%(!"#

-

𝑿

%(!"#

-./

𝑿

%(!"#

/

𝑿

%(!"#

)

...

...

...

(b) Decoding Process of Bi-Directional Consistent Diffusion (BCDiff)

Bi-LSTM

𝒗

!"#

Gate

Gate

Bi-LSTM

Fusion

Fusion

Transformer

𝑾

$%&

𝑾

%(!"#

𝒈

%(!"#

'

𝒄

0

!"#

'

𝒄

0

"$%&'

'

(c) Structure of BCDUnit

(a) Enc-Dec Architecture

𝝐

$%&

'

𝝐

%(!"#

'

𝑿

$%&

'

𝑿

%(!"#

'

𝑿

$%&

'

𝑿

%(!"#

'

𝓛

$%&

𝓛

%(!"#

𝝓

!"#$%

𝜽

!"#$%

𝒆

BCDUnit

BCDUnit

BCDUnit

𝝓

&!'

𝜽

&!'

models based method to generate both previous unobserved historical and future trajectories, so that more temporal information can be captured to better predict future trajectories, given only two observed frames. Since we utilize the diffusion model, we denote the maximum diffusion steps as M , and use X m fut and X m unobs to represent the future and unobserved trajectories following a diffusion of m steps or a denoising of M -m steps, respectively. Note that X unobs = X 0 unobs and X fut = X 0 fut .

3.2 Overall Architecture

The overall architecture consists of an encoder and a decoder, as shown in Figure 1 (a). The encoder encodes X obs as v obs , and captures social and scene context e . The decoder is our proposed BCDiff framework, as shown in Figure 1 (b), which simultaneously generates previous observed historical trajectories and future trajectories step-by-step through the Bidirectional Consistent Denoising Unit (BCDUnit). The BCDUnit comprises two coupled diffusion models, As shown in Figure 1 (c), the diffusion model { ϕ fut , θ fut } marked in red color, is responsible for generating future trajectories, while the diffusion model { ϕ unobs , θ unobs } , depicted in yellow, is used for generating unobserved historical trajectories. The two diffusion models are coupled through a mutual guidance mechanism. To be specific, as the m th step, the network ϕ = { ϕ fut , ϕ unobs } leverages two bidirectional LSTMs to encode unobserved trajectories and future trajectories as mutual guidance g m fut and g m unobs used for generating each other in the m -1 th step. Considering the guidance information contains relatively high noise during the initial steps, we introduce a gate mechanism to balance the contributions between the observed guidance v obs and future guidance g m fut , as well as the observed guidance v obs and unobserved guidance g m unobs , ultimately producing the appropriate guidance c ϕ,fut and c ϕ,unobs . Then the network θ = { θ fut , θ unobs } fuses the guidance with both predicted unobserved historical and future trajectories at current steps to generate the denoise intensities, which is used to simultaneously generate X m -1 fut and X m -1 unobs of the next step. It is noteworthy that both previous unobserved historical trajectories and future trajectories contain the information of the same pedestrian at different timesteps, thus we adopt a parameter-shared transformer across θ fut and θ unobs .

3.3 Bidirectional Consistent Diffusion

In this section, we introduce our proposed BCDiff framework, which contains two coupled diffusion models to consistently generate trajectories in two directions, i.e., simultaneously predicting previous unobserved trajectories and future trajectories. In this paper, we utilize DDPM [16] as our basic diffusion model, because of its excellent performance in various tasks. For the generation of each direction, the diffusion model executes diffusion and conditional denoising processes. The diffusion process aims to intentionally add a series of noises to a ground-truth trajectory, while the conditional denoising process recovers the trajectory from noise inputs conditioned on the guidance.

Diffusion Process. The diffusion process is defined as a Markov chain, conditioned on the groundtruth trajectories X 0 fut , X 0 unobs . To write conveniently, we omit the subscripts, allowing X 0 to represent unobserved historical trajectories or future trajectories. The diffusion process generates the sequence { X i } i M =1 by accumulating noise M times, i.e.,

where N denotes the Gaussian distribution, and α m represents the noise intensity from X m to X m -1 . Typically, α m is equal to 1 -β m , where β m is a pre-defined value belonging to the interval [0,1]. Due to the additivity of Gaussian distributions, we are able to directly obtain X m from X 0 :

where α m = ∏ m i =1 α i . Note that as M becomes sufficiently large, α approaches to zero, making q ( X M | X 0 ) converge to the standard Gaussian distribution. By employing the above process, the ground-truth trajectory is transformed into the Gaussian noise X M ∼ N (0 , I ) .

Conditional Denoising Process. In this process, we aim to generate X 0 from the Gaussian noise X M ∼ N (0 , I ) . We can reverse the aforementioned diffusion process, to gradually denoise from the Gaussian noise X M and reconstruct X 0 . Based on the proof in [10]: If q ( X m | X m -1 ) follows a Gaussian distribution and β m is sufficiently small, q ( X m -1 | X m ) also satisfies a Gaussian distribution. Therefore, we formulate q ( X m -1 | X m ) as a Gaussian Markov process. However, it is intractable to directly obtain q ( X m -1 | X m ) . Consequently, we employ a denoising neural network to estimate its mean and variance:

where Θ = { ϕ, θ } is the parameter of the neural network. µ Θ and Σ Θ are the predicted mean and variance by Θ . c m ϕ is produced by network ϕ , serving as conditions to guide the denoising (we will introduce it in detail later). Our ultimate goal in the denoising process is to ensure that the denoising step becomes the inverse process of the diffusion step, thus enabling p Θ ( X m -1 | X m , c m ϕ ) and q ( X m -1 | X m ) to have the same distribution. For more details, please refer to Section 3.4.

Bidirectional Consistent Denoising Unit. To utilize more temporal complementary information in trajectories, we propose the BCDUnit to couple two diffusion models. One is called the backward model, used to generate previous unobserved historical trajectories. The other, named as the forward model, is employed to produce future trajectories. We design a mutual guidance mechanism in the two diffusion models: At each step, the predicted unobserved historical trajectories and observed trajectories are jointly utilized to guide the forward model to generate future trajectories of the next step. Likewise, the predicted future trajectories together with observed trajectories are responsible for guiding the backward model to generate the unobserved historical trajectories. In this way, the BCDUnit continuously refines the future and unobserved historical trajectories, ultimately leading to accurate predictions.

We denote the network in BCDUnit as Θ = {{ ϕ fut , θ fut } , { ϕ unobs , θ unobs }} . As illustrated in Figure 1 (c), the forward model { ϕ fut , θ fut } , marked as red, is used to generate future trajectories, while the backward model { ϕ unobs , θ unobs } , depicted in yellow, is responsible for generating

unobserved historical trajectories. Here, the guidance information can be obtained by the network ϕ = { ϕ fut , ϕ unobs } . At the m th step, ϕ first generates future guidance g m fut and unobserved guidance g m unobs by sending the unobserved historical trajectories X m unobs and future trajectories X m fut to Bidirectional Long-Short Term Memory (Bi-LSTM), respectively:

We further incorporate observed trajectory guidance v obs into future and unobserved guidance to obtain more informative guidance c m ϕ,fut and c m ϕ,unobs , c m ϕ,fut = [ g m fut , v obs , m ] , c m ϕ,unobs = [ g m unobs , v obs , m ] , where [ · , · ] represents the concatenation operation. However, considering that it contains relatively high noise in the initial steps due to the inherent property of the diffusion model, it is not appropriate to directly concatenate them in the beginning. To this end, we adopt a gating mechanism to automatically learn the weights for balancing the contributions between two kinds of guidance information. We first calculate the weights for the future guidance g m fut and observed trajectory guidance v obs to produce appropriate guidance. Formally,

where γ fut is the learnt weight. Gate is a two layers MLP with the Sigmoid activation in this paper. The guidance c m can be then obtained by:

Similarly, c m ϕ,unobs can be obtained in the same way. After obtaining c m ϕ,fut and c m ϕ,unobs , the network θ = { θ fut , θ unobs } utilizes them to perform mutual guidance. They fuse the guidance with the predicted trajectories at the m th step. Then, a transformer is leveraged to capture temporal dependencies in the fused features. Finally, the outputs of the transformer are passed through two MLPs, i.e., W fut and W unobs , to obtain denoising intensities ϵ m fut and ϵ m unobs , respectively. These denoising intensities are employed to generate trajectories of the next steps, i.e. , X m -1 fut and X m -1 unobs .

In this way, we can bidirectionally and consistently generate unobserved historical trajectories and future trajectories step-by-step, effectively utilizing the complementary information between them.

3.4 The Objective Function

We define the objective as the negative log-likelihood of the model p Θ under X 0 fut and X 0 unobs as

Here, we also omit the subscript for convenient writing. By minimizing the objective L , the original trajectories X 0 fut , and X 0 unobs can be recovered through the denoising process. However, it is difficult to directly compute L . Therefore, we employ the variational methods to derive the Variational Lower Bound (VLB) [20] of the expectation, denoted as:

The first term of L V LB approximates to 0, as both q ( X M | X 0 ) and p Θ ( X M | c M ϕ ) are approximate to N (0 , I ) . The second term can be formulated as a special case of the third term when m = 1 . The third term computes the KL divergence between the estimated distribution p Θ ( X m -1 | X m , c m ϕ ) and the true posterior distribution q ( X m -1 | X m , X 0 ) , aiming to lower the error between the estimated distribution and the ground-truth posterior distribution. To determine the q ( X m -1 | X m , X 0 ) , we apply the Bayes formula as follows:

m

ϕ,fut

By applying the Bayes formula, we observe each term can be calculated with Equation 2. We then substitute the results of Equation 2 to Equation 9 to obtain the mean and variance of the posterior distribution q ( X m -1 | X m , X 0 ) as:

Note that ˜ σ m is a constant value related to β m , thus the KL in the third term can be further derived as:

By reparameterizing µ Θ and substituting it into Equation 11, we can further simplify the expression and finally obtain the diffusion loss.

where ϵ ∼ N (0 , I ) , X m fut and X m unobs are calculated by Equation 2, and ϵ Θ represents the aforementioned BCDUnit.

3.5 Mutual Guidance based Optimizing and Inference

When optimizing the model, we take the following four steps: Firstly, we sample future groundtruth trajectories X 0 fut , instantaneous observations X obs , and unobserved historical ground-truth trajectories X 0 unobs from the training dataset. We also sample a timestep m ∼ uniform (1 , M ) and a Gaussian noise ϵ ∼ N (0 , I ) . Secondly, we use the encoder to encode X obs as the v obs , and employ the Equation 2 to obtain the X m fut and X m unobs . Thirdly, we utilize the Equation 5,6, and 7 to obtain mutual guidance c m ϕ,unobs and c m ϕ,fut . Finally, we calculate the loss defined in Equation 12 and take the gradient descent to optimize the model until it converges.

For inference, we first sample instantaneous observations X obs from the testing data and two Gaussian noises ˆ X M fut and ˆ X M unobs from N (0 , I ) . Following the third step in the optimizing stage, we obtain mutual guidance c M ϕ,fut and c M ϕ,unobs . Finally, we execute the following two updates from m = M to 1 so that the predicted trajectories can be continuously refined until ˆ X 0 fut and ˆ X 0 unobs are generated,

where ϵ is a random noise sampled from the standard Gaussian distribution. The details of training and inference procedures are provided in Appendix.

4 Experiments

4.1 Experiment Settings

Dataset. We verify the effectiveness of our proposed method on the widely used ETH/UCY [37, 26] and Stanford Drone [38] Dataset (SDD). ETH/UCY is a dataset group. It consists of 5 different scenes, among which 2 scenes (ETH, HOTEL) are from the ETH dataset, and the other three scenes (UNIV, ZARA1, and ZARA2) come from the UCY dataset. The whole dataset includes more than 1500 pedestrians. We follow the widely used leave-one-scene-out protocol, i.e., the models are trained on 4 scenes and tested on the remaining one [15, 42]. SDD is a large scale dataset consisting of 20 scenes. It contains various agents such as pedestrians, bicycles, and vehicles.

Evaluation Metrics. Following previous works [43, 19, 14, 53, 48], we employ the Average Displacement Error (ADE) and Final Displacement Error (FDE) as metrics to evaluate the performance of future trajectory predictions. In the instantaneous trajectory prediction setting, the observations are

reduced to 2 frames, and the length of future predictions is 12 frames. Following previous works [46, 15, 31], we sample 20 future predicted trajectories, and report the final error by the minimum error over all predicted trajectories.

Backbone and Baselines. To demonstrate the compatible ability of our BCDiff, we apply it to three popular trajectory prediction models, Trajectron++ [42], PCCSNet [47], SGCN [43], and SocialVAE [50] by replacing their decoders with our BCDiff. Moreover, we compare BCDiff with the following baselines: Instantaneous means directly predicting the trajectories of the next 12 frames conditions on 2 frames of observations using the above three backbones. Additionally, we take MOE [46] and DTO [34] as two baselines for instantaneous trajectory prediction. Since the original MOE employs image semantic information, we also implement MOE without using image semantic information, denoted as MOEw/o Image , in order to fairly compare with other methods.

4.2 Experiment Results and Analysis

Performance on Instantaneous Trajectory Prediction. The overall performance are listed in Table 1 and 2. The results by applying our BCDiff to three different backbones, consistently outperform the baseline methods on all the datasets. This demonstrates the effectiveness of our proposed method for instantaneous trajectory prediction. Meanwhile, it also illustrates our method can be well compatible with different trajectory prediction models. Note that the performance of MOE declines when image information is not included, which shows that MOE heavily depends on image semantic information. In contrast, our proposed method does not rely on any additional image information.

Ablation Study. We perform the ablation studies, as listed in Table 3. We first utilize two separate diffusion models to predict previous unobserved trajectories and future trajectories, respectively, denoting it as BCDiff-w/o.Guidance. Then, we only utilize unobserved and observed trajectories to guide the generation of future trajectories, denoted as BCDiff-Uni.Guidance. BCDiff-Uni.Guidance is better than BCDiff-w/o.Guidance, meaning that the guidance from previous unobserved historical trajectories is helpful for improving the prediction performance of future trajectories. Moreover, by

incorporating bidirectional guidance, called BCDiff-Bi.Guidance, we observe the performance is further improved. This illustrates the guidance from future trajectories is beneficial for predicting previous unobserved trajectories, thereby improving the final prediction performance. Finally, we integrate the gating mechanism into our model, named as BCDiff-Bi.Guidance & Gate. It achieves the best performance, showing the gating mechanism is effective for trajectory prediction.

Analysis on Length of Predicted Unobserved Points. We investigate the impact of the number of predicted unobserved points on trajectory prediction. We use the Trajectron++ encoder [42], as listed in Table 4. As the number of predicted points increases, the accuracy of future predictions gradually improves, which can be attributed to the additional unobserved points providing more useful information. When T = 5 , the prediction performance starts to decline. This is because it becomes unprecise when predicting previous unobserved historical points with a larger length, thus introducing noise into our method.

Qualitative Analysis. We visualize the predicted trajectories in four different scenarios: Walking side-by-side, walking along, turning and yielding, as shown in Figure 2. BCDiff can accurately predict future trajectories, compared to MOE and DTO in all four scenarios. This is because BCDiff utilizes temporal information of unobserved trajectories to aid the prediction of future trajectories. Note that when pedestrians are walking side-by-side, as depicted in the first column of Figure 2, BCDiff utilizes the predicted unobserved historical trajectories to capture the walking patterns between two pedestrians, thereby precisely forecasting future trajectories. However, DTO and MOE predict deviated trajectories, due to only using two observed frames. Moreover, we visualize m m

Figure 3: Visualization of weights γ m fut and γ m unobs . Darker colors represent larger values.

the gating weights γ fut and γ unobs in the denoising process from m = 40 to m = 1 . As shown in Figure 3, the weights are gradually increased. It indicates our gating mechanism can adaptively learn weights, assigning lower weights to the predicted guidance in the initial steps.

a)

b)

c)

5 Conclusion

We proposed a diffusion model based framework for the task of instantaneous trajectory prediction. The proposed framework simultaneously generated both unobserved historical and future trajectories by designing a mutual guidance mechanism to couple two diffusion models, such that more temporal information can be leveraged for future trajectory prediction. We introduced a gating strategy, automatically balancing the contributions of different guidance information. Experiments demonstrated our proposed framework achieved superior performance to the state-of-the-art methods.

Acknowledgments and Disclosure of Funding

This work was supported by the NSFC under Grants 62122013, U2001211. This work was also supported by the Innovative Development Joint Fund Key Projects of Shandong NSF under Grants ZR2022LZH007.

References

Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 961-971, 2016.

Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems , 34:17981-17993, 2021.

Inhwan Bae, Jin-Hwi Park, and Hae-Gon Jeon. Non-probability sampling network for stochastic human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6477-6487, 2022.

Changan Chen, Yuejiang Liu, Sven Kreiss, and Alexandre Alahi. Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning. In 2019 international conference on robotics and automation (ICRA) , pages 6015-6022. IEEE, 2019.

Guangyi Chen, Zhenhao Chen, Shunxing Fan, and Kun Zhang. Unsupervised sampling promoting for stochastic human trajectory prediction. arXiv preprint arXiv:2304.04298 , 2023.

Guangyi Chen, Junlong Li, Nuoxing Zhou, Liangliang Ren, and Jiwen Lu. Personalized trajectory prediction via distribution discrimination. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15580-15589, 2021.

Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713 , 2020.

Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202 , 2022.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

William Feller. On the theory of stochastic processes, with particular reference to applications. In Selected Papers I , pages 769-798. Springer, 2015.

Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamics from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11525-11533, 2020.

Shansan Gong, Mukai Li, Jiangtao Feng, Zhiyong Wu, and LingPeng Kong. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933 , 2022.

Junru Gu, Chen Sun, and Hang Zhao. Densetnt: End-to-end trajectory prediction from dense goal sets. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15303-15312, 2021.

Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17113-17122, 2022.

Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2255-2264, 2018.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res. , 23(47):1-33, 2022.

Xin Huang, Stephen G McGill, Jonathan A DeCastro, Luke Fletcher, John J Leonard, Brian C Williams, and Guy Rosman. Diversitygan: Diversity-aware vehicle motion prediction via latent semantic sampling. IEEE Robotics and Automation Letters , 5(4):5089-5096, 2020.

Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. Stgat: Modeling spatial-temporal interactions for human trajectory prediction. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6272-6281, 2019.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761 , 2020.

Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. Advances in Neural Information Processing Systems , 32, 2019.

Vladimir Kulikov, Shahar Yadin, Matan Kleiner, and Tomer Michaeli. Sinddm: A single image denoising diffusion model. arXiv preprint arXiv:2211.16582 , 2022.

Mihee Lee, Samuel S Sohn, Seonghyeon Moon, Sejong Yoon, Mubbasir Kapadia, and Vladimir Pavlovic. Muse-vae: multi-scale vae for environment-aware long term trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2221-2230, 2022.

Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 336-345, 2017.

Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by example. In Computer graphics forum , volume 26, pages 655-664. Wiley Online Library, 2007.

Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. Advances in neural information processing systems , 33:19783-19794, 2020.

Xin Li, Xiaowen Ying, and Mooi Choo Chuah. Grip++: Enhanced graph-based interactionaware trajectory prediction for autonomous driving. arXiv preprint arXiv:1907.07792 , 2019.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2837-2845, 2021.

Yecheng Jason Ma, Jeevana Priya Inala, Dinesh Jayaraman, and Osbert Bastani. Likelihoodbased diverse sampling for trajectory forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13279-13288, 2021.

Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 759-776. Springer, 2020.

Weibo Mao, Chenxin Xu, Qi Zhu, Siheng Chen, and Yanfeng Wang. Leapfrog diffusion model for stochastic trajectory prediction. arXiv preprint arXiv:2303.10895 , 2023.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2021.

Alessio Monti, Angelo Porrello, Simone Calderara, Pasquale Coscia, Lamberto Ballan, and Rita Cucchiara. How many observations are enough? knowledge distillation for trajectory forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6553-6562, 2022.

Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. arXiv preprint arXiv:2207.05844 , 2022.

Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting future trajectories of multiple agents. In International Conference on Learning Representations , 2022.

Stefano Pellegrini, Andreas Ess, and Luc Van Gool. Improving data association by joint modeling of pedestrian trajectories and groupings. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part I 11 , pages 452-465. Springer, 2010.

Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human trajectory understanding in crowded scenes. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14 , pages 549-565. Springer, 2016.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1349-1358, 2019.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16 , pages 683-700. Springer, 2020.

Liushuai Shi, Le Wang, Chengjiang Long, Sanping Zhou, Mo Zhou, Zhenxing Niu, and Gang Hua. Sgcn: Sparse graph convolution network for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8994-9003, 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Hao Sun, Zhiqun Zhao, and Zhihai He. Reciprocal learning networks for human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7416-7425, 2020.

Jianhua Sun, Yuxuan Li, Liang Chai, Hao-Shu Fang, Yong-Lu Li, and Cewu Lu. Human trajectory prediction with momentary observation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6467-6476, 2022.

Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu. Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13250-13259, 2021.

Conghao Wong, Beihao Xia, Ziming Hong, Qinmu Peng, Wei Yuan, Qiong Cao, Yibo Yang, and Xinge You. View vertically: A hierarchical network for trajectory prediction via fourier spectrums. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII , pages 682-700. Springer, 2022.

Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6498-6507, 2022.

Pei Xu, Jean-Bernard Hayet, and Ioannis Karamouzas. Socialvae: Human trajectory prediction using timewise latents. In European Conference on Computer Vision , pages 511-528. Springer, 2022.

Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16 , pages 507-523. Springer, 2020.

Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9813-9823, 2021.

Jiangbei Yue, Dinesh Manocha, and He Wang. Human trajectory prediction via neural social physics. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV , pages 376-394. Springer, 2022.

Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. In Conference on Robot Learning , pages 895-904. PMLR, 2021.

Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12126-12134, 2019.

Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5826-5835, 2021.

Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8823-8833, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a  diffusion-based model specifically for instantaneous trajectory prediction. It leverages the consistency of the prediction for unobserved historical trajectory and the future trajectory to improve the accuracy of the future trajectory. Regarding the noise in the prediction wh, especially in the early steps, the paper introduced a gating mechanism to guide the fusion.\n\nSTRENGTHS:\n1. The paper is written-well and the method section is generally straightforward and coherent. \n2. The idea is intuitive. Since longer trajectories would provide more dynamic information and motion hints to help the system make better predictions, even those longer trajecotires are also predicted and contain a certain amount of noise. \n3. The experiments and comparison are thorough and have covered two popular datasets in pedestrian trajectory forecasting and three SOTA methods. \n4. The improvements are significant compared to the existing instantaneous trajectory forecasting methods.\n\nWEAKNESSES:\nMaybe there is some misunderstanding but my major concern is that the motivation of the method may not align with the motivation of the task very well. \n\nThe task of instantaneous trajectory prediction wants the system to extract valuable information when the objects' dynamic information is extremely limited.  As the authors mentioned in the related works L110-L111, the lack of temporal information is an inherent challenge of the task. Therefore, MOE[1] tries to use map information as an informative reference to help the system conduct more accurate predictions, and the motivation of DTO[2] is to do a thorough analysis of how limited observed information can lead to comparatively accurate predictions. \n\nBut in this paper, the authors let the system access the unobserved historical trajectories during the training time as one of the supervision signals. This data-augmentation-like design may make the comparison with other methods not fair enough. It would be necessary to rule out the influence of data augmentation when evaluating the boost brought by the system and prove the performance boost all comes from that the system has fully dug the potential of the limited observation. \n\n[1] Jianhua Sun, Yuxuan Li, Liang Chai, Hao-Shu Fang, Yong-Lu Li, and Cewu Lu. Human trajectory prediction with momentary observation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6467–6476, 2022.\n\n[2] Alessio Monti, Angelo Porrello, Simone Calderara, Pasquale Coscia, Lamberto Ballan, and Rita Cucchiara. How many observations are enough? knowledge distillation for trajectory forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6553–6562, 2022.\n\nQUESTIONS:\n1. Given the first point in [Weakness], for a fair comparison, I was curious to know the evaluation results if the existing methods also utilize unobserved trajectories as a prediction target for data augmentation. For instance, in Table 1, during the training of Trajectron++, Trajectron++ with MOE, and Trajectron++ with DTO, in addition to supervising future trajectories, we reverse the observed historical trajectories, use them as input to predict unobserved trajectories and evaluate the models. Another approach to demonstrate this could be seen in Table 3, BCDiff-w/o.Guidance, where instead of employing two diffusion models, only one model is used to predict both reversed unobserved and future trajectories.\n\n2. Regarding the gating mechanism, it is intuitive that the early steps should have lower weights given the noise level. Have authors tried simpler ways to handle the issue? Such as manually assigning lower weights to the early steps, and gradually increasing the weight till the last steps. \n\n3. Since the behavior of pedestrians can be very spontaneous, can you provide the evaluation comparison with other methods on some hard cases, especially for instantaneous trajectory prediction? Such as the cases where there are large contrasts between the unobserved trajectories and the future trajectories. \n\n4. Table 4 is very interesting. Can you provide some more quantitative results related to it? For instance, the numeric results of the accuracy on unobserved trajecotires and the relationship between which and the performance of the future predictions.\n\nLIMITATIONS:\nThe authors have not discussed the limitation. From my point of view, the limitation of the work is that the input information is so little that it can hardly provide valid predictions for samples in the wild or more complex situations. Furthermore, the dynamic or interaction types of the tested datasets are also very limited, the results can be biased without testing the methods on more diverse datasets, such as some autonomous driving datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a diffusion framework for instantaneous trajectory prediction (only observing two frames in history). Specifically, they develop two coupled diffusion models by designing a mutual guidance mechanism that can bidirectionally and consistently generate unobserved historical trajectories and future trajectories step-by-step, to utilize the complementary information between them. The experimental results demonstrate the proposed solution improves the accuracy of instantaneous trajectory prediction.\n\nSTRENGTHS:\n1. The intuition of the proposed solution is compelling. As mentioned in line 60, \"Both previous unobserved historical trajectories and future trajectories contain information of the same pedestrian at different timesteps, and thereby they provide complementary information to each other. It will be beneficial for the prediction of future trajectories if we can design an elegant method to simultaneously generate previous unobserved historical trajectories and future trajectories by fully leveraging the complementary information between them.\"\n2. The proposed solution (that simultaneously generates both future and unobserved historical trajectories in a consistent manner) sounds solid and should work in theory. \n3. The experimental results on multiple datasets and models prove the validity of the proposed solution. \n4. This paper is well-organized and easy to understand. \n5. The supplementary material provides more details about the model and experiments, which makes it easier for the reader to understand their work.\n\nWEAKNESSES:\n1. The proposed solution is time-consuming (4 seconds for predicting trajectories once). This problem appears to be particularly serious, especially in the area of autonomous driving. The long computation time should be caused by the model design (diffusion model and using an iterative way to generate predicted trajectories). \n2. Some details are missing. For example, what are the social and scene context e, and how to generate them for the Encoder is not clear. \n3. It will be better to show some failure cases. Considering that the trajectory prediction task is usually used in autonomous driving, failure cases are more valuable to study.\n\nQUESTIONS:\n1. Providing more information about failure cases will help readers better understand the performance of the proposed solution. \n2. Discussing some potential ways to speed up the proposed solution will be better.\n\nLIMITATIONS:\nThe long computation time should be the biggest limitation of the proposed solution. The authors discussed the issues in the supplementary material, but no potential doable solution. It may need more exploration.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the trajectory prediction in an instantaneous observation by generative diffusion models. Specifically, the authors propose a bidirectional consistent diffusion framework, named BCDiff, to address the issue of temporal information scarcity. BCDiff employs two coupled diffusion models are devised to predict unobserved historical trajectories and future trajectories with a mutual guidance mechanism, which ensures the consistency between predicted unobserved trajectories and predicted future trajectories.  The authors conduct extensive experiments to show the effectiveness of proposed method.\n\nSTRENGTHS:\nInstantaneous trajectory prediction is a common challenge in traffic scene. It is promising that employ the diffusion model in instantaneous trajectory prediction.  The conducted experimental results show proposed BCDiff is effective in the scenes of ETH-UCY and SDD. \nMoreover, the authors claim that the proposed method has friend compatibility with existing trajectory prediction models. Also, the organization of paper is well.\n\nWEAKNESSES:\nThe experiments is insufficient.\n1: The compared baselines are not the newest methods.  The authors should add more baselines with high accuracy, such as SocialVAE[1], SIT[2], MemoNet[3]. These baselines could be trained under the setting of the instantaneous trajectory prediction, such as observing 2 steps and predicting the next 12 steps. \n2: The inference time is missing. Instantaneous trajectory prediction often occurs on the emergency scene. The inference time is an important metrics to evaluate model. \n\n[1] Pei Xu, Jean-Bernard Hayet, and Ioannis Karamouzas. Socialvae: Human trajectory prediction using timewise latents. In ECCV, pages 511–528. Springer, 2022\n[2] Liushuai Shi, Le Wang, Chengjiang Long, Sanping Zhou, Fang Zheng, Nanning Zheng, and Gang Hua. Social interpretable tree for pedestrian trajectory prediction. In AAAI, pages 2235–2243, 2022\n[3] Chenxin Xu, Weibo Mao, Wenjun Zhang, and Siheng Chen. Remember intentions: Retrospective-memory-based trajectory prediction. In CVPR, pages 6488–6497, 2022.\n\nQUESTIONS:\nAs shown in WEAKNESS,\n1)\tI suggest the authors provide more comparison with the latest baselines in the setting of instantaneous trajectory prediction.\n2)\tStandard diffusion model suffer from expensive inference time, the author should provide detailed inference speed to evaluate the proposed model comprehensively.\n\nLIMITATIONS:\nRefer to the weakness and questions. My concerns mainly lies on the comparison with the latest baselines and the inference time.  Providing above information could be helpful to improve this paper comprehensively.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a bi-directional consistent diffusion approach for handling instantaneous trajectory prediction, i.e. trajectory prediction without requiring a large number of observation time. It relies on two coupled diffusion models to generate historical and future trajectories. The forward diffusion model predicts future trajectories conditioned on the predicted trajectories of the other model and the limited observed trajectories, while the backward model predicts the historical trajectories guided by the predicted future trajectories and the observed trajectories. Since the amount of noise in the first few diffusion steps is inherently relatively high, a gating mechanism is introduced to control the contribution of the two types of trajectories. In the evaluation, the approach achieves state-of-the-art performance on the TH/UCY and Stanford Drone datasets by replacing the decoder of three popular trajectory prediction models with the proposed one.\n\n\n\nHaving read the reviews and the rebuttal, I am going to maintain my positive assessment. The rebuttal addressed a lot of concerns, especially the experimental ones.\n\nSTRENGTHS:\n- The idea and method of the paper are easy to follow.\n\n- The use of diffusion models for trajectory generation is novel and interesting.\n\n- The approach works even with very sparse observations based on the experimental evaluation.\n\n- There are extensive experiments with different backbone architectures and datasets. \n\n- The proposed approach beats the current state-of-the-art in almost every experiment.\n\nWEAKNESSES:\n- The task of instantaneous trajectory prediction is not standard. It would value for the approach to be evaluated for the standard trajectory prediction protocol. Then it would be allowed to be compared with a wide range of methods. This is a particularly interesting point since the diffusion models are heavy in terms of parameters. It would allow for a comparison with the current state-of-the-art methods. \n\n- It is not clear how many observations are being used during the test time, for example, if it is $M = 7$, then the approach does not seem to improve the current state-of-the-art in trajectory. prediction. Moreover, it would be interesting to have an ablation study on the influence of step M on performance.\n\nQUESTIONS:\n- Clarify on the possibility to perform standard trajectory prediction and also discus the hyper-parameter selection.\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a bidirectional consistent diffusion framework for instantaneous trajectory prediction. Specifically, two coupled diffusion models are employed to predict unobserved historical and future trajectories jointly. In addition, a gating mechanism is presented to deal with the noisy situation at the initial steps. Experiments on ETH-UCY and SDD show that the proposed method is able to achieve the promising performance.\n\nSTRENGTHS:\n1. Clear-written and organized.\n2. Promising results on several datasets.\n\nWEAKNESSES:\n1. I am concerned about the consistency between the discussed motivation and the experiments. In the Abstract and Introduction Section, the motivation is that \"there are many real-world situations where the model lacks sufficient time to observe\". However, in the Experiments Section, the evaluation protocol is the commonly used leave-one-out setting, which is somewhat inconsistent with motivation. I assume the training and the inference phase both take only 2 frames as the input (this is another unclear point), which makes the  \"real-world situation\" that the training scenario lacks adequate observation and the testing scenario also lacks adequate observation.\n\n2. In addition to the above point, another concern is that only human trajectories are predicted. While such real-world situations often happen in the driving scenario.\n\n3. To better demonstrate the motivation, some motivated experiments could be included, i.e., existing methods fail to make accurate predictions when the observations are not adequate.\n\n4. In Lines 60-62, it claims that jointly predicting unobserved historical trajectories and future trajectories can be beneficial to each other. This claim is very similar to [1]. However, there is no experiment on this point, such as conducting these two jobs separately, or first predicting unobserved historical trajectories and then predicting future trajectories based on the predicted historical results.\n\n5. In the Method Section, LSTM is used as in Eq.5, why don't use the Bi-directional LSTM?\n\n6. Some details are missing. In my opinion, there is a gap between the training and the inference phase. During the training, the method has access to the future gound-truth trajectory, in other words, there could have a forward and backward process. While in the inference phase, the method mustn't have access to future information, and the details of how to evaluate the trained model are missing.\n\n\n\n[1] Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction. CVPR'23\n\nQUESTIONS:\nMy questions are raised in the above Weaknesses Section.\n\nLIMITATIONS:\nI didn't see any limitations discussed in the submission.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}