SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rongqing Li
lirongqing99@gmail.com
Changsheng Li *
lcs@bit.edu.cn
Dongchun Ren
ALLRIDE.AI
Dongchun. ren@allride.ai
Guangyi Chen
CMU & MBZUAI
guangyichen1994@gmail.com
Ye Yuan
yuan-ye@bit.edu.cn
Guoren Wang
wanggrbit@126.com
The objective of pedestrian trajectory prediction is to estimate the future paths of pedestrians by leveraging historical observations, which plays a vital role in ensuring the safety of self-driving vehicles and navigation robots. Previous works usually rely on a sufficient amount of observation time to accurately predict future trajectories. However, there are many real-world situations where the model lacks sufficient time to observe, such as when pedestrians abruptly emerge from blind spots, resulting in inaccurate predictions and even safety risks. Therefore, it is necessary to perform trajectory prediction based on instantaneous observations, which has rarely been studied before. In this paper, we propose a Bi-directional Consistent Diffusion framework tailored for instantaneous trajectory prediction, named BCDiff. At its heart, we develop two coupled diffusion models by designing a mutual guidance mechanism which can bidirectionally and consistently generate unobserved historical trajectories and future trajectories step-by-step, to utilize the complementary information between them. Specifically, at each step, the predicted unobserved historical trajectories and limited observed trajectories guide one diffusion model to generate future trajectories, while the predicted future trajectories and observed trajectories guide the other diffusion model to predict unobserved historical trajectories. Given the presence of relatively high noise in the generated trajectories during the initial steps, we introduce a gating mechanism to learn the weights between the predicted trajectories and the limited observed trajectories for automatically balancing their contributions. By means of this iterative and mutually guided generation process, both the future and unobserved historical trajectories undergo continuous refinement, ultimately leading to accurate predictions. Essentially, BCDiff is an encoder-free framework that can be compatible with existing trajectory prediction models in principle. Experiments show that our proposed BCDiff significantly improves the accuracy of instantaneous trajectory prediction on the ETH/UCY and Stanford Drone datasets, compared to related approaches.
*Changsheng Li (lcs@bit.edu.cn) is the corresponding author
Pedestrian trajectory prediction aims to predict future trajectories conditioned on their past movements, which is an important task for autonomous driving [25, 57] and navigation robot [4]. Previous pedestrian trajectory prediction approaches usually rely on long enough observation time (typically, 2 to 3 seconds) for a pedestrian to precisely predict the future trajectories [53, 48, 54, 13]. However, in many real-world situations, e.g., when pedestrians suddenly emerge from blind spots and are in close proximity to autonomous vehicles, traditional trajectory prediction methods do not have ample time to collect a sufficient number of locations. This leads to sub-optimal prediction performance and potentially unsafe behaviors in the decision-making of autonomous vehicles and robots. Therefore, it is quite necessary to forecast future trajectories based on limited or instantaneous observations.
The prediction of instantaneous trajectories for pedestrians is a highly challenging task due to the limited observation time. In some cases, as extreme as it can be, merely two frames of locations can be observed. In the face of such a challenging task, there have been only a few academic works to date. MOE [46] is the first to propose the problem of instantaneous trajectory prediction, and thus is the most relevant to ours. MOE incorporates scene context information into limited observations and introduces the masked trajectory complement and context restoration as self-supervised tasks to pretrain the model. However, since MOE only utilizes instantaneous temporal information acquired from limited trajectories, it might be hard to accurately predict the future trajectories of a pedestrian with complex behavior such as turning and yielding. DTO [34] focuses on lowering the influence of noise introduced by incorrect detection and tracking, and attempts to employ limited observed trajectory to alleviate this problem. It utilizes the knowledge distillation technique to distill knowledge from a teacher model trained with an ample amount of long observations, and transfer the knowledge to a student model receiving fewer observations as input. Although these approaches have shown some effectiveness in instantaneous trajectory prediction, the representation of a pedestrian is restricted to two frames of locations, which contains extremely limited temporal information.
In this paper, we propose BCDiff, a bidirectional consistent diffusion framework specifically designed for the instantaneous trajectory prediction task. As we know, the diffusion model is a generative model, which has been successfully applied to various generation tasks, including image synthesis [39, 33], image denoising [23], etc. Different from them, we leverage the diffusion model for instantaneous trajectory prediction. We devise two coupled diffusion models to bidirectionally generate previous unobserved trajectories and future trajectories from random noises, which can address the issue of temporal information scarcity in limited observations. The underlying intuition behind this idea is: Both previous unobserved historical trajectories and future trajectories contain information of the same pedestrian at different timesteps, and thereby they provide complementary information to each other. It will be beneficial for the prediction of future trajectories if we can design an elegant method to simultaneously generate previous unobserved historical trajectories and future trajectories by fully leveraging the complementary information between them.
To accomplish this, we devise a step-by-step mutual guidance mechanism in two coupled diffusion models to simultaneously generate previous unobserved historical trajectories and future trajectories. Specifically, at each step, the predicted unobserved historical trajectories, together with the limited observed trajectories serve as a guidance for one diffusion model to predict a denoising intensity, which is then used to generate future trajectories of the subsequent step. Likewise, the predicted future trajectories, together with the observed trajectories, guide the other diffusion model to predict unobserved historical trajectories of the next step. Meanwhile, considering there exists relatively high noise in the generated trajectories during the initial steps, we devise a gating mechanism to learn the weights between the predicted trajectories and the limited observed trajectories for automatically controlling the proportion of the guidance information from two kinds of trajectories during each generation step. Through this iterative and mutually guided generation process, the future and unobserved historical trajectories are continuously refined, ultimately leading to precise predictions. Notably, our proposed BCDiff is encoder-free and is compatible with existing trajectory prediction encoders in principle, allowing them to gracefully handle cases with instantaneous observations.
Our contributions can be summarized as follows: 1) We propose BCDiff, a diffusion model based framework tailored for instantaneous trajectory prediction. BCDiff can simultaneously generate both future and unobserved historical trajectories in a consistent manner, which can effectively leverage complementary information between them. 2) We devise a step-by-step mutual guidance mechanism to couple two diffusion models for trajectory generation, and present a gating strategy to adaptively
adjust the contributions of the guidance information between two kinds of trajectories. 3) Experiments demonstrate our proposed BCDiff significantly improves the accuracy of instantaneous prediction and outperforms the state-of-the-art methods on ETH/UCY and Stanford Drone datasets.
2.1 Traditional Trajectory Prediction
Traditional trajectory prediction methods aim to predict future trajectories given sufficient observation time. To capture complex interactions between pedestrians, many methods have been proposed [1, 15, 40]. These models utilize a social mechanism to aggregate neighboring actors and broadcast information to each actor. In addition, graph neural networks [49, 11, 19, 22, 27, 28] and transformer architectures [52, 36, 35, 51] are introduced to encapsulate implicit interactions among pedestrians. To resolve the problem of high uncertainty in pedestrians, researchers propose stochastic generative models, such as GAN [15, 22, 40, 45, 55], VAE [25, 24, 31, 53], and Diffusion Models [14, 32], to better capture the variability in future trajectories. Various sampling strategies are designed to avoid purely random sampling [6, 3, 18, 5, 30]. However, the aforementioned methods can accurately predict trajectories only when a sufficient amount of long observation trajectories are available. The accuracy cannot be guaranteed given the limited observation time. In contrast to these methods, our goal is to address the trajectory prediction problem under instantaneous observation scenarios.
2.2 Instantaneous Trajectory Prediction
Instantaneous trajectory prediction aims to predict future trajectories given a limited number of observed trajectory points. In the most extreme scenarios, merely two frames of locations can be observed. This task poses significant challenges due to the exceedingly short observation period. MOE [46] and DTO [34] are two trajectory prediction methods based on limited observed trajectory points. MOE proposes a feature extractor to incorporate image semantic information and develops a self-supervised task to enhance the representational ability of instantaneous observations. Meanwhile, DTO investigates the influence of noise introduced by detection or tracking to trajectory prediction, and intends to use fewer trajectory points in a knowledge distillation framework to address the issue. Despite these advances, they fail to address the inherent lack of temporal information in instantaneous trajectory prediction. In our study, we attempt to predict and leverage previous unobserved trajectories to capture more temporal information, thereby improving the prediction accuracy of future trajectories.
2.3 Diffusion Models
The diffusion model is a class of stochastic generation models, which exhibits amazing performance in a diverse range of fields such as image synthesis [9, 39, 33], audio synthesis [7, 21] and text generation [2, 12, 8]. Among these works, a typical diffusion model is the Denoising diffusion probabilistic model (DDPM) [44, 16]. DDPM is inspired by the non-equilibrium thermodynamics, in which a forward Markov process perturbs real data into noise, and a reverse Markov process converts noise back to real data. DDPM has been widely used in various tasks, including image super-resolution [41, 17], 3D point cloud generation [29, 56] etc. For instance, CDM [17] cascades multiple DDPM models to generate images of increasing resolution. The work in [29] employs a heat bath mechanism on DDPM to facilitate the generation of 3d point clouds. Different from these works, we leverage DDPM to solve the problem of instantaneous trajectory prediction.
In this work, we aim to tackle the task of instantaneous trajectory prediction, where we assume only two frames are observed, i.e., the most extreme case. We denote  $X_{obs} = \{x_1,x_2\}$  as the observation locations. The ground-truth future trajectory is symbolized as  $X_{fut} = \{x_{3},x_{4},\dots ,x_{T_{fut} + 2}\}$ , where  $T_{fut}$  is the prediction length, and  $x_{i}\in \mathbb{R}^{2}$  is the 2D coordinate of a trajectory location. Moreover, we characterize the previous unobserved trajectories as  $X_{unobs} = \{x_{1 - T_{unobs}},\ldots ,x_{-1},x_0\}$ , where  $T_{unobs}$  represents the length of the unobserved trajectories. Our objective is to develop a diffusion
(a) Enc-Dec Architecture
(c) Structure of BCDUnit
Figure 1: An illustration of our BCDiff framework. (a) The overall architecture comprises an encoder and a decoder. The encoder is utilized to generate features of trajectories by incorporating social and scene context. The decoder is depicted in Figure 1 (b). The decoder, i.e., our proposed BCDiff, generates previous unobserved historical trajectories and future trajectories step-by-step through two coupled diffusion models. (c) The BCDUnit describes the details of two diffusion models marked in red and yellow. We design a mutual guidance mechanism and predict denoise intensities that are used for generating unobserved historical trajectories and future trajectories at the next step.
models based method to generate both previous unobserved historical and future trajectories, so that more temporal information can be captured to better predict future trajectories, given only two observed frames. Since we utilize the diffusion model, we denote the maximum diffusion steps as  $M$ , and use  $X_{fut}^{m}$  and  $X_{unobs}^{m}$  to represent the future and unobserved trajectories following a diffusion of  $m$  steps or a denoising of  $M - m$  steps, respectively. Note that  $X_{unobs} = X_{unobs}^{0}$  and  $X_{fut} = X_{fut}^{0}$ .
3.2 Overall Architecture
The overall architecture consists of an encoder and a decoder, as shown in Figure 1 (a). The encoder encodes  $X_{obs}$  as  $v_{obs}$ , and captures social and scene context  $e$ . The decoder is our proposed BCDiff framework, as shown in Figure 1 (b), which simultaneously generates previous observed historical trajectories and future trajectories step-by-step through the Bidirectional Consistent Denoising Unit (BCDUnit). The BCDUnit comprises two coupled diffusion models, As shown in Figure 1 (c), the diffusion model  $\{\phi_{fut},\theta_{fut}\}$  marked in red color, is responsible for generating future trajectories, while the diffusion model  $\{\phi_{unobs},\theta_{unobs}\}$ , depicted in yellow, is used for generating unobserved historical trajectories. The two diffusion models are coupled through a mutual guidance mechanism. To be specific, as the  $m^{th}$  step, the network  $\phi = \{\phi_{fut},\phi_{unobs}\}$  leverages two bidirectional LSTMs to encode unobserved trajectories and future trajectories as mutual guidance  $g_{fut}^{m}$  and  $g_{unobs}^{m}$  used for generating each other in the  $m - 1^{th}$  step. Considering the guidance information contains relatively high noise during the initial steps, we introduce a gate mechanism to balance the contributions between the observed guidance  $v_{obs}$  and future guidance  $g_{fut}^{m}$ , as well as the observed guidance  $v_{obs}$  and unobserved guidance  $g_{unobs}^{m}$ , ultimately producing the appropriate guidance  $\mathbf{c}_{\phi ,fut}$  and  $\mathbf{c}_{\phi ,unobs}$ . Then the network  $\theta = \{\theta_{fut},\theta_{unobs}\}$  fuses the guidance with both predicted unobserved historical and future trajectories at current steps to generate the denoise intensities, which is used to simultaneously generate  $X_{fut}^{m - 1}$  and  $X_{unobs}^{m - 1}$  of the next step. It is noteworthy that both previous unobserved historical trajectories and future trajectories contain the information of the same pedestrian at different timesteps, thus we adopt a parameter-shared transformer across  $\theta_{fut}$  and  $\theta_{unobs}$ .
3.3 Bidirectional Consistent Diffusion
In this section, we introduce our proposed BCDiff framework, which contains two coupled diffusion models to consistently generate trajectories in two directions, i.e., simultaneously predicting previous unobserved trajectories and future trajectories. In this paper, we utilize DDPM [16] as our basic diffusion model, because of its excellent performance in various tasks. For the generation of each direction, the diffusion model executes diffusion and conditional denoising processes. The diffusion process aims to intentionally add a series of noises to a ground-truth trajectory, while the conditional denoising process recovers the trajectory from noise inputs conditioned on the guidance.
Diffusion Process. The diffusion process is defined as a Markov chain, conditioned on the ground-truth trajectories  $X_{fut}^{0}, X_{unobs}^{0}$ . To write conveniently, we omit the subscripts, allowing  $X^{0}$  to represent unobserved historical trajectories or future trajectories. The diffusion process generates the sequence  $\{X^i\}_{i=1}^M$  by accumulating noise  $M$  times, i.e.,
$$
q \left(X ^ {1: M} \mid X ^ {0}\right) = \prod_ {m = 1} ^ {M} q \left(X ^ {m} \mid X ^ {m - 1}\right), \quad q \left(X ^ {m} \mid X ^ {m - 1}\right) = \mathcal {N} \left(X ^ {m}; \sqrt {\alpha^ {m}} X ^ {m - 1}, (1 - \alpha^ {m}) \mathbf {I}\right), \tag {1}
$$
where  $\mathcal{N}$  denotes the Gaussian distribution, and  $\alpha^m$  represents the noise intensity from  $X^m$  to  $X^{m-1}$ . Typically,  $\alpha^m$  is equal to  $1 - \beta^m$ , where  $\beta^m$  is a pre-defined value belonging to the interval [0,1]. Due to the additivity of Gaussian distributions, we are able to directly obtain  $X^m$  from  $X^0$ :
$$
q \left(X ^ {m} \mid X ^ {0}\right) = \mathcal {N} \left(X ^ {m}; \sqrt {\bar {\alpha} ^ {m}} X ^ {0}, \left(1 - \bar {\alpha} ^ {m}\right) \mathbf {I}\right), \quad \bar {\alpha} ^ {m} = \prod_ {i = 1} ^ {m} \alpha^ {i}, \tag {2}
$$
where  $\overline{\alpha}^m = \prod_{i=1}^m \alpha^i$ . Note that as  $M$  becomes sufficiently large,  $\overline{\alpha}$  approaches zero, making  $q(X^M | X^0)$  converge to the standard Gaussian distribution. By employing the above process, the ground-truth trajectory is transformed into the Gaussian noise  $X^M \sim \mathcal{N}(0, I)$ .
Conditional Denoising Process. In this process, we aim to generate  $X^0$  from the Gaussian noise  $X^M \sim \mathcal{N}(0, I)$ . We can reverse the aforementioned diffusion process, to gradually denoise from the Gaussian noise  $X^M$  and reconstruct  $X^0$ . Based on the proof in [10]: If  $q(X^m | X^{m-1})$  follows a Gaussian distribution and  $\beta^m$  is sufficiently small,  $q(X^{m-1} | X^m)$  also satisfies a Gaussian distribution. Therefore, we formulate  $q(X^{m-1} | X^m)$  as a Gaussian Markov process. However, it is intractable to directly obtain  $q(X^{m-1} | X^m)$ . Consequently, we employ a denoising neural network to estimate its mean and variance:
$$
p _ {\Theta} \left(X ^ {0: M} \mid \mathbf {c} _ {\phi} ^ {m}\right) = p \left(X ^ {M} \mid \mathbf {c} _ {\phi} ^ {m}\right) \prod_ {m = 1} ^ {M} p _ {\Theta} \left(X ^ {m - 1} \mid X ^ {m}, \mathbf {c} _ {\phi} ^ {m}\right), \tag {3}
$$
$$
p _ {\Theta} \left(X ^ {m - 1} \mid X ^ {m}, \mathbf {c} _ {\phi} ^ {m}\right) = \mathcal {N} \left(X ^ {m - 1} \mid \mathbf {c} _ {\phi} ^ {m}; \mu_ {\Theta} \left(X ^ {m}, m, \mathbf {c} _ {\phi} ^ {m}\right), \boldsymbol {\Sigma} _ {\Theta} \left(X ^ {m}, m, \mathbf {c} _ {\phi} ^ {m}\right)\right), \tag {4}
$$
where  $\Theta = \{\phi, \theta\}$  is the parameter of the neural network.  $\mu_{\Theta}$  and  $\Sigma_{\Theta}$  are the predicted mean and variance by  $\Theta$ .  $\mathbf{c}_{\phi}^{m}$  is produced by network  $\phi$ , serving as conditions to guide the denoising (we will introduce it in detail later). Our ultimate goal in the denoising process is to ensure that the denoising step becomes the inverse process of the diffusion step, thus enabling  $p_{\Theta}(X^{m-1}|X^{m}, \mathbf{c}_{\phi}^{m})$  and  $q(X^{m-1}|X^{m})$  to have the same distribution. For more details, please refer to Section 3.4.
Bidirectional Consistent Denoising Unit. To utilize more temporal complementary information in trajectories, we propose the BCDUnit to couple two diffusion models. One is called the backward model, used to generate previous unobserved historical trajectories. The other, named as the forward model, is employed to produce future trajectories. We design a mutual guidance mechanism in the two diffusion models: At each step, the predicted unobserved historical trajectories and observed trajectories are jointly utilized to guide the forward model to generate future trajectories of the next step. Likewise, the predicted future trajectories together with observed trajectories are responsible for guiding the backward model to generate the unobserved historical trajectories. In this way, the BCDUnit continuously refines the future and unobserved historical trajectories, ultimately leading to accurate predictions.
We denote the network in BCDUnit as  $\Theta = \{\{\phi_{fut},\theta_{fut}\} ,\{\phi_{unobs},\theta_{unobs}\} \}$ . As illustrated in Figure 1 (c), the forward model  $\{\phi_{fut},\theta_{fut}\}$ , marked as red, is used to generate future trajectories, while the backward model  $\{\phi_{unobs},\theta_{unobs}\}$ , depicted in yellow, is responsible for generating
unobserved historical trajectories. Here, the guidance information can be obtained by the network  $\phi = \{\phi_{fut},\phi_{unobs}\}$ . At the  $m^{th}$  step,  $\phi$  first generates future guidance  $g_{fut}^{m}$  and unobserved guidance  $g_{unobs}^{m}$  by sending the unobserved historical trajectories  $X_{unobs}^{m}$  and future trajectories  $X_{fut}^{m}$  to Bidirectional Long-Short Term Memory (Bi-LSTM), respectively:
$$
g _ {f u t} ^ {m} = \mathbf {B i} - \mathbf {L S T M} \left(X _ {u n o b s} ^ {m}\right), g _ {u n o b s} ^ {m} = \mathbf {B i} - \mathbf {L S T M} \left(X _ {f u t} ^ {m}\right). \tag {5}
$$
We further incorporate observed trajectory guidance  $v_{obs}$  into future and unobserved guidance to obtain more informative guidance  $\mathbf{c}_{\phi, fut}^{m}$  and  $\mathbf{c}_{\phi, unobs}^{m}$ ,  $\mathbf{c}_{\phi, fut}^{m} = [g_{fut}^{m}, v_{obs}, m]$ ,  $\mathbf{c}_{\phi, unobs}^{m} = [g_{unobs}^{m}, v_{obs}, m]$ , where  $[\cdot, \cdot]$  represents the concatenation operation. However, considering that it contains relatively high noise in the initial steps due to the inherent property of the diffusion model, it is not appropriate to directly concatenate them in the beginning. To this end, we adopt a gating mechanism to automatically learn the weights for balancing the contributions between two kinds of guidance information. We first calculate the weights for the future guidance  $g_{fut}^{m}$  and observed trajectory guidance  $v_{obs}$  to produce appropriate guidance. Formally,
$$
\gamma_ {f u t} ^ {m} = \mathbf {G a t e} _ {f u t} ([ g _ {f u t} ^ {m}, v _ {\text {o b s}}, m ]), \tag {6}
$$
where  $\gamma_{fut}^{m}$  is the learnt weight. Gate is a two layers MLP with the Sigmoid activation in this paper.
The guidance  $\mathbf{c}_{\phi ,f_{ut}}^{m}$  can be then obtained by:
$$
\mathbf {c} _ {\phi , f u t} ^ {m} = \gamma_ {f u t} ^ {m} g _ {f u t} ^ {m} + (1 - \gamma_ {f u t} ^ {m}) v _ {o b s}. \tag {7}
$$
Similarly,  $\mathbf{c}_{\phi ,unobs}^{m}$  can be obtained in the same way. After obtaining  $\mathbf{c}_{\phi ,fut}^{m}$  and  $\mathbf{c}_{\phi ,unobs}^{m}$ , the network  $\theta = \{\theta_{fut},\theta_{unobs}\}$  utilizes them to perform mutual guidance. They fuse the guidance with the predicted trajectories at the  $m^{th}$  step. Then, a transformer is leveraged to capture temporal dependencies in the fused features. Finally, the outputs of the transformer are passed through two MLPs, i.e.,  $W_{fut}$  and  $W_{unobs}$ , to obtain denoising intensities  $\epsilon_{fut}^{m}$  and  $\epsilon_{unobs}^{m}$ , respectively. These denoising intensities are employed to generate trajectories of the next steps, i.e.,  $X_{fut}^{m - 1}$  and  $X_{unobs}^{m - 1}$ .
In this way, we can bidirectionally and consistently generate unobserved historical trajectories and future trajectories step-by-step, effectively utilizing the complementary information between them.
3.4 The Objective Function
We define the objective as the negative log-likelihood of the model  $p_{\Theta}$  under  $X_{fut}^{0}$  and  $X_{unobs}^{0}$  as
$$
\mathcal {L} = \mathbb {E} \left[ - \operatorname {l o g p} _ {\Theta} \left(X ^ {0}\right) \right]. \tag {8}
$$