SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here, we also omit the subscript for convenient writing. By minimizing the objective  $\mathcal{L}$ , the original trajectories  $X_{fut}^{0}$ , and  $X_{unobs}^{0}$  can be recovered through the denoising process. However, it is difficult to directly compute  $\mathcal{L}$ . Therefore, we employ the variational methods to derive the Variational Lower Bound (VLB) [20] of the expectation, denoted as:
$$
\begin{array}{l} \mathcal {L} \leq - \mathcal {L} _ {V L B} = \mathbb {E} _ {q} \left[ l o g \frac {q (X ^ {1 : M} | X ^ {0})}{p _ {\Theta} (X ^ {0 : M})} \right] = \mathbb {E} _ {q} [ K L (q (X ^ {M} | X ^ {0}) | | p _ {\Theta} (X ^ {M} | \mathbf {c} _ {\phi} ^ {M}) \\ - \log p _ {\Theta} (X ^ {0} | X ^ {1}, \mathbf {c} _ {\phi} ^ {1}) + \sum_ {m = 2} ^ {M} K L (q (X ^ {m - 1} | X ^ {m}, X ^ {0}) | | p _ {\Theta} (X ^ {m - 1} | X ^ {m}, \mathbf {c} _ {\phi} ^ {m})) ]. \\ \end{array}
$$
The first term of  $\mathcal{L}_{VLB}$  approximates to 0, as both  $q(X^{M}|X^{0})$  and  $p_{\Theta}(X^{M}|\mathbf{c}_{\phi}^{M})$  are approximate to  $\mathcal{N}(0,I)$ . The second term can be formulated as a special case of the third term when  $m = 1$ . The third term computes the KL divergence between the estimated distribution  $p_{\Theta}(X^{m - 1}|X^m,\mathbf{c}_\phi^m)$  and the true posterior distribution  $q(X^{m - 1}|X^m,X^0)$ , aiming to lower the error between the estimated distribution and the ground-truth posterior distribution. To determine the  $q(X^{m - 1}|X^m,X^0)$ , we apply the Bayes formula as follows:
$$
q \left(X ^ {m - 1} \mid X ^ {m}, X ^ {0}\right) = q \left(X ^ {m} \mid X ^ {m - 1}, X ^ {0}\right) \frac {q \left(X ^ {m - 1} \mid X ^ {0}\right)}{q \left(X ^ {m} \mid X ^ {0}\right)} = q \left(X ^ {m} \mid X ^ {m - 1}\right) \frac {q \left(X ^ {m - 1} \mid X ^ {0}\right)}{q \left(X ^ {m} \mid X ^ {0}\right)}. \tag {9}
$$
By applying the Bayes formula, we observe each term can be calculated with Equation 2. We then substitute the results of Equation 2 to Equation 9 to obtain the mean and variance of the posterior distribution  $q(X^{m - 1}|X^m,X^0)$  as:
$$
\widetilde {\sigma} ^ {m} = \frac {1 - \bar {\alpha} ^ {m - 1}}{1 - \bar {\alpha} ^ {m}} \cdot \beta^ {m}, \quad \widetilde {\mu} ^ {m} \left(X ^ {m}, X ^ {0}\right) = \frac {\sqrt {\alpha^ {m}} \left(1 - \bar {\alpha} ^ {m - 1}\right)}{1 - \bar {\alpha} ^ {m}} X ^ {m} + \frac {\sqrt {\bar {\alpha} ^ {m - 1}} \beta^ {m}}{1 - \bar {\alpha} ^ {m}} X ^ {0}. \tag {10}
$$
Note that  $\widetilde{\sigma}^m$  is a constant value related to  $\beta^m$ , thus the KL in the third term can be further derived as:
$$
K L \left(q \left(X ^ {m - 1} \mid X ^ {m}, X ^ {0}\right) \| p _ {\Theta} \left(X ^ {m - 1} \mid X ^ {m}, \mathbf {c} _ {\phi} ^ {m}\right)\right) \propto \left\| \tilde {\mu} ^ {m} \left(X ^ {m}, X ^ {0}\right) - \mu_ {\Theta} \left(X ^ {m}, m, \mathbf {c} _ {\phi} ^ {m}\right) \right\| _ {2}. \tag {11}
$$
By reparameterizing  $\mu_{\Theta}$  and substituting it into Equation 11, we can further simplify the expression and finally obtain the diffusion loss.
$$
\mathcal {L} _ {u n o b s} = \mathbb {E} _ {X _ {u n o b s} ^ {0}, \epsilon , m} | | \epsilon - \epsilon_ {\Theta} \left(X _ {u n o b s} ^ {m}, m, \mathbf {c} _ {\phi , u n o b s} ^ {m}\right) | | _ {2}
$$
$$
\mathcal {L} _ {f u t} = \mathbb {E} _ {X _ {f u t} ^ {0}, \epsilon , m} | | \epsilon - \epsilon_ {\Theta} (X _ {f u t} ^ {m}, m, \mathbf {c} _ {\phi , f u t} ^ {m}) | | _ {2}
$$
$$
\mathcal {L} _ {d} = \mathcal {L} _ {f u t} + \mathcal {L} _ {u n o b s} \tag {12}
$$
where  $\epsilon \sim \mathcal{N}(0,I)$ ,  $X_{fut}^{m}$  and  $X_{unobs}^{m}$  are calculated by Equation 2, and  $\epsilon_{\Theta}$  represents the aforementioned BCDUnit.
3.5 Mutual Guidance based Optimizing and Inference
When optimizing the model, we take the following four steps: Firstly, we sample future ground-truth trajectories  $X_{fut}^{0}$ , instantaneous observations  $X_{obs}$ , and unobserved historical ground-truth trajectories  $X_{unobs}^{0}$  from the training dataset. We also sample a timestep  $m \sim \text{uniform}(1, M)$  and a Gaussian noise  $\epsilon \sim \mathcal{N}(0, I)$ . Secondly, we use the encoder to encode  $X_{obs}$  as the  $v_{obs}$ , and employ the Equation 2 to obtain the  $X_{fut}^{m}$  and  $X_{unobs}^{m}$ . Thirdly, we utilize the Equation 5,6, and 7 to obtain mutual guidance  $\mathbf{c}_{\phi,unobs}^{m}$  and  $\mathbf{c}_{\phi,fut}^{m}$ . Finally, we calculate the loss defined in Equation 12 and take the gradient descent to optimize the model until it converges.
For inference, we first sample instantaneous observations  $X_{obs}$  from the testing data and two Gaussian noises  $\hat{X}_{fut}^{M}$  and  $\hat{X}_{unobs}^{M}$  from  $\mathcal{N}(0,I)$ . Following the third step in the optimizing stage, we obtain mutual guidance  $\mathbf{c}_{\phi ,fut}^{M}$  and  $\mathbf{c}_{\phi ,unobs}^{M}$ . Finally, we execute the following two updates from  $m = M$  to 1 so that the predicted trajectories can be continuously refined until  $\hat{X}_{fut}^{0}$  and  $\hat{X}_{unobs}^{0}$  are generated,
$$
\hat {X} _ {f u t} ^ {m - 1} := \frac {1}{\sqrt {\alpha^ {m}}} \left(\hat {X} _ {f u t} ^ {m} - \frac {\beta^ {m}}{\sqrt {1 - \overline {{\alpha}} ^ {m}}} \epsilon_ {\Theta} \left(\hat {X} _ {f u t} ^ {m}, m, \mathbf {c} _ {\phi , f u t} ^ {m}\right)\right) + \widetilde {\sigma} ^ {m} \epsilon , \tag {13}
$$
$$
\hat {X} _ {u n o b s} ^ {m - 1} := \frac {1}{\sqrt {\alpha^ {m}}} \left(\hat {X} _ {u n o b s} ^ {m} - \frac {\beta^ {m}}{\sqrt {1 - \bar {\alpha} ^ {m}}} \epsilon_ {\Theta} \left(\hat {X} _ {u n o b s} ^ {m}, m, \mathbf {c} _ {\phi , u n o b s} ^ {m}\right)\right) + \widetilde {\sigma} ^ {m} \epsilon , \tag {14}
$$
where  $\epsilon$  is a random noise sampled from the standard Gaussian distribution. The details of training and inference procedures are provided in Appendix.
Dataset. We verify the effectiveness of our proposed method on the widely used ETH/UCY [37, 26] and Stanford Drone [38] Dataset (SDD). ETH/UCY is a dataset group. It consists of 5 different scenes, among which 2 scenes (ETH, HOTEL) are from the ETH dataset, and the other three scenes (UNIV, ZARA1, and ZARA2) come from the UCY dataset. The whole dataset includes more than 1500 pedestrians. We follow the widely used leave-one-scene-out protocol, i.e., the models are trained on 4 scenes and tested on the remaining one [15, 42]. SDD is a large scale dataset consisting of 20 scenes. It contains various agents such as pedestrians, bicycles, and vehicles.
Evaluation Metrics. Following previous works [43, 19, 14, 53, 48], we employ the Average Displacement Error (ADE) and Final Displacement Error (FDE) as metrics to evaluate the performance of future trajectory predictions. In the instantaneous trajectory prediction setting, the observations are
Table 1: Comparisons of different methods on the ETH/UCY dataset. The metrics are presented as ADE/FDE (m).
Model	Methods	Dataset
ETH	HOTEL	UNIV	ZARA1	ZARA2	AVG
Trajectory++	Instantaneous	0.76/1.43	0.30/0.56	0.36/0.74	0.22/0.42	0.18/0.34	0.36/0.70
MOE [46]	0.64/1.12	0.20/0.33	0.33/0.62	0.22/0.42	0.17/0.32	0.31/0.56
MOE w/o Image	0.68/1.22	0.25/0.49	0.35/0.68	0.22/0.42	0.18/0.33	0.34/0.63
DTO [34]	0.70/1.23	0.22/0.45	0.32/0.62	0.22/0.42	0.17/0.33	0.33/0.61
BCDiff	0.61/1.09	0.16/0.28	0.28/0.53	0.22/0.41	0.18/0.33	0.29/0.53
PCCSNet	Instantaneous	0.34/0.65	0.14/0.25	0.31/0.63	0.23/0.46	0.16/0.37	0.24/0.47
MOE [46]	0.31/0.57	0.13/0.21	0.25/0.53	0.20/0.41	0.14/0.31	0.20/0.41
MOE w/o Image	0.32/0.61	0.14/0.24	0.28/0.57	0.21/0.45	0.15/0.34	0.22/0.44
DTO [34]	0.33/0.64	0.14/0.24	0.31/0.62	0.22/0.46	0.15/0.35	0.23/0.46
BCDiff	0.30/0.56	0.13/0.20	0.25/0.52	0.18/0.37	0.14/0.31	0.19/0.39
SGCN	Instantaneous	0.88/1.66	0.55/1.16	0.38/0.71	0.30/0.54	0.25/0.46	0.47/0.91
MOE [46]	0.74/1.41	0.45/0.85	0.38/0.71	0.29/0.54	0.25/0.45	0.42/0.79
MOE w/o Image	0.79/1.52	0.50/1.05	0.38/0.71	0.30/0.54	0.25/0.46	0.44/0.85
DTO [34]	0.80/1.56	0.49/1.02	0.38/0.71	0.30/0.54	0.25/0.46	0.44/0.86
BCDiff	0.66/1.18	0.34/0.62	0.38/0.70	0.30/0.54	0.25/0.44	0.39/0.72
SocialVAE	Instantaneous	0.64/1.10	0.21/0.34	0.27/0.51	0.22/0.39	0.18/0.34	0.30/0.54
MOE [46]	0.57/1.01	0.17/0.29	0.26/0.44	0.22/0.36	0.17/0.32	0.28/0.48
MOE w/o Image	0.59/1.06	0.20/0.33	0.27/0.49	0.22/0.38	0.18/0.33	0.29/0.52
DTO [34]	0.61/1.06	0.18/0.31	0.25/0.43	0.22/0.38	0.17/0.33	0.29/0.50
BCDiff	0.53/0.91	0.17/0.27	0.24/0.40	0.21/0.37	0.16/0.26	0.26/0.44
reduced to 2 frames, and the length of future predictions is 12 frames. Following previous works [46, 15, 31], we sample 20 future predicted trajectories, and report the final error by the minimum error over all predicted trajectories.
Backbone and Baselines. To demonstrate the compatible ability of our BCDiff, we apply it to three popular trajectory prediction models, Trajectory++ [42], PCCSNet [47], SGCN [43], and SocialVAE [50] by replacing their decoders with our BCDiff. Moreover, we compare BCDiff with the following baselines: Instantaneous means directly predicting the trajectories of the next 12 frames conditions on 2 frames of observations using the above three backbones. Additionally, we take MOE [46] and DTO [34] as two baselines for instantaneous trajectory prediction. Since the original MOE employs image semantic information, we also implement MOE without using image semantic information, denoted as MOE w/o Image, in order to fairly compare with other methods.
4.2 Experiment Results and Analysis
Performance on Instantaneous Trajectory Prediction. The overall performance are listed in Table 1 and 2. The results by applying our BCDiff to three different backbones, consistently outperform the baseline methods on all the datasets. This demonstrates the effectiveness of our proposed method for instantaneous trajectory prediction. Meanwhile, it also illustrates our method can be well compatible with different trajectory prediction models. Note that the performance of MOE declines when image information is not included, which shows that MOE heavily depends on image semantic information. In contrast, our proposed method does not rely on any additional image information.
Ablation Study. We perform the ablation studies, as listed in Table 3. We first utilize two separate diffusion models to predict previous unobserved trajectories and future trajectories, respectively, denoting it as BCDiff-w/o.Guidance. Then, we only utilize unobserved and observed trajectories to guide the generation of future trajectories, denoted as BCDiff-Uni.Guidance. BCDiff-Uni.Guidance is better than BCDiff-w/o.Guidance, meaning that the guidance from previous unobserved historical trajectories is helpful for improving the prediction performance of future trajectories. Moreover, by
Table 2: Comparisons of different methods on the Stanford Drone dataset. The metrics are presented as ADE/FDE (m).
Methods	Trajectory++	PCCSNet	SGCN	SocialVAE
Instantaneous	13.07/22.88	9.19/17.71	15.40/25.69	9.56/16.10
MOE[46]	11.71/19.54	8.40/16.08	14.45/24.88	9.12/14.98
MOE w/o Image	12.41/21.46	8.87/16.80	15.02/25.13	9.39/15.87
DTO[34]	12.32/20.79	8.93/16.92	14.99/25.07	9.28/15.58
BCDiff	11.56/19.32	8.32/15.87	13.67/23.92	9.05/14.86
Table 3: Ablation Studies using Trajectory++ on the ETH/UCY and Stanford Drone datasets.
Method	ETH/UCY	SDD
BCDiff-w/o.Guidance	0.36/0.70	13.07/22.88
BCDiff-Uni.Guidance	0.33/0.60	12.57/21.26
BCDiff-Bi.Guidance	0.32/0.57	11.96/19.88
BCDiff-Bi.Guidance & Gate	0.29/0.53	11.56/19.32
incorporating bidirectional guidance, called BCDiff-Bi.Guidance, we observe the performance is further improved. This illustrates the guidance from future trajectories is beneficial for predicting previous unobserved trajectories, thereby improving the final prediction performance. Finally, we integrate the gating mechanism into our model, named as BCDiff-Bi.Guidance & Gate. It achieves the best performance, showing the gating mechanism is effective for trajectory prediction.
Analysis on Length of Predicted Unobserved Points. We investigate the impact of the number of predicted unobserved points on trajectory prediction. We use the Trajectory++ encoder [42], as listed in Table 4. As the number of predicted points increases, the accuracy of future predictions gradually improves, which can be attributed to the additional unobserved points providing more useful information. When  $T = 5$ , the prediction performance starts to decline. This is because it becomes unprecise when predicting previous unobserved historical points with a larger length, thus introducing noise into our method.
Table 4: Analysis of different unobserved point lengths  ${T}_{\text{unobs }}$
Dataset	Direction	Tunobs = 1	Tunobs = 2	Tunobs = 3	Tunobs = 4	Tunobs = 5	Tunobs = 6
ETH/UCY	Future	0.31/0.55	0.30/0.54	0.30/0.53	0.29/0.53	0.32/0.57	0.34/0.60
Unobserved	0.006/0.006	0.034/0.033	0.055/0.063	0.086/0.089	0.108/0.133	0.149/0.212
SDD	Future	11.86/19.78	11.81/19.71	11.68/19.50	11.56/19.32	11.92/20.01	12.33/20.98
Unobserved	0.311/0.312	1.564/1.492	2.626/2.678	3.552/3.884	4.828/5.423	6.177/7.894
Qualitative Analysis. We visualize the predicted trajectories in four different scenarios: Walking side-by-side, walking along, turning and yielding, as shown in Figure 2. BCDiff can accurately predict future trajectories, compared to MOE and DTO in all four scenarios. This is because BCDiff utilizes temporal information of unobserved trajectories to aid the prediction of future trajectories. Note that when pedestrians are walking side-by-side, as depicted in the first column of Figure 2, BCDiff utilizes the predicted unobserved historical trajectories to capture the walking patterns between two pedestrians, thereby precisely forecasting future trajectories. However, DTO and MOE predict deviated trajectories, due to only using two observed frames. Moreover, we visualize
the gating weights  $\gamma_{fut}^{m}$  and  $\gamma_{unobs}^{m}$  in the denoising process from Figure 3, the weights are gradually increased. It indicates our gating weights, assigning lower weights to the predicted guidance in the i
Figure 3: Visualization of weights  $\gamma_{fut}^{m}$  and  $\gamma_{unobs}^{m}$ . Darker colors represent larger values.
Figure 2: Visualization of predicted trajectories on the ETH/UCY Dataset. Given the instantaneous observed trajectories (red), we predict the future trajectories (green) by (a) MOE, (b) DTO and (c) Our BCDiff. The ground-truth future trajectories are shown in orange color. In addition, we also draw the predicted unobserved trajectories (blue) by our BCDiff and ground-truth (cyan). Our predicted future trajectories are closer to the ground-truth, compared to other methods.
We proposed a diffusion model based framework for the task of instantaneous trajectory prediction. The proposed framework simultaneously generated both unobserved historical and future trajectories by designing a mutual guidance mechanism to couple two diffusion models, such that more temporal information can be leveraged for future trajectory prediction. We introduced a gating strategy, automatically balancing the contributions of different guidance information. Experiments demonstrated our proposed framework achieved superior performance to the state-of-the-art methods.
This work was supported by the NSFC under Grants 62122013, U2001211. This work was also supported by the Innovative Development Joint Fund Key Projects of Shandong NSF under Grants ZR2022LZH007.
[1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social lstm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961–971, 2016.
[2] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17993, 2021.
[3] Inhwan Bae, Jin-Hwi Park, and Hae-Gon Jeon. Non-probability sampling network for stochastic human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6477-6487, 2022.
[4] Changan Chen, Yuejiang Liu, Sven Kreiss, and Alexandre Alahi. Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning. In 2019 international conference on robotics and automation (ICRA), pages 6015–6022. IEEE, 2019.
[52] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9813-9823, 2021.
[53] Jiangbei Yue, Dinesh Manocha, and He Wang. Human trajectory prediction via neural social physics. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV, pages 376-394. Springer, 2022.
[54] Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. In Conference on Robot Learning, pages 895-904. PMLR, 2021.
[55] Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12126-12134, 2019.
[56] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826–5835, 2021.
[57] Zikang Zhou, Luyao Ye, Jianping Wang, Kui Wu, and Kejie Lu. Hivt: Hierarchical vector transformer for multi-agent motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8823-8833, 2022.