SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and discrete external properties (e.g., departure time and regions) are derived. A wide network is utilized for embedding the numerical motion attributes, whereas a deep network is employed for the discrete categorical attributes. The wide network consists of a fully connected layer with 128 outputs, capturing linear relationships among motion attributes. In regard to the discrete categorical attributes, the deep network initially embeds them into multiple 128-dimensional vectors, concatenates these vectors, and subsequently employs two fully connected layers to learn intricate feature interactions. Finally, the outputs generated by the Wide & Deep networks are combined and integrated into each Resnet block.
4.3 Ensuring Generation Diversity
In practice, it should be avoided that conditional information results in a model with overly smooth or deterministic behavior patterns, which could undermine the intended privacy protections. To regulate the diversity of the generated trajectories and prevent the DiffTraj from following the conditional guidance too closely, we employ the classifier-free diffusion guidance method [12]. Specifically, we jointly train conditional and unconditional diffusion models with a trade-off between sample quality and diversity by using a guiding parameter  $\omega$ . That is, the noise prediction model can be written as:
$$
\epsilon_ {\theta} = (1 + \omega) \epsilon_ {\theta} \left(\boldsymbol {x} _ {t} ^ {\mathrm {s}}, t \mid \boldsymbol {x} _ {0} ^ {\mathrm {c o}}\right) - \omega \epsilon_ {\theta} \left(\boldsymbol {x} _ {t} ^ {\mathrm {s}}, t \mid \varnothing\right), \tag {7}
$$
where  $\varnothing$  denotes a vector in which all conditional information is 0, essentially not encoding any information. When  $\omega = 0$ , the model is a general diffusion model with conditional information only. By increasing the  $\omega$ , the model will focus more on unconditionally predicting noise (i.e., more diversity) when generating trajectories. Therefore, we can strike a balance between maintaining the quality of the generated trajectories and increasing their diversity, ensuring that the model does not generate overly deterministic behavior patterns.
4.4 Sampling Speed up
As described in Sec. 3.2, DiffTraj relies on a large Markov process to generate high-quality trajectories, rendering a slow reverse diffusion process. Therefore, it is a significant challenge for the proposed model to generate valid and usable trajectories in a reasonable time frame. To address this issue, we adopt a non-Markov diffusion process approach with the same optimization objective [28], thus allowing the computation of efficient inverse processes. Specifically, we can sample every  $[T / S]$  steps with the skip-step method presented in [23]. The corresponding set of noise trajectories changes to  $\{\tau_1,\dots ,\tau_S\} ,\tau_i\in [1,T]$ . Through this approach, the sampling steps during trajectory generation can be significantly reduced from  $T$  steps to  $S$  steps. Compared to the typical diffusion model [11], this method can generate higher quality samples in fewer steps [28].
4.5 Discussion on Privacy
The DiffTraj inherently protects privacy in GPS trajectory generation due to its generative nature and the manner in which it reconstructs synthetic data from random noise. It offers effective privacy protection through two key aspects. (1) Generative Approach: The DiffTraj generates trajectories by sampling from a learned distribution, rather than directly relying on or perturbing real-world data. By reconstructing trajectories from random noise during the reverse diffusion process, the model effectively decouples synthetic data from specific real data points. This ensures that the generated trajectories do not contain personally identifiable information or reveal sensitive location details, thus protecting their privacy. (2) Learning Distribution: The Traj-UNet learns the distribution of real trajectories and generates synthetic ones that exhibit similar patterns and behaviors, which avoids direct replicating individual real trajectories. The diverse and realistic synthetic trajectory generation ensures that sensitive information about specific individuals or their movement patterns is not inadvertently leaked. Moreover, it prevents the possibility of reverse engineering real trajectories from synthetic ones. By combining these aspects, the DiffTraj offers robust privacy protection for GPS trajectory generation, addressing concerns about data privacy while maintaining utility.
We conduct extensive experiments on two real-world trajectory datasets to show the superior performance of the proposed DiffTraj in GPS trajectory generation. In this section, we only include
the basic setup, comparison of the main experimental results, and visualization analysis. Owing to space limitation, we leave other details (dataset, baselines, network structure, hyperparameters, etc.) and more results (conditional generation, case studies, etc.) in the Appendix. All experiments are implemented in PyTorch and conducted on a single NVIDIA A100 40GB GPU. The code for the implementation of DiffTraj is available for reproducibility<sup>2</sup>.
Dataset and Configuration. We evaluate the generative performance of DiffTraj and baselines on two real-world GPS trajectory datasets of Chinese cities, namely, Chengdu and Xi'an. A detailed description of the dataset and statistics is available in Appendix B.1, and experiment about Porto dataset is shown in Appendix C.1.
Evaluation Metrics. To fully evaluate the quality of the generated trajectories, we follow the design in [8] and use four utility metrics at different levels. Specifically, 1) Density error measures the geo-distribution between the entire generated trajectory and the real one, which measures the quality and fidelity of the generated trajectories at the global level. For trajectory level, 2) Trip error measures the distributed differences between trip origins and endpoints, and 3) Length error focuses on the differences in real and synthetic trajectory lengths. Moreover, we use 4) Pattern score to calculate the pattern similarity of the generated trajectories (see Appendix B.2 for details).
Baseline Methods. We compare DiffTraj with a series of baselines, including three typical generative models (VAE [34], TrajGAN [9] and DP-TrajGAN [38]) and a range of variants based on the diffusion model approach. Specifically, Diffwave is a diffusion model based on the WaveNet structure [14], while Diff-LSTM replaces the 1D-CNN-based temporal feature extraction module with an LSTM. In addition, Diff-scatter and Diff-wo/UNet can be considered as ablation studies of the proposed method to evaluate the effectiveness of diffusion models and UNet structures, respectively. Furthermore, DiffTraj-wo/Con indicates no integration of the conditional information embedding module, which is used to evaluate the efficiency of the Wide & Deep module for trajectory generation. Finally, we also compare Random Perturbation (RP) and Gaussian Perturbation (GP), two classes of non-generative approaches, to compare the superiority of generative methods in privacy preservation. The details of the implementation are presented in Appendix B.3.
5.2 Overall Generation Performance
Table 1 presents the performance comparison of DiffTraj and the selected baseline methods on two real-world datasets. Specifically, we randomly generate 3,000 trajectories from each generative method and then calculate all metrics. From the results, we observe several trends that highlight the advantages of the proposed DiffTraj method. Non-generative methods, RP and GP, exhibit poor performance due to their reliance on simple perturbation techniques, which fail to capture the complex spatial-temporal dependencies and motion properties of the real GPS trajectories. This leads to lower-quality synthetic trajectories compared to generative models. Generative models, VAE and TrajGAN, show better performance than RP and GP but are still inferior to DiffTraj (or DiffTraj-wo/Con). This is because VAE and TrajGAN primarily focus on spatial distribution and may not adequately capture the temporal dependencies within trajectories. Additionally, the DiffTraj is designed to reconstruct the data step-by-step from random noise, which makes it more suitable for generating real GPS trajectory that reflect the inherent stochastic nature of human activities.
Diff-LSTM achieves good results in some metrics compared to the model without UNet, but falls short of DiffTraj due to the differences in the backbone network. The LSTM-based approach in Diff-LSTM is inherently designed for sequence processing and may be less effective in capturing spatial dependencies in the data compared to the CNN-based approach used in DiffTraj. The CNN-based backbone in DiffTraj can better model local spatial patterns and multi-channel dependencies more effectively, leading to higher-quality generated trajectories. Nevertheless, such results advocate the employment of diffusion models in future studies related to spatial-temporal data generation.
Ablation Study. The other diffusion model variants, Diffwave, Diff-scatter, and Diff-wo/UNet, do not perform as well as DiffTraj due to differences in their network structures or the absence of the UNet architecture. The Traj-UNet structure in DiffTraj is specifically designed for spatial-temporal
$^{2}$ https://github.com/Yasoz/DiffTraj
Table 1: Performance comparison of different generative approaches.
Methods	Chengdu	Xi'an
Density (↓)	Trip (↓)	Length (↓)	Pattern (↑)	Density (↓)	Trip (↓)	Length (↓)	Pattern (↑)
RP	0.0698	0.0835	0.2337	0.493	0.0543	0.0744	0.2067	0.381
GP	0.1365	0.1590	0.1423	0.233	0.0928	0.1013	0.2164	0.233
VAE	0.0148	0.0452	0.0383	0.356	0.0237	0.0608	0.0497	0.531
TrajGAN	0.0125	0.0497	0.0388	0.502	0.0220	0.0512	0.0386	0.565
DP-TrajGAN	0.0117	0.0443	0.0221	0.706	0.0207	0.0498	0.0436	0.664
Diffwave	0.0145	0.0253	0.0315	0.741	0.0213	0.0343	0.0321	0.574
Diff-scatter	0.0209	0.0685	-	-	0.0693	0.0762	-	-
Diff-wo/UNet	0.0356	0.0868	0.0378	0.422	0.0364	0.0832	0.0396	0.367
DiffTraj-wo/Con	0.0072	0.0239	0.0376	0.643	0.0138	0.0209	0.0357	0.692
Diff-LSTM	0.0068	0.0199	0.0217	0.737	0.0142	0.0195	0.0259	0.706
DiffTraj	0.0055	0.0154	0.0169	0.823	0.0126	0.0165	0.0203	0.764
Bold indicates the statistically best performance (i.e., two-sided t-test with  $p < 0.05$ ) over the best baseline.
(a) VAE
(b) TrajGAN
(c) Diffwave
Figure 2: Geographic visualization of generated trajectory in Chengdu (Larger view in Appendix D).
(d) Diff-LSTM
(e) DiffTraj
(f) Real
modeling and multi-scale feature fusion, enabling the accurate prediction of noise levels during denoising and resulting in high-quality generated trajectory. It is also noteworthy that satisfactory results can still be accomplished when only generating scattered locations or discarding the Traj-UNet structure. Finally, the conditional version of DiffTraj outperforms the unconditional version (DiffTraj-wo/Con) because it incorporates conditional information into the trajectory generation process. This allows the model to better account for external factors such as traffic conditions, departure time, and regions, which significantly influence human mobility patterns. This result can also be verified by the performance on the pattern indicator, where this indicator significantly outperforms other models.
Geographic Visualization. We also visualized the generated results to better present the performance of different generative models. Fig. 2 shows the trajectory distributions generated by baseline methods for Chengdu (Please refer to Fig. 10 and Fig. 11 in Appendix D for Xi'an and larger view). From visualizations, we can see that all the generated trajectories show geographic profiles of the city. Notably, DiffTraj and Diff-LSTM generate trajectories almost identical to the original trajectories and significantly surpass the other methods. However, Diff-LSTM performs poorly compared to DiffTraj on sparse roads, suggesting that CNNs are skilled in capturing local and global spatial features, making them more suitable for handling sparse road scenes. In addition, developing a tailor-made UNet architecture is also essential. Comparing Fig. 2(c) and Fig. 2(e), we can observe that the latter is able to generate a visually more realistic trajectory distribution.
5.3 Utility of Generated Data
As the generated trajectory serves downstream tasks, its utility is critical to determine whether the data generation method is indeed practical. In this section, we evaluate the utility of generated trajectory through an inflow (outflow) prediction task, which is one of the important applications of trajectory data analysis [13, 39]. Specifically, we train multiple advance prediction models using the original and generated trajectory separately, and then compare the prediction performance (detail experimental setup refer to Appendix C.3). As concluded in Table 2, the performance of the models when trained on the generated data is comparable to their performance on the original data. This result shows that our generative model, DiffTraj, is able to generate trajectory that retain the key statistical properties of the original one, hence making them useful for downstream tasks such as inflow and outflow prediction. However, there is a slight increase in error metrics when using the generated data, implying there's still room for further improvements in trajectory generation. Nevertheless, this result encourages the use of generated, privacy-free alternatives to the original data in future studies related to trajectory data analysis.
Table 2: Data utility comparison by inflow/outflow prediction.
Task	Inflow (origin/generated)	Outflow (origin/generated)
Methods	AGCRN	GWNet	DCRNN	MTGNN	AGCRN	GWNet	DCRNN	MTGNN
MSE	4.33/4.55	4.42/4.50	4.45/4.62	4.28/4.56	4.45/4.70	4.64/4.72	4.72/5.01	4.39/4.77
RMSE	2.08/2.13	2.10/2.12	2.11/2.15	2.07/2.13	2.11/2.16	2.15/2.17	2.17/2.24	2.10/2.18
MAE	1.49/1.52	1.50/1.50	1.51/1.53	1.48/1.51	1.50/1.54	1.53/1.54	1.53/1.57	1.49/1.53
(a) Chengdu
(b) Xi'an
Figure 3: Additional Experiment. (a) and (b) Transfer learning; (c) and (d) Speed up sampling.
(c) Chengdu
(d) Xi'an
5.4 Additional Experiment
Transfer Learning. Transfer learning is an essential aspect to investigate in the context of trajectory generation, as it can demonstrate the robustness and generalization capabilities of the proposed DiffTraj framework. In this paper, we pre-train the DiffTraj model in one city, and then continue the training in another city using a different percentage  $(5\% \sim 100\%)$  of data. We also compared these results with only training DiffTraj on different percentages of data. As shown in Fig. 3, 'origin' indicates that transfer learning is not applicable, while 'transfer' indicates applying transfer learning. Notably, even with just  $5\%$  of the data, the transfer learning model achieves a significantly lower error compared to the original one. As the percentage of data increases, the gap between the two models narrows. The results show that the DiffTraj model exhibits strong adaptability and generalization capabilities when applied to different urban scenarios. This allows the model to quickly adapt to new urban contexts and capture the inherent mobility patterns with less data, thereby reducing training time and improving applicability to diverse real-world scenarios.
Speed up Sampling. As discussed in Sec. 3.2, the diffusion model requires numerous sampling steps for gradual denoising, resulting in a slow trajectory generation process. By incorporating the sampling speed-up technique outlined in Sec. 4.4, we examine the efficiency of generating  $50 \times 256$  trajectories using a well-trained DiffTraj on two datasets. The results are summarized in Fig. 3, where the X-axis represents the total sample steps after speed-up, and the two Y-axes indicate the time spent and Density error, respectively. All models trained with  $T = 500$  diffusion steps consistently yield high-quality trajectories resembling the original distribution. Moreover, the model matches the outcomes of the no skipped steps method at  $T = 100$ , saving  $81\%$  of the time cost. However, too few sampling steps ( $S < 100$ ) result in a significant divergence between the generated and real trajectory. This occurs because fewer sample steps lead to more skipped reverse operations, causing Diff-Traj to discard more noise information during denoising. Nonetheless, these results highlight the substantial efficiency improvements achieved by the speed-up methods presented in Sec. 4.4.
In this work, we propose a new GPS trajectory generation method based on diffusion model and spatial-temporal data mining techniques. This method, named DiffTraj, leverages the data generation ability of the diffusion model and learning from spatial-temporal features by Traj-UNet. Specifically, real trajectories are gradually transformed into random noise by a forward trajectory noising process. After that, DiffTraj adopts a reverse trajectory denoising process to recover forged trajectories from the noise. Throughout the DiffTraj framework, we develop a Traj-UNet structure and incorporate Wide & Deep conditional module to extract trajectory features and estimate noise levels for the reverse process. Extensive experiments validate the effectiveness of DiffTraj and its integrated Traj-UNet. Further experiments prove that the data generated by DiffTraj can conform to the statistical properties of the real trajectory while ensuring utility, and provides transferability and privacy-preserving.
This work is supported by the Stable Support Plan Program of Shenzhen Natural Science Fund under Grant No. 20220815111111002. This research was partially supported by Research Impact Fund (No. R1015-23), APRC - CityU New Research Initiatives (No.9610565, Start-up Grant for New Faculty of City University of Hong Kong), CityU - HKIDS Early Career Research Grant (No.9360163), Hong Kong ITC Innovation and Technology Fund Midstream Research Programme for Universities Project (No.ITS/034/22MS), Hong Kong Environmental and Conservation Fund (No. 88/2022), SIRG - CityU Strategic Interdisciplinary Research Grant (No.7020046, No.7020074), Tencent (CCF-Tencent Open Fund, Tencent Rhino-Bird Focused Research Fund), Huawei (Huawei Innovation Research Program), Ant Group (CCF-Ant Research Fund, Ant Group Research Fund) and Kuaishou.
[1] M. P. Armstrong, G. Rushton, and D. L. Zimmerman. Geographically masking health data to preserve confidentiality. Statistics in medicine, 18(5):497-525, 1999.
[2] O. Avrahami, D. Lischinski, and O. Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208-18218, 2022.
[3] J. Bergh and J. Lofstrom. Interpolation Spaces: An Introduction. Springer Berlin Heidelberg, 2012.
[4] C. Cao and M. Li. Generating mobility trajectories with retained data utility. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2610-2620, 2021.
[5] H. Cao, C. Tan, Z. Gao, G. Chen, P.-A. Heng, and S. Z. Li. A survey on generative diffusion model. arXiv preprint arXiv:2209.02646, 2022.
[6] H.-T. Cheng, L. Koc, J. Harmsen, T. Shaked, T. Chandra, H. Aradhye, G. Anderson, G. Corrado, W. Chai, M. Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7-10, 2016.
[7] P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[8] Y. Du, Y. Hu, Z. Zhang, Z. Fang, L. Chen, B. Zheng, and Y. Gao. Ldptrace: Locally differentially private trajectory synthesis. arXiv preprint arXiv:2302.06180, 2023.
[9] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, volume 27, 2014.
[10] X. Han, X. Zhao, L. Zhang, and W. Wang. Mitigating action hysteresis in traffic signal control with traffic predictive reinforcement learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, page 673-684, 2023.
[11] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[12] J. Ho and T. Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[13] R. Jiang, D. Yin, Z. Wang, Y. Wang, J. Deng, H. Liu, Z. Cai, J. Deng, X. Song, and R. Shibasaki. DL-traff: Survey and benchmark of deep learning models for urban traffic prediction. In Proceedings of the 30th ACM international conference on information & knowledge management, pages 4515-4525, 2021.
[14] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2020.
[34] T. Xia, X. Song, Z. Fan, H. Kanasugi, Q. Chen, R. Jiang, and R. Shibasaki. Deeprailway: A deep learning system for forecasting railway traffic. In 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR), pages 51-56, 2018.
[35] L. Yang, Z. Zhang, Y. Song, S. Hong, R. Xu, Y. Zhao, Y. Shao, W. Zhang, B. Cui, and M.-H. Yang. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796, 2022.
[36] Y. Yuan, J. Ding, H. Wang, D. Jin, and Y. Li. Activity trajectory generation via modeling spatiotemporal dynamics. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4752-4762, 2022.
[37] P. A. Zandbergen. Ensuring confidentiality of geocoded health data: assessing geographic masking strategies for individual-level data. Advances in medicine, 2014, 2014.
[38] J. Zhang, Q. Huang, Y. Huang, Q. Ding, and P.-W. Tsai. Dp-trajgan: A privacy-aware trajectory generation model with differential privacy. Future Generation Computer Systems, 2022.
[39] Z. Zhang, X. Zhao, H. Miao, C. Zhang, H. Zhao, and J. Zhang. Autostl: Automated spatio-temporal multi-task learning. arXiv preprint arXiv:2304.09174, 2023.
[40] X. Zhao, W. Fan, H. Liu, and J. Tang. Multi-type urban crime prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 4388-4396, 2022.
[41] Y. Zheng. Trajectory data mining: an overview. ACM Transactions on Intelligent Systems and Technology (TIST), 6(3):1-41, 2015.
[42] Y. Zheng, L. Capra, O. Wolfson, and H. Yang. Urban computing: concepts, methodologies, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 5(3):1-55, 2014.
[43] Y. Zheng, L. Zhang, X. Xie, and W.-Y. Ma. Mining interesting locations and travel sequences from gps trajectories. In Proceedings of the 18th international conference on World wide web, pages 791-800, 2009.
[44] Y. Zhu, Y. Ye, Y. Liu, and J. J. Q. Yu. Cross-area travel time uncertainty estimation from trajectory data: a federated learning approach. IEEE Transactions on Intelligent Transportation Systems, 23(12):24966-24978, 2022.
A DiffTraj Details & Hyperparameters
In this section, we cover the specific details of DiffTraj, including the DiffTraj framework, the Traj-UNet structure, and the implementation details.
A.1 Architecture
Figure 4: The network architecture used by DiffTraj in modeling  $\epsilon_{\theta}\left(x_{t}^{\mathrm{s}}, t \mid x_{0}^{\mathrm{co}}\right)$  is divided into two modules, down-sampling and up-sampling, each containing multiple Resnet blocks.