SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuanshao Zhu $^{1,2}$ , Yongchao Ye $^{1}$ , Shiyao Zhang $^{1}$ , Xiangyu Zhao $^{2,*}$ ; James J.Q. Yu $^{3,*}$
Southern University of Science and Technology
$^{2}$  City University of Hong Kong
<sup>3</sup>University of York
{zhuys2019, 12032868}@mail.sustech.edu.cn
zhangsy@sustech.edu.cn
xianzhao@cityu.edu.hk
james.yu@york.ac.uk
Pervasive integration of GPS-enabled devices and data acquisition technologies has led to an exponential increase in GPS trajectory data, fostering advancements in spatial-temporal data mining research. Nonetheless, GPS trajectories contain personal geolocation information, rendering serious privacy concerns when working with raw data. A promising approach to address this issue is trajectory generation, which involves replacing original data with generated, privacy-free alternatives. Despite the potential of trajectory generation, the complex nature of human behavior and its inherent stochastic characteristics pose challenges in generating high-quality trajectories. In this work, we propose a spatial-temporal diffusion probabilistic model for trajectory generation (DiffTraj). This model effectively combines the generative abilities of diffusion models with the spatial-temporal features derived from real trajectories. The core idea is to reconstruct and synthesize geographic trajectories from white noise through a reverse trajectory denoising process. Furthermore, we propose a Trajectory UNet (Traj-UNet) deep neural network to embed conditional information and accurately estimate noise levels during the reverse process. Experiments on two real-world datasets show that DiffTraj can be intuitively applied to generate high-fidelity trajectories while retaining the original distributions. Moreover, the generated results can support downstream trajectory analysis tasks and significantly outperform other methods in terms of geo-distribution evaluations.
GPS trajectory data play a crucial role in numerous spatial-temporal data mining applications, including urban traffic planning, business location selection, and travel time estimation [43, 42]. Despite the substantial progress in urban applications through GPS trajectory analysis, few attention was paid to data accessibility and privacy concerns [41]. For example, real-world trajectory data, which contain sensitive personal geolocation information, raise significant privacy issues when using directly [38, 44]. Additionally, the time-consuming and labor-intensive nature of trajectory data collection leads to challenges in obtaining accessible, publicly available datasets that preserve privacy. Therefore, it is essential to develop methods that facilitate the efficient use of trajectory data in urban applications while safeguarding personal privacy. One promising approach involves generating synthetic trajectories by learning from real trajectory distributions and replacing privacy-sensitive
real trajectories with these synthetic counterparts. The generated trajectories allow for equivalent data analysis outcomes and support high-level decision-making without compromising privacy.
However, generating GPS trajectory that accurately reflect real-world distributions will encounter the following challenges in practice. First, most cities contain various regions with diverse functions and population densities, resulting in non-independent and identically distributed trajectories among these regions [44, 40]. This complex distribution makes it difficult to learn a global trajectory model that captures regional nuances. Second, the inherent stochastic nature of human activities makes each trajectory unique, posing challenges in modeling and predicting individual movement patterns [41, 16]. Third, external factors, such as traffic conditions, departure time, and local events, significantly influence personal travel schedules [36]. Accounting for these factors and understanding their impact on the correlations between individual GPS locations within a trajectory adds to the complexity of modeling trajectory generation accurately.
To address the above challenges, we propose a trajectory generation method based on the spatial-temporal diffusion probabilistic model, which can effectively capture the complex behaviors of real-world activities and generates high-quality trajectories. The core idea behind this method is to perturb the trajectory distribution with noise through a forward trajectory diffusion process and then recover the original distribution by learning the backward diffusion process (denoising), resulting in a highly flexible trajectory generation model [11, 28]. We propose this framework based on three primary motivations: (i) The diffusion model is a more reliable and robust method of generation than canonical methods [35]. (ii) Human activities in the real world exhibit stochastic and uncertain characteristics [18, 10], and the diffusion model reconstructs data step-by-step from random noise, making it suitable for generating more realistic GPS trajectory. (iii) Since the diffusion model generates trajectories from random noise, it eliminates the risk of privacy leakage.
While the diffusion model shows promise for generating high-quality trajectories, it is a non-trivial task when applied to GPS trajectory generation. Firstly, the spatial-temporal nature of trajectory data presents complexity to the diffusion process, requiring the model to consider the spatial distribution of GPS points and the temporal dependencies among them. Secondly, external factors such as traffic conditions, departure time, and regions can significantly impact human mobility patterns [44], posing challenges for modeling these factors within the diffusion framework. Finally, the inherently stochastic nature of human activities demands that the diffusion model be capable of capturing a wide range of plausible trajectory patterns, which can be difficult due to the need to balance diversity and realism in the generated trajectories. Addressing these challenges is crucial for successfully applying the diffusion model to GPS trajectory generation and ensuring privacy-preserving, accurate, and computationally efficient outcomes.
Building on the identified motivations and addressing the limitations of applying the diffusion model to trajectory generation, we propose a Diffusion probabilistic model Trajectory generation (DiffTraj) framework. This framework can generate infinite trajectories while preserving the real-world trajectory distribution. DiffTraj incorporates spatial-temporal modeling of raw trajectories without the need for additional operations, allowing for direct application to trajectory generation tasks. Crucially, the synthetic trajectories generated by DiffTraj ensure high generation accuracy while maintaining their utility, offering a promising solution to privacy-preserving GPS trajectory generation. To summarize, the contributions of this work are concluded as follows:
- We propose a DiffTraj framework for trajectory generation, leveraging the diffusion model to simulate real-world trajectories while preserving privacy. In addition, this framework can be applied to generate high-fidelity trajectory data with high efficiency and transferability. To the best of our knowledge, our work is the first exploration of trajectory generation by the diffusion model.
- We design a novel denoising network structure called Traj-UNet, which integrates residual blocks for spatial-temporal modeling and multi-scale feature fusion to accurately predict the noise level during denoising. Meanwhile, Traj-UNet integrates Wide and Deep networks [6] to construct conditional information, enabling controlled trajectory generation.
- We validate the effectiveness of DiffTraj on two real-world trajectory datasets, showing that the proposed methods can generate high-fidelity trajectories and preserve statistical properties. At the same time, the generated trajectories can support downstream trajectory analysis tasks with replaceable utility and privacy-preserving.
Trajectory Data Synthesizing. Existing methods for trajectory data can be generally divided into two main categories: non-generative and generative [4]. Non-generative methods include perturbing real trajectories [1, 37] or combining different real trajectories [21]. Adding random or Gaussian perturbations protects privacy but compromises data utility by altering the spatial-temporal characteristics and data distribution. Striking a balance between trajectory utility and privacy protection is challenging [4]. Although Mehmet et al. generated forged trajectories by mixing different trajectories, this method relies on massive data and sophisticated mixing processes [21].
Besides, the principle of the generative method is to leverage deep neural networks that learn the spatial-temporal distribution underlying the real data. New trajectory data are therefore generated by sampling from the learned distribution. Liu et al. proposed a preliminary solution that employs generative adversarial networks (GAN) for trajectory generation, yet it failed to go further towards a detailed design[33]. Subsequently, some works divided the city map into grids and performed trajectory generation by learning the distribution of trajectories among the grids [38, 24]. However, there is a trade-off between generation accuracy and grid size. Meanwhile, researchers used the image generation capability of GAN to convert the trajectory data into images for time-series generation [4, 31], while the transformation between images and trajectories imposed an additional computational burden. Compared with previous methods, DiffTraj uses the diffusion model for trajectory generation, which can better explore the spatial and temporal distributions without additional data manipulation.
Diffusion Probabilistic Model. The diffusion model is a probabilistic generative model, which was first proposed by Sohl-Dickstein et al. [27] and then further improved by Ho et al. [11] and Song et al. [29]. A typical diffusion model generates synthetic data via two sequential processes, i.e., a forward process that gradually perturbs the data distribution by adding noise on multiple scales and a reverse process that learns how to recover the data distribution[35]. In addition, researchers have made extensive attempts to improve generative sample quality and sampling speed. For example, Song et al. proposed a non-Markovian diffusion process to reduce the sampling steps [28], Nichol et al. proposed to learn the variance of reverse processes allowing fewer sampling steps [22], and Dhariwal et al. searched for the optimal structure of the reverse denoising neural network to obtain better sampling quality. As a new type of advanced generative model, diffusion models have achieved superior performance over alternative generative models in various generative tasks, such as computer vision [25], natural language processing [15], multi-modal learning [2, 19], and traffic forecasting [32]. Nevertheless, the diffusion probabilistic model calls for efforts in spatial-temporal trajectory data generation. To the best of our knowledge, this work is a pioneering attempt that uses diffusion probabilistic model to generate GPS trajectory.
In this section, we first introduce the definitions and notations we use in this paper and then briefly present the fundamentals of the diffusion probabilistic model.
Definition 1: (GPS Trajectory). A GPS trajectory is defined as a sequence of continuously sampled private GPS location points, denoted by  $\mathbf{x} = \{p_1, p_2, \dots, p_n\}$ . The  $i$ -th GPS point is represented as a tuple  $p_i = [\mathrm{lat}_i, \mathrm{lng}_i]$ , where  $\mathrm{lng}_i$  and  $\mathrm{lat}_i$  denote longitude and latitude, respectively.
Problem Statement: (Trajectory Generation). Given a set of real-world GPS trajectories,  $\mathcal{X} = \{\pmb{x}^1, \pmb{x}^2, \dots, \pmb{x}^m\}$ , where each  $\pmb{x}^i = \{p_1^i, p_2^i, \dots, p_n^i\}$  is a sequence of private GPS location points. The objective of the trajectory generation task is to learn a generative model,  $G$ , that can generate synthetic GPS trajectories,  $\mathcal{Y} = \{\pmb{y}^1, \pmb{y}^2, \dots, \pmb{y}^k\}$ , where each  $\pmb{y}^i = \{q_1^i, q_2^i, \dots, q_n^i\}$ . such that:
- Similarity: The generated trajectories  $\pmb{y}^i$  preserve the spatial-temporal characteristics and distribution of the real trajectories  $\pmb{x}^i$ . In practice, there are various physical meanings, such as the regional distribution of trajectory points, the distance between successive points, etc.
- Utility: The generated trajectories  $y^{i}$  maintain utility for downstream applications and analyses.
- Privacy: Privacy is preserved, meaning that the generated trajectories  $\pmb{y}^i$  do not reveal sensitive information about the individuals associated with the real trajectories  $\pmb{x}^i$ .
3.2 Diffusion Probabilistic Model
The diffusion probabilistic model is a powerful and flexible generative model, which has gained increasing attention in recent years for its success in various data generation tasks [11, 28, 29]. In general, the diffusion probabilistic model consists of two main processes: a forward process that gradually perturbs the data distribution with noise, and a reverse (denoising) process that learns to recover the original data distribution.
Forward process. Given a set of real data samples  $\boldsymbol{x}_0 \sim q(\boldsymbol{x}_0)$ , the forward process adds  $T$  time-steps of Gaussian noise  $\mathcal{N}(\cdot)$  to it, where  $T$  is an adjustable parameter. Formally, the forward process can be defined as a Markov chain from data  $\boldsymbol{x}_0$  to the latent variable  $\boldsymbol{x}_T$ :
$$
q \left(\boldsymbol {x} _ {1: T} \mid \boldsymbol {x} _ {0}\right) = \prod_ {t = 1} ^ {T} q \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t - 1}\right); \quad q \left(\boldsymbol {x} _ {t} \mid \boldsymbol {x} _ {t - 1}\right) = \mathcal {N} \left(\boldsymbol {x} _ {t}; \sqrt {1 - \beta_ {t}} \boldsymbol {x} _ {t - 1}, \beta_ {t} \mathbf {I}\right). \tag {1}
$$
Here,  $\{\beta_t\in (0,1)\}_{t = 1}^T (\beta_1 < \beta_2 < \dots < \beta_T)$  is the corresponding variance schedule. Since it is impractical to back-propagate the gradient by sampling from a Gaussian distribution, we adopt a reparameterization trick to keep the gradient derivable [11] and the  $\pmb{x}_t$  can be expressed as  $\pmb {x}_t = \sqrt{\bar{\alpha}_t}\pmb {x}_0 + \sqrt{1 - \bar{\alpha}_t}\pmb {\epsilon},$  where  $\pmb {\epsilon}\sim \mathcal{N}(0,\mathbf{I})$  and  $\bar{\alpha}_t = \prod_{i = 1}^t (1 - \beta_i)$
Reverse process. The reverse diffusion process, also known as the denoising process, aims to recover the original data distribution from the noisy data  $\boldsymbol{x}_T \sim \mathcal{N}(0, \mathbf{I})$ . Accordingly, this process can be formulated by the following Markov chain:
$$
p _ {\theta} \left(\boldsymbol {x} _ {0: T}\right) = p \left(\boldsymbol {x} _ {T}\right) \prod_ {t = 1} ^ {T} p _ {\theta} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t}\right); \quad p _ {\theta} \left(\boldsymbol {x} _ {t - 1} \mid \boldsymbol {x} _ {t}\right) = \mathcal {N} \left(\boldsymbol {x} _ {t - 1}; \boldsymbol {\mu} _ {\theta} \left(\boldsymbol {x} _ {t}, t\right), \boldsymbol {\sigma} _ {\theta} \left(\boldsymbol {x} _ {t}, t\right) ^ {2} \mathbf {I}\right) \tag {2}
$$
where  $\pmb{\mu}_{\theta}(x_{t},t)$  and  $\sigma_{\theta}(x_{t},t)$  are the mean and variance parameterized by  $\theta$ , respectively. Based on the literature [11], for any  $\tilde{\beta}_t = \frac{1 - \bar{\alpha}_{t - 1}}{1 - \bar{\alpha}_t}\beta_t(t > 1)$  and  $\tilde{\beta}_1 = \beta_1$ , the parameterizations of  $\pmb{\mu}_{\theta}$  and  $\pmb{\sigma}_{\theta}$  are defined by:
$$
\boldsymbol {\mu} _ {\theta} \left(\boldsymbol {x} _ {t}, t\right) = \frac {1}{\sqrt {\alpha_ {t}}} \left(\boldsymbol {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \boldsymbol {\epsilon} _ {\theta} \left(\boldsymbol {x} _ {t}, t\right)\right), \text {a n d} \boldsymbol {\sigma} _ {\theta} \left(\boldsymbol {x} _ {t}, t\right) = \tilde {\beta} _ {t} ^ {\frac {1}{2}}. \tag {3}
$$
4 DiffTraj Framework
In this section, we present the details of the DiffTraj framework shown in Fig. 1, which applies the diffusion model to trajectory generation. The primary goal of DiffTraj is to estimate the real-world trajectory distribution  $q(\boldsymbol{x}_0 \mid \boldsymbol{x}_0^{\mathrm{co}})$  using a parameterized model  $p_{\theta}(\boldsymbol{x}_0^{\mathrm{s}} \mid \boldsymbol{x}_0^{\mathrm{co}})$ . Given a random noise  $\boldsymbol{x}_T \sim \mathcal{N}(0, I)$ , DiffTraj generates a synthetic trajectory  $\boldsymbol{x}_0^{\mathrm{s}}$ , conditioned on the observations  $\boldsymbol{x}_0^{\mathrm{co}}$ . The reverse (generation) process from the Eq. (2) can be reformulated for DiffTraj as:
$$
p _ {\theta} \left(\boldsymbol {x} _ {t - 1} ^ {\mathrm {s}} \mid \boldsymbol {x} _ {t} ^ {\mathrm {s}}, \boldsymbol {x} _ {0} ^ {\mathrm {c o}}\right) := \mathcal {N} \left(\boldsymbol {x} _ {t - 1} ^ {\mathrm {s}}; \boldsymbol {\mu} _ {\theta} \left(\boldsymbol {x} _ {t} ^ {\mathrm {s}}, t \mid \boldsymbol {x} _ {0} ^ {\mathrm {c o}}\right), \boldsymbol {\sigma} _ {\theta} \left(\boldsymbol {x} _ {t} ^ {\mathrm {s}}, t \mid \boldsymbol {x} _ {0} ^ {\mathrm {c o}}\right) ^ {2} \mathbf {I}\right). \tag {4}
$$
In this context,  $\mu_{\theta}$  serves as a conditional denoising function that takes the conditional observation  $x_0^{\mathrm{co}}$  as input. We then extend the Eq. (3) to account for the conditional observation as follows:
$$
\left\{ \begin{array}{l} \boldsymbol {\mu} _ {\theta} \left(\boldsymbol {x} _ {t} ^ {\mathrm {s}}, t \mid \boldsymbol {x} _ {0} ^ {\mathrm {c o}}\right) = \boldsymbol {\mu} _ {\theta} \left(\boldsymbol {x} _ {t} ^ {\mathrm {s}}, t, \boldsymbol {\epsilon} _ {\theta} \left(\boldsymbol {x} _ {t} ^ {\mathrm {s}}, t \mid \boldsymbol {x} _ {0} ^ {\mathrm {c o}}\right)\right) \\ \boldsymbol {\sigma} _ {\theta} \left(\boldsymbol {x} _ {t} ^ {\mathrm {s}}, t \mid \boldsymbol {x} _ {0} ^ {\mathrm {c o}}\right) = \boldsymbol {\sigma} _ {\theta} \left(\boldsymbol {x} _ {t} ^ {\mathrm {s}}, t\right) \end{array} \right. \tag {5}
$$
Training. In practice, the reverse trajectory denoising process can be summarized as learning the Gaussian noise  $\epsilon_{\theta}(\pmb{x}_{t}^{\mathrm{s}}, t \mid \pmb{x}_{0}^{\mathrm{co}})$  through  $\pmb{x}_{t}^{\mathrm{s}}$ ,  $t$  and  $\pmb{x}_{0}^{\mathrm{co}}$  (refer to Fig. 1). Then solving  $\mu_{\theta}(\pmb{x}_{t}^{\mathrm{s}}, t \mid \pmb{x}_{0}^{\mathrm{co}})$  according to Eq. (3). This process can be trained by following objective [11]:
$$
\min  _ {\theta} \mathcal {L} (\theta) := \min  _ {\theta} \mathbb {E} _ {t, x _ {0}, \epsilon} \| \boldsymbol {\epsilon} - \boldsymbol {\epsilon} _ {\theta} \left(\sqrt {\bar {\alpha} _ {t}} \boldsymbol {x} _ {0} + \sqrt {1 - \bar {\alpha} _ {t}} \boldsymbol {\epsilon}, t\right) \| ^ {2}. \tag {6}
$$
The above equations show that the core of training the diffusion model is to minimize the mean squared error between the Gaussian noise  $\epsilon$  and predicted noise level  $\epsilon_{\theta}\left(\boldsymbol{x}_{t}^{\mathrm{s}}, t \mid \boldsymbol{x}_{0}^{\mathrm{co}}\right)$ .
Sampling. Given the reverse process presented in Sec. 3.2, the generative procedure can be summarized as sampling the  $\boldsymbol{x}_T^{\mathrm{s}} \sim \mathcal{N}(0,\mathbf{I})$ , and iterative sampling based on  $\boldsymbol{x}_{t - 1} \sim p_{\theta}\left(\boldsymbol{x}_{t - 1}^{\mathrm{s}} \mid \boldsymbol{x}_t^{\mathrm{s}},\boldsymbol{x}_0^{\mathrm{co}}\right)$ . The final sampled output is  $\boldsymbol{x}_0^{\mathrm{s}}$ .
Figure 1: An illustration for trajectory generation with diffusion model. (Left) Forward and the reverse process (multiple GPS trajectories presented). (Right) Coupled the neural network model structure for reverse denoising.
4.1 Traj-UNet Architecture
While developing the DiffTraj framework, accurately predicting the noise level at each diffusion time step is non-trivial. A model must capture these complex spatial-temporal dependencies to accurately predict the noise level at each diffusion time step. Inaccurate noise predictions can lead to excessively noisy trajectories, resulting in unrealistic patterns. Thus, we employ Traj-UNet to predict the noise of each diffusion time step in DiffTraj, i.e.,  $\epsilon_{\theta}\left(\boldsymbol{x}_{t}^{\mathrm{s}}, t \mid \boldsymbol{x}_{0}^{\mathrm{co}}\right)$ . Specifically, we construct a network based on the UNet architecture [26], which has been extensively used in generative applications involving diffusion models [35, 20, 5]. The Traj-UNet is composed of two parts: down-sampling and up-sampling, each featuring multiple 1D-CNN-based stacked residual network blocks (Resnet block) with a kernel size of 3 in each layer. An attention mechanism-based transitional module is integrated between the two parts [30] (Detailed information about Traj-UNet can be found in Fig. 4 of the Appendix A). To enhance the learning of noise for each diffusion time step, Traj-UNet incorporates time step embedding and then fed to each block. Specifically, we employ Sinusoidal embedding [30] to represent each  $t$  as a 128-dimensional vector. Subsequently, we apply two shared-parameter fully connected layers and add them to the input of the Resnet block.
Input trajectory. It is important to note that the input/output data for this model is a two-dimensional trajectory tensor ([2, length]), with each dimension representing the longitude and latitude of the trajectory, respectively. Considering CNN-based models can only accept fixed-shape data, we initially up-sample or down-sample the training trajectories using linear interpolation methods [3]. The generated trajectories are then transformed to the desired length using the same approach.
Why use CNN based structure? In DiffTraj, we adopt a CNN-based UNet structure over other temporal processing modules such as recurrent neural networks (RNNs) and WaveNet for several reasons. First, CNN-based methods can alleviate the vanishing gradient problem often encountered in RNNs, enabling more efficient learning of long-term dependencies. Second, compared to models like RNNs, CNNs hold a smaller number of parameters and offer greater computational efficiency, which is crucial during the reverse process. Finally, the UNet architecture excels in various generative tasks, particularly when combined with diffusion models [28, 11, 7]. Its multi-level structure with skip connections enables multi-scale feature fusion, helping the model better capture local and global contextual information in GPS trajectory data. We found that RNN and WaveNet-based architectures led to worse trajectory quality, as will be presented in Sec. 5.
4.2 Conditional Generation
Various external factors, such as the region of the trip and departure time, influence real-world trajectories. This conditional information should provide meaningful guidance for the generation process, ensuring that synthetic trajectories exhibit similar patterns and behaviors. In the proposed DiffTraj framework, a Wide & Deep network [6] structure is employed to effectively embed conditional information, enhancing the capabilities of the Traj-UNet model. The wide component emphasizes memorization, capturing the significance of individual features and their interactions, while the deep component concentrates on generalization, learning high-level abstractions and feature combinations. This combination effectively captures both simple and complex patterns in the trajectory. For a given trajectory, several numerical motion properties (e.g., velocity and distance)