Traj2Former: A Local Context-aware Snapshot and Sequential Dual Fusion Transformer for Trajectory Classification

Authors: yuan  xie
yichen  zhang
yifang  yin
sheng  zhang
ying  zhang
rajiv  ratn
roger  zimmermann
guoqing  xiao
ratn  shah
guoqing xiao 2024 traj2former
yicheng  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 15:54:21

Abstract: The wide use of mobile devices has led to a proliferated creation of extensive trajectory data, rendering trajectory classification increasingly vital and challenging for downstream applications. Existing deep learning methods offer powerful feature extraction capabilities to detect nuanced variances in trajectory classification tasks. However, their effectiveness remains compromised by the following two unsolved challenges. First, identifying the distribution of nearby trajectories based on noisy and sparse GPS coordinates poses a significant challenge, providing critical contextual features to the classification. Second, though efforts have been made to incorporate a shape feature by rendering trajectories into images, they fail to model the local correspondence between GPS points and image pixels. To address these issues, we propose a novel model termed Traj2Former to spotlight the spatial distribution of the adjacent trajectory points (\emph{i.e.}, contextual snapshot) and enhance the snapshot fusion between the trajectory data and the corresponding spatial contexts. We propose a new GPS rendering method to generate contextual snapshots, but it can be applied from a trajectory database to a digital map. Moreover, to capture diverse temporal patterns, we conduct a multi-scale sequential fusion by compressing the trajectory data with differing rates. Extensive experiments have been conducted to verify the superiority of the Traj2Former model.

Main paper: 
 introduction: 
 The widespread use of mobile devices has led to the production of vast amounts of trajectory data, typically in the form of twodimensional GPS point sequences. Classification is a crucial and fundamental challenge within the domain of spatio-temporal analytics. Applications of trajectory classification extend to various downstream tasks, such as providing trip recommendations [22] and enhancing smart transportation systems [2, 24, 37] .Existing research on trajectory classification can be roughly divided into trajectory-based methods, image-based methods, and their fusion. For trajectory-based methods, early work manually extracts speed, acceleration, and bearing as features [6] and employs traditional machine learning models such as SVM, Bayes, KNN, and Random Forest as the classifier [36] . Recent efforts mostly adopt deep learning models such as CNN and RNN [6, 27] to learn more robust trajectory representations from large-scale training datasets. For example, Liang et al. proposed TrajODE [20] to capture the representation of the continuous-time dynamic in trajectory data inherently. Further, they proposed TrajFormer [19] to encode the spatio-temporal intervals of the continuous trajectory sequences to deal with GPS noise and sparsity issues. Meanwhile, a parallel research direction for trajectory classification is to render trajectories into images [8, 13] . This strategy captures the trajectory shape information, constrained by the underlying road network and thus complementary to the traditional trajectory representation. For instance, the Estimator model [13] was proposed to transmute the target trajectory into an image, and then fuse it with the original trajectory data to achieve performance gains. Yuki et al. [8] extracted features from the formulated image by the trajectory for the transportation mode estimation. Moreover, Kontopoulos et al. [14] proposed TraClets to fine-tune the VGG16 model on the trajectory image features for the trajectory classification. However, these methods primarily focus on rendering a single target trajectory without fully exploring other trajectories in a large historical database. Furthermore, the fusion of trajectory-and image-based features is mostly performed before the output layer without exchanging information locally.As illustrated in Fig. 1 , the historical trajectory distribution is an important feature in revealing the transportation mode of the target trajectory. For example, if the nearby samples of a target trajectory are majorly labeled as "Walk" rather than "Car" or "Train", then the target trajectory is more likely to follow a sidewalk that is highly correlated with the label "Walk" rather than a highway which is high related to "Car". However, such semantic information [25] has not been fully investigated in previous work.Motivated by the above observations, we propose a novel framework termed Traj2Former, which first transforms trajectories into global maps and then performs local context-aware snapshots and multi-scale sequential dual fusion. Specifically, our framework extracts a comprehensive global map enhanced by semantics through the rendering of a large-scale historical trajectory database. This conversion leads to the creation of two distinct maps: 1) a physical feature map, which captures the concrete environmental characteristics, and 2) a class distribution map, which depicts the probabilistic spread of different categorical elements across the global map. Thereafter, we propose to extract local contextual snapshots (see Fig. 2 ) from the global map to reduce irrelevant information, which is next enhanced by a novel transformer-based map encoder to further reduce the data noise. The contextual representation generated by the map encoder and the corresponding trajectory representation generated by the trajectory encoder are next fused hierarchically by progressively condensing the input trajectory, aiming to capture both neighborhood and sequential patterns in multiscale. Finally, we would like to emphasize that our proposed Traj2Former is a unified fusion framework, which is capable of accommodating maps generated from different data sources, e.g., trajectory database, crowdsourced map, or satellite imagery. Extensive experiments have been performed on two public benchmark datasets, namely Geolife and MTL, to evaluate our model design and effectiveness. Here we summarize our contributions as follows.• To our knowledge, we are the first to present a unified fusion framework that performs local context-aware snapshot fusion by simultaneously modeling multiscale correlations between trajectory and neighborhood representations. • We propose to generate an enhanced global map by rendering from a large-scale trajectory database to support contextual snapshot extraction, which provides vital and complementary information to the traditional trajectory representations. • We introduce a new and general Transformer-based Map Encoder, which is capable of coping with diverse map sources, to further reduce the impact of noise in crowdsourced data.• We compare our model to the state-of-the-art trajectory classification methods, where significant performance gain has been obtained on both Geolife and MTL benchmark datasets. 
related work: 
 We review related works which can be roughly categorized into point-aware and image-aware trajectory classification methods.Point-aware Trajectory Classification. A trajectory can be represented via multiple data formats [28] [29] [30] 34] . The usual formats of trajectories are comprised of a sequence of 2-dimensional GPS points, e.g., spatial locations, and temporal timestamps. Point-aware methods [1, 7, 17] are proposed to capture the spatio-temporal features [16] among the trajectory that is comprised of the separated GPS points. Zheng et al. [36] proposed four different methods including KNN [11] , Bayes [21] , SVM [4] and Random Forest [10] , to classify a user's transportation mode. Furthermore, Zheng et al. [35] proposed a supervised learning approach to identify the sophisticated features and designed a graph-based postprocessing algorithm to further improve the inference performance. Additionally, Liang et al. [19] adapted transformers to model trajectories to embed the spatio-temporal intervals of the continuous trajectory points and squeeze the points to speed up the representation learning. Lee et al. [18] explored the region-and the trajectory-based features to overcome the discriminative parts of the trajectory identification. However, these methods cannot easily capture the coarse-grained spatial features and the nearing distribution of the trajectories. Image-aware Trajectory Classification. To boost classification performance, some existing studies focus on mapping trajectories to images to capture the shape of the objective trajectory. Hu et al. [13] developed CNN-TCN to utilize the shape of a trajectory and the time embedding for the trajectory classification. Yuki et al. [8] converted a trajectory to a grayscale image and integrated it with hand-extracted features to classify trajectories. Kontopoulos et al. [15] identified a vessel's mobility patterns by fusing the vessel's trajectory and the image of the corresponding trajectory. Furthermore, Kontopoulos et al. [14] proposed TraClets that fine-tune the VGG16 for image-based trajectory classification. These existing approaches directly introduce deep learning methods to fuse the trajectory embedding [9] with the mapped trajectory images to solve the trajectory classification problem. However, the semantic distribution information of nearby trajectories has not been fully investigated in previous works.  
traj2former architecture: 
 The overview framework of our proposed Traj2Former model is illustrated in Fig. 3 , which consists of 1) a Trajectory Encoder that encodes the input trajectory with robust point-level representations; 2) a Local Context-aware Fusion Network that effectively fuses trajectory and map representations with varying spatial granularity; and 3) a Transformer-based Map Encoder that enhances the local contexts and updates the global map with self-attentions.By using the Trajectory Encoder as an initial step, the Traj2Former model gains a refined initial representation of the trajectory data. This improves its performance during the subsequent integration with image-based trajectory data.4.1.1 Sequential Trajectory Feature Extraction. Following previous work [6] , we extract velocity, acceleration, jerk, heading, and its changing rate from GPS trajectories, which are the key features for training an effective trajectory classifier. To avoid obtaining unstable features for each track point, we implement a moving average window to approximate the features at each point by moving through the whole trajectory. The time span of the sliding window is set to △𝑡. For a sample point 𝑖, we set it to be the starting point of the window and calculate the distance 𝑑 from the first sample point to the last sample point. Then, the velocity, acceleration, and jerk of sample point 𝑖 are computed as 𝑣 𝑖 =𝑑/△𝑡, 𝑟 𝑖 =|𝑣 𝑗 -𝑣 𝑖 |/△𝑡, and 𝑎 𝑖 =|𝑟 𝑗 -𝑟 𝑖 |/△𝑡, respectively. Similarly, we also compute the heading and its changing rate at each track point.Encoding. The extracted features described above are then encoded into a high-dimensional vector space using a sophisticated CNN-based neural network [6] . This learned trajectory embeddings comprise point-level robust representations, which will be fed into the Trajectory Compressor to generate informative segment-level representations. Formally, the embedding of a trajectory 𝑇 𝑗 is expressed as:EQUATIONwhere 𝑓 𝑒𝑚𝑏 refers to the trajectory encoder, and 𝜉 denotes the dimensionality of the feature space. By embracing the Trajectory Encoder as a preliminary step, the Traj2Former model benefits from a nuanced initial representation of the trajectory data, leading to improved performance and accuracy in the subsequent fusion with spatial trajectory features.Though efforts have been made on fusing the sequential and spatial trajectory features [13] , the modeling of the local correspondences is largely overlooked. Therefore, we propose a novel fusion method to highlight both global and local fusion at a multi-scale level.A naive fusion approach involves directly merging the self-image feature representation with the trajectory embedding [13] to potentially improve classification outcomes. The mathematical formulation of this coarse-grained naive fusion technique can be described as follows:EQUATIONwhere 𝐸 𝑝 represents the trajectory embedding defined in Eq. 1, 𝐸 𝑚 signifies the spatial trajectory embedding which will be introduced in Section 5.1.2, and ⊗ is the concatenation operation. By merging different forms of data representations, the global segment-to-image fusion approach generates a richer and more comprehensive feature set. Moreover, merging at the coarse-grained level allows for better flexibility in the development process, such as optimizing or modifying each component independently. However, naive fusion overlooks the local correlations between different data modalities and limits the model's ability to capture the interactions on fine-grained level features, which could lead to a decrease in performance.To explore the fine-grained information, we propose a fusion method that primarily aggregates the point-level representations with the corresponding local image representation, shown as follows,EQUATIONwhere𝑒 𝑝 1 , • • • , 𝑒𝑝 𝑛 are the point-wise embedding from the raw trajectory, 𝑒 𝑚 1 , • • • , 𝑒 𝑚 𝑛 are the local spatial trajectory representation, and 𝑛 is the length of the target trajectory. 𝐿_𝐹𝑢𝑠𝑖𝑜𝑛 is a general fusion method with mean-pooling, max-pooling, or other fusion methods to aggregate the features. Here, we adopt concatenation as the fusion method to generate the segment-level embedding. Point-to-pixel fusion exploits the correlation among fine-grained features that are overlooked in segment-to-image fusion. It also extracts consistent features across multiple data sources to gain a deeper understanding of the contextual information. However, multiple points in a trajectory may correspond to the same pixel in the map, resulting in duplicate local contexts retrieved in the point-wise fusion. These redundant local contexts offer minimal new or valuable information and should therefore be excluded from further processing to improve the system's efficiency. Mapping TrajectoryEQUATION)where F𝑝2𝑝 denotes the multi-scale point-to-pixel fusion across 𝐿 layers and 𝐹 𝑠2𝑖 is the segment-to-image fusion as shown in Eq. 2.Trajectory Compressor. To prevent generating redundant information and incurring higher time costs when extracting local maps from the global map, the initial trajectory is compressed to obtain a segment-level feature concentrated with essential information. Particularly, we divide the trajectory into 𝑆 (referred to as the compression ratio) segments recursively, and extract the segment-level representations based on the average pooling of the features from the previous level. Thereafter, we project the location of the center EQUATIONwhere 𝐺 𝑙 and 𝐺 𝑔 represent the local and global contexts, 𝐿𝐶𝐹 denotes the Local Context Fetcher module, 𝑐 𝑝 designates the crop size, and {𝑖, 𝑗 } signifies the pixel coordinates. The local contexts are tailored to offer the distinct advantage of mitigating noise while maintaining a sharp focus on the relevant spatial context.To cope with maps generated from different sources, we would like to design a generalized map encoder with the capability of extracting robust features regardless of the map type. Different types of maps may have their own limitations. For example, the global map generated from historical GPS trajectories may suffer from GPS intrinsic noise and sparsity. OpenStreetMap is crowdsourced and thus may contain duplicate or inaccurate information. Motivated by the above observations, we propose a Transformer-based Map Encoder to aggregate information from other regions to enhance the point-level fusion and segment-level fusion.Refinement for Point-to-Pixel Fusion. To refine the local context for enhancing the point-level fusion, we adopt the multi-head attention model [26] to train the local context map embedding. Especially, we process the two-dimensional global map image and reshape it from R 𝐶 ×𝐻 ×𝑊 into a sequence of two-dimension patches 𝑝 𝑖 ∈𝑃, 𝑃 ∈R 𝑁 × (𝐶 •𝑟 2 ) , where 𝐻 ×𝑊 is the resolution of the global map, 𝑟 2 is the resolution of every single patch, 𝐶 is the number of feature channels, and 𝑁 is the number of patches (e.g., 𝑁 =𝐻𝑊 /𝑟 2 ). Next, the patches are flattened to 𝑑-dimensional vectors with a trainable linear projection 𝑝 𝑖 ∈R 𝑁 ×𝑑 . The traditional attention mechanism is represented as𝑀𝑢𝑙𝑡𝑖𝐻𝑒𝑎𝑑 (𝑄, 𝐾, 𝑉 )=𝐶𝑜𝑛𝑐𝑎𝑡 (ℎ𝑒𝑎𝑑 1 , • • •, ℎ𝑒𝑎𝑑 ℎ )𝑊 𝑜 , ℎ𝑒𝑎𝑑 𝑖 =𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝑄𝑊 𝑄 𝑖 , 𝐾𝑊 𝐾 𝑖 , 𝑉𝑊 𝑉 𝑖 ),where the local maps are the query, and the flattened global maps are the key and value in our applied multi-head attention module. More specifically, the query is given as 𝑄 ∈R 𝑣× (𝐶 •𝑟 2 ) and the key and value are defined as 𝐾=𝑉 ∈R 𝑁 × (𝐶 •𝑟 2 ) , where 𝑣 is the number of local contexts returned by the Local Context Fetcher.Map Enhancement for Segment-to-Image Fusion. To refine the global maps and extract highly representative features, we employ a multi-head self-attention model to iteratively update the global map at each layer within the Traj2Former framework.The global map enhancement is proposed to enhance the segmentlevel fusion. For the self-attention model, the query, the key, and the value share the same sequence of patches from the global map. The relationship is defined as 𝑄=𝐾=𝑉 ∈R 𝑁 × (𝐶 •𝑟 2 ) . For optimization efficiency and feature consistency, the multi-head self-attention model shares the same parameters of the local context refinement module introduced in Section 4.3.1. The same transformative weights are applied when updating the global map and extracting the local contexts. Thus, the global map and local contexts are mutually enhanced, benefiting from the consistent set of learned features.Our Traj2Former model consists of 𝐿 layers where MLPs are applied as decoders to convert trajectory representations to class distributions. The 𝐿 𝑡ℎ layer produces the final predictions. Building on the observation that sub-trajectories share the same class label, we train our Traj2Former model by jointly minimizing the cross-entropy loss on all layers as given belowEQUATIONwhere ŷ and 𝑦 are the predicted class and true label, respectively. L 𝑖 is the cross-entropy on output of the 𝑖-th layer and L 𝐿 is the cross-entropy on the final output of the last layer. 
map acquisition: 
 Features such as speed, acceleration rate, and bearing play a decisive role in trajectory classification [6] . To obtain complementary information from the historical trajectories, we extract not only the aforementioned physical statistics but also the transportation mode class distributions to generate a global map in our work.Efforts have been made to convert GPS trajectories [13, 32] and crowdsourced map data [31] into images, enriching trajectory data with visual information (see Section 4.1.1). While sequential features analyze movement patterns over time, trajectory images highlight the distribution of nearby trajectories, offering insights into the road environment. The model name, Traj2Former, reflects this dual approach, encompassing both sequential and spatial features.5.1.1 Self-image generation from a single trajectory. To enable visual representation of the trajectory in a more intuitive and accessible format, the initial trajectory is converted to a self-image by transforming GPS data points into pixel coordinates in an image. We compute the latitude and longitude span of the whole trajectory and obtain the pixel index of each GPS point by setting the image size to 𝑊 ×𝐻 . In particular, for each sample point {⟨𝑙𝑎𝑡 𝑖 , 𝑙𝑜𝑛 𝑖 ⟩, 𝑡 𝑖 }, we compute the pixel index as follows,EQUATIONwhere 𝑚𝑖𝑛 𝑎 and 𝑚𝑖𝑛 𝑜 are the minimal latitude and longitude of the trajectory, 𝑙𝑎𝑡 𝑝 𝑖 and 𝑙𝑜𝑛 𝑝 𝑖 are the obtained pixel index. Based on the given trajectory, we extract five important features, including speed, acceleration, angle, acceleration rate, and angle rate. Since multiple GPS points can be projected to the sample pixel on the map, we compute their average as the final feature.To create a global map with nearby trajectories, we aggregate trajectories from the historical database and apply the self-image formulation method, replacing the individual trajectory with a set of neighboring trajectories. Two types of maps are converted from the historical trajectory database: physical statistic maps and class distribution maps, collectively called global maps. The region covered by the global map is defined to be the same as the self-image introduced above. Subsequently, we retrieve all trajectories within that region from the historical database to generate the global map. Besides, we have enriched the original features by incorporating the detailed statistical descriptors of speed, acceleration, and angle, specifically the maximal, minimal, and average values of each attribute, and the number of GPS points. The physical statistic features 𝐶 𝑝 are formulated based on channel-wise concatenation as,EQUATIONwhere 𝐶 𝑖 𝑓 , 1≤𝑖≤𝑚, is the extracted features from the adjacent trajectories, and 𝑚 is the number of physical feature channels. However, the physical statistic features neglect the class distribution of the neighboring trajectories, which intuitively captures the road environment. To enrich the global map features, we further generate the class distribution features as one component of the global map. Assume that there are 𝑘 classes in the dataset, then 𝑘 channels will be created, each of which records the normalized count of the GPS points belonging to a specific class. Formally, we haveEQUATIONwhere 𝐶 𝑖 𝑠 is the distribution channel for the 𝑖-th class, 1≤𝑖≤𝑘. The final global map representation 𝐸 𝑚 is comprised of the physical feature map and the class distribution map as 𝐸 𝑚 =𝐶 𝑝 ⊙ 𝐶 𝑑 .Public map sources can also be utilized in our Traj2Former model, where APIs are usually provided for downloading map visualizations such as digital map [33] and satellite map [32] .In real-world scenarios, we would like to leverage additional map resources that are publicly available. This is important since our model has the ability to fuse different map features from multiple map sources to boost classification performance. Digital maps, such as HERE map and OpenStreetMap, can be rendered in different styles (i.e., Cycling Map, Transportation Map, Topography Map, and Humanitarian Relief Map), which capture different road attributes to further enhance the model performance.Imagery. Our Traj2Former model exhibits remarkable flexibility, proving effective not only with self-generated global maps but also with satellite imagery. To assess the performance and reliability of Traj2Former when applied to satellite maps, the HERE map API is utilized. This tool enables us to precisely extract satellite images corresponding to target trajectories. Table 6 presents the results of the ablation study on using different map sources. 
experiments 6.1 experimental settings: 
 Existing works divide trajectory data into segments by counting a specific number of valid GPS points [5] . However, the time duration of each segment is critical in practical applications, as the transportation mode typically remains constant over a continuous time span. To achieve this, we inserted a new point at one-second intervals between two consecutive points, resulting in segments composed of 600 points (i.e., 10 minutes) in Geolife and inserted a new point at five-second intervals of 650 points in MTL. Geolife and MTL datasets. The Geolife dataset contains 27,795 trajectory segments and six transportation modes. The MTL includes four selected modes of transportation and contains 23,406 segments. These two datasets, with a division of training and testing data, were maintained at an 8:2 ratio across each transportation mode category. Details of the dataset are illustrated in the Appendix. Implementation Details. We apply the Adam optimizer during training and adopt a learning rate of 0.001 with a weight decay of 0.001. The batch size is set to 64 and all generated images are 50×50 in size. The kernel size for the sequential and spatial trajectory embedding is set to 3 and 6, respectively. The Traj2Former model has 3 layers, and we set 𝛼=0.3, 𝛽=0.4 in the loss function. All experiments are conducted on a single NVIDIA RTX A6000 with 48GB memory. Metrics. For the trajectory classification problem, we follow [3] to report the sub-class classification accuracy and the overall accuracy to evaluate our model's effectiveness. Here, the accuracy is the ratio of the number of correct samples to the number of the whole samples. Besides, we also list the per-class accuracy to show the detailed effectiveness of each class. In the presented tables, we also use the notation Δ to indicate the relative improvement of accuracy compared with the state-of-the-art method.We compared our proposed method to the following state-of-art methods and report the results in Table 1 .• SVM, KNN, RF [36] applies the machine learning methods to solve the trajectory prediction problem. • RNN, GRU, STGRU [3, 27] focuses on solving the variable length sequence to address the constraints of topological structure on trajectory modeling. • LSTM, Bi-LSTM [12, 23] proposes a classifier that automatically processes the features from trajectories. • TrajFormer, TrajODE [19, 20] considers spatio-temporal intervals to generate trajectory embeddings.• SECA [6] automatically extracts relevant features from trajectories and exploits useful information in unlabeled data. • DNN, TraClets [8, 14] extracts the self-images feature and fuses it with the trajectory embedding. • Estimator, Estimator* [13] integrates the self-images with time-interval embedding and frozen partial parameters by [6] . As can be seen, an obvious trend is that Traj2Former outshines all baseline contenders. Particularly, Traj2Former shows a 10.54% enhancement over trajectory-only methods, i.e., SVM, KNN, RF, RNN, GRU, LSTM, Bi-LSTM, TrajFormer, TrajODE, and SECA, for the Geolife dataset, and an 8.57% increase on the MTL dataset. This improvement suggests that the Traj2Former model adeptly captures the distribution of nearby trajectories, thereby enhancing its performance relative to models considering only pure trajectories. Specifically, in comparison to image-related methods such as DNN, TraClets, Estimator, and Estimator*, Traj2Former exhibits advancements of at least 8.15% and 2.99% on the Geolife and MTL datasets, respectively. These figures underscore the effectiveness of the generated global map, revealing the fact that considering nearing trajectories improves the classification performance. In specific sub-classes like cars, buses, and subways within the Geolife dataset, Traj2Former achieves notable improvements. For the MTL dataset, Traj2Former significantly outperforms all baselines in all public transportation classes. Such findings demonstrate that Traj2Former has the capability to identify similar modes of transportation, by leveraging the consistent relationship between the compressed trajectory embedding and the corresponding local contexts.We conduct ablation studies to evaluate the effectiveness of each design strategy. Experiments are implemented on the Geolife dataset.Components in Proposed Model. Table 2 presents the effectiveness of each designed components. Firstly, the transformer-based map encoder is replaced with the CNN, resulting in the most decreased accuracy by 3.11%. This fact reveals the designed attention module can reduce the noise from global maps and focus on the local context information. Besides, we remove the trajectory compressor module but directly concatenate the trajectory embedding to the global map, leading to a 1.96% decrease. This shows the merits of the compressor module since it effectively reduces the duplicated local context features from the global map. In the experiment, i.e., w/o Auxiliary Losses, we only compute the loss of the final layer and exclude the loss of the middle layers. The performance is decreased by 0.97%, demonstrating the dependency of the model's performance on its internal layer-wise interactions. This cumulative evidence underscores the effectiveness of each component in the Traj2Former model. 4 . When we decrease the compress rate from 30 to 5, the overall trend is that the performance increases first and then decreases. In particular, reducing the compress rate first results in more local context images from the global map, and the accuracy increases. When the compress rate is increased further, the accuracy decreases. That is, a small compress rate results in duplicated local contexts, especially when we have a dense GPS distribution under a small interpolation size. Fig. 4 demonstrates the accuracy trends for different subclasses at assorted compression rates. We observe that when the compress In contrast, Car, Bus, and Train initially show improved accuracy, probably of their stop-and-go moving pattern, but this benefit is lost as accuracy drops when compression rates excessively.Size 𝑐 𝑝 . To detect the impact of the local context size 𝑐 𝑝 , we adjust 𝑐 𝑝 from 3 to 9. The experiment results are shown in Table 5 . With increasing the local context size, the accuracy increases first and then decreases. That is when expanding 𝑐 𝑝 , more information on the distribution of nearby trajectories is provided to distinguish different mode categories. However, this benefit is lost since further expending 𝑐 𝑝 results in the distribution of all local contexts being similar and difficult to distinguish.To evaluate the effectiveness of our proposed Traj2Former model on the actual map dataset, we extract the actual map from the HERE map. Global maps are the original self-built maps, including physical features and class distribution features. Satellite maps are the 3-channel images from the real map. To better compare the baseline models with our proposed Traj2Former model, we implement the CNN model as the baseline model. Table 6 presents Traj2Former performing better than CNN on the global map, demonstrating the model's effectiveness. Similar results can be obtained on the satellite map dataset. When fusing the global map and satellite map, Traj2Former performs better than using the global map and satellite map individually.6.3.6 Impact of the Filtering Methods. To further reduce the noise and irrelevant information in the generated global map, it is feasible to retrieve only nearby candidates around the target trajectory instead of gathering all trajectories within the self-image region. In this way, the global map is purged of noise data, thereby sharpening the relevance of the information to the target trajectory and Noted that our method generates a complete offline map only once. Thus, the major overhead of our method is the multi-localmap cropping process, which is currently computed in a serialized manner but can be accelerated with parallel computing. The processing time per sample is 0.05 s for Traj2Former without applying filtering in the global map generation. The balance between accuracy and efficiency should be determined based on the specific application needs, e.g., to enhance efficiency by increasing the compression ratio or to boost accuracy by generating a filtered global map, with a compromise on the other factor. 
conclusion: 
 In this paper, the Traj2Former framework is designed to solve the trajectory classification problem. This framework generates global maps from the historical trajectory database to serve as image-based feature embedding and subsequently concatenated with trajectory embedding. To address the advantages of segment-to-image fusion and point-to-pixel fusion, we design a multi-scale fusion framework to generate an enhanced representation. Additionally, to improve the quality of global maps, the transformer-based map encoder is developed to focus on the alignment between the compressed trajectory and the corresponding local context snapshot. Extensive experiments are conducted on two real-world datasets to demonstrate the effectiveness of our developed framework. 
 
 Appendix:  


Decision: The authors propose Traj2Former, a framework for trajectory classification. In particular, this framework has the goal of improving trajectory classification tasks by using local contextual information and integrating trajectory data and spatial contexts. As noticed by one of the reviewer, the paper incorporates an interesting GPS rendering method to generate contextual snapshots as well as a multi-scale sequential fusion methodology to capture diverse temporal patterns. The experimental work is solid and conducted extensively on two datasets, Geolife and MTL, in the original paper and on more datasets in the rebuttal. Also the comparisons with other approaches are adequate: various baseline and alternative models were tested in the original paper and in the rebuttal. However, as pointed out by Reviewer vvAG, several of the components of the models are well established one in computer vision and also multimedia communities, so the suggestion is to highlight better the novel aspects of the proposed approach. Finally, the authors conducted a detailed rebuttal where they target the various concerns raised by the reviewers.

Reviews: review: paper_summary: paper_summary: The paper introduces Traj2Former, which leverages both local context and sequential information for trajectory classification. The authors have identified key issues in existing methods, such as the inability to effectively utilize the distribution of nearby trajectories and the loss of local correspondence information when trajectories are rendered into images.
strength_weakness: strength_weakness: 1. The paper presents a transformer-based model that considers both the local context and the sequential nature of trajectory data.

2. The idea of generating contextual snapshots from a global map to provide critical contextual features is new enhances the model's ability to classify trajectories accurately.

3. This paper conducted comprehensive experiments on two real-world datasets, demonstrating the superiority of Traj2Former over existing state-of-the-art methods.
limitations: limitations: 1. The paper is lack of discussion of the computational complexity of the proposed method, especially considering the use of transformer models which can be computationally intensive.

2. While the method shows promising results on the tested datasets, it would be beneficial to evaluate its performance on other types of trajectory data and across different domains.
paper_summary: paper_summary: The paper introduces a novel model, Traj2Former,  for trajectory classification using deep learning techniques. Generating contextual snapshots to capture the spatial distribution of nearby trajectories. Implementing a multi-scale sequential fusion technique to compress trajectory data and capture diverse temporal patterns. Extensive experiments demonstrate that Traj2Former achieves state-of-the-art classification accuracy on two real-world datasets, validating its effectiveness in trajectory classification tasks.
strength_weakness: strength_weakness: 1. The paper introduces a novel framework, Traj2Former, which combines local context-aware snapshots with a multi-scale sequential dual fusion transformer for trajectory classification. The approach of generating contextual snapshots and using a transformer-based map encoder to handle noise and refine local contexts is innovative.  Previous methods either did not fully explore the spatial distribution of nearby trajectories or failed to model the local correspondence between GPS points and image pixels. Traj2Former bridges this gap by integrating both local and global spatial contexts, providing a more holistic view of the trajectory data.

2. The use of a multi-scale sequential fusion technique to compress trajectory data at different rates is theoretically robust. This method allows for capturing diverse temporal patterns, which is critical for accurate trajectory classification. The transformer-based map encoder is a technically sound choice for handling large-scale data with noise and sparsity. The attention mechanisms used in transformers are well-suited for identifying relevant spatial features in the context of trajectory data.

3. The paper is well-organized and clearly written, making it easy to follow the methodology and understand the contributions.
Figures and tables are used effectively to illustrate the model architecture, the experimental setup, and the results.
limitations: limitations: 1. Some components of the model, such as the use of transformer-based encoders and multi-scale fusion, are well-established techniques in the field of deep learning and computer vision. These methods have been widely used in various other contexts, which may reduce the perceived novelty of the paper. Although the specific application of these techniques to trajectory classification is innovative, the individual components themselves are not new. For instance, transformers and attention mechanisms are standard in many current deep learning models (Vaswani et al., 2017).

2. The paper focuses heavily on comparisons with other deep learning models but provides limited evaluation against traditional machine learning methods. This could give a skewed perspective of the model's performance. Including comparisons with well-known traditional methods, such as SVMs, decision trees, or KNN, would provide a more comprehensive understanding of the improvements offered by Traj2Former. 

3. The evaluation is conducted on only two datasets, Geolife and MTL. While these are valuable datasets, the model’s robustness and generalizability could be better demonstrated with a more diverse set of datasets from different domains or geographic regions.

4. The proposed model involves multiple complex components, such as the transformer-based map encoder and the multi-scale sequential fusion, which might be computationally intensive and challenging to scale for very large datasets. Although not directly addressed in the paper, the computational requirements for training and deploying Traj2Former could be significant. A discussion on the model's scalability, resource requirements, and potential optimization strategies would be beneficial.

5. While the paper provides a clear high-level overview, some of the implementation details are sparse. For example, specific hyperparameters, the exact structure of the transformer layers, and the computational costs are not thoroughly discussed. Detailed implementation information is crucial for replicability and practical application of the research. Including more comprehensive details would help practitioners and researchers to better understand and implement the model.
paper_summary: paper_summary: The paper proposes Traj2Former,  which addresses challenges in trajectory classification arising from noisy and sparse GPS data. The methodology is robust, with a clear explanation of the dual fusion mechanism. The experimental results are impressive, showing clear benefits over traditional models. However, it needs some revisions based on the limitation part.
strength_weakness: strength_weakness: 1. The paper is well structured, clearly explaining the motivation, challenges, proposed architecture and experimental findings.
2. The design decisions of the architecture are well explained and supported with strong ablation studies.
3.The strong baselines including recent methods are provided.
4. The code is released which helps the community to easily build upon the contributions.
limitations: limitations: 1. The author did not specify the settings for the sliding window in the text, which I consider a critical hyperparameter. Specific experiments or descriptions should be added.
2. Section 4.1.2 does not clearly define what function or method f_emb refers to, the author should provide a more detailed explanation.
3. The class ratio in the appendix is crucial, it provides further insights into the experimental results and should be moved to the main text.
4. Table 1 shows that Traj2Former without attention performs poorly on the subway category in the Geolife dataset, with only a 51.2% accuracy. Why is this the case?
5. In Table 1, should the 'car' category in the MTL dataset be labeled as 'bus' based on the information in the appendix? Is this a typographical error?
6. An odd phenomenon is that, in the MTL dataset, there are more data points for the Bike category than for the Walk category, yet the Traj2Former performs worse on Bike and is weaker than the state of the art. Normally, with more data, the model should learn more and thus perform better.
7. Regarding the ablation experiments, some settings in Section 6.3.2-5 perform better than the best metrics in Table 1, which are 93.49%. Specifically, why isn't the best ACC result in Table 1 the highest at 96.09% on Table 5?
paper_summary: paper_summary: The authors propose a novel model termed Traj2Former to address challenges in trajectory classification. The model aims to improve the detection of nuanced variances in trajectory classification tasks by leveraging local contextual information and enhancing the integration between trajectory data and spatial contexts. This paper incorporates a new GPS rendering method to generate contextual snapshots and employs a multi-scale sequential fusion to capture diverse temporal patterns. The model is designed to be adaptable to various context sources, such as trajectory databases, digital maps, or satellite imagery. Extensive experiments demonstrate that Traj2Former achieves state-of-the-art classification accuracy on two real-world datasets, Geolife and MTL.
strength_weakness: strength_weakness: 1: The paper proposes a unified fusion framework, which effectively combines trajectory and map data, leveraging both point-level and segment-level embeddings to capture complex spatial and temporal patterns.

2: The model's ability to work with various context sources (trajectory databases and digital maps) enhances its applicability across different domains and scenarios.

3: The paper conducts sufficient ablation studies to evaluate the effectiveness of each design strategy.
limitations: limitations: 1: Although the model shows superior performance, it lacks comparison with more extensive and state-of-the-art baselines comparison as follows.

[1]. TIF（2023）—— “Cristiano Landi, Riccardo Guidotti, Mirco Nanni, and Anna Monreale. 2023. The Trajectory Interval Forest Classifier for Trajectory Classification. In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems (SIGSPATIAL ’23). Article 67, 4 pages.”

[2]. TrajODE（2021）—— “Liang Y, Ouyang K, Yan H, et al. Modeling Trajectories with Neural Ordinary Differential Equations[C]//IJCAI. 2021: 1498-1504.”

[3]. CIF（2020）—— “Middlehurst M, Large J, Bagnall A. The canonical interval forest (CIF) classifier for time series classification[C]//2020 IEEE international conference on big data (big data). IEEE, 2020: 188-195.”

[4]. ST-GRU（2019）—— “Liu H, Wu H, Sun W, et al. Spatio-temporal GRU for trajectory classification[C]//2019 IEEE International Conference on Data Mining (ICDM). IEEE, 2019: 1228-1233.”

2: While the model performs well on the Geolife and MTL datasets, it is suggested to conduct more experiments on another two datasets: TDrive and NYCTaxi, to see the generality of the proposed method.

3: The proposed model's complexity might pose challenges for practical implementation, the paper lacks experiments related to computational resources and efficiency required for model training.