TrajCLIP: Pedestrian trajectory prediction method using contrastive learning and idempotent networks

Authors: pengfei  yao
yinglong  zhu
huikun  bi
tianlu  mao
zhaoqi  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-25 11:06:03

Abstract: The distribution of pedestrian trajectories is highly complex and influenced by the scene, nearby pedestrians, and subjective intentions. This complexity presents challenges for modeling and generalizing trajectory prediction. Previous methods modeled the feature space of future trajectories based on the high-dimensional feature space of historical trajectories, but this approach is suboptimal because it overlooks the similarity between historical and future trajectories. Our proposed method, TrajCLIP, utilizes contrastive learning and idempotent generative networks to address this issue. By pairing historical and future trajectories and applying contrastive learning on the encoded feature space, we enforce same-space consistency constraints. To manage complex distributions, we use idempotent loss and tightness loss to control over-expansion in the latent space. Additionally, we have developed a trajectory interpolation algorithm and synthetic trajectory data to enhance model capacity and improve generalization. Experimental results on public datasets demonstrate that TrajCLIP achieves state-of-the-art performance and excels in scene-to-scene transfer, few-shot transfer, and online learning tasks.

Main paper: 
 introduction: 
 Pedestrian trajectory prediction involves predicting future paths based on observed historical trajectories. This task is crucial in real-world applications such as autonomous driving [4, 13] , and robotics [11] . It is challenging to model diverse trajectory distributions across different scenes (e.g., roundabouts and intersections), various patterns (e.g., staggering and strolling), and different interactions (e.g., walking together or avoiding strangers) due to their complexity.The existing methods [31, 21, 6] assume that the training and test motions follow the same distribution to effectively model complex distributions within a specific range. These approaches focus on modeling the high-dimensional feature space distribution of trajectories within a particular dataset. They use generative models, such as Conditional Variational AutoEncoder [23, 24] and diffusion, to change the feature space of past trajectories into the feature space of future trajectories. This process aids in generating predicted trajectories and fitting the distribution of pedestrian trajectories to specific datasets. However, as shown in Fig. 1 , this approach has limitations. It involves reducing The existing methods use an encoder to sample from a normal distribution in order to generate predicted trajectories. However, this approach leads to different feature spaces for historical and future trajectories, causing problems during sampling, such as sharp turns or predictions of positions far exceeding normal speeds. Our method ensures consistency between the two feature spaces, enabling position prediction through affine transformations and preserving spatio-temporal continuity.the feature space to match the distribution of a specific dataset to improve prediction accuracy. Nonetheless, this projection operation creates a gap in the feature space between the historical and future trajectories, limiting the model's ability to make predictions across different scenarios and to adapt to new scenarios.We believe that predicting future trajectories in trajectory prediction tasks is significantly different from tasks such as classification [32] and tracking [30] , where the input and output are inconsistent. In trajectory prediction, the feature space of the input and output remains consistent. Therefore, mapping historical trajectory features to future trajectory features in the same feature space can effectively bridge the feature space gap and model complex feature space distribution more accurately.We have developed a method called TrajCLIP for predicting trajectories, which is based on contrastive learning and idempotent neural networks. This method was inspired by the approach used in the paper by [18] , which involves limiting the feature space of heterogeneous data. To achieve this, we created a trajectory feature encoder that utilizes frequency domain additivity and Fourier transforms reversibility to combine trajectory Interaction Features (STIF) in the time domain and Interaction Features (SAIF) between agents in the frequency domain. We utilized cross-entropy loss for constructive learning between historical trajectory and future trajectory encoders to ensure that the feature spaces of the two encoders are the same. Subsequently, we employed an idempotent generation network (IGN) as a global feature mapping framework to map the same set of features from past trajectory features to future trajectory features. Additionally, we constrained the growth of the specific distribution to allow for more general usage. Finally, the mapped future trajectory features were decoded into predicted trajectories. We also introduced a neighbor-based interpolation algorithm and used a trajectory generation model to create a synthetic trajectory to enhance the model's ability to describe complex distributions. Furthermore, we trained a lightweight version (TrajCLIP-tiny) without these augmentations. We conducted experimental comparisons with existing trajectory prediction methods on various tasks and datasets. The experiments demonstrated that our method achieves state-of-the-art accuracy and outperforms existing methods in cross-scene generalization, small sample transfer, and online tasks.The contributions of our work are summarized as follows: (1) In this study, TrajCLIP is presented as a proposed method. The model effectively represents and generalizes complex feature distributions by integrating the feature spaces of historical and future trajectories. An idempotent neural network with global feature mapping capabilities is utilized, enabling strong cross-scene generalization abilities. (2) We proposed a novel trajectory feature encoder combines features from both the time domain and the frequency domain using STIF and SAIF to model pedestrian trajectories. (3) We have created a novel method for measuring trajectory similarity, which we use as a label for contrastive learning. This helps to narrow down the feature space of past and future trajectories. To our knowledge, this is the first application of contrastive learning to trajectory prediction. This is the first work that considers the feature space alignment between the history and future trajectory. Our experiments demonstrate that TrajCLIP performs exceptionally well across different tasks and datasets. 
pedestrian trajectory prediction: 
 Pedestrian trajectory prediction is approached as a sequence-to-sequence challenge. Social-LSTM [1] first uses LSTM to predict trajectories deterministically. Social-GAN [7] uses a Generative Adversar-ial Network (GAN) to make many possible future predictions for an input trajectory while taking into account the inherent multimodality of human motion. Trajectron++ [21] takes in motion trajectories and heterogeneous data, and it handles multi-modality by using CVAE to sample latent variables in high-dimensional space. AgentFormer [31] proposes agent-aware attention to simultaneously capture the spatiotemporal features of trajectories and improves the sampling method of the CVAE framework to model more intricate trajectory modalities. MID [6] was the first to use diffusion, which gradually gets rid of uncertainty in all walkable areas by using a high-dimensional denoiser until the desired path is reached. A new model for predicting trajectories called Flow-Chain [14] uses normalizing flow to quickly figure out the probability distribution for each point on the trajectory. TUTR [22] employs a unified transformer to model trajectories, directly providing the predicted trajectory and the probability of each mode, which effectively eliminates the need for post-processing. Long-tail analysis [15] uses Kalman filter to directly predict output trajectory errors as a basis, distinguishing difficult and easy samples. It then conducts contrastive learning on difficult (long-tail part) historical trajectories to enhance the feature representation capabilities of the long-tail part of the dataset. FEND [25] uses VAE to extract features from the entire trajectory and performs offline clustering on the features to obtain labels; during training, it uses offline labels for contrastive learning on historical trajectories to guide the feature generation of historical trajectories and enhancing the prediction of the long-tail part. However, projecting past trajectory features into separate feature spaces creates a feature space gap that restricts the model's capacity to generalize and accurately represent complex trajectory distributions.Compared with the deterministic framework, the generative framework focuses on modeling data distribution in the dataset and is well-suited for tasks involving one input and multiple outputs. GAN [5] introduces a generative adversarial network, allowing the generator to model the real data distribution through adversarial learning with the discriminator. VAE [12] introduces latent variables, uses the encoder to construct the latent feature space representation of the data, and uses the decoder to restore and reconstruct the feature tensor. CVAE [24] adds a control signal on top of VAE [12] . This signal changes the variational lower bound in the conditional probability state, which lets the model control the generated data distribution based on the signal that goes into it. Diffusion [9] models the data distribution by simulating the diffusion process in high-dimensional space. Adding high-dimensional denoisers gradually lowers uncertainty about the data distribution, leading to samples that are similar to the training data.In these methods, there is a separation between the feature spaces of the control signal and reconstructed data. However, future and historical trajectory features should be in the same feature space, and modeling these independently will lead to a natural gap. IGN [23] aims to model global feature mapping, which constrains the distribution of specific datasets on the expected manifold through idempotent loss and compaction loss. This approach prevents the model from being overfitted to the distribution of specific datasets and enables data generation within the same feature space. DALLE [19] uses CLIP [18] to align the different feature spaces of text and images. It then uses the encoded feature vector of the text as a control signal to make different kinds of images using GLIDE [17] . We utilize contrastive learning to ensure consistency between the feature spaces of historical trajectory and future trajectory, based on the concept of aligning feature spaces. Subsequently, we employ IGN mapping to generate trajectories. 
problem formulation: 
 We generalize the trajectory prediction task as follows. Given a scenario comprising N agents, the historical information for each agent includes the two-dimensional coordinate position p n t = (x n t , y n t ) of agent n at time t. The trajectory prediction task is to predict future position sequence pn t0+1∼t0+T f based on the observed historical agent position sequence p n t0-T h +1∼t0 .We propose a trajectory prediction method called TrajCLIP, which uses contrastive learning and and idempotent generative network, as illustrated in Fig. 2 . Our TrajCLIP comprises four parts: (1) . A trajectory encoder that models interactions between agents, scene-agent interactions in the frequency domain, and trajectory spatiotemporal characteristics in the time domain. (2) . A CLIP pairing learning module that uses cross-entropy loss to maintain feature space consistency between past and future trajectories. (3) . A feature manifold predictor that employs idempotent and tightness loss to train an IGN for mapping past trajectory features into future trajectory features in the same feature space. (4). A trajectory manifold decoder that uses a pre-trained task decoder from the trajectory encoder to generate predicted trajectories. We will provide detailed introductions to each part in the subsequent sections. Specific implementation details can be found in Sec. 4.2.Various data augmentation techniques are used in current methods to enhance the performance of pedestrian trajectory prediction models, including rotation for robustness, scale alignment for training simplicity, and trajectory origin alignment for addressing absolute position ambiguity. In our research, we incorporate minor perturbations (as discussed in [20] ) into trajectories to generate more reliable results. We achieve this by simulating scenarios using available datasets, as demonstrated in the study conducted by [33] . This approach allows us to enhance the existing dataset. Additionally, we have developed an interpolation technique that increases the model capacity by integrating neighboring trajectory sample points.We formulate the adjacent point interpolation algorithm as illustrated in Fig. 4 . If the vectors ----→ p t-1 p t and -----→ p t+1 p t+2 are on the same side of ----→ p t p t+1 for four consecutive positions p t-1∼t+2 , we can calculate the acute angle between the two adjacent trajectories. To derive the interpolated trajectory, the data is divided into three equal segments. Then, we find the intersection point of the third line and connect them. When the vectors are on opposite sides or at the beginning and end of the sequence, the trisect points are chosen as the interpolation points. In the rest of this paper, we will use the symbol p n t-T h +1∼t to represent the interpolated trajectory, for simplicity. Despite the potential decrease in accuracy, the generated and interpolated data can still be used to improve the model's capabilities and capture more complex distributions, considering the multimodal nature of the pedestrian trajectory prediction task.In our approach, we use two structurally identical encoders, as shown in Fig. 3 . These encoders represent the historical trajectory H and the future trajectory F . We utilize the agent-aware attention φ Att from Agentformer [31] to extract the STIF, and employ a transformer based on Fourier Transform to extract the SAIF. By fusing these features, we train both encoders to decode future trajectories. Additionally, the historical trajectory encoder aims to accurately predict the subsequent frame position to maintain spatio-temporal continuity.We define the time-series set T H = {t 0 -T h + 1, ..., t 0 -1, t 0 } for the historical trajectory encoder, and T F = {t 0 -1, t 0 , t 0 + 1, ..., t 0 + T f } for future trajectory encoders. To obtain the spatio-temporal zF  X n X nh X nl X t φ i n p k n  A i n X t +3 X t +2 X t +1position sequence velocity sequence interaction feature (STIF), we arrange the trajectories in a scene intoX t = ((p n t | n = 1, ..., N ) | t ∈ T ), where time is used as the index. At each time step, we use a straightforward linear mapping to tokenize the X matrix and include positional encoding to maintain the temporal information of the trajectory token sequence. We employ agent-aware attention to extract both the temporal features of each individual trajectory and the interaction features between different trajectories simultaneously. The feature is denoted as z t and can be formally described as follows:EQUATIONwhere W T 1 and W 2 represent trainable matrices, φ Att denotes agent-aware attention, Γ denotes sine and cosine position encoding [31] , and M LP denotes a multi-layer perceptron. We can easily derive the velocity V t and acceleration A t of the sequence using X t . Similarly, we employ equation 1 to extract features for both of these sequences. Parameters are not shared among different sequences.The Scene-Agent Interaction Feature (SAIF) characterizes the shared representation of the scene by the agents. We believe that the interactions between agents and the scene are implicit and shared. These interaction features should be represented uniformly, without including the individual characteristics of the agents. As there is no direct way to merge the agents' trajectories in the time domain, we leverage the additivity in the frequency domain to eliminate the fine-grained features of the agents' trajectories, thereby emphasizing the interaction information of the scene.We organize the trajectories in a scene into X n = ((p n t | t ∈ T ) | n = 1, ..., N ) with agent as the index. We apply a Fast Fourier Transform (FFT) to each agent's trajectory based on Equation 2. Utilizing conjugate symmetry, we can derive the cosine expression for both amplitude and phase.EQUATIONWe employ a Butterworth filter to separate high-frequency information (p n k , A n i , ϕ n i ) to enhance modeling efficiency. Then we superimpose and concatenate the low-frequency signals X nl and high-frequency signals X nh of all agents in the scene respectively, and employ self-attention to extract feature.EQUATIONwhere W 3 and W 4 denote trainable matrices, φ g att denotes multi-head self-attention, and IF F T denotes Inverse Fast Fourier Transform (IFFT). IFFT is a lossless and reversible transformation, so the extracted features can be inversely transformed into time-domain features without any information loss and fused with STIF in the same feature space. The final output of the trajectory encoder is the result of a linear transformation of the sum of z t and z a .We aim to transform the historical trajectory features into estimated future trajectory features, displaying temporal continuity to aid in trajectory prediction. It is important to maintain consistency between the feature spaces of past and future trajectories to satisfy spatio-temporal constraints. Therefore, it is necessary to retrain the historical trajectory encoder to align its feature space with that of the future trajectory encoder. We utilize CLIP [18] , a method for aligning feature spaces of different types of information representation, such as text and images, using contrastive learning. We apply this concept to ensure consistency between the feature spaces in the past and future. The specific algorithm is outlined in Algorithm 1. The similarity matrix within each training batch of trajectories is calculated using Equation 4. The mean included angle cosine of the velocity direction vectors between trajectories i, j ∈ {1, 2, ..., batch_size} is computed at each corresponding sample point in both the historical and future data.EQUATIONSimultaneously, we calculate features z i H and z i F for each agent using the historical trajectory encoder and future trajectory encoder, and then combine the corresponding encoder features. Next, we align the dimensions using a trainable matrix W. Finally, we use the dot product to obtain the similarity scores and optimize them using the cross-entropy loss with the similarity matrix. Contrastive learning ensures that the feature space from the two encoders is consistent, which makes subsequent feature space mapping possible. Following the CLIP pairing learning module, the historical and future trajectory embeddings are constrained to share the same feature space. Future trajectory features can be derived by applying a linear transformation to the historical trajectory features. Because there is inherent uncertainty surrounding the intentions of agents, the model's projected future trajectory should be represented as a probability distribution of the agents' positions. Therefore, it is necessary to estimate a manifold for the distribution of future trajectory features.We have noticed that there is an idempotent phenomenon in this consistent feature space. Specifically, the training instances should have fixed points in the feature space as their future trajectories. Our goal is to create an estimated manifold within which the future historical trajectory will be located. This statement aligns with the fact that the distribution of predicted future trajectories, sampled in real space, includes the original trajectory. Therefore, the loss function used to train the feature transformation network should encompass both the reconstruction loss and the idempotent loss, as shown in the equation.EQUATIONWe additionally incorporate Gaussian noise sampling into the input to model the agents' intentions. Additionally, since the feature space is subject to consistency constraints, trajectories from multiple source domains correspond to different manifolds within this space. If, on the other hand, the model runs into complex distributions, the idempotent loss can cause representation collapse, also known as overfitting [23] . To mitigate this, a tightness loss is also needed to constrain the manifold expansion.EQUATION) where θ denotes the MLP parameter, and θ ′ denotes freezing the MLP parameter at training. The detailed reason and approach for adopting this training strategy can be found in the [23] .We employ an auto-regressive decoder with the standard Transformer architecture. No additional training tasks are required at this stage. During the trajectory encoding phase, both the historical trajectory feature encoder and the future trajectory feature encoder aim to output position coordinates for the period from t -1 to t + T f . Two additional frames of historical trajectory are included to better model spatiotemporal continuity. The former is similar to a prediction task, while the latter resembles a reconstruction task. The future trajectory feature decoder, trained during the encoder phase, is directly used for trajectory decoding at this stage.EQUATION4 Experiments 
datasets and metrics: 
 Datasets. We evaluate our method using three popular trajectory prediction datasets: These include the ETH-UCY dataset, with its five scenes named ETH, HOTEL, UNIV, ZARA1, and ZARA2, which is considered the standard for predicting the paths of pedestrians. Additionally, we used the SDD dataset, the first large-scale dataset that collects agent trajectories in eight different outdoor scenes, and the SNU dataset, which includes different patterns such as walking straight, strolling, and staggering in indoor scenes.Metrics. We use a standard evaluation approach for pedestrian trajectory prediction. We observe 8 frames of trajectories to predict 12 frames of future trajectories. Each time, we sample 20 predicted trajectories and compare the prediction results closest to the ground truth [31] . Our evaluation metrics of choice are Average Displacement Error (ADE) and Final Displacement Error (FDE).EQUATIONFor training process, our batch size is set to 64, epochs to 100, and the learning rate is 0.01, which is half-formed every 25 epochs. We use the Adam optimizer. Our model is trained on an RTX 3090, with the encoder and contrastive learning pre-training requiring approximately 8 GPU hours, and the full framework training taking about 9.6 GPU hours. Our method is divided into three stages: offline training, inference, and online adaptation.Offline Training. First, we apply data augmentation to process trajectory data. Next, we pre-train historical and future trajectory encoders. Once these two encoders have reached convergence in their training, we use contrastive learning to impose constraints on their feature spaces for spatio-temporal constraints. During contrastive learning, we fine-tune the future trajectory decoders using pre-training tasks. It's important to note that during this step, the backpropagation gradient from the pre-train task is truncated only in the decoder section to adapt the future trajectory decoder to changes in the future trajectory encoder and maintain its decoding capabilities. Finally, we train the trajectory manifold predictor. The gradient for this training task is also truncated, meaning the parameters of the encoder are no longer updated.Inference. During the inference stage, we do not engage in comparative learning or future trajectory encoding, unlike the offline training stage. Instead, we conduct interpolation and alignment on the input historical pedestrian trajectory that needs to be predicted. We then utilize the historical trajectory encoder and trajectory manifold predictor to extract future trajectory features. Finally, we use the trajectory manifold decoder to generate the predicted trajectory.Online Adaption. During the online adaptation stage, we use a lightweight version of our model (TrajCLIP-tiny) for real-time inference. We freeze the trajectory encoder and manifold decoder and only fine-tune the parameters of the updated trajectory manifold predictor with a small learning rate based on newly observed trajectories. By adjusting the mapping between historical trajectory and future trajectory, the manifold predictor can map the same historical trajectories into future trajectories adapted to the current environment.We conduct comparative experiments with state-of-the-art methods on four common trajectory prediction tasks: standard trajectory prediction, scene-to-scene transfer, few-shot transfer, and online adaptation. These experiments provide a comprehensive evaluation of our method's prediction accuracy, generalization ability across different scenarios, learning capability with few samples, and online learning ability in dynamic environments. Additionally, we perform ablation studies on our proposed data augmentation method and various modules to verify the effectiveness of our approach.Standard Trajectory Prediction Task. Follows the standard trajectory prediction setting, we compare our model, TrajCLIP with six recent models with the best performance (1) Traj++ [21] ,(2) AgentFormer [31] , (3) MID [6] , (4) GP-Graph [2] , (5) EqMotion [27] (6) TUTR [22] and ( 7) SingularTrajectory [3] on the three datasets ETH-UCY, SDD, and SNU in Table . 1. From the table, our TrajCLIP model outperforms all other models, achieving the smallest ADE/FDE value and reducing the prediction error by up to 17%. Additionally, we develop TrajCLIP-tiny, which eliminates trajectory interpolation and reduces the attention matrix in the model, resulting in a 76.92% reduction in parameters. This allows TrajCLIP-tiny to adapt to various trade-offs between computational cost and prediction performance in real-world application scenarios. Moreover, TrajCLIP-tiny demonstrates competitive or superior performance in most of the dataset evaluation scenes.Scene-to-Scene Transfer Task. In this experiment, our goal is to assess the model's ability to generalize across different scenes by training and evaluating it on different scenes within the ETH-UCY dataset. Following the approach in [29] , we train the model on one scene and test it on another.To ensure a fair comparison with existing baselines in this new setup, we train the baselines using the training data from the source scene and the validation data from the target scene. For example, taking column A2B from right part of Table 2 , we train the baselines using the training data from dataset A and the validation data from dataset B, and then evaluate the model's performance using ADE/FDE on the test data from dataset B. It's important to note that the test and evaluation sets are independent, so the model only has access to trajectories from the validation set during evaluation.We compare them with the existing trajectory prediction methods: the transfer learning work T-GNN [29] based on domain adaptation, the meta-learning work K0 [10] using ALPaCA [8] adaptation, and tra2tra [28] which performs better in transfer learning. The experiment result is shown in right part of Table 2 . We discover that TrajCLIP-tiny exhibits competitive performance in the majority of scenes thanks to strong generalization capability from manifold predictor. In certain situations, it even outperforms trajectory prediction methods specifically designed for transfer learning. Moreover, TrajCLIP has achieved optimal or sub-optimal accuracy in the five scene-to-scene transfer tasks. Few-shot Transfer Task. In this experiment, different from the scene-to-scene transfer, we train the model with the entire ETH-UCY dataset and four trajectories in the object scene in SNU dataset and evaluate with the other trajectories from the object scene. This experimental setting is called 4-shot transfer learning. We compare our TrajCLIP and TrajCLIP-tiny with several models: MemoNet [26] , performs trajectory prediction with a memory-driven mechanism; Social-STGCNN [16] , which claims data efficiency and performs well with 20% data in a dataset. T-GNN [29] and K0 [10] are models designed especially for the transfer learning task. The experiment results on ADE/FDE are shown in right part of Table 3 . Only K0 outperforms our TrajCLIP-tiny 2.5%, but TrajCLIP-tiny performance remains competitive with other methods. With the ability to model more complex distributions, the TrajCLIP method outperforms all other methods on each SNU scene 8.7% to 40.6%. Ablation Studies. In left part of Table 4 , we present the performance contributions of our proposed key modules: the SAIF, the CLIP pairing learning, and the manifold predictor (IGN generation module). The results show that combining the CLIP pairing method and the IGN module significantly enhances prediction performance, surpassing all other generation methods. This is because the CVAE framework involves an encoder obtaining a feature and then randomly sampling to get the mean and variance, which are used as inputs for the decoder. The diffusion method starts with a random initialization of an input and then gradually denoises. This randomness makes alignment work less effective (that is, even if aligned, the presence of random sampling still makes the generated results uncontrollable). The comparative experiments (c)(d) and (e)(f) illustrate that the two random methods of CVAE and diffusion methods make the improvement effect of alignment work not significant. On the ETH_UCY dataset, the addition of CLIP only improved ADE/FDE by 0.01/0.03 and 0.01/0.01. However, the network structure of IGN is an MLP, which is an affine transformation and does not have this issue; hence, as in our method, alignment is significantly effective on IGN. When CLIP is removed, as shown in experiments (b)(c)(e), since the two feature spaces cannot be connected through IGN without alignment, the performance drops by 0.06/0.17 compared to when CLIP is added.We also conduct experiments to test different data augmentation methods and find that our method's improved performance is due to TrajCLIP's strong modeling ability to represent complex trajectory distributions, which shown in right part of Table 4 . The results of applying the data augmentation methods are shown in the rightmost column of Table . When we train the latest state-of-the-art methods EqMotion and TUTR with our proposed data augmentation methods, we observe minimal changes in their performance, whereas our model's performance improved significantly. Additionally, experiments on each of the three core data augmentation methods demonstrate their effectiveness. Among these methods, Adjacent Point Interpolation (API) is proved to be the most effective in enhancing the model's prediction accuracy. Infer Speed and Computation Consumption. As the table 5 illustrates, we have contrasted our approach with alternative techniques in terms of model size, computational complexity, and inference speed. Our medium-sized model meets the requirements for a real-time prediction task in terms of both inference speed and model size, as it can predict trajectories in 0.0615 seconds. Additionally, our lightweight model is only 3.45MB in size, and its computational complexity is relatively low compared to its model size, making it deployable on most hardware platforms.Qualitative Results. The left figure of Table 3 shows the visualization results of trajectory prediction for complex history trajectory using TrajCLIP and other state-of-the-art methods EqMotion and MID on UNIV scene of ETH-UCY dataset. Two trajectories in the figure include interference of surrounding pedestrians and scene environment and several turnings. We see that our TrajCLIP could predict these unusual turns, and the other two methods only predict a common smooth trajectory pattern, which shows a better ability to model complex trajectories. Given the observed historical trajectories (yellow), our method predicts future trajectories (green). In complex interaction scenarios, our method occasionally exhibits failures such as deviating from the intended endpoint and colliding (red) with other agents.Limitation Analysis. as illustrated in Figure 5 , our method falls short in ensuring collision avoidance in complex, high-density environments, necessitating further research. 
conclusion: 
 In our work, we introduce TrajCLIP, a method for predicting pedestrian trajectories. We use contrastive learning and an idempotent generative network to model a complex trajectory feature space. TrajCLIP offers five key contributions: the ability to model intricate trajectory distributions, CLIP pairing learning to combine the feature spaces of historical and future trajectories, a trajectory feature encoder based on the fusion of time domain and frequency domain, an idempotent generative network for mapping the same feature space, and an adjacent point interpolation algorithm to enhance the model's expression ability. We evaluate TrajCLIP on four prediction tasks, including standard trajectory prediction, scene-to-scene transfer, few-shot transfer, and online adaption, using three datasets. Our method achieves state-of-the-art prediction performance. 
acknowledgement: 
 This work was supported by the National Natural Science Foundation of China under Grant 62002345. 
 
 Appendix:  


Decision: The paper presents TrajCLIP, a novel approach to pedestrian trajectory prediction using contrastive learning and idempotent networks. The method shows promise in addressing the challenges of modeling complex trajectory distributions and demonstrates strong performance across various datasets and tasks.

Initially, the paper received mixed concerns especially in terms of (1) limited discussion of limitations, (2) computational complexity, and (3) implementation details. After the rebuttal stage, most of the reviewers' concerns have been addressed and Reviewer iNsW raised his score to weak accept. 

Given the novelty of the approach, strong empirical results, and the authors' thorough response to reviewers' concerns, this paper appears to make a significant contribution to the field of pedestrian trajectory prediction. The AC would like to recommend the paper for acceptance. Congratulations! Please be aware that the authors should integrate the response in the rebuttal and adequately address the reviewers' concerns in the final camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper presents TrajCLIP, a novel method for pedestrian trajectory prediction that utilizes contrastive learning and idempotent neural networks. The authors propose an interesting approach to address some limitations of existing methods, particularly in terms of generalization and modeling complex trajectory distributions.
strength_weakness: strength_weakness: 1 The paper lacks a discussion on the computational complexity and runtime performance of the proposed method compared to existing approaches.

2 More analysis of the limitations of the proposed method and potential failure cases would strengthen the paper.
questions: questions: 1 Provide more detailed explanations of the idempotent neural network implementation and training process.

2 Add a section discussing the computational requirements and runtime performance of TrajCLIP.

3 Include a discussion on the limitations of the proposed method and potential scenarios where it might not perform well.
limitations: limitations: The limitations discussed in the Experiments seem too easy and do not fully reflect the limitations of the work.
paper_summary: paper_summary: This paper utilizes idempotent generative network to perform multiple tasks in pedestrian trajectory prediction and achieves state-of-the-art performance in those tasks, showing its great representation and generalization ability. 

The proposed model has the following main components:
1. Spatio-Temporal Interaction Feature (STIF) and Scene-Agent Interaction Feature (SAIF) based encoder
2. CLIP for past and future feature alignment
3. IGN for mapping from past to future in aligned feature space.

The authors also provide a theoretical analysis of IGN, CLIP and data augmentation, but some issues need to be addressed.
strength_weakness: strength_weakness: 1. This is not the first work to use constructive learning in pedestrian trajectory prediction, since it was used in long-tailed pedestrian trajectory prediction for forming better latent representations [1, 2]. The author should cite them and clarify the difference. 
2. Missing citation and comparison for SingularTrajectory[3], which is also a universal model for pedestrian trajectory prediction. It defines a singular space that can well represent trajectory features for multiple tasks without alignment between past and future.
3. Some questions need to be addresses. See questions.



[1] Wang, Yuning, Pu Zhang, Lei Bai, and Jianru Xue. "Fend: A future enhanced distribution-aware contrastive learning framework for long-tail trajectory prediction." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1400-1409. 2023.

[2] Makansi, Osama, Özgün Cicek, Yassine Marrakchi, and Thomas Brox. "On exposing the challenging long tail in future prediction of traffic actors." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13147-13157. 2021.

[3] Bae, Inhwan, Young-Jae Park, and Hae-Gon Jeon. "SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17890-17901. 2024.
questions: questions: 1. Alignment is quite useful for IGN but not as much useful as IGN when applying to CVAE and diffusion according to ablative experiments. Why was that? This issue needs more analysis and clarification. 
From the ablative experiments, when CLIP is not used, IGN is worse than CVAE and diffusion (from b, d, f in Tab.4). Meanwhile, CLIP cannot provide pronounced improvement as IGN when applied on CVAE and diffusion. The author just says combining them achieves the best results but motivation of combining IGN and CLIP is not very clear. Is it due to some intrinsic characters of IGN that requires feature alignment while other models do not drop as much as IGN when removing it? More detailed analyses of the phenomena is needed.

2. Is the affine transformation mentioned in Fig.1 same as the trainable matrix W in CLIP?
limitations: limitations: Not seen in main content.
paper_summary: paper_summary: This paper proposes to utilize contrastive learning for pedestrian trajectory prediction. A STIF encoder is used to extract spatial-temporal features and is trained with data augmentation. A SAIF utilizes the Fast Fourier Transform to extract the interaction information among the agents. The authors incorporate the idempotent loss for training. Experiments on three datasets show the efficacy of the proposed method.
strength_weakness: strength_weakness: 1. I have some doubts about the intuition of using CLIP between history and future trajectories. See questions below.
2. There is no speed or computational cost comparison. Since the method involves multiple stages, a comparison of training/inference costs is needed. 
3. The proposed method does not consider visual scene information.
4. There is no error analysis. When does the method fail and why?

Minor comments:
Section 2.2 Generalization Framework -> Generative Framework. They are different things.
The text in Figure 2 is too small to read.
Line 263 “We conducted comparative ”, Line 270 “we compare our model”. Better to use a consistent present tense when describing your method.
questions: questions: 1. I have some doubts about the intuition of using CLIP learning between the historical feature and the future trajectory feature. The authors propose to align historical feature space with future feature space (Line 186). Does this lead to identical trajectory prediction, i.e., copying the historical trajectory as the future trajectory? Different from aligning images and text that have the same semantic meaning, the historical trajectory and future trajectory of the same agent may be different due to the difference in the spatial location (hence the surroundings are different).
2. Since the method involves multiple stages, how does the model compare to the baselines in terms of inference speed and training cost?
3. There is no error analysis. When does the method fail and why?
limitations: limitations: The authors did not discuss the limitations of the method and failure cases.
paper_summary: paper_summary: The paper proposes a trajectory prediction approach for multi-agent configurations. The historical and future trajectories are encoded in terms of spatio-temporal interaction features (STIF) using Agentformer [28] and scene-agent interaction features (SAIF) using a transformer based on Fourier transform. The historical and future trajectory encoders share the same architecture but have different parameters. The encoded features are then summed and passed to a generative model [21] to represent  the trajectory space. Finally, a transformer predicts the next position. The encoded features for historical and future trajectories (STIF and SAIF) are trained using CLIP [16] to force the historical and future encoded features to share the same space and help achieve temporal continuity for the trajectory prediction task. The approach is evaluated on the ETH-UCY, SDD and SNU datasets and shows promising results compared to related methods.
strength_weakness: strength_weakness: - The paper makes various solid and small contributions, but it lacks a novel idea. This is a limitation, but it's not a major point.
- The related work would benefit from a more direct comparison of the proposed method with the related works. 
- The motivation for relying on an idempotent generative network over other generative models, e.g. GANs, is not well explained.
- According to the literature, the article "Human Trajectory Prediction via Neural Social Physics" by Yue et. al. (2022) achieves average ADE and FDE of 0.17 and 0.24 respectively on ETH, while 6.52 ADE and 10.61 FDE are achieved on SDD. The comparisons should include all state-of-the-art methods.
questions: questions: - For the Online Adaptation task, prediction times are another crucial factor to consider alongside performance metrics. Is there any reason not to report them?
- Could the authors provide details of the hyper-parameters used during training? Implementation details are missing
- There are significant discrepancies between the results you report in Table 1 for Traj++ and the results reported in other publications, such as Traj++ in "Human Trajectory Prediction via Neural Social Physics". Could the authors provide more details on the experimental setup?
limitations: limitations: The paper does not dedicate space to discuss the limitations. This is a clear missing point.

Comments: comment: The author's response has resolved some of my questions, but I still have questions about the Computational Complexity and Inference Speed of the paper. Compared to the previous state-of-the-art papers, this paper has significantly increased the number of parameters, but the performance improvement is limited. So I will maintain my "Borderline accept" score for this paper.
Thank you for the author's response. The reply addressed most of my concerns, and I am inclined to maintain my original score.
The rebuttal does a very good job of addressing my concerns and most of the other reviewers' points. For this reason, I am moving to a positive rating (updated scores).