TrajCLIP: Pedestrian trajectory prediction method using contrastive learning and idempotent networks
Authors: Pengfei Yao
Yinglong Zhu
Huikun Bi
Tianlu Mao
Zhaoqi Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: trajectory prediction
constrative learning
Primary area: generative_models

Abstract: The distribution of pedestrian trajectories is highly complex and influenced by the scene, nearby pedestrians, and subjective intentions. This complexity presents challenges for modeling and generalizing trajectory prediction. Previous methods modeled the feature space of future trajectories based on the high-dimensional feature space of historical trajectories, but this approach is suboptimal because it overlooks the similarity between historical and future trajectories. Our proposed method, TrajCLIP, utilizes contrastive learning and idempotent generative networks to address this issue. By pairing historical and future trajectories and applying contrastive learning on the encoded feature space, we enforce same-space consistency constraints. To manage complex distributions, we use idempotent loss and tightness loss to control over-expansion in the latent space. Additionally, we have developed a trajectory interpolation algorithm and synthetic trajectory data to enhance model capacity and improve generalization. Experimental results on public datasets demonstrate that TrajCLIP achieves state-of-the-art performance and excels in scene-to-scene transfer, few-shot transfer, and online learning tasks.

TrajCLIP: Pedestrian Trajectory Prediction Method Using Contrastive Learning and Idempotent Networks
Pengfei Yao!
Yinglong Zhu 2. 2.yinggoongebupt. edu. en
yaopengfei22emaill ucas. cn
Huikun Bi
Tianlu Ma A++
Zhaoqi Wang!
Shhukkun@senseatto com
Stm@ict.ac.
zqwangeict. ac en
Institute of Computing Technology, University of Chinese Academy of Sciences *Beijing University of Posts and Telecommunications "SenseTime
Beijing Key Laboratory of Mobile Computing and Pervasive Device
Abstract
The distribution of pedestrian trajectories is highly complex and influenced by the scene, nearby pedestrians, and subjective intentions. This complexity presents challenges for modeling and generalizing trajectory prediction. Previous methods modeled the feature space of future trajectories based on the -ich-iimensional feature space of historical trajectories, but this approach is suboptimal because it overlooks the similarity between historical and future trajectories. Our proposed method, TrajCLIP, utilizes contrastive learning and idempotent generative networks address this issue. By pairing historical and future trajectories and applying contrastive learning on the encoded feature space, we enforce same-space con- sistency constraints. To manage complex distributions, we use idempotent loss and tightness loss to control over-expansion in the latent space. Additionally, we have developed trajectory interpolation algorithm and synthetic trajectory data to enhance model capacity and improve generalization. Experimental results on public datasets demonstrate that TrajCLIP achieves state-of-the-an performance and excels in scene-to-scene transfer, few-shot transfer, and online learning tasks.
Introduction
Pedestrian trajectory prediction involves predicting future paths based on observed historical tra- jectories. This task is crucial in real-world applications such as autonomous driving 4 [3]. and robotics (II]. It is challenging (c.g., roundabouts and interscctions), various patterns (c.g., staggering and strolling), and different interactions (e.g., walking together avoiding strangers) due to their complexity.
model diverse trajectory distributions across different scenes
The existing methods 21 assume that the training and test motions follow the same distribution to effectively model complex distributions within specific range. These approaches focus on modeling the high-dimensional feature space distribution of trajectories within a particular dataset.
They use generative models, such as Conditional Variational AutoEncoder 23 24 and diffusion, to change the feature space of past trajectories into the feature space of future trajectories. This process aids in generating predicted trajectories and fitting the distribution of pedestrian trajectories to specific datascts. However, as shown in Fig. I this approach has limitations. It involves reducing
denotes the corresponding author. contributed equally to this work co-first author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Bilterze Space Sampling
Escode:
5y A65a) Decoda:
Histoicaal Tajectoc Space
Fature Trijattry Fenture Space
SpntioTTenpeaal Discontinitt
by Sy
Whese will Gane?
Saace Space Aoffire Trevelmmrttee
Space Tereporal Coniinit
Figure 1: The existing methods use an encoder to sample from a normal distribution order to generate predicted trajectories. However, this approach leads to different feature spaces for historical and future trajectories, causing problems during sampling, such as sharp turs predictions of positions far exceeding normal speeds. Our method ensures consistency between the two feature spaces, enabling position prediction through affine transformations and preserving spatio-temporal continuity.
the feature space match the distribution of specific dataset to improve prediction accuracy. Nonetheless, this projection operation creates a gap in the feature space between the historical and future trajectories, limiting the model's ability to make predictions across different scenarios and to adapt to new scenarios.
We believe that predicting future trajectories in trajectory prediction tasks is significantly different from tasks such as classification [32 and tracking [30], where the input and output are inconsistent. In trajectory prediction, the feature space of the input and output remains consistent. Therefore, mapping historical trajectory features to future trajectory features in the same feature space can effectively bridge the feature space gap and model complex feature space distribution more accurately.
We have developed a method called TrajCLIP for predicting trajectories, which is based on contrastive learning and idempotent neural networks. This method was inspired by the approach used in the paper by [18], which involves limiting the feature space heterogeneous data. To achieve this, we created a trajectory feature encoder that utilizes frequency domain additivity and Fourier transforms reversibility to combine trajectory Interaction Features (STIF) in the time domain and Interaction Features (SAIF) between agents in the frequency domain. We utilized cross-entropy loss for constructive learning between historical trajectory and future trajectory encoders to ensure that the feature spaces of the two encoders are the same. Subsequently, a global feature mapping framework to map the same set of features from past trajectory features to future trajectory features. Additionally, to allow for more general usage. Finally, the mapped future trajectory features were decoded into predicted trajectories. We also introduced a neighbor-based interpolation algorithm and used trajectory generation model to create a synthetic trajectory to enhance the model's 's ability to describe complex distributions. Furthermore, we trained a lightweight version (TrajCLIP-tiny)) without these augmentations. We conducted experimental comparisons with existing trajectory prediction methods on various tasks and datasets. The experiments demonstrated that our method achieves ofte-offtheeart accuracy and outperforms existing methods in cross-scene generalization, small sample transfer, and online tasks.
employed an idempotent generation network (IGN) as
constrained the growth of the specific distribution
The contributions of our work are summarized as follows: (1) In this study, TrajCLIP is presented as a proposed method. The model effectively represents and generalizes complex feature distributions by integrating the feature spaces of historical and future trajectories. An idempotent neural network with global feature mapping capabilities is utilized, enabling strong cross-scene generalization abilities. (2) We proposed novel trajectory feature encoder combines features from both the time domain and the frequency domain using STIF and SAIF to model pedestrian trajectories.
(3) We have created a novel method for measuring trajectory similarity, which we use as a label for contrastive learning. This helps to narrow down the feature space of past and future trajectories. To our knowledge, this the first application of contrastive learning to trajectory prediction. This is the first work that considers the feature space alignment between the history and future trajectory. Our experiments demonstrate that TrajCLIP performs exceptionally well across different tasks and datasets.
2 Related Work
2.1 Pedestrian Trajectory Prediction
Pedestrian trajectory prediction is approached as a sequence-to-sequeee challenge. Social-L.STM first uses LSTM to predict trajectories deterministically. Social-GAN uses Generative Adversar-

---

ial Network (GAN) to make many possible future predictions for an imput trajectory while taking into account the inherent multimodality of human motion. Trajectron++ [2] takes in motion trajectories and heterogeneous data, and it handles multi-modality by using CVAE to sample latent variables in high-dimensional space. AgentFormer [31 proposes agent-aware attention to simultaneously capture the spatiotemporal features of trajectories and improves the sampling method of the CVAE framework to model more intricate trajectory modalities. MID [6] was the first to use diffusion, which gradually gets rid of uncertainty in all walkable areas by using high-dimensional denoiser until the desired path is reached. A new model for predicting trajectories called Flow-Chain 14) uses normalizing flow quickly figure out the probability distribution for each point on the trajectory. TUTR [22 employs a unified transformer to model trajectories, directly providing the predicted trajectory and the probability of each mode, which effectively eliminates the need for post post-processing. Long-tail analysis [5] uses Kalman filter to directly predict output trajectory errors as a basis, distinguishing difficult and easy samples. It then conducts contrastive learning on difficult (long-tail part) historical trajectories to enhance the feature representation capabilities of the long-tail part of the dataset. FEND 25 uses VAE to extract features from the entire trajectory and performs offline clustering on the features to obtain labels; during training, it uses offline labels for contrastive learning on historical trajectories to guide the feature generation of historical trajectories and enhancing the prediction of the long-tail part. However, projecting past trajectory features into separate feature spaces creates a feature space gap that restricts the model's capacity to generalize and accurately represent complex trajectory distributions.
2.2 Generative Framework
Compared with the deterministic framework, the generative framework focuses on modeling data distribution in the dataset and is well-suited for tasks involving one input and multiple outputs. GAN [5] introduces a generative adversarial network, allowing the generator to model the real data distribution through adversarial learning with the discriminator. VAE 112 introduces latent variables, uses the encoder to construct the latent feature space representation of the data, and uses the decoder to restore and reconstruct the feature tensor. CVAE (24] adds control signal on top of VAE [2]. This signal changes the variational lower bound in the conditional probability state, which lets the model control the generated data distribution based on the signal that goes into Diffusion [9] models the data distribution by simulating the diffusion process in high high-dimensional space. Adding dibenssinnii denoisers gradually lowers uncertainty about the data distribution, leading to samples that are similar to the training data.
In these methods, there is separation between the feature spaces of the control signal and recon- structed data. However, future and historical trajectory features should be in the same feature space, and modeling these independently will lead natural gap. IGN (23) aims to model global feature mapping, which constrains the distribution of specific datasets on the expected manifold through idempotent loss and compaction loss. This approach prevents the model from being overfitted the distribution of specific datasets and enables data generation within the same feature space. DALLE [9 uses CLIP [8] to align the different feature spaces of text and images. It then uses the encoded feature vector of the text as control signal to make different kinds of images using GLIDE [17]. We utilize contrastive learning to ensure consistency between the feature spaces of historical trajectory and future trajectory, based on the concept of aligning feature spaces. Subsequently, we employ IGN mapping to generate trajectories.
3 Methodology
3.1 Problem Formulation
We generalize the trajectory prediction task as follows. Given a scenario comprising N agents, the historical information for each agent includes the two-dimensional coordinate position pp = (x7 vi of agent at time t. The trajectory prediction task is to predict future position sequence 1% Pig+1~to777 Ty based on the observed historical agent position sequence Pio-TT+1o00"
3.2 Overall Framework
We propose a trajectory prediction method called TrajCLIP, which uses contrastive learning and and idempotent generative network, as illustrated Fig. 2 Our TrajCLIIP comprises four parts: (1). A trajectory encoder that models interactions between agents, scene-agent interactions in the
3

---

frequency domain, and trajectory spatiotemporal characteristics in the time domain. (2). A CLIP pairing learning module that uses cross- cross-entropy loss to maintain feature space consistency between past and future trajectories.
(3). A feature manifold predictor that employs idempotent and tightness loss train an IGN for mapping past trajectory features into future trajectory features in the same feature space. (4). A trajectory manifold decoder that uses pre-trained task decoder from the trajectory encoder to generate predicted trajectories. We will provide detailed introductions to each part in the subsequent sections. Specific implementation details can be found in Sec. 4.2
3.3 Data Augmentation
Various data augmentation techniques are used in current methods to enhance the performance of pedestrian trajectory prediction models, including rotation for robustness, scale alignment for training simplicity, and trajectory origin alignment for addressing absolute position ambiguity. In our research, we incorporate minor perturbations (as discussed in (20] into trajectories to generate more reliable results. We achieve this by simulating scenarios using available datasets, demonstrated in the study conducted by [33]. This approach allows us to enhance the existing dataset. Additionally, we have developed an interpolation technique that increases the model capacity by integrating neighboring trajectory sample points.
We formulate the adjacent point interpolation algorithm as illustrated in Fig. If the vectors Pt- 1Pt and Pt+1Pt--2 are on the same side of PrP:+1 for four consecutive positions Pt I--1- we can calculate the acute angle between the two adjacent trajectories. To derive the interpolated trajectory, the data is divided into three equal segments. Then, we find the intersection point of the third line and connect them. When the vectors are on opposite sides or at the beginning and end of the sequence, the trisect points are chosen as the interpolation points. In the rest this paper, we will use the symbol +i represent the interpolated trajectory. for simplicity. Despite the potential decrease in accuracy, the generated and interpolated data can still be used to improve the model's 's capabilities and capture more complex distributions, considering the multimodal nature of the pedestrian trajectory prediction task.
3.4 Trajectory Encoder
In our approach, we use two structurally identical encoders, as shown Fig. 3 These encoders represent the historical trajectory H and the future trajectory F. We utilize the agent-aware attention AAt from Agentformer [31 to extract the STIF, and employ a transformer based on Fourier Transform to extract the SAIF. By fusing these features, train both encoders decode future trajectories. Additionally, the historical trajectory encoder aims to accurately predict the subsequent frame position maintain spatio-temporal continuity.
We define the time-series set TH = (lo - Th + I,.,, lo 1, Lo} for the historical trajectory encoder, and TF {to 1, to, to .... to +Ty } for future trajectory encoders. To obtain the spatio-temporal
Trajectory Encoder
CLIP Pairing Learning
Trajectory Manifold Predictor L L + L Eigs
STIF
ZF
Historical Trajectory
Predicted Future
SAIF
Trajectory Mainfold
Trajecoory
|
Trajectory Manifold Decoder
Criss Eneeeyy Liss
STIF
Eyp
Zr
Decoder
Trjectory Feature
Auto-rogressive
SAIF
P. T, 1MA... Brr...
Future Trijectory
Predicted Trajectory
STIF
Cemtio-Tempaal Interacioon Feature
Decoder
TransSemee Dosoder
Concoloratic
Trin & Inferenee
SAIF
ScenesAgen Intesaction Fosture
Mulillyer Peroeption
Elessnttoisec Add
Train Only
Figure 2: Overall framework of our proposed TrajCLIP. Introduction to each part can be found in Section 3.2 and specific implementation details can be found in Sec. 4.2

---

Spatio-Temporaa Interaction Feature (STIF)
Scene-Agent Interaction Feature (SAIF)
X,.
X
X"
P
Buteerwrthh filter
X
I
IFFT
Z
Xr.
IFFT
position sequence velocity sequence
acceleration sequence
ngent sequence
position Embedding
+ element-wise
MLP
Agent-aware attention
self-attention
Figure 3: The left side shows the Architecture of Temppoall Interaction Feature, while the right side shows the Scene-Agent Interaction Feature.
interaction feature (STIF), we arrange the trajectories in a scene into X; = ((p) l T), where time is used the index. At each time step, use a straightforward linear mapping to tokenize the X matrix and include positional encoding to maintain the temporal information of the trajectory token sequence. We employ agent-aware attention to extract both the temporal features of each individual trajectory and the interaction features between different trajectories simultaneously. The feature is denoted as 24 and can be formally described follows:
f M AM (W M + F))
2t MLP(f(X)) f(V.) + f(A:))
where WT and W2 represent trainable matrices, AA denotes agent- attention, I' denotes sine and cosine position encoding [II, and MP denotes multi-layer perceptron. We can easily derive the velocity V, and acceleration A, of the sequence using X, Similarly, we employ equation to extract features for both of these sequences. Parameters are not shared among different sequences.
The Scene-Agent Interaction Feature (SAIF) characterizes the shared representation of the scene by the agents. We believe that the interactions between agents and the scene are implicit and shared. These interaction features should be represented uniformly, without including the individual characteristics of the agents. As there is no direct way to merge the agents' trajectories in the time domain, we leverage the additivity in the frequency domain to eliminate the fine- fine-grained features of the agents' trajectories, thereby emphasizing the interaction information of the scene.
We organize the trajectories in a scene into X = ((pī t TT)) index. We apply Fast Fourier Transform (FFT) to each agent's trajectory based on Equation 2 Utilizing conjugate symmetry, can derive the cosine expression for both amplitude and phase.
1,..., N) with agent as the
2kM Th
Pk
P
A COS
(2)
TET
SET
We employ a Butterworth filter to separate high-frequency information (7). AF. o enhance modeling efficiency. Then we superimpose and concatenate the low-frequency signals X and high-frequency signals x of all agents in the scene respectively, and employ self-attention to extract feature.
WW + D), (h,4) 2n MLP (X" Ple (X")))
(3)
where W3 and W, denote trainable matrices, y denotes multi-heas heat- self-attention, and IFFT denotes Inverse Fast Fourier Transform (IFFT). IFFT is lossless and reversible transformation, so the extracted features can inversely transformed into time-domain features without any information loss and fused with STIF in the same feature space. The final output of the trajectory encoder is the result of a linear transformation of the sum of 2 and Zq-
3.5 CLIP Pairing Learning Module
We aim to transform the historical trajectory features into estimated future trajectory features, displaying temporal continuity to aid in trajectory prediction. It is important to maintain consistency between the feature spaces of past and future trajectories to satisfy spatio-temporal constraints. Therefore, it necessary retrain the historical trajectory encoder to align its feature space with that the future trajectory encoder. We utilize CLIP [88] a method for aligning feature spaces of different types of information representation, such as text and images, using contrastive learning. We apply this concept to ensure consistency between the feature spaces in the past and future. The
5

---

specific algorithm is outlined in Algorithm 1 The similarity matrix within each training batch of trajectories is calculated using Equation 4 The mean included angle cosine of the velocity direction
vectors between trajectories i, E {1, 2. 2..., batcbbottiaaee size] is computed each corresponding sample
point in both the historical and future data.
To Σ
-
sim(t, Ty Th Io +, IIppPh 1Ilpppi
(4)
Simultaneously, we calculate features z and z/p- for each agent using the historical trajectory encoder and future trajectory encoder, and then combine the corresponding encoder features. Next, we align the dot product to obtain the similarity
the dimensions using trainable matrix W. Finally, scores and optimize them using the cross-entropy loss with the similarity matrix. Contrastive learning ensures that the feature space from the encoders is consistent, which makes subsequent feature space mapping possible.
Algorithm 1 CLIP Pairing Learning
Input: agent index in batch: i=l, batch size
groued poir inserplbitie
1: Calculate similarity matrix S using equation.
2: z ML + zig] for batch.
proand butk trg
3: 2 t- MLP + zirl for in batch.
4: ZH <- concate([2]y for in batch])
5: Ze concate([2 for i in batch))
6: Iæ + normalize(W iW azis 1)
7: IF f normalize( wWrZ azrs = 1)
8: LH + cross_entropp_loass(S 1 I2, avis 0)
9: LF <- cross croas_entropy_l400 lossss, I awis = 1)
Figure 4: Schematic diagram
10: backpropagation with Leti +- (CH L2)
of Adjacent Point
Interpolation.
3.6
Trajectory Manifold Predictor
Following the CLIP pairing learning module, the historical and future trajectory embeddings constrained to share the same feature space. Future trajectory features can be derived by applying linear transformation to the historical trajectory features. Because there is inherent uncertainty surrounding the intentions of agents, the model's 's projected future trajectory should be represented as a probability distribution of the agents' positions. Therefore, it is necessary estimate a manifold for the distribution of future trajectory features.
We have noticed that there is an idempotent phenomenon in this consistent feature space. Specifically, the training instances should have fixed points in the feature space as their future trajectories. Our goal is to create an estimated manifold within which the future historical trajectory will be located. This statement aligns with the fact that the distribution of predicted future trajectories, sampled in real space, includes the original trajectory. Therefore, the loss function used to train the feature transformation network should encompass both the reconstruction loss and the idempotent loss, as shown in the equation.
ZP MLLPZZ;; c), e~ 1),
Lree 22 -z 1 ll2
(5)
Lidem (MMP&
MLPo
We additionally incorporate Gaussian noise sampling into the input to model the agents' intentions. Additionally, since the feature space is subject to consistency constraints, trajectories from multiple source domains correspond to different manifolds within this space. If, on the other hand, the model runs into complex distributions, the idempotent loss can cause representation collapse, also known as overfitting [23 To mitigate this, a tightness loss is also needed to constrain the manifold expansion.
Ltight = II L.MMPPMMPPP [Z]]] MLP [ZF]lz
(6)
where 0 denotes the MLP parameter, and 8' denotes freezing the MLP parameter at training. The detailed reason and approach for adopting this training strategy can be found in the 23
3.7 Trajectory Manifold Decoder
We employ an auto-regressive decoder with the standard Transformer architecture. No additional training tasks are required at this stage. During the trajectory encoding phase, both the historical

---

trajectory feature encoder and the future trajectory feature encoder aim to output position coordinates for the period from f to t T Two additional frames of historical trajectory are included to better model spatiotemporal continuity. The former is similar to a prediction task, while the latter resembles a reconstruction task. The future trajectory feature decoder, trained during the encoder phase, is directly used for trajectory decoding at this stage.
MLP(za + 21) Pi 1I-1T Decoder(2)
L+Ty
L Nx x (Ty +
1
Σ
Σ
1 - -piila
Experiments
4.1 Datasets And Metrics
Datasets. We evaluate our method using three popular trajectory prediction datasets: These include the ETH-UCY dataset, with its five scenes named ETH, HOTEL, UNIV, ZARA1, and ZARA2, which is considered the standard for predicting the paths of pedestrians. Additionally, we used the SDD dataset, the first laage-scale dataset that collects agent trajectories in eight different outdoor scenes, and the SNU dataset, which includes different patterns such as walking straight, strolling, and staggering in indoor scenes.
Metrics. We use a standard evaluation approach for pedestrian trajectory prediction. We observe 8 frames of trajectories to predict 12 frames of future trajectories. Each time, we sample 20 predicted trajectories and compare the prediction results closest to the ground truth [I]. Our evaluation metrics of choice are Average Displacement Error (ADE) and Final Displacement Error (FDE).
4+T
ADE
1
Σ
Σ
1
FDE
IiitT, - Pi+T, ll2
(8)
N
1=t1
4.2 Implementation Detail
For training process, our batch size is set 64, epochs to 100, and the learning rate 0.01, which half-formed every 25 epochs. We use the Adam optimizer. Our model is trained on an RTX 3090, with the encoder and contrastive learning pre-training requiring approximately GPU hours, and the full framework training taking about 9.6 GPU hours. Our method is divided into three stages: offline training, inference, and online adaptation.
Offline Training. First, we apply data augmentation to process trajectory data. Next, we pre-train historical and future trajectory encoders. Once these two encoders have reached convergence their training, we use contrastive learning to impose constraints their feature spaces for constraints. During contrastive learning, we fine-tune the future trajectory decoders using pre-training tasks. It's important to note that during this step, the backpropagation gradient from the pre-train task truncated only in the decoder section to adapt the future trajectory decoder to changes in the future trajectory encoder and maintain its decoding capabilities. Finally, we train the trajectory manifold predictor. The gradient for this training task is also truncated, meaning the parameters of the encoder are no longer updated.
During the inference stage, we do not engage in comparative learning or future trajectory encoding, unlike the offline training stage. Instead, we conduct interpolation and alignment on the input historical pedestrian trajectory that needs be predicted. We then utilize the historical trajectory encoder and trajectory manifold predictor to extract future trajectory features. Finally, we use the trajectory manifold decoder to generate the predicted trajectory.
Inference.
Online Adaption. During the online adaptation stage, we use lightweight version of our model (TrajCLIP-tiny) for real-time inference. We freeze the trajectory encoder and manifold decoder and only fine-tune the parameters of the updated trajectory manifold predictor with a small learning rate based on newly observed trajectories. By adjusting the mapping between historical trajectory and future trajectory, the manifold predictor can map the same historical trajectories into future trajectories adapted the current environment.
4.3 Experiment and Analysis
We conduct comparative experiments with state-of-the-art methods on four common trajectory prediction tasks: standard trajectory prediction, scene-to-scent transfer, few-shot transfer, and online adaptation. These experiments provide a comprehensive evaluation of our method's prediction
7

---

ETH
HOTEL
UNIV
ZARAI
ZARA2
AVG
SDD
SNU
Trajectron++
0.61/1.02
0.19/0.28
0.30/0.54
0,24/0.42
0.18/0.31
0.30/0.51
10.65/16,74
0.35/0.50
AgentFormer
0.45/0.75
0.14/0.22
0.25/0.45
0.18/0.30
0.14/0.24
0.23/0.39
8.24/15.21
0.32/0.46
MID
0.39/0.66
0.13/0.22
0.22/0.45
0.17/0.30
0.1340.27
0.21/0.38
7.61/14.32
0.30/0.45
GP Graph
0.43/0663
0.18/0.30
0.24/0.42
0.17/0.31
0.15/0.29
0.23/0.39
9.10/13.76
0.31/0.47
EgMotion
0,40/066
0.12/0.18
0.23/0.43
0.18/0.32
0.13/0.23
0.21/0.35
7.52/12.88
0.28/0.42
TUTR
0.400061
0.11/0.18
0.23/0.42
0.18/0.34
0.13/0.25
0.21/0.36
7.76/12.69
0.29/0.43
SingularTraecctory
0,35/0,42
0.13/0.19
0.25/0.44
0.19/0.32
0.15/0.25
0.21/0.32
7.26/12.58
0.29/0.41
Ours-tiny
0.39/0.62
0.13/0.19
0.22/0.42
0.19/0.32
0.13/0.24
0.21/0.36
7.04/13.40
0.29/0.42
Ouss(Iaj(LLP))
0.36/0.57
0.10/0.17
0.19/0.41
0.16/0.28
0.11/0.20
0.18/0.33
6.29/11.79
0.25/0.39
Table Prediction performance on the ETH-UCY, SDD, SNU dataset using ADE/FDE. We sample 20 possible trajectories and calculate metries based on the trajectory that is closest to the ground truth. The bold/underlined font denotes the best/second-best result.
accuracy, generalization ability across different scenarios, learning capability with few samples, and online learning ability in dynamic environments. Additionally, we perform ablation studies on our proposed data augmentation method and various modules to verify the effectiveness of our approach.
Standard Trajectory Prediction Task. Follows the standard trajectory prediction setting, we compare our model, TrajCLIP with six recent models with the best performance (1) Traj++ (2]]. (2) AgentFormer BT, (3) MID [6], (4) GP-Graph 12), (5) EqMotion (27 (6) TUTR [22] and (7) SingularTrajectory G on the three datasets ETH-UCY, SDD, and SNU in Table. T From the table, our TrajCLIP model outperforms all other models, achieving the smallest ADE/FDE value and reducing the prediction error by up to 17% Additionally, we develop TrajCLIP-tiny which eliminates trajectory interpolation and reduces the attention matrix in the model, resulting in 76.92% reduction in parameters. This allows TrajCLIP-tiny to adapt to various trade-offs between computational cost and prediction performance in real-world application scenarios. Moreover, TrajCL.P- demonstrates competitive or superior performance in most of the dataset evaluation scenes.
Exene-to-Seent Transfer Task. In this experiment, our goal is to assess the model's ability to generalize across different scenes by training and evaluating on different scenes within the ETH- UCY dataset. Following the approach in (29), we train the model on one scene and test it on another. To ensure a fair comparison with existing baselines in this new setup, we train the baselines using the training data from the source scene and the validation data from the target scene. For example, taking column A2B from right part of Table 2 we train the baselines using the training data from dataset and the validation data from dataset B, and then evaluate the model's performance using ADE/FDE on the test data from dataset B. It's important to note that the test and evaluation sets are independent, so the model only has access to trajectories from the validation set during evaluation.
We compare them with the existing trajectory prediction methods: the transfer learning work T-GNN (2 based on domain adaptation, the meta-learning work KO (10) using ALPaCA [8] adaptation, and tra2tra (2 which performs better in transfer learning. The experiment result is shown in right
part of Table 2 We discover that TrajCLIP- -iny exhibits competitive performance in the majority of scenes thanks to strong generalization capability from manifold predictor. In certain situations, even outperforms trajectory prediction methods specifically designed for transfer learning. Moreover, TrajCLIP has achieved optimal or sub-optimal accuracy in the five scene-to-scene transfer tasks.
Trjjctrone SoEEAAATGONN
B2C
1.36
0.52
1.03
Agentter mer
T-GNN
1.001
K3
KD
0.87
1.08
60-
KS Aliiistuue
Ours
Oun-tiny
Qus(TTgjCLIP)
0.55
0.79
base
40-
B2C
D2B
20
T-GNN
tmftra
2:18
3.29
1,80
2.50
1.78
1,30
0.65
0.92
2.18
1,86
2:15
1.56
Ka
188
1.67
0.92
1.01
2.14
1.51
Ours
1.66
0.96
0.96
1.99
1.70
1,61
0.91
0.23
1.36
60
Table 2: Left is [ETH-UCY >>
DeathCircle 0 in SDD] Comparison of ADE reduction of each method for updating models based on online observed data. Right is prediction performance in scene-to-scene transfer experiment setting on UTH-UCY using ADE and FDE, A, B. C, D. denote ETH, Hotel, Univ, Zaral, Zara2. A2B experiment denotes a training model with a training set from A as well as the validation set from B, and reports test performance on test set from B.
8

---

Few-shot Transfer Task. In this experiment, different from the scene-to-scene transfer, we train the model with the entire ETH-UCY dataset and four trajectories in the object scene in SNU dataset and evaluate with the other trajectories from the object scene. This experimental setting is called 4-shot transfer learning. We compare our TrajCLIP and TrajCLIP-tiny with several models: MemoNet (26), performs trajectory prediction with a memory-driven mechanism; claims data efficiency and performs well with 20% data in a dataset. T-GNN (29] and KO [00 are models designed especially for the transfer learning task. The experiment results ADE/FDE are shown in right part of Table 3 Only KO outperforms our TrajCLIP-tiny 2.5%, but TrajCLIP-tiny performance remains competitive with other methods. With the ability to model more complex distributions, the TrajCLIP method outperforms all other methods on each SNU scene 8.7% to 40.6%.
[16], which
Tract
Oneway
Stagger
Stroll
AVG
MemoNet
0.31/0.62
0.34/0.65
0,32X0.59
0.32/0.62
Social STGCNN
0,34/0.69
0.37/0.72
0.40M0.79
0.37/0.73
T-GNN
0.28/0.51
0.32/0.58
0.34/0.57
0.31/0.55
KD
0,27/0.48
0.33/0.61
0.31/9.53
0.30/0.54
Ours-tiay
0.29/0.53
0.33/0.59
0,32/0.54
0.31/0,55
Ours(TrajCLIP)
0,26/0.47
0.31/0.56
0,28/0.44
0.28/0.49
Table 3: Left is qualitative results of prediction results on UNIV scene (top) of ETH-UCY dataset, little 3
scene (left) and coupa scene (right) of SDD dataset. Right ADE/FDE comparison of 4-shot transfer learning performance on each SNU scene.
Online Adaption Task. We evaluate the model performance of the online adaptation task. All methods trained on the ETH-UCY dataset initially, with batch size of to simulate an online situation where data are observed sequentially. Then, all models are online adapted and tested. Our model adopts the online adaptation in Sec. [4.2 and compares it with the meta- meta-learning- gearning-based based KO with/without additional online fine-tuning, as well as other methods that have better performance in online tasks. As shown in the left figure of Table 2 we take DeathCircle_ O in SDD as an example to give our comparison results on the online adaptation task. We select DeathCirele_ in SDD as the target scene and compare the online adaptation effects of each model in the left figure of Table 2 The term "base" in the figure denotes the results of offline training for each model on DeathCircle O. Using this prediction error as benchmark, we calculate the ADE reduction rate of each model during the online adaptation process. Our model performance decreases the least at first and is improving fastest as the training step increases. After 100 steps, our model performs 20% better than the base model, which proves it could adapt to predict trajectories in new scenes quickly.
new scenario
SAIF
CLIP
Generatoz framework
BTH UCY
SDD
TyMotin
9
X
API
PTT PIAACY
02MB:3
(4
(2)
X
x
x
CVAR
IGN
IGN
0.248477
0.240.50
0.200.36
7.55/14.66
791155.3
6,34/12.8
TUTR
/
x x
/
7397227
7411241
(8)
x
Diffusion
CVAE
0.231.4
0.220.39
7.39133.5
7.8614.62
TryCUI
/ x
x
x
/
x
0.106.34
0.190:35
65111233
6.8812.95
6.901305
16
Diffusioo
0.1/0138
7.25/13844
x
/
0.180:31
6.0012065
Onts
/
IGN
0.150133
6.2911.79
O
V
0.100.3
6.2011799
Table 4: Ablation study on each key modulc of our TrajCLIP model (left). Ablation study on each data
augmentation (right). SP denotes Small Perturbations, API denotes Adjacent Point Interpolation, PTG denotes Plausible Trajectories Generation.
Ablation Studies. In left part of Table we present the performance contributions of our proposed key modules: the SAIF, the CLIP pairing learning, and the manifold predictor (IGN generation module). The results show that combining the CLIP pairing method and the IGN module significantly enhances prediction performance, surpassing all other generation methods. This is because the CVAE framework involves an encoder obtaining a feature and then randomly sampling to get the mean and variance, which are used as inputs for the decoder. The diffusion method starts with a random initialization of an input and then gradually denoises. This randomness makes alignment work less effective (that is, even if aligned, the presence of random sampling still makes the generated results uncontrollable). The comparative experiments (c)(d) and (e)(f) illustrate that the two random methods of CVAE and diffusion methods make the improvement effect of alignment work not significant. On the ETH UCY dataset, the addition of CLIP only improved ADE/FDE by 0.01/0.03 and 0.01/0.01. However, the network structure of IGN is an MLP, which an affine transformation and does not have this issue; hence, as in our method, alignment is significantly effective on IGN. When CLIP is removed, as shown in experiments (b), since the two feature spaces cannot be connected through IGN without alignment, the performance drops by 0.06/0.17 compared to when CLIP is added.
9

---

We also conduct experiments to test different data augmentation methods and find that our method's improved performance due TrajCLIP's strong modeling ability represent complex trajectory distributions, which shown in right part of Table The results of applying the data augmentation methods are shown in the rightmost column of Table. When we train the latest state-of-the-ard methods Eq and TUTR with our proposed data augmentation methods, we observe minimal changes their performance, whereas our model's performance improved significantly. Additionally, experiments on each of the three core data augmentation methods demonstrate their effectiveness. Among these methods, Adjacent Point Interpolation (API) is proved to be the most effective in enhancing the model's prediction accuracy.
Model Size (MB) Computational Complexity (GFlops)
Inter Speed (S)
trajectro+++
0.53
2.48
0.0223
AgentFormer
6.78
12.79
0.1874
MID Y-net
2.42
9.06
0.8802
203.23
35.70
1.0716
TUTR
0.46
3.51
0.0577
Ours-tiny
3.45
5.26
0.0615
Ours TrajCCLIP)
14,94
18.96
0.2108
Table 5: Comparison of method with other methods in terms of model size, computational complexity, and infer speed. Infer speed refers the time required predict the next 12 frames using 8 frames.
Infer Speed and Computation Consumption. As the table 5 iillustaates we have contrasted our approach with alternative techniques in terms of model size, computational complexity, and inference speed. Our medium-sizee both inference speed and model size, as it can predict trajectories in 0.0615 seconds. Additionally, our lightweight model is only 3.45MB in size, and its computational complexity relatively low compared to its model size, making it deployable on most hardware platforms.
model meets the requirements for a real-time prediction task terms of
Qualitative Results. The left figure of Table 3 shows the visualization results of trajectory prediction for complex history trajectory using Tra and other state-of-the-art methods EqMotion and MID on UNIV scene of ETH-UCY dataset. Two trajectories in the figure include interference of surrounding pedestrians and scene environment and several turnings. We see that our TrajCLIP could predict these unusual turns, and the other two methods only predict common smooth trajectory pattern, which shows a better ability to model complex trajectories.
Figure 5: Visualization of failure cases on the ETH/UCY dataset. Given the observed historical trajectories (yel- low), our method predicts future trajectories (green). In complex interaction scenarios, our method occasionally cxhibits failures such as deviating from the intended endpoint and colliding (red) with other agents.
Limitation Analysis. as illustrated in Figure our method falls short in ensuring collision avoidance in complex, high-density environments, necessitating further research.
Conclusion
In our work, we introduce TrajCLIP, a method for predicting pedestrian trajectories. We use con- trastive learning and an idempotent generative network to model complex trajectory feature space. TrajCLIP offers five key contributions: the ability to model intricate trajectory distributions, CLIP pairing learning combine the feature spaces of historical and futurre trajectories, trajectory feature encoder based on the fusion of time domain and frequency domain, an idempotent generative network for mapping the same feature space, and an adjacent point interpolation algorithm enhance the model's 's expression ability. We evaluate TrajCLIP on four prediction tasks, including standard trajec- tory prediction, scene-to to sceneene transfer, few-shot transfer, and online adaption, using three datasets. Our method achieves state-of-the-ee prediction performance.
Acknowledgement
This work was supported by the National Natural Science Foundation of China under Grant 62002345.
10

---

References
[1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robiequet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 961-971, 2016.
[2] Inhwan Bae, Jin- Jin-Wwi Park, and Hae-Gon Jeon. Learning pedestrian group representations for modal trajectory prediction. In European Conference on Computer Vision, pages 270-289. Springer, 2022.
[3] Inhwan Bae, Young-Jac Jae Park, and Hae-Gon Jeon. Singulartrajecorry: Universal trajectory predictor using diffusion model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 17890-17901 2024.
[4] Haoyu Bai, Shaojun Cai, Nan Ye, David Hsu, and Wee Sun Lee. Intention-aware online pomdp planning for autonomous driving in a crowd. In 2015 ieee international conference on robotics and automation (icra), pages 454-460. IEEE, 2015.
[5] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[6] Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17113-17122,
2022.
[7] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks. Proceedings of the IEEE conference computer vision and pattern recognition, pages
2018.
[8] James Harrison, Apoorva Sharma, and Marco Pavone. Meta-learning priors for efficient online bayesian regression. In Algorithmic Foundations of Robotics XIII: Proceedings of the 13th Workshop on the Algorithmic Foundations of Robotics 13, pages 318-33 118-337. Springer, 2020.
[9] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6857) 2020.
[10] Boris Ivanovic, James Harrison, and Marco Pavone. Expanding the deployment envelope of behavior prediction via adaptive meta-learning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7786-7793. IEEE, 2023.
[11] Takayuki Kanda, Hiroshi Ishiguro, Tetsuo Ono, Michita Imai, and Ryohei Nakatsu. Develop- ment and evaluation of an interactive humanoid robot" robovie". In Proceedings 2002 IEEE international conference on robotics and automation (Cat. No. 02CH37292), volume 2, pages 1848-1855. IEEE, 2002.
[12] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 131iv:1312.61 2013.
[13] Yuanfu Luo, Panpan Cai, Aniket Bera, David Hsu, Wee Sun Lee, and Dinesh Manocha. Porca: Modeling and planning for autonomous driving among many pedestrians. IEEE Robotics and Automation Letters, 3(4):3418 04253, 2018.
[14] Takahiro Maeda and Norimichi Ukita. Fast inference and update of probabilistic density estimation on trajectory prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9795-9805, 2023.
[15] Osama Makansi, Özgün Çięek, Yassine Marrakchi, and Thomas Brox. On exposing the challenging long tail in future prediction traffic actors. In 2021 IEEE/CVF International Conference Computer Vision (ICCV), pages 13127-13137, 2021.

---

[16] Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgen:: A social ppatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14424-14432, 2020.
[17] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12. 2021.
[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
[19] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2)3, 2022.
[20] Saeed Sandatnejad, Mohammadhossein Bahari, Pedram Khorsandi, Mohammad Saneian, Seyed- Mohsen Moosavi-Dezfoole and Alexandre Alahi. Are socially-awar trajectory prediction mod- els really socially-aware? Transportation research part C: emerging technologies, 141:103705, 2022.
[21] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- trajectory forecasting with heterogeneous data. In European Conference on Computer Vision, pages 683-700. Springer, 2020.
[22] Liushuai Shi, Le Wang, Sanping Zhou, and Gang Hua. Trajectory unified transformer for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9675-9684, 2023.
[23] Assaf Shocher, Amil Dravid, Yossi Gandelsman, Inbar Mosseri, Michael Rubinstein, and Alexei A Efros. Idempotent generative network. arXiv preprint arXiv:2311.01466 2023,
[24] Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 28, 2015.
[25] Yuning Wang, Pu Zhang, Lei Bai, and Jianru Xue. Fend: A future enhanced distribution- aware contrastive learning framework for long-tail trajectory prediction. In 2023 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1400-1409, 2023.
[26] Chenxin Xu, Weibo Mao, Wenjun Zhang, and Siheng Chen. Remember intentions: cinoopeetivee-mmry--eerre trajectory prediction. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 6488 6497, 2022.
[27] Chenxin Xu, Robby T Tan, Yuhong Tan, Siheng Chen, Yu Guang Wang. Xinchao Wang, and Yanfeng Wang. Eqmotion: Equivariant multi-agent motion prediction with invariant interaction reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1410 1420, 2023.
[28] Yi Xu, Dongchun Ren, Mingxia Li, Yuehai Chen, Mingyu Fan, and Huaxia Xia. Tra2tra: dnnecctory-totraaectotn prediction with a global social spatial- trappalltemporal attentive neural network. IEEE Robotics and Automation Letters, 6(2):1574-1555 1581,, 2021.
[29] Yi Xu, Lichen Wang, Yizhou Wang, and Yun Fu. Adaptive trajectory prediction via transferable gnn. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6520-6533 6531, 2022.
[30] Alper Yilmaz, Omar Javed, and Mubarak Shah. Object tracking: A survey. Acm computing surveys (CSUR), 38(4):13-es, 2006.
12

---

[31] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani. Agentformer: Agent-aware transform- ers for socio-temporal aggett forecasting. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9813-9823, 2021.
[32] Guoqiang Peter Zhang. Neural networks for classification: a survey. IEEE Transactions Systems, Man, and Cybernetics, Part C (Applications and Reviews), 30(4):451 462, 2000.
[33] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3560-3566.
IEEE, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The thesis proposition has been elaborated in the abstract and introduction.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed in the experiment the disadvantages of our proposed TrajCLIP in terms of computational speed.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The theoretical results involved the paper have all provided relevant refer- ences.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We are in methodology and Sec. 4.2 complete training process and detailed training details are provided in ref implement, and the code will be open source in the future.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Although did not submit the code in the submission materials, confident that the description the article is sufficiently reproducible, and we will open source the code and dataset on GitHub in the futurr.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have elaborated in detail on the reasons, methods, and results of each experiment the experimental section.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experiments random seeds on the robustness of the experiment.
randomly initialized multiple times to avoid the impact
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Question:
14

---

Answer: [Yes]
Justification: We have provided the relevant details of our training.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of
Answer: [Yes]
Justification: Our research complies with relevant ethical guidelines.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our research provides perspective for modeling abstract and complex pedestrian trajectories, and its generalization ability is beneficial for the online deployment of downstream planning tasks.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The data we use are all commonly used benchmark trajectory datasets and do not involve any safety risks.
12. Lieenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The dataset used in the paper and other studies have provided references.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The model mentioned in the article is not yet open source, and will open source the code on GitHub as soon as possible.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: not involve.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: invlove.
15


Reviews:
[{"review_id": "OfYPj1AhuX", "reviewer": "Reviewer_Xn2A", "summary": "This paper presents TrajCLIP, a novel method for pedestrian trajectory prediction that utilizes contrastive learning and idempotent neural networks. The authors propose an interesting approach to address some limitations of existing methods, particularly in terms of generalization and modeling complex trajectory distributions.", "questions": "1 Provide more detailed explanations of the idempotent neural network implementation and training process.\n\n2 Add a section discussing the computational requirements and runtime performance of TrajCLIP.\n\n3 Include a discussion on the limitations of the proposed method and potential scenarios where it might not perform well.", "limitations": "The limitations discussed in the Experiments seem too easy and do not fully reflect the limitations of the work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The paper introduces an innovative idea of using contrastive learning to align the feature spaces of historical and future trajectories. This approach has the potential to improve the model's ability to generalize across different scenarios.\n\n2 The use of idempotent neural networks for global feature mapping is a creative solution to prevent overfitting to specific dataset distributions.\n\n3 The combination of time-domain and frequency-domain features in the trajectory encoder is an interesting approach that could capture more comprehensive trajectory information.", "weaknesses": "1 The paper lacks a discussion on the computational complexity and runtime performance of the proposed method compared to existing approaches.\n\n2 More analysis of the limitations of the proposed method and potential failure cases would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Add a section discussing the computational requirements and runtime performance of TrajCLIP.\n\nWe appreciate you bringing up the model performance experiment. As the table illustrates, we have contrasted our approach with alternative techniques in terms of model size, computational complexity, and inference speed. Our medium-sized model meets the requirements for a real-time prediction task in terms of both inference speed and model size, as it can predict trajectories in 0.0615 seconds. Additionally, our lightweight model is only 3.45MB in size, and its computational complexity is relatively low compared to its model size, making it deployable on most hardware platforms. \n\n|                | Model Size (MB) | Computational Complexity (GFlops) | Infer Speed (s) |\n|----------------|:---------------:|:---------------------------------:|:---------------:|\n| trajectron++   |       0.53      |               2.48               |      0.0223     |\n| AgentFormer    |       6.78      |              12.79               |      0.1874     |\n| MID            |       2.42      |               9.06               |      0.8802     |\n| Y-net          |      203.23     |              35.70               |      1.0716     |\n| TUTR           |       0.46      |               3.51               |      0.0577     |\n| Ours-tiny      |       3.45      |               5.26               |      0.0615     |\n| Ours(TrajCLIP) |      14.94      |              18.96               |      0.2108     |\n\n*Comparison of our method with other existing methods in terms of model size, computational complexity, and inference speed. Inference speed refers to the time required to input an 8-frame trajectory and predict the next 12 frames.*\n\n> Q2:More analysis of the limitations of the proposed method and potential failure cases would strengthen the paper.\n\nWe express our gratitude for the feedback. Due to spatial limitations, a summary of the limitations of our methodology will be presented in the camera-ready version. Our approach primarily focusses on the modelling of trajectory prediction tasks, in line with common practices that omit scene imagery as input. Therefore, empirical validation of the model's performance in practical scenarios is essential. As illustrated in Figure 1 in attached PDF, our method falls short in ensuring collision avoidance in complex, high-density environments, necessitating further research.\n\n> Q3: Provide more detailed explanations of the idempotent neural network implementation and training process.\n\nWe will provide a detailed summary of the model's experimental details and training process in the camera-ready version and will also release the source code. The experimental details of the idempotent neural network can be summarised as follows: First, we froze the pre-trained historical trajectory encoder and trained the manifold predictor, that is, the idempotent neural network, as well as the manifold decoder. The manifold predictor was trained using reconstruction loss, idempotent loss, and tightness loss, while the manifold decoder was trained using L2 loss. Training the idempotent and tightening losses requires gradient clipping. For clarity, we provide the training code as follows. Moreover, an RTX 3090 was used to train our model. Our training parameters included a 64-batch size, 100 epochs, and a 0.01 learning rate that was half-formed every 25 epochs. The Adam optimizer was employed.\n\n```python\ndef ign_train(f, f_copy, Z_H):\n    # f, f_copy : MLP_θ\n    predict_Z_F = f(Z_H)\n      \n    f_z = Z_F.detach()  \n    ff_z = f(f_z) \n    f_fz = f_copy(fz) \n\n    # calculate losses\n    loss_rec = (predict_Z_F - Z_F).pow(2)  \n    loss_idem = (f_fz - fz).pow(2)  \n    loss_tight = -(ff_z - f_z).pow(2)  \n\n    # optimize for losses\n    loss = loss_rec + loss_idem + loss_tight * 0.1\n    opt.zero_grad()\n    loss.backward()\n    opt.step()\n```\n\n[1] Shocher, Assaf, et al. \"Idempotent Generative Network.\" The Twelfth International Conference on Learning Representations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6f49IGebGd", "reviewer": "Reviewer_AYPL", "summary": "This paper utilizes idempotent generative network to perform multiple tasks in pedestrian trajectory prediction and achieves state-of-the-art performance in those tasks, showing its great representation and generalization ability. \n\nThe proposed model has the following main components:\n1. Spatio-Temporal Interaction Feature (STIF) and Scene-Agent Interaction Feature (SAIF) based encoder\n2. CLIP for past and future feature alignment\n3. IGN for mapping from past to future in aligned feature space.\n\nThe authors also provide a theoretical analysis of IGN, CLIP and data augmentation, but some issues need to be addressed.", "questions": "1. Alignment is quite useful for IGN but not as much useful as IGN when applying to CVAE and diffusion according to ablative experiments. Why was that? This issue needs more analysis and clarification. \nFrom the ablative experiments, when CLIP is not used, IGN is worse than CVAE and diffusion (from b, d, f in Tab.4). Meanwhile, CLIP cannot provide pronounced improvement as IGN when applied on CVAE and diffusion. The author just says combining them achieves the best results but motivation of combining IGN and CLIP is not very clear. Is it due to some intrinsic characters of IGN that requires feature alignment while other models do not drop as much as IGN when removing it? More detailed analyses of the phenomena is needed.\n\n2. Is the affine transformation mentioned in Fig.1 same as the trainable matrix W in CLIP?", "limitations": "Not seen in main content.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. First-time usage of IGN in pedestrian trajectory prediction task, which is novel.\n2. The paper is well-written and easy to follow.\n3. Experiment on various kinds of pedestrian trajectory prediction tasks is sufficient and a universal model for those tasks is the main tendency.", "weaknesses": "1. This is not the first work to use constructive learning in pedestrian trajectory prediction, since it was used in long-tailed pedestrian trajectory prediction for forming better latent representations [1, 2]. The author should cite them and clarify the difference. \n2. Missing citation and comparison for SingularTrajectory[3], which is also a universal model for pedestrian trajectory prediction. It defines a singular space that can well represent trajectory features for multiple tasks without alignment between past and future.\n3. Some questions need to be addresses. See questions.\n\n\n\n[1] Wang, Yuning, Pu Zhang, Lei Bai, and Jianru Xue. \"Fend: A future enhanced distribution-aware contrastive learning framework for long-tail trajectory prediction.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1400-1409. 2023.\n\n[2] Makansi, Osama, Özgün Cicek, Yassine Marrakchi, and Thomas Brox. \"On exposing the challenging long tail in future prediction of traffic actors.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13147-13157. 2021.\n\n[3] Bae, Inhwan, Young-Jae Park, and Hae-Gon Jeon. \"SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17890-17901. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: This is not the first work to use constructive learning.\n\nWe thanks reviewers for providing a new perspective. The mentioned two works, long-tail analysis[1] and FEND[2], both utilize contrastive learning, but our work differs in the problems it addresses with contrastive learning. These two mostly use contrastive learning to capture features from historical trajectories. They aim to get better representations of historical trajectory encodings in latent space, which will improve their ability to predict trajectories for the long-tail part of the dataset.}\n\nFor example, long-tail analysis uses a Kalman filter to directly predict output trajectory errors as a basis, distinguishing difficult and easy samples. It then conducts contrastive learning on difficult (long-tail part) historical trajectories to enhance the feature representation capabilities of the long-tail part; FEND uses VAE to extract features from the entire trajectory and performs offline clustering on the features to obtain labels; during training, it uses offline labels for contrastive learning on historical trajectories to guide the feature generation of historical trajectories (making features extracted from more similar historical trajectories closer), enhancing the prediction effect of the long-tail part.\n\nHowever, our work, not only uses contrastive learning for better encoding of historical trajectories but also acts on future trajectories. While ensuring historical and future motion consistency, it achieves a more unified latent space representation of future and historical trajectories, thereby enhancing the model's predictive generalization capability.\n\n> Q2: Missing citation and comparison for SingularTrajectory.\n\nSingularTrajectory[3] introduces a diffusion model for multimodal pedestrian trajectory prediction, utilizing singular space rather than trajectory coordinates to convey motion information and capture the dynamics of trajectories. It also employs an adaptive anchor-based strategy for dynamically adjusting trajectory anchors. We acknowledge the merits of this technique, but our approach surpasses this work in terms of performance and adaptability. The summary is as follows:\n\n(1) In the performance comparison on the ETH-UCY dataset as shown in Table 3, our model achieves a 14\\% lower Average Displacement Error (ADE) than the SingularTrajectory model (0.18 for our model versus 0.21 for SingularTrajectory), with the difference in Final Displacement Error (FDE) being negligible.\n\n(2) For transfer learning and few-shot learning tasks, given the differences in task settings and data processing between our method and SingularTrajectory, we conducted experiments following the experimental framework described in their paper. The results of these experiments are presented in Tables. 2 and 3. Observations indicate that our model attains comparable ADE performance to SingularTrajectory in transfer learning tasks and slightly outperforms in FDE. In few-shot learning tasks, our model demonstrates superior performance in both ADE and FDE.\n\n(3) Regarding the task of online adaptability, the architecture of SingularTrajectory, which incorporates complex diffusion models and scene semantic segmentation techniques, results in slower inference speed, making online learning impractical. In contrast, our model consistently outperforms in this task.\n\n> Q3:More detailed analyses for ablation experimental phenomena about CLIP and generative framework (IGN, CVAE, diffusion). \n\nWe further explain the analysis of the ablation study. The CVAE framework involves an encoder obtaining a feature and then randomly sampling to get the mean and variance, which are used as inputs for the decoder. The diffusion method starts with a random initialization of an input and then gradually denoises. This randomness makes alignment work less effective (that is, even if aligned, the presence of random sampling still makes the generated results uncontrollable). As shown in Table 4 in main content, the ablation experiments of CLIP and the generation framework were conducted. The comparative experiments (c)(d) and (e)(f) illustrate that the two random methods of CVAE and diffusion methods make the improvement effect of alignment work not significant. On the ETH\\_UCY dataset, the addition of CLIP only improved ADE/FDE by 0.01/0.03 and 0.01/0.01. However, the network structure of IGN is an MLP, which is an affine transformation and does not have this issue; hence, as in our method, alignment is significantly effective on IGN. When CLIP is removed, as shown in experiments (b)(c)(e), since the two feature spaces cannot be connected through IGN without alignment, the performance drops by 0.06/0.17 compared to when CLIP is added.\n\n> Q4:Is the affine transformation mentioned in Fig.1 same as the trainable matrix W in CLIP?\n\nIn our approach, CLIP is only utilized to facilitate the alignment of feature spaces during the pre-training phase. The affine transformation of the idempotent generative network operates independently of the parameters of CLIP. However, using CLIP's trainable matrix W as an initialization for the network's parameters could enhance the generative framework. Since CLIP's trainable matrix W can project the feature space into a unified fusion space, it is possible to employ this matrix W as an initial parameter within the idempotent generative framework and subsequently further train the idempotent generative network based on it. This is a perspective worth exploring, and we will conduct further research on this attempt in the subsequent studies.\n\n[1] Makansi, Osama, et al. \"On exposing the challenging long tail in future prediction of traffic actors.\" ICCV. 2021.\n\n[2] Wang, Yuning, et al. \"Fend: A future enhanced distribution-aware contrastive learning framework for long-tail trajectory prediction.\" CVPR. 2023.\n\n[3] Bae, Inhwan, et al. \"SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model.\" CVPR. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jrOsVoN5PH", "reviewer": "Reviewer_NFeW", "summary": "This paper proposes to utilize contrastive learning for pedestrian trajectory prediction. A STIF encoder is used to extract spatial-temporal features and is trained with data augmentation. A SAIF utilizes the Fast Fourier Transform to extract the interaction information among the agents. The authors incorporate the idempotent loss for training. Experiments on three datasets show the efficacy of the proposed method.", "questions": "1. I have some doubts about the intuition of using CLIP learning between the historical feature and the future trajectory feature. The authors propose to align historical feature space with future feature space (Line 186). Does this lead to identical trajectory prediction, i.e., copying the historical trajectory as the future trajectory? Different from aligning images and text that have the same semantic meaning, the historical trajectory and future trajectory of the same agent may be different due to the difference in the spatial location (hence the surroundings are different).\n2. Since the method involves multiple stages, how does the model compare to the baselines in terms of inference speed and training cost?\n3. There is no error analysis. When does the method fail and why?", "limitations": "The authors did not discuss the limitations of the method and failure cases.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using contrastive learning and idempotent loss is interesting.\n2. The performance is good. The proposed method achieves significantly better results than SOTA on SDD and others.", "weaknesses": "1. I have some doubts about the intuition of using CLIP between history and future trajectories. See questions below.\n2. There is no speed or computational cost comparison. Since the method involves multiple stages, a comparison of training/inference costs is needed. \n3. The proposed method does not consider visual scene information.\n4. There is no error analysis. When does the method fail and why?\n\nMinor comments:\nSection 2.2 Generalization Framework -> Generative Framework. They are different things.\nThe text in Figure 2 is too small to read.\nLine 263 “We conducted comparative ”, Line 270 “we compare our model”. Better to use a consistent present tense when describing your method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:I have some doubts about the intuition of using CLIP between history and future trajectories.\n\nRegarding further clarification on the unified encoding of historical and future trajectories using CLIP. In our method, the trajectory encoders are designed to capture motion characteristics rather than static spatial data, enabling synchronization of feature spaces between the future and historical trajectory encoders. Considering that inputs for pedestrian trajectory prediction consist of sequential relative coordinates, the encoders focus on the agent's trajectory, surrounding dynamics, and their interactions. The decoder then infers future locations from these features. For datasets like ETH\\_UCY, where 20 frames equal 5 seconds, spatial consistency is maintained by analyzing the same two frames. The use of STIF and SAIF is for modeling trajectory and interaction features, respectively, aligning them in the feature space to prevent redundancy in trajectory predictions that could arise from spatial alignment.\n\n> Q2: There is no speed or computational cost comparison. Since the method involves multiple stages, a comparison of training/inference costs is needed.\n\nFor training process, our batch size was set to 64, epochs to 100, and the learning rate was 0.01, which was half-formed every 25 epochs. We used the Adam optimizer. Our model was trained on an RTX 3090, with the encoder and contrastive learning pre-training requiring approximately 8 GPU hours, and the full framework training taking about 9.6 GPU hours. \n\nFor inference process, as shown in Table 1, our medium-sized model meets the requirements for a real-time prediction task in terms of both inference speed and model size, as it can predict trajectories in 0.0615 seconds. Additionally, our lightweight model is only 3.45MB in size, and its computational complexity is relatively low compared to its model size, making it deployable on most hardware platforms. \n\n> Q3:The proposed method does not consider visual scene information.\n\nThank you for the new perspective provided by the reviewer. Our focus lies in how to align the trajectory features and interaction features of agents in the historical and future spaces within trajectory prediction, and we are concerned with whether this method can be generalized to various subtasks at a low cost. Due to the high real-time requirements of autonomous driving, we chose not to introduce image information. Admittedly, using the features of visual information from map scenes to improve the performance of trajectory prediction is indeed another angle worth paying attention to. We will consider how to efficiently integrate visual information to further enhance the predictive effect of the model in our subsequent work.\n\n> Q4: Minor comments: Section 2.2 Generalization Framework -> Generative Framework. They are different things. The text in Figure 2 is too small to read. Line 263 “We conducted comparative ”, Line 270 “we compare our model”. Better to use a consistent present tense when describing your method.\n\nWe will address these issues (e.g., typos, grammar mistakes) in the camera-ready version.\n\n> Q5:There is no error analysis. When does the method fail and why?\n\nWe express our gratitude for the feedback. Due to spatial limitations, a summary of the limitations of our methodology will be presented in the camera-ready version. Our approach primarily focusses on the modelling of trajectory prediction tasks, in line with common practices that omit scene imagery as input. Therefore, empirical validation of the model's performance in practical scenarios is essential. As illustrated in Figure 1 in attached PDF, our method falls short in ensuring collision avoidance in complex, high-density environments, necessitating further research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "atN2zoH1dX", "reviewer": "Reviewer_iNsW", "summary": "The paper proposes a trajectory prediction approach for multi-agent configurations. The historical and future trajectories are encoded in terms of spatio-temporal interaction features (STIF) using Agentformer [28] and scene-agent interaction features (SAIF) using a transformer based on Fourier transform. The historical and future trajectory encoders share the same architecture but have different parameters. The encoded features are then summed and passed to a generative model [21] to represent  the trajectory space. Finally, a transformer predicts the next position. The encoded features for historical and future trajectories (STIF and SAIF) are trained using CLIP [16] to force the historical and future encoded features to share the same space and help achieve temporal continuity for the trajectory prediction task. The approach is evaluated on the ETH-UCY, SDD and SNU datasets and shows promising results compared to related methods.", "questions": "- For the Online Adaptation task, prediction times are another crucial factor to consider alongside performance metrics. Is there any reason not to report them?\n- Could the authors provide details of the hyper-parameters used during training? Implementation details are missing\n- There are significant discrepancies between the results you report in Table 1 for Traj++ and the results reported in other publications, such as Traj++ in \"Human Trajectory Prediction via Neural Social Physics\". Could the authors provide more details on the experimental setup?", "limitations": "The paper does not dedicate space to discuss the limitations. This is a clear missing point.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow. In addition, the approach is well presented and the evaluation is clearly described. \n- The proposed approach achieves promising results on several standard benchmarks. The results are convincing. \n- The paper provides a wealth of ablation studies. \n- The proposed combination of coding and decoding is interesting and makes a solid contribution. In particular, the adaptation of CLIP from image-text pairs to the historical-future trajectory feature space is novel.", "weaknesses": "- The paper makes various solid and small contributions, but it lacks a novel idea. This is a limitation, but it's not a major point.\n- The related work would benefit from a more direct comparison of the proposed method with the related works. \n- The motivation for relying on an idempotent generative network over other generative models, e.g. GANs, is not well explained.\n- According to the literature, the article \"Human Trajectory Prediction via Neural Social Physics\" by Yue et. al. (2022) achieves average ADE and FDE of 0.17 and 0.24 respectively on ETH, while 6.52 ADE and 10.61 FDE are achieved on SDD. The comparisons should include all state-of-the-art methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1:The motivation for relying on an idempotent generative network over other generative models, e.g. GANs, is not well explained.\n\nFurther explanation on the starting point for choosing the idempotent generation framework. Our motivation is that, for trajectory prediction task, the trajectory features of an agent should be consistently aligned in the feature space of both historical and future trajectories. However, GANs and CVAEs (SocialGAN) both have separate encoding-decoding structures, and diffusion inference samples the initial space for generation, which leads to a certain degree of inconsistency in the feature space. The idempotent generation framework is an affine transformation in the feature space, which can effectively bridge the gap in the feature space caused by the generation framework. In addition, the idempotent neural framework has the characteristic of global projection, which can more effectively improve generalization ability, allowing our method to perform well on different tasks without fine-tuning. Our extensive quantitative and qualitative experiments have fully demonstrated this point. Please refer to Reviewer  AYPL Q3 for more experiential analysis.\n\n> Q2: article NSP-SFM[1] performs better. The comparisons should include all state-of-the-art methods.\n\nFurther explanation on the comparison with the SOTA methods, we will compare and analyze with the methods mentioned by the reviewers in the final version. (1) We acknowledge that the model in the article NSP-SFM[1] has better predictive quantitative performance than TrajCLIP. However, the NSP-SFM method first relies on a CNN-based network to extract goal information from the map, which is not feasible for many complex scene prediction tasks as the goal information is not available. Secondly, the method's estimation capability for multimodal trajectories is not achieved by a generative model and does not consider the diversity of pedestrian intentions during the prediction process; moreover, the network has a large number of parameters, with its model parameter count being more than 50 times that of other trajectory prediction networks. In contrast, TrajCLIP's predictions do not need to rely on goal information and can be predicted independently, without being limited by the prediction anchor points, and with a smaller number of parameters. We are more focused on comparing with similar methods to demonstrate the model's performance and capabilities in the context of the predictive task's focus points. \n\nWe compared four recent trajectory prediction state-of-the-art method from CVPR 2024 in Table 4.  our method achieves the best ADE/FDE results on the ETH-UCY dataset. \n\n> Q3:For the Online Adaptation task, prediction times are another crucial factor to consider alongside performance metrics. Is there any reason not to report them?\n\nWe have included implementation details in Table 1 in attached PDF. Our tiny model meets the requirements for a real-time prediction, as it can predict trajectories in 0.0615 seconds. Additionally, our lightweight model is only 3.45MB in size, and its computational complexity is relatively low compared to its model size, making it deployable on most hardware platforms. \n\n> Q4:Could the authors provide details of the hyper-parameters used during training? Implementation details are missing.\n\nFor training process, our batch size was set to 64, epochs to 100, and the learning rate was 0.01, which was half-formed every 25 epochs. We used the Adam optimizer. Our model was trained on an RTX 3090, with the encoder and contrastive learning pre-training requiring approximately 8 GPU hours, and the full framework training taking about 9.6 GPU hours. \n\n> Q5: There are significant discrepancies between the results you report in Table 1 for Trajectron++ and the results reported in other publications, such as Trajectron++ in \"Human Trajectory Prediction via Neural Social Physics\". Could the authors provide more details on the experimental setup?\n\nExplanation of the reproduction details for Trajectron++. When reproducing Trajectron++[3], there was a data leakage issue in the original released code implementation, leading to better metrics. The original author also acknowledged this on GitHub issue #26. \"The function used for differentiating is **derivative of** and calls **np.gradient**. np.gradients approximates in different ways for boundary points and non-boundary points. For an example of calculating velocities, np.gradients will give $v _x[t] = (x[t+1]-x[t-1])/(2*dt)$ at time t.\" This method inadvertently includes future trajectory information in the last frame of historical data, which significantly affects the model's predictive performance. The author of Trajectron++ suggests using the correct differentiation method, such as **np.ediff1d**, to preprocess velocity and acceleration inputs to avoid data leakage caused by using information beyond the last observed time step. (Due to conference policy restrictions, we cannot provide a link.) After resolving this data issue, we reproduced the experiment using the method suggested by the author, which resulted in differences in the cited values when compared with other papers. In addition, the recent CVPR 2024 paper: SingularTrajectory [2] and many other works also adopt the same practice when citing the Trajectron++ work, and the metrics are consistent.\n\n[1] Yue, Jiangbei, Dinesh Manocha, and He Wang. \"Human trajectory prediction via neural social physics.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.\n\n[2]  Bae, Inhwan, et al. \"SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model.\" CVPR. 2024.\n\n[3] Salzmann, Tim, et al. \"Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVIII 16. Springer International Publishing, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all reviewers for your valuable comments and recognition of the novelty of our work. We have provided further elaboration and clarification in response to the reviewers' feedback, along with additional experiments to supplement the explanation. We will address each reviewer's comments and queries individually to assist in understanding this work. And we will release the source code and address all the reviewers' issues in the camera-ready version.

We have included more experiment results in the attached file, to facilitate the reviewers understanding of our responses. 

Here, we list the additional experiments according to each reviewer's comments.

**[Xn2A-Q2, AYPL, NFeW-Q5, iNsW]** limitation and potential failure discussion:  we have conducted supplemental experiments and discussed the limitation and failure cases of our method as shown in Figure 1.

**[Xn2A-Q1, NFeW-Q2]** model computational complexity discussion: We have conducted additional experiments and discussed the comparison of model size, computational complexity and inference speed between our TrajCLIP and other recent popular pedestrian trajectory prediction models, as shown in Table 1.

**[AYPL-Q2]** comparison for SingularTrajectory[1]: We compare our work with SingularTrajectory from transfer learning and few-shot learning, as shown  in Table 2 and Table 3.

**[iNsW-Q3]** comparisons with all state-of-the-art methods: we compare our work with multiple SOTA methods[1-4], as shown in Table 4. 

|                | Model Size (MB) | Computational Complexity (GFlops) | Infer Speed (s) |
|:--------------:|:---------------:|:---------------------------------:|:---------------:|
| trajectron++   |       0.53      |               2.48               |      0.0223     |
| AgentFormer    |       6.78      |              12.79               |      0.1874     |
| MID            |       2.42      |               9.06               |      0.8802     |
| Y-net          |      203.23     |              35.70               |      1.0716     |
| TUTR           |       0.46      |               3.51               |      0.0577     |
| Ours-tiny      |       3.45      |               5.26               |      0.0615     |
| Ours(TrajCLIP) |      14.94      |              18.96               |      0.2108     |

*Tab1. Comparison of our method with other existing methods in terms of model size, computational complexity, and inference speed. Inference speed refers to the time required to input an 8-frame trajectory and predict the next 12 frames.*

| ADE                |  A2B  |  A2C  |  A2D  |  A2E  |  AVG  |
|:------------------:|:-----:|:-----:|:-----:|:-----:|:-----:|
| SingularTrajectory |  0.29 |  0.59 |  0.51 |  0.42 |  0.45 |
| Ours(TrajCLIP)     |  0.30 |  0.59 |  0.49 |  0.43 |  0.45 |

| FDE                |  A2B  |  A2C  |  A2D  |  A2E  |  AVG  |
|:------------------:|:-----:|:-----:|:-----:|:-----:|:-----:|
| SingularTrajectory |  0.57 |  1.19 |  1.08 |  0.81 |  0.91 |
| Ours(TrajCLIP)     |  0.59 |  1.20 |  0.99 |  0.83 |  0.90 |

*Tab 2. Comparison of ADE/FDE for transfer learning on the ETH-UCY dataset between our method and SingularTrajectory. The ETH, HOTEL, UNIV, ZARA1, and ZARA2 scenes are denoted as A, B, C, D, and E, respectively.*

| Few-Shot           |  ETH      |  HOTEL    |  UNIV     |  ZARA1    |  ZARA2    |  AVG      |
|:------------------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| SingularTrajectory | 0.35/0.46 | 0.14/0.21 | 0.26/0.44 | 0.21/0.36 | 0.18/0.31 | 0.23/0.35 |
| Ours(TrajCLIP)     | 0.34/0.42 | 0.15/0.23 | 0.24/0.39 | 0.21/0.34 | 0.17/0.30 | 0.22/0.34 |

*Tab 3. Comparison of ADE/FDE for few-shot learning on ETH-UCY with SingularTrajectory.*

|                   |    ETH    |   HOTEL   |   UNIV    |   ZARA1   |   ZARA2   |    AVG    |
|:-----------------:|:---------:|:---------:|:---------:|:---------:|:---------:|:---------:|
| LMTraj-SUP        | 0.41/0.51 | 0.12/0.16 | 0.22/0.34 | 0.20/0.32 | 0.17/0.27 | 0.22/0.32 |
| MSN-SC            | 0.27/0.39 | 0.13/0.18 | 0.22/0.45 | 0.18/0.34 | 0.15/0.27 | 0.19/0.33 |
| HighGraph   | 0.33/0.56 | 0.13/0.21 | 0.23/0.47 | 0.19/0.33 | 0.15/0.25 | 0.21/0.36 |
| SingularTrajectory| 0.35/0.42 | 0.13/0.19 | 0.25/0.44 | 0.19/0.32 | 0.15/0.25 | 0.21/0.32 |
| Ours(TrajCLIP)    | 0.36/0.57 | 0.10/0.17 | 0.19/0.41 | 0.16/0.28 | 0.11/0.20 | 0.18/0.33 |

*Tab 4. Comparison of ADE/FDE for performance with SOTA methods on the ETH-UCY dataset.*



[1] Bae, Inhwan, Young-Jae Park, and Hae-Gon Jeon. "SingularTrajectory: Universal Trajectory Predictor Using Diffusion Model." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17890-17901. 2024.

[2] Bae, Inhwan, Junoh Lee, and Hae-Gon Jeon. "Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Wong, Conghao, et al. "SocialCircle: Learning the Angle-based Social Interaction Representation for Pedestrian Trajectory Prediction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Kim, Sungjune, et al. "Higher-order Relational Reasoning for Pedestrian Trajectory Prediction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Meta-review: The paper presents TrajCLIP, a novel approach to pedestrian trajectory prediction using contrastive learning and idempotent networks. The method shows promise in addressing the challenges of modeling complex trajectory distributions and demonstrates strong performance across various datasets and tasks.

Initially, the paper received mixed concerns especially in terms of (1) limited discussion of limitations, (2) computational complexity, and (3) implementation details. After the rebuttal stage, most of the reviewers' concerns have been addressed and Reviewer iNsW raised his score to weak accept. 

Given the novelty of the approach, strong empirical results, and the authors' thorough response to reviewers' concerns, this paper appears to make a significant contribution to the field of pedestrian trajectory prediction. The AC would like to recommend the paper for acceptance. Congratulations! Please be aware that the authors should integrate the response in the rebuttal and adequately address the reviewers' concerns in the final camera-ready version.
Decision: Accept (poster)