Towards Predicting Any Human Trajectory In Context
Authors: Ryo Fujii
Hideo Saito
Ryo Hachiuma
Venue: NeurIPS 2025 poster ()
Keywords: In-Context Learning
pedestrian trajectory forecasting
trajectory prediction
Primary area: applications
TL;DR: We introduce TrajICL, a novel in-context learning (ICL) framework for pedestrian trajectory forecasting that enables rapid adaptation without costly fine-tuning.

Abstract: Predicting accurate future trajectories of pedestrians is essential for autonomous systems but remains a challenging task due to the need for adaptability in different environments and domains. A common approach involves collecting scenario-specific data and performing fine-tuning via backpropagation. However, the need to fine-tune for each new scenario is often impractical for deployment on edge devices. To address this challenge, we introduce TrajICL, an In-Context Learning (ICL) framework for pedestrian trajectory prediction that enables adaptation without fine-tuning on the scenario-specific data at inference time without requiring weight updates. We propose a spatio-temporal similarity-based example selection (STES) method that selects relevant examples from previously observed trajectories within the same scene by identifying similar motion patterns at corresponding locations. To further refine this selection, we introduce prediction-guided example selection (PG-ES), which selects examples based on both the past trajectory and the predicted future trajectory, rather than relying solely on the past trajectory. This approach allows the model to account for long-term dynamics when selecting examples. Finally, instead of relying on small real-world datasets with limited scenario diversity, we train our model on a large-scale synthetic dataset to enhance its prediction ability by leveraging in-context examples. Extensive experiments demonstrate that TrajICL achieves remarkable adaptation across both in-domain and cross-domain scenarios, outperforming even fine-tuned approaches across multiple public benchmarks.

Towards Predicting Any Human Trajectory In Context
Ryo Fujii
Hideo Saito 1.2
Ryo Hachiuma
Keio University
Keio Al Research Center
3NVIDIA
Abstract
Predicting accurate future trajectories of pedestrians is essential for autonomous systems but remains challenging task due to the need for adaptability in different environments and domains. common approach involves collecting scenario- specific data and performing fine-tuning via backpropagation. However, the need to fine-tune for each new scenario is often impractical for deployment on edge devices. To address this challenge, we introduce TrajICI. In-Context Learning (ICL) framework for pedestrian trajectory prediction that enables adaptation with- out fine-tuning on the scenario-specific data at inference time without requiring weight updates. We propose a spatio-temporal similarity- example selection (STES) method that selects relevant examples from previously observed trajecto- ries within the same scene by identifying similar motion patterns at corresponding locations. To further refine this selection, we introduce prediction -uided example selection (PG-ES), which selects examples based on both the past trajectory and the predicted future trajectory, rather than relying solely on the past trajectory. This approach allows the model to account for long-term dynamics when selecting examples. Finally, instead of relying on small real-world datasets with limited scenario diversity, we train our model on large-scale synthetic dataset to enhance its prediction ability by leveraging in-context examples. Extensive experiments demonstrate that TrajICL achieves remarkable adaptation across both in-domain and cross-domain scenarios, outperforming even fine-tuned approaches across multiple public benchmarks. Project Page
1 Introduction
Predicting pedestrian trajectories is crucial for applications such as autonomous driving 10 robot navigation [26] [7). and surveillance systems 13] Recent learning based methods have shown strong
42 22 (3) 19, [59]. However, most existing approaches trained and evaluated within specific environments domains, limiting their generalizability. For real-world deployment, autonomous systems must handle diverse scenarios, requiring trajectory prediction models to be robust across varying environments and domains (e.g., map layouts, camera positions, and sensor types). The lack of adaptability to these factors significantly hinders their practical applicability. A common solution involves collecting scenario-specific data and fine-tuning models [60] 355 [8). However, this approach requires on-device backpropagation for adaptation using the collected data the target environment, which is often impractical on edge devices due to the high computational and memory costs associated with training. Furthermore, managing multiple models tailored different scenarios increases system complexity. Therefore. developing a single, adaptable model capable of generalizing across diverse real-world environments without fine-tuning remains an open challenge.
performance in trajectory prediction 2 55 77 28
41
To address this challenge, we explore the use of In-Context Learning (ICL) 7 68 54 prediction (Figure [] ICL enables models to adapt examples provided as part of the input, without requiring updates to the model weights. Unlike fine-tuning, which modifies model weights through backpropagation, ICL operates solely via forward
for trajectory tasks using only few demonstration
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In-Context Training with Synthetic Data
Model
Training and Inference on Scenario-Specifie Data
Model
In-Context Inference on Scenario-Specffi Data
Model
Model
Model
Model
Past Trajectory
Predicted Future Trajectory
Model
Predicted Future Trajectory
In-Context Example Past Trajectory
(a) Traditional Scenario-Specific Training
(b) TrajiCL (ours)
Illustration of real-world trajectory prediction scenarios and the adaptation pipeline. (a) The adaptation pipeline of traditional methods, where models are trained on -cenario-specific data. (b) The adaptation pipeline of our proposed TrajICL, which automatically selects examples and adapts to novel scenarios by leveraging the scenario-specific examples without requiring training on scenario-specific data.
Figure 1:
passes, keeping the model weights fixed. This key difference means that adaptation occurs during inference without requiring computationally intensive training phase or storing gradients, making highly suitable for resource-constrained edge devices. Leveraging this capability, we utilize small set of examples to enable single model to quickly adapt to diverse scenarios. This approach facilitates the reuse of trajectory models across a wide range of scenarios, climinating the need for costly fine-tuning.
However, incorporating ICL into trajectory prediction presents three major challenges: (1) Emerging ICL capability for trajectory prediction: Randomly selected examples provide minimal ICL capability to the model. Even when examples selected from the same scene, if they represent different spatial locations or exhibit divergent movements, the effectiveness in improving the model's 's adaptation is limited. Such variations fail to provide the model with sufficiently relevant context generalize effectively to new scenarios. (2) Suboptimal selection of examples based on past trajectory input: Existing methods select examples based on similarity the input query (i.e., past trajectory). However, short past trajectory segments often fail to capture long-term intentions. Moreover, pedestrian motion inherently multi-modal, where similar past trajectories can lead divergent future trajectories due to subtle influencing factors. Consequently, relying solely on past trajectories for the example selection, without accounting for long-term dynamics, can result in misleading examples and hinder effective in- nn-context adaptation. (3) Adaptability to diverse scenarios: Existing real- world datasets [8]] have limited scenario diversity, often focusing on specific environments and interaction patterns. Training on these small-scale datasets restricts ICL's ability to generalize to unseen scenarios, reducing the model's adaptability to broader range of real-world situations.
49 30
53
5
75 45)
To address these challenges, we propose TrajICI. an ICL. framework for trajectory prediction. First, we introduce a spatio-temporal Limilarity-based example selection (STES) that identifies relevant examples exhibiting similar motions at comparable locations in the past. By training the model using these spatially and temporally similar examples selected through STES, we enable the development of ICL capability, allowing the model to effectively adapt to new scenarios with minimal examples and without the need for additional training. Second, address the suboptimal selection based on past trajectory, we propose prediction-gui -uided example selection (PG-ES) that refines the example selection process using prediction results. PG-ES consists two phases. First, the model predicts the future trajectory based on its own past. trajectory and examples selected according to their similarity with the query past trajectory using the proposed STES. In the second phase, predicted future trajectory, in addition to the past trajectory, is utilized for example selection. This refinement process

---

allows the model to select more relevant examples by incorporating long-term dynamics. Finally, instead of relying on small real-world datasets with limited scenario diversity, we train the model on a large-scale synthetic dataset [14] to learn predictive capabilities using in-context examples. This allows the trained model be directly applicable to edge devices in various environments, enhancing its adaptability to real-world conditions.
We propose an ICL framework for trajectory prediction, enabling adaptation to diverse scenarios without training on the scenario-specific data, Our approach leverages auge-scale synthetic datasets to enhance generalization to real-world conditions, overcom- ing the limitations of small-scale real-world datasets to learn predictive capabilities using in-context examples. 2) We introduce ppatio-temporal similarity- baasee example selection (STES) that allows the model to acquire ICL capability. 3) We present prediction delfiction-uidedd guided example selection (PG-ES), which utilizes both past and predicted future trajectories to select more relevant examples, rather than relying solely on past trajectory input.
Our main contributions are as follows: 1)
2 Related Work
Adapting Trajectory Prediction. Pedestrian trajectory prediction aims to predict future positions based on past trajectories. Deep learning methods demonstrate strong performance due to their
19. [8]. Despite the significant advancements, previous trajectory prediction models are often tailored to specific training domains, new scenarios. To address this, recent research has explored lightweight adaptation techniques for pretrained models. Some approaches focus on cross-domain transfer 25 35), while others emphasize online adaptation 48 [0) continual learning
representational capabilities 0 23
55
70
28
41 42
22
4
56
limiting their generalizability
4E 69
58). or prompt tuning-based strategies [600. While these methods enhance forecasting performance, they often incur high computational costs due to backpropagation on the scenario-specific samples and require multiple models for different scenarios. In contrast, we leverage a small number of examples seamlessly adapt diverse scenarios, eliminating the need for costly fine-tuning.
In-Context Learning. In-Context Learning (ICL) 7 68 S4] enables large language models (LLMs) to perform new tasks by providing few input-output examples, or demonstrations, alongside the task input. ICL offers several advantages over traditional model adaptation methods, which typically involve pre-training followed by fine-tuning. key benefit of ICL is that it circumvents the need for fine-tuning, which can be limited by computational resource constraints. Compared to parameter- efficient fine-tuning (PEFT) methods (24 4. ICL is computationally cheaper and preserves the model's generality by leaving the model parameters unchanged. Following its success in Natural Language Processing (NLP), ICL has been extended to various domains within Computer Vision (CV), including images 4
63 64 80 3 50 76 62 3T video (27)) point clouds [6] 57]. and skeleton sequences 665. showcasing its ability to generalize across diverse, unseen tasks. Recent studies 22 38 have explored the use of LLMs in vehicle trajectory prediction, highlighting their effectiveness in in-context learning [38. In contrast, our approach prioritizes equipping lighter trajectory prediction models with in-context learning capability, offering greater efficiency and practicality for real-world applications.
Prompt Selection. The effectiveness of In-Context Learning (ICL) is highly influenced by the selection of relevant examples 377 47 32) Previous studies have shown that selecting in-context examples that are closer to the query improves performance [37 67 (7]. To address this challenge, several methods have been proposed
select examples that are more similar to the query for ICL [37 (78 A simple approach is to retrieve the nearest neighbors of the input query based on To enhance the robustness of ICL, some studies have employed iterative 31 36]. However, in trajectory prediction tasks, a typical example selection based solely on past trajectory input can be suboptimal. In this work, we introduce prediction guided prompt selection approach that selects in-context examples based on their similarity to both the input past trajectory and the predicted future trajectory, aiming to choose more effective examples considering long-term dynamics.
their similarities 37 5T).
methods 27
3

---

TrajICL
Example Selection
Tapa Agenta
KLP
Pradiae loar (rat Tapecary
aday
Eangle
Exariple
Gamnaccing Agenna!
Past Tragetoric
Colledio
Par Tepedrr aimarry
Thoeetory
x2"
ViP
Tupi:
ThalCt
Darde
X2
Rncader
Contect
Ripeat V/ Coreed Conned
Feiding Ecorpi Selection
Prodictie- Ecceqple Seltcation
Figure 2: An illustration of our TrajICL framework. (a) The overall architecture includes an embedding layer, trajectory encoder, in-context-awar trajectory predictor, and multi- decoder. (b) Rather than relying solely on past trajectories for the example selection, we introduce prediction-guidee example selection, which leverages both past and predicted future trajectories to identify more relevant examples.
TrajICL
3.1 Problem Formulation
Trajectory Prediction with In-Context Learning. Trajectory prediction aims to forecast the future positions of a target agent based on its own past trajectory and the trajectories of surrounding agents.
Formally, let X (X,, X,...., N E I NxTaxx x2 denote the past trajectories of N pedestrians over Tobs time steps. The trajectory of the n-th pedestrian is defined as X,, (z , Th..)
TTaxx2 where each position x = (u,', of E TR² indicates the location at time We consider the primary X1 the target pedestrian. Let Y1 (Y y?,...,, YK) denote the set of K possible future trajectories of the target pedestrian over T pred time steps, where each predicted trajectory is y = (vi, v....., w E T and y! = (u), x()) E RR2 Both and are preprocessed such that the last observed position of the primary agent at time step Tabs is shifted to the origin. The goal learn mapping function FF from the past trajectories to the of future trajectories Yi, such that Y1 = F(X).
In this study, propose leveraging in-context learning to enable trajectory prediction models to adapt to diverse scenarios without the need for updating model parameters through backpropagation. To predict Y1, we the past trajectories X in conjunction with an in-context set C: Yi C), where C (X M contains M pairs of past trajectories and the primary agent's ground-truth future trajectory. These example pairs are selected from the example pool, D=, which consists of pairs of past trajectories and their corresponding ground-truth future trajectories, all coming from the same scene as the one to which X belongs.
3.2 Spatio-temporal Similarity-based Example Selection (STES)
While random selection of in-context learning examples represents the most straightforward approach, experiments empirically demonstrate that this method leads to suboptimal performance for trajectory prediction tasks. To address this limitation, we introduce a novel approach called spatio- temporal similarity example selection (STES) that automatically identifies the most relevant examples for enhanced trajectory prediction.
The core insight driving our proposed STES method is that trajectories with similar past patterns are likely to exhibit similar future hehaviors. Building on this intuition, we propose retrieving the top- M past trajectories along with their corresponding ground-truth future trajectories based on carefully designed similarity metric. Formally, given past trajectory query X, and an example pool Dd, we identify the most relevant in-context examples
C = Top-Mx;eD!
[(X,, x1),
[C| M,
(1)

---

where C. represents the set of top-M relevant trajectories selected from the example pool D* according to the similarity metric (i),, This metric quantifies the similarity between the target agent's past trajectory and each candidate primary agent's past trajectory in the example pool.
In our introduced STES approach, define spatio-temporal similarity S(X,, a (S,(X1, x()) + (X1, Xi)). Sp and So are the spatial and temporal similarities defined as follows:
(p(X 1+ dx XD
1 1 (XI,
where .)) and (:)) represent the mean squared emrors (MSE) of position and velocity, re- spectively, capturing both spatial proximity and motion similarity between trajectories. a(.) is the normalization function that scales the similarity value within the range [-1, via min-max normal- ization across the entire set of similarities between the target agent's past trajectory and all candidate trajectories in the example pool.
3.3 Prediction-Guided Example Selection (PG-ES)
To address the suboptimal selection based solely on past trajectory, we propose prediction-guidee example selection (PG-ES), which utilizes the trajectory prediction results refine the example selection process. PG-ES consists of two steps, as shown in Figure (b). In the first step, we predict multiple future trajectories based on the past trajectory and examples selected using past trajectory similarity, as outlined in Equation (1) yielding the prediction result Y; F(X, C). In the second step, these predicted future trajectories, Yi. are used to further refine the context selection, as shown in the following equation:
C' ( XA,tEED
min S (1 (E()....,K)
IC'| M.
The similarity is computed between the concatenated past trajectory and each of the multiple predicted future trajectories of the target agent, well as the concatenated past and ground-truth future trajectory of the example. The minimum similarity value across all K predictions is then used as the selection metric. By employing this two-step selection process, can identify more relevant examples for in-context learning, taking into account both past and future trajectories.
3.4 Model Architecture
Our model consists of an embedding layer, trajectory encoder, an in-context-aware trajectory predictor, and multi-modal decoder, as presented in Figure (a). (a). Initially, the embedding layer is applied to the agent's 's past trajectory to obtain d-dimensional features. We then concatenate G(X) with learnable query tokens Q E F Tims xd and pass them through the trajectory encoders, Encoder obtain the trajectory feature (e.g.. spatio-temporal features are extracted using a Q). Similarly, we use the embedding layer and trajectory encoder to obtain the trajectory features of M in-context examples. Here, instead of employing learnable queries, we apply & to both the in-context past trajectories X and future (xx)odeer(("),)(())))) where i= M.
encoder (55)) as follows: [H' Q'] =
trajectories Y for all M examples: [H*, 2
Since the coordinates of the agents in the context examples are normalized the position of each primary agent, it is crucial to integrate the relative position information of the target agent into the features of the in-context examples. We refer to this as the relative context position encoding (RCPE). RCPE is implemented using a simple one-layer MLP as follows: RCPE = MLP( Brel), where (Frel, Mrel) represent the relative position of the primary agent of a context example with respect to the target agent. To incorporate information about which context example's 's primary agent is more similar to the target agent, we introduce similarity rank position encoding (SRPE), which encodes the similarity ranking of each context example's primary agent relative to the target agent. This is implemented using the original sinusoidal positional encoding from [11] Both RCPE and SRPE are applied to the features of the context examples. These features are then fed into the in-context-awan trajectory predictor (Predictor), which consists of three-layer Transformer encoder 61 allowing to leverage the context examples for prediction as follows:
5

---

Q1, Q1, 21................. H -2,1 Predictor[[HHi 0i 11 + RCPE(1) + SRPE(1), z - RCPE((1) + SRPE(I), H + RCPE(M) + SR) ZMM RCPE(MM) + SR))
Finally, we implement the multimodal decoder, MuliiMoaalDevoder, which consists of K simple one-layer MLPs for multimodal prediction. The decoding process can be formulated as follows:
V MultiMModallDecooe(((((
Vi E R X
3.5 Training and Inference
Our framework comprises two sequential training phases phases vanilla trajectory prediction (VTP) training and in-context folling-flllowed by inference phase.
Training. The VTP training phase equips TrajICI. with foundational trajectory prediction capabilities. Analogous to conventional trajectory prediction models, TrajICI. SrnjICI.lean learns to forecast the future trajectory of target agent based on historical path and the trajectories of surrounding agents. The subsequent in-context training phase enables TrajICL to perform effective in-context learning. During this phase, the model trains on examples where similar instances are selected using the proposed metric as detailed in Section 3.2 Both training phases utilize MSE loss, implementing winner-take-all
strategy that optimizes only the most accurate prediction: only employ a large-scale synthetic dataset to train the model.
ming (}((()) Y2 Note that we
Inference. During inference, we adopt the prediction-guided example selection from the scenario- specific samples introduced in Section 3.3 allowing the model to adapt to environmental changes and domain shifts without any additional parameter updates.
4 Experiments
4.1 Experimental Settings
Datasets. We train our model using the MOTSynth [4 dataset, a synthetic pedestrian detection and tracking dataset, consisting of over 700 90-second videos captured from various camera viewpoints in diverse outdoor environments. For our experiments, we use a subset of 424 scenes for training and 107 scenes for evaluation, all captured with a static camera. In addition to in-domain evaluation on MOTSynth, we assess our method on five widely adopted datasets for cross-domain evaluation: JRDB 45 (in both image coordinates, JRDB- Image, and world coordinates, JRDB-World), Wild- Track 8. SDD [53], and JTA [15]. It is important to note that the model is not trained on these datasets for cross-domain evaluation; only few examples are used for inference. Following standard practice, we predict 12 future timesteps based on the previous 9 timesteps. For consistency with the human seconds protocol adopted by (60), we the N identities chronologically based their initial appearance in the scene. The earliest 80% of identities are used to construct the example pool, while the remaining 20% reserved for evaluation.
Evaluation Metries. We evaluate the performance of different trajectory prediction methods using two standard metrics: minADE and minFDEx. minADEx calculates the minimum average displacement error over time among the K 20 predicted trajectories and the ground-truth future trajectory following the standard protocol [41 22 43). minFDEx measures the minimum final displacement error, computing the distance between the closest predicted endpoint among the K 20 predictions and the ground-truth endpoint.
Baselines. We combine TrajICL with the transformer-bant based model Social-Transmotin Transmotion 55 Al- though
is designed to process multiple modalities, we use only trajectory data as input in all experiments to ensure fair comparisons. In addition, following recent works 21 [8], we incorporate multiple forecasting heads to generate K possible future predictions. To verify the effectiveness of our method, we compare with versions of Docial-Transmotionn that have been fine- tuned using various methods, including full fine-tuning (Full FT), PEFT methods such as LoRA 24]

---

Table 1: Comparison with baseline methods the MOTSynth, JRDB, WildTrack, SDD, and JTA datasets. minADE /minFDEx X are reported. The unit for MOTSynth, WildTrack, and SDD is pixels, while the unit for -oB-Worldd and JTA is meters. Bold and underlined fonts represent the best and second-best results, respectively. The difference (A) represents the percentage improvement achieved by TrajICL over the vanilla
In. In-Domain
Cross- Domain
Method
Truining-fre free
MOTSyath
JRDB-Image
WiddTack
SDD
JRDB-Worl
Social Transmation
55
2,88/3.32
34.7/36.3
10.2/1899
+Head Tuning
17.1/22.6
2,70/3.13
23.9/34.7
9.81/18.3
0.TINII6
0.68/1.37
4VPT Shallow
17 87.8/33.5
2,803333
24.3/37.4
8.73/14.8
1.1
0.61/1922
Deep
17.7/23.9
24.4/37.9
8.84/15.6
0,109.15
0.60/0.0
+LoRA [r
16.9/22.2
2.64/3.02
23.8/34.5
9.02/16.6
0.10
0.61/0.93
LoRA [r 64)
16 8/2.2
2.65/2.98
23.6/35.6
9.11/16.8
0.10
0.60A193
16.0/2099
7.96/136
0.09/0.14
0.52/0.76
+TajICL (Ours)
15.3/17.5
2.61/2.68
21.1/28.3
0.1380221
0.59N0KS
A
14.29%/-23.9%
-7.6%%--99.5%
-1..6%/.22.0%%
-17.6%/221.7
3.3%%/-9..28
44 15555/96669
Table 2:
Comparisons on JRDB-World and JTA under few-shot evaluation setting. mrinADDg/minDDDDD are reported for different percentages of labeled real data available for the example pool for TrajCCL and fine-tuning for the fine-tuning methods.
Method
Traening-fre
MUT'Syuh
IRDB-Imaae
WildTrack
2055
20%
30%
1056
10%
20%
Social Transmitionn
55
17.6/33
2.88/3.32
24.7/36.3
10.2/18.9
1,18/1 97
-Hoad Tosing
17:5/230
17.4/221)
10031.50
2:34
24:57553
24.57352
10:2715.1
102/1911
0.781132
1176/1.25
-VPT Shallow 1
21. 21.3/30.2
20.5/38.7
3.514.25
2.98/.75
25.7441.
24.9/39.5
11.2/30.4
105//8.11
0.77/1.17
0,701.03
Deep
21.3/24.2
19.8/268
3.48/.166
3.16/3000
24.4235.6
24.5/37.2
10.2/18.3
10.3/18.1
0.68/100
-ToRA
17.4/33.0
1744/22.9
2.75/.34
243335.0
23.9/34.0
10.1/7.0
10/1/88
0.75/225
0.201.14
(r
17.4/2.9
17.5/22.8
23.8/31
0.201.13
17.0222
16.8218
2.783.16
10.//166
933317
0.640.99
TaaiiC[(0mmm (Ckm)
16.7200
16,4199
1353324
168294
2XA3350
22.0300
9.76/7.4
9.00163
0.62436
VPT 4 and head tuning, which adjusts the prediction heads. The data from the example pool is used for fine-tuning methods ensure that all models have access to the same information.
Implementation Details. Our training process is divided into stages: VTP training and in-context training. In the first stage, we train the model using the Adam optimizer [40] with a base learning rate of 10 for 100 epochs. We perform a 3-epoch warmup and decay the learning rate throughout training using the cosine annealing scheduler [39]. In the second stage, we train the model for 400 epochs, with a 12-epoch warmup and the cosine annealing scheduler, following the same setup as in the first stage. We set M (number of in-context examples) to eight. The hyperparameters were determined through standard coarse-to-fine grid search or step-by-step tuning. We set the batch size to 16 and train the model using one NVIDIA RTX A6000 GPU. The model configuration for Predictor consists three layers and four attention heads, with model dimension employ Leaky ReLU functions the activation function. Data augmentation techniques, including rotation, noise addition, and masking, as adapted from 18 are applied.
128. We
4.2 Comparison with Baseline Methods
Table shows that TrajICL consistently outperforms Social-Trann Transmotion by a significant margin across all datasets, highlighting its adaptability to wide range of scenarios. On the MOTSynth, IRDB-Image, WildTrack, and SDD datasets, TrajICL. exceeds the performance of the best fine- tuned methods, including full fine-tuning, in terms of minFDEx. Furthermore, the IRDB-Words and JTA datasets-where pedestrian positions are provided in real-world coordinates-Traile. remains competitive, demonstrating its generalizability across various sensor configurations. Despite being trained solely on synthetic data and without requiring any additional fine-tuning, these results showcase TrajICL's effectiveness diverse scenarios, overcoming the third challenge of adaptability to new environments.
shows
4.3 Comparison with Limited Pool Size
In Section 4.2
we validate the effectiveness of TrajICL using sufficiently large example pool. How- ever, real-world scenarios, acquiring annotations manually incurs collection costs, and even when using detection and tracking algorithms, gathering sufficient example pool takes time. Therefore,
7

---

Table 3: Ablation study of the prediction guided STES for inference. minFDEx is reported. The subscript percentage denotes relative minFDE reduction over random selection. Spatial
guidde
Temporal
Prediction-guiie guidded
MOTSynth
WildTrack
JTA
SDD
21.8
35.5
1,08
16.4
18.0 17.4%
28.9
18.6%
0.85
21.3%
15.1
7.9%
16.7 23.49%
29.4
-1.22%
0.89
-1.66
15.9
-3.0%
19.2 11.9%5
32.5
8.50%
0.91
16.7%
16.2 1.2%
17.5
28.3
20.3%
0.85
14.8
99.8%
13.10
12.00
Randen
23.00
Nuree
handen
1.30
hundem
11.50
17.00
16.50
20:00
14.00
18:00
SurtereFuurpho
Lorpla
sloarphs
(a) MOTSyath
(b) JTA
Figure 3: Performance of random example selection and the proposed STES varying numbers of in-context examples.
we evaluate the effectiveness of TrajICL with limited pool size, which is a realistic and challenging scenario in real-world applications. We selected subsets of 5% and 10% from the example pool, using each subset as the example pool in TrajICL and as the fine-tuning data for the fine-tuning methods. The models were then evaluated the same test set. As shown in Table 2 while PEFT methods demonstrate their effectiveness over full fine-tuning in the 100% example setting (Section 4.2) our method consistently outperforms both challenging scenario, with only 10% of annotations, TrajICL achieves improvements of 8.8%, 7.0%, 4.9%, and 11.9% on MOTSynth, JRDB-Image, SDD, and JTA, respectively, compared the best- performing fine-tuning model in terms of minFDE,. The results of ETH-UCY 49 30 and NBA SportVU 74 are provided in the supplementary.
and fine-tuning methods. the most
4.4 Ablation Studies
Effectiveness of STES. We first evaluate the impact of incorporating our STES into in-context learning. As shown in Figure 3 STES consistently results in improvements as the number of examples increases across different datasets. In contrast, randomly selected examples yields minimal performance gains, even as the number of examples increases in the MOTSynth dataset. In JTA, no improvement is observed despite the increase in examples. These results highlight the effectiveness of STES, which efficiently equips the model with ICL capabilities, overcoming the first limitation.
Effectiveness of the PG-STES. We conduct ablation experiments to evaluate the impact of our proposed PG-STES on inference. As shown in Table PG-STES consistently outperforms random selection in terms of minFDE across all datasets. By incorporating both the past trajectory and the predicted future trajectory for example selection, PG-STES improves over STES, which only uses past trajectory for selection, by 8.1 8.1%, 7.4%, 6.6%, and 8.6% on the MOTSynth, WildTrack, JTA, and SDD datasets in terms of minFDE respectively. These results demonstrate the effectiveness of combining predicted future trajectories with past trajectories to enhance performance, addressing the second limitation. We also investigate the impact of spatial and temporal components in the STES
Randomiy Selected Examples
Prediction
Examples Selected PG-STES
Prediction
Trejectory
Trajectory
Trajectory
Figure Qualitative comparison between random example selection and our proposed PG-STES.

---

Table 4: Ablation study of first-stage VTP training.
Table 5; Ablation study of RCPE and SRPE.
RCPE
SRPE
MOTSynth
WildTrack
JRDB World
JTA
minFDE is reporeted.
15.5/18.1
21.9/31.0
0.14/0.23
0.61/0.91
MOTSyath
WalTrack
JTA
15.4/17.6
2128/99.9
0,14/0.24
0.59/0.88
15.4/17.6 15.3/17.5
21,9304
0.60/0.89
15.3/17.6
21/8/29.6
0,14/0.23
0.59/0.89
21.128.3
0.59/0.85
15.3/17.5
21.1 /28.3
0.13/0.21
0.59/0.85
Past Trajectory
GT Future Trajectory
Social-Transmotion
TrajICL (Ours)
Figure 5: Qualitative results on MotSynth, JRDB, WildTrack, and SDD. These examples demonstrate scenarios where our TrajICL outperforms the Social- TranismTraoooooonn baseline. TrajICL effectively learns the plausible motion patterns from examples.
example selection process. The results indicate that temporal similarity yields greater improvements than spatial similarity on the JTA and SDD datasets. However, on the MOTSynth dataset, spatial similarity provides more substantial gains. This suggests that the optimal selection dimension for demonstration examples is dataset-dependent When both spatial and temporal similarities are utilized together, our method achieves the best performance across most datasets, particularly when dnffiction-uidder retrieval applied.
Effectiveness of VTP Training, Our experiments demonstrate that VTP training consistently delivers performance improvements across multiple datasets, as shown Table
Effectiveness of RCPE and SRPE. We investigate the impact of RCPE, which encodes the relative position with respect to the target agent, and SRPE, which encodes the similarity rank of the context examples' primary agents in relation to the target agent. As shown in Table [ our ablation study demonstrates that both RCPE and SRPE contribute improved performance across various datasets. On the in-domain MOTSynth dataset, RCPE alone achieves slight improvement in performance. However, on the cross-dee doosan datasets, WildTrack and JRDB- World, the combination of both RCPE and SRPE results in more significant improvements. Further ablation studies and analysis are available in the supplementary material.
Effect of Architecture. Even a weak base model with a shallow encoder and small dimensions exhibits ICI. capabilities, shown in Tables 6 and While deeper encoder and larger dimensions offer negligible benefits for the zero-shot case, they significantly boost performance in ICI.
Table 6: Effect of model dimension MotSynth. minFDEx is reporeted.
Table 7: Effect of pre- dictor depth MotSynth. minFDE is reporeted.
Table Performance of Tra- jICL with ForecastMAE backbone. minFDE is reporeted.
Dim. zero-shot
8-shots
Depth zero-shot
8-shots
MotSynth
JTA
32
19.2
17.7
19.1
16.6
ForecastMAE +TaiICLL (Ours)
18.4
0.76
64
17.9
16.5
18.8
16.9
12.5%
-005%
128
18.1
15.3
3
18.1
15.3

---

Result of Different Backbone. We selected Social- Tianslm-rnnooooonn our backbone due to its state-of- the-art performance and generalized architecture. Our TrajICL framework is then integrated with this backbone to enable ICI. capabilities. To confirm the framework's s generalizability, we replaced the backbone with ForecastMAF, [9] (disabling its lane encoder for pedestrian prediction). The results, summarized in Table verify that our framework is generalizable and can be successfully applied to different backbones.
4.5 Qualitative Results
We compare randomly selected examples with those chosen by our PG-STES in Figure f along with their prediction results. PG-STES effectively selects spatially and temporally similar examples, enabling our model to generate more plausible predictions, such as a pedestrian riding down an esca- lator, with better understanding of 3D structures compared to random selection. Figure 5 highlights the qualitative results of TrajICL and Tranima-Tansootionn across various datasets, showcasing our method's s adaptability in predicting future trajectories across domains. Unlike the baseline, which often predicts pedestrians floating in the air, our model aligns closely with the ground truth, even non-planar surfaces like stairs. Furthermore, map awareness, avoiding obstacles like trees and respecting constraints (e.g., not crossing fences), while capturing behavioral trends such walking sidewalks instead of roads.
approach incorporates finer-grained
Conclusion
In this paper, we introduce TrajICL, a novel in-context learning (ICL) framework for pedestrian trajectory prediction that enables adaptation without the need for fine-tuning on the domain -epecii data. We address the challenges of incorporating ICL into trajectory prediction by employing spatio- temporal similarity- example selection, prediction-guided example selection, and leveraging large-scale synthetic trajectory dataset. In our experiments, we thoroughly validate that our approach effectively adapts environmental variations and domain shifts. Despite these promising results, there remains work to be done. While increasing the number of in-context examples improves accuracy, it also raises computational costs during inference. We plan to explore this further in our future work.
References
[1] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei- Fei, and Silvio Savarese. Social LSTM: Human Trajectory Prediction in Crowded Spaces. In CVPR, 2016.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikot aj Binkkossii, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: Visual Language Model for Few-Shot Learning. In NewrlPS, 2022.
[3] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L. Yuille, Trevor Darrell, Jitendra Malik, and Alexei A. Efros. Sequential Modeling Enables Scalable Learning for Large Vision Models, In CVPR, 2024.
[4] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. NeurIPS, 2022.
[5] Julian Bock, Robert Krajewski, Tobias Moers, Steffen Runde, Lennart Vater, and Lutz Eckstein. The inD Dataset: A Drone Dataset of Naturalistic Road User Trajectories German Intersections. In /V, 2020.
[6] Daniel Bolya, Cheng Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffmann. Token Merging: Your VIT But Faster. In ICLR, 2023.
[7] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are sho- leamers. NewrIPS, 2020.
10

---

[8] Tatjana Chavdarova, Pierre Baqué, Stéphane Bouquet. Andrii Maksai, Cijo Jose, Timur Bagautdinov. Louis Lettry, Pascal Fua, Luc Van Gool, and François Fleuret. WILDTRACK: A Multi-Camera HDD Dataset for Dense Unscripted Pedestrian Detection. In CVPR, 2018
[9] Jie Cheng, Xiaodong Mei, and Ming Liu. Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders. In ICCV, 2023.
[10] Yujiao Cheng, Weiye Zhao, Changliu Liu, and Masayoshi Tomizuka. Human Motion Prediction using Semi-adaptable Neural Networks. In ACC, 2019.
[11] Henggang Cui, Vladan Radosavljevic, Fang-Chich Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang. Jeff Schneider, and Nemanja Djurie. Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks. In ICRA, 2019.
[12] Yonghao Dong, Le Wang, Sanping Zhou, Gang Hua, and Changyin Sun. Recurrent Aligned Network for Generalized Pedestrian Trajectory Prediction. IEEE TCSVT, 35(3):2188-2201. 2025.
[13] Omar Elharrouss, Noor Almaadeed, and Somaya Al-Maadeed. A Review of Video Surveillance Systems. JVCIR, 77:103116, 2021.
[14] Matteo Fabbri, Guillem Brasó, Gianluca Maugeri, Orcun Cetintas, Riccardo Gasparini, Aljoša OSep. Simone Calderara, Laura Teax- and Rita Cucchiara. MOTSynth: How Can Synthetic Data Help Pedestrian Detection and Tracking? In ICCV, 2021.
[15] Matteo Fabbri. Fabio Lanzi, Simone Calderara, Andrea Palazzi. Roberto Vezzani, and Rita Cucchiara. Learning to Detect and Track Visible and Occluded Body Joints in Virtual World. In ECCV, 2018.
[16] Zhonghin Fang, Xiangtai Li, Xia Li, Joachim M Buhmann, Chen Change Loy, and Mengyuan Liu. Explore In-Context Learning for 3D Point Cloud Understanding. NeurlPS, 2024.
[17] Amalia Foka and Panos Trahanias. Probabilistic Autonomous Robot Navigation in Dynamic Environments with Human Motion Prediction. IJSR, 2:79 94, 2010.
[18] Ryo Fujii, Hideo Saito, and Ryo Hachiuma. RealTraj: Towards Real-World Pedestrian Trajectory Forecast- ing, arXiv preprint 1..iv:2411.1737777 2024.
[19] Ryo Fujii, Jayakorn Vongkulhhisal, Ryo Hachiuma, and Hideo Saito. A Two-Block RNN-Based Trajectory Prediction From Incomplete Trajectory. IEEE Access, 9:56140-5611 -61511, 2021.
[20] Jun Gao, Qian Qiao, Tianxiang Wo. Zili Wang, Ziqiang Cao, and Wenjie Li. AIM: Let Any Multimodal Large Language Models Embrace Efficient In Contexxt Learning, 2025.
[21] Yang Gao, Po-Chien Luan, and Alexandre Alahi. Tulii-Transsotion: Pre-trained Model for Human Motion Prediction. In CoRL, 2024.
[22] Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion. In CVPR, 2022.
[23] Agrim Gupta, Justin Johnson, Li Fei-Fei. Silvio Savarese, and Alexandre Alahi. Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks. In CVPR, 2018
[24] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 2022.
[25] Boris Ivanovic, James Harrison, and Marco Pavone. Expanding the Deployment Envelope of Behavior Prodiction via Adaptive Meta-L.carning. In ICRA, 2023.
[26] Nikolay Jetchev and Marc Toussaint. Trajectory Prediction: Learning to Map Situations to Robot Trajecto- ries. In ICMI, 2009.
[27] Kangsan Kim, Geon Park, Youngwan Lee, Woongyeong Yeo, and Sung Ju Hwang. VideoICL: Confidence- based Iterative In-context Learning for Diito---iiiibbttion Video Understanding. In CVPR, 2025.
[28] Vincet Kosaraju, Amir Sadeghian, Roberto Hartin-Martin, Ian Reid, S. Hamid Rezatofighi, and Silvio Savarese. Social-BiGAT: Multimodal Trajectory Forecasting Using Bicycle-GAN and Graph Attention Networks. In NewrlPS, 2019.
[29] Zhengxing Lan, Lingshan Liu, Bo Fan, Yisheng Lv. Yilong Ren, and Zhiyong Cui. Traj-LLM: A New Exploration for Empowering Trajectory Prediction With Trainre Large Language Models. T-IV. 10(2):794 (122)794-807,2200 2024.
11

---

[30] Laura Leal-Taixé, Michele Fenzi, Alina Kuznetsova, Bodo Rosenhahn, and Silvio Savarese. Learning an Image-Basee Motion Context for Multiple People Tracking, In CVPR, 2014.
[31] Seung Hyun Lee. Jijun Jiang, Yiran Xu, Zhuofang Li. Junjie Ke, Yinxiao Li, Junfeng He, Steven Hickson, Katie Datsenko, Sangpil Kim, Ming-Hsuan Yang, Irfan Essa, and Feng Yang. Cropper: Vision-Language Model for Image Cropping through In-Context Learning. In CVPR, 2025.
[32] Junlong Li, Jinyuan Wang, Zhuosheng Zhang, and Hai Zhao. Self-Prompting Large Language Models for Zero-Shot Open-Domain QA. In NAACI, 2024.
[33] Maosen Li, Siheng Chen, Yanning Shen, Genjia Liu, Ivor W Tsang, and Ya Zhang. Online Multi-Agent Forecasting With Interpretable Collaborative Graph Neural Networks. TNNLS, 35(4):4768 4782, 2024.
[34] Xiaohe Li, Feilong Huang, Zide Fan, Fangli Mou, Yingyan Hou, Chen Qian, and Lijie Wen. MetaTra: Meta-Learning for Generalized Trajectory Prediction in Unseen Domain, 2024.
[35]
Yiheng Li, Seth Z. Zhao, Chenfeng Xu, Chen Tang, Chenran Li, Mingyu Ding, Masayoshi Tomizuka, and Wei Zhan. Pre- tre-training on Synthetic Driving Data for Trajectory Prediction. In IROS, 2024.
[36] Haoyu Liu, Jianfeng Liu, Shaohan Huang, Yuefeng Zhan, Hao Sun, Weiwei Deng, Furu Wei, and Qi Zhang. Sea: Sequential Example Selection for In-Context Learning. In ACL, 2024.
[37] Jiachang Liu. Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What Makes Good In-Context Examples for GPT-3? In DeeLIO, 2022.
[38] Jiawei Liu, Yanjiao Liu, Xun Gong, Tingting Wang, Hong Chen, and Yunfeng Hu. Harnessing and Evaluating the Intrinsic Extrapolation Ability Large Language Models for Vehicle Trajectory Prediction. In NAACI,, 2025.
[39] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts. ICLR, 2017.
[40] Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In ICLR, 2019.
[41] Karttikeya Mangalam, Yang An, Harshayu Girase, and Jitendra Malik. From Goals, Waypoints & Paths to Long Term Human Trajectory Forecasting. In ICCV, 2021.
[42] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It Is Not the Journey But the Destination: Endpoint Conditioned Trajectory Prediction. In ECCV, 2020.
[43] Weibo Mao, Chenxin Xu, Qi Zhu, Siheng Chen, and Yanfeng Wang. Leapfrog Diffusion Model for Stochastic Trajectory Prediction. In CVPR, 2023.
[44] Francesco Marchetti, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo. MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction. In CVPR, 2020.
[45] Roberto Martin-Martin Mihir Patel, Hamid Rezatofighi, Abbiject Shenoi, Jun Young Gwak, Eric Frankel, Amir Sadeghian, and Silvio Savarese. JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments. TPAMI, 45(6))6788-6665, 2021.
[46] Anton Milan, Laura Leal-Taixé, Ian D. Reid, Stefan Roth, and Konrad Schindler. MOTI6: A Benchmark for Multi-Object Tracking. CoRR, 2016.
[47] Sewon Min, Xinxi Lyu. Ari Holtzman. Mikel Artetxe, Mike Lewis. Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In-Contewe learingg Work? In EMNLP, 2022.
[48] Dachee Park, Jaeseok Jeong, Sung-Hoon Yoon, Jaewoo Jeong, and Kuk-Jin Yoon. T4P: Test- Training of Trajectory Prediction via Masked Autoencoder and Actor-specific Token Memory. In CVPR, 2024.
[49] Stefano Pellegrini, Andreas Ess, and Luc Van Gool. Improving Data Association by Joint Modeling of Podestrian Trajectories and Groupings. In ECCV, 2010.
[50] Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingho Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, et al. UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild. NeurIPS, 2023.
[5]] Chengwei Qin. Aston Zhang, Chen Chen, Anirudh Dagar, and Wenming Ye. In- In-Contex Learning with Iterative Demonstration Selection. In EMNLP, 2024.
12

---

[52] Shaoqing Ren, Kaiming He. Ross Girshick. and Jian Sun. Faster R-CNN: Towards Real-Tine Object Detection with Region Proposal Networks. In NewrIPS, 2015.
[53] Alexandre Rohiequet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes. In ECCV, 2016.
[54] Ohad Jonathan Herzig, and Jonathan Berant. Learning To Retrieve Prompts for In-Context Learning. In NAACL, 2022.
[55] Saeed Saadatnejad, Yang Gao, Kaouther Messaoud, and Alexandre Alahi. Trans--rrnsmooion:: Promptable Human Trajectory Prediction. In ICLR, 2024.
[56] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++4: Dynamically- Feasible Trajectory Forecasting with Heterogeneous Data. ECCV, 2020.
[57] Feifei Shao, Ping Liu, Zhao Wang, Yawei Luo, Hongwei Wang, and Jun Xiao. MICAS: Multi-grained
In In-Context Adaptive Sampling for 3D Point Cloud Processing. In CVPR, 2025.
[58] Jianhua Sun, Yuxuan Li, Hao Shu Fang, and Cewu Lu. Three Steps to Multimodal Trajectory Prediction: Modality Clustering. Classification and Synthesis. In ICCV, 2021.
[59] Hiromu Taketsugu, Takeru Oba, Takahiro Maeda, Shohei Nobuhara, and Norimichi Ukita. Physical Plausibility-wware Trajectory Prediction via Locomotion Embodiment. In CVPR, 2025.
[60] Neerja Thakkar, Karttikeya Mangalam, Andrea Bajesy, and Jitendra Malik. Adaptive Human Trajectory Prediction via Latent Corridors. In ECCV, 2024.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In NewrIPS, 2017.
[62]
Chaoyang Wang, Xiangtai Li, Henghui Ding, Lu Qi, Jiangning Zhang. Yunhai Tong, Chen Change Loy, and Shuicheng Yan. Explore In In-Context Segmentation via Latent Diffusion Models. In AAAI, 2024.
[63] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images Speak in Images: A Generalist Painter for In-Context Visual Learning. In CVPR, 2023.
[64] Xinlong Wang. Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. SegGPT: Towards Segmenting Everything In Context. In ICCV, 2023.
[65] Xinshun Wang. Zhongbin Fang, Xia Li, Xiangtai Li, Chen Chen, and Mengyuan Liu. Skeleton-in-Contxt: Unified Skeleton Sequence Modeling with In-Context Learning. In CVPR, 2024.
[66] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with deep association metric. In ICIP, 2017.
[67] Zhenyu Wu, Yaoxiang Wang. Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Jingjing Xu, and Yu Qiao. OpenICL: An OpeneSource Framework for In-context Learning. In ACL, 2023.
[68] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An Explanation of In-context Learning as Implicit Bayesian Inference. ICLR, 2022.
[69] Chenxin Xu, Weiho Mao, Wenjun Zhang, and Siheng Chen. Remember Intentions: Retrospective-Memoyy based Trajectory Prediction. In CVPR, 2022.
[70] Chenxin Xu, Robby Tan, Yuhong Tan, Siheng Chen, Yu Guang Wang. Xinchao Wang, and Yanfeng Wang. EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning. In CVPR, 2023.
[71] Pei Xu, Jean-Bernard Hayet, and loannis Karamouzas. SocialVAE: Human Trajectory Prediction using Timewise Latents. In ECCV, 2022.
[72] Yi Xu, Armin Bazarjani, Hyung-gun Chi, Chiho Choi, and Yun Fu, Uncovering the Missing Pattern: Unified Framework Towards Trajectory Imputation and Prediction. In CVPR, 2023.
[73] Yi Xu, Lichen Wang, Yizhou Wang, and Yun Fu. Adaptive Trajectory Prediction via Transferable GNN. In CVPR, 2022.
[74] Yisong Yue, Patrick Lucey, Peter Car,, Alina Bialkowski, and lain Matthews. Learning Fine-Grained Spatial Models for Dynamic Sports Play Prediction. In IEEE ICDM, 2014.
13

---

[75] Eric Zhan, Stephan Zheng. Yisong Yue, Long Sha, and Patrick Lucey. Generating Multi-Agent Trajectories using Programmatic Weak Supervision. In ICLR, 2019.
[76] Jiahao Zhang. Bowen Wang, Liangzhi Li, Yuta Nakashima, and Hajime Nagahara. Instruct Me More! Random Prompting for Visual In-Context Learning. In WACV, 2024.
[77] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What Makes Good Examples for Visual In-Context Learning? In NeurlPS, 2023.
[78] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic Chain of Thought Prompting in Large Language Models. In ICLR, 2023.
[79] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang. Wenjuan Han, and Baobao Chang. MMICL: Empowering Vision-language Model with Multi-Modal In Context Learning. ICLR, 2024.
[80] Zhen Zhao, Jingqun Tang, Chunhui Lin, Binghong Wu, Can Huang, Hao Liu, Xin Tan, Zhizhong Zhang, and Yuan Xie. Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer. In CVPR, 2024.
[81] Bolei Zhou, Xiaogang Wang, and Xiaoou Tang. Understanding collective crowd behaviors: Learning Mixture model of Dynamic Aaaeestrian-Agente In CVPR, 2012.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
the paper introduces the In-Context Learning framework for pedestrian jectory prediction, which facilitates adaptation without requiring fine-tuning. This claim is clearly stated in the abstract.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the limitations are discussed in the Conclusion section in the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification: this paper is primarily an application-focused work without significant theoret- ical contributions. It leverages the established theoretical framework of In-Context Learning without introducing any theoretical extensions. As such, no new assumptions or proofs are presented.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: the paper provides all necessary details to reproduce the methods and results presented, in the main text.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
we will open source the code and models.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
public/guides/Coeeee wtwpess/ooooeSuumiiiiiiineneeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips Siiaps.cc/publie wwtppbbiccgguueeeeeoooooocppcped
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Due page length constraints, we provide detailed explanation of the training process in the Implementation Details of Experimental Settings section in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: make the consistent benchmarking practice, we do not report the variance of the evaluation results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: we provide details about the computational resources used for the experiments in the Implementation Details section of the Experiment part.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: we have read the code of ethics and conform with it during the work of this paper.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: there is no societal impact associated with this work. While we use human trajectory data, private sensitive information are utilized.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: this paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
codebase, which is licensed under the GNU Affero GPLv3. We evaluate our method on the MOTSynth, JRDB, Wild'Track, JTA, ETH/UCY, and SDD datasets. MOTSynth, JRDB, and SDD datasets are licensed under CC BY-NC-SA 3.0, the JTA dataset licensed under CC BY-NC 4.0, and the ETH/UCY dataset is licensed under CC BY-SA 4.0.
Justification: our code is partially based on the
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thoode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: we use the MIT License for our code and provide detailed instructions on how reproduce the experimental results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing or research involving human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing or research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: the core methodology and development of this research do not involve the use of LLMss important, original, non-standard component.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neuri)) hutte:://eearrrsscc/oooofereerr ccC///ofeeeeeeeee/222///... for what should or should not be described.
21

---

A Appendix
A.1 Implementation Detail
MOTSynth and WildTrack datasets are processed 2.0 FPS, the 9-step observed past corresponds to 4.5 seconds, and the 12-step predicted future to 6.0 seconds. JRDB, JTA, and SDD datasets are processed 2.5 FPS; these durations are 3.6 seconds for the past and 4.8 seconds for the future, respectively.
A.2 Experiments Additional Datasets
We also provide the results on ETH-UCY 49 30 and NBA SportVU 77]. For NBA SportVU, we extracted two sub-setasets to use as benchmarks, named Rebounding and Scoring subsets, following the previous methods [71 12] Table 9 presents a performance comparison on the ETH-UCY. Across all subsets of ETH-UCY, TrajICL consistently surpasses Seeiial-Tansmotion by a notable margin and achieves competitive performance compared to fine-tuning methods, demonstrating its strong adaptability and robustness in diverse scenarios. Furthermore, we present a performance comparison on the challenging NBA SportVU dataset (74) in Table 10 When the full 100% data pool is available, TrajICL outperforms Social-Transing Transmotion in both subsets. However, on the rebounding subset, Social- Transmotion achieves better performance over TrajICL in terms of minFDE, when only 10% of the pool available. The reason that the original scene segments in the NBA dataset are short, so the default pool size is already small. Maintaining strong performance even with a small data pool is crucial, and we plan to explore this further in our future work.
Table 9: Comparison with baseline methods on the ETH-UCY 49 30].
are reported. The unit is meters. Bold and underlined fonts represent the best and second-best results, respectively. The difference (A) represents the percentage improvement achieved by TrajICL over the vanilla Docial-Transmotionn
Training-fre
HOTEL
UNIV
Social- Transmotion
53
0:42/0.79
0.11/0.19
0.33/0.59
0,30/0.58
0,26/0.46
0,28/0.52
+Head Tuning
0.4660.85
0.17AL71
0,30N1.33
0,25V1147
0,230142
0,23/0.52
+VPT Shallow
0,4380882 82
0.11/0177
0.22/0.46
0.21/0.40
0.24/0.43 43
+VPT Doop
+LaRA ir 16
24
X
0.38/0.78
0:46/0.78
0.09/0.13
0.09/0.13
0.24K0.00
0.25/0.44
0.19/0.36
0.19/0.35
0.17/0.30
0.17/0.29
0,21// 39
0,23/0.40
#LaRA
0.4481.78
0.09NL13
0,25/0.43
0.21V0.37
0.17/1.31
0.23/140
+Full FT
0.364066
0.0900133
0.22/0.39
0.19/0.37
0.18/0.32
0.21/0.37
+TajICL (Ours)
0.344066
O. 0,101113
0,28/0.48
0,21//040
0,24/0.40
0,23/0.41
19.00%/-9.118
-15.199/18866%
-36.7%/-31.0%
-7.7%/f13.0%
-17.9%/21.28
Table 10: Comparisons NBA SPr minADE is reported for different percentages of labeled real data available for the example pool for TrajICL.
Rebounding Subset
Scoring Subset
Method
Training-fres free
10%
100%
10%
100%
Toniiml-ransmotionn + TrajICI. (Ours)
55
1.03/1.57
1.02/1.91
0.93/1.61
0.90/1.35
0.98/1.78 0.94/1.73
A.3 Comparison with Adapting Trajectory Prediction Methods
Most existing adaptive trajectory prediction methods exhibit tight coupling between the adaptation module and the prediction model architecture (60 12 34] Moreover, these methods often employ varied training settings, such as different datasets and target domains, which complicates fair, direct comparison. The evaluations in these prior works are also typically limited to baselines or commonly used prediction models. To establish direct comparison, our main experiments include the conceptual methodology from a recent adaptation paper [60] (see Table in the main paper, "+VPT Shallow"). This was achieved by integrating their visual prompt tuning approach into our TransooTra base model. For additional context, the performance of other recent
adaptive methods [12 34] the same evaluation datasets is Table II
The results indicate that our
22

---

Table 11: Comparison with adaptive trajectory prediction methods. minADEx is reported.
Table 12: Different prompt selection meth- ods. minFDEx reported.
Model
SDD
NBA (Rebounding)
Selection Method
SDD
MotSynth
Latent Corridors 660
8.73
Random 20
79
16.6
21.8
RAN [12
11.0
1.28
Feature Sim. 2.
16.0
19.2
MetaTTra 34
10.1
Motion Sim.
65
19.2
32.5
Ours
8.40
0.92
PG-STES (Ours)
14.8
17.5
approach outperforms these methods. However, we note that direct comparison is challenging due to significant differences in their underlying model architectures and training datasets.
A.4 Detailed Ablation Study
Effectiveness of the PG-STES, We compare PG-STES with in-context sample selection strategies from other computer vision fields Table [2 The results confirm that our selection strategy achieves the best performance the trajectory prediction task.
Effectiveness of the STES, Figure demonstrates the impact of integrating our STES into ICL on the WildTrack SDD, JRDB-World, and JRDB-Image datasets. The inclusion of in-context examples consistently improves accuracy across all datasets, underscoring the effectiveness of STES in efficiently enhancing the model's ICI capability. Furthermore, as shown in Figure 7 we validate the effectiveness of STES with larger number of examples. While STES continues to yield improvements the number of examples increases across different datasets, the accuracy gains tend to saturate beyond certain point.
34.0
28.50
20:0
22.0
12
5184
37.00
21.0
176
01.01
(a) WildTrack
(b) SDD
R24
330
6:14
loncan
320
Z02x
6.2
622
280
Navoer
Sumbe Lamphs
JRDB
(b) JRDB-Image
Figure 6: Performance of random example selection and the proposed STES at varying numbers of in-context examples.
Kilenn
Kaven
2.0
2.32
3.1
3.2
Narda: scarglo
Marroi Guargic
Harder korple
MOTSynth
(b) JTA
Figure 7: Performance of random example selection and the proposed STES at varying numbers of in-context examples with larger number of examples.
Effect of Pool Size. We next investigate the impact of varying the size of the in-context pool. As shown in Figure experiments on MotSynth reveal that increasing the number of examples
23

---

in the in-context pool leads to improved performance. TrajICL consistently outperforms all fine- tuning methods on MotSynth. On the other hand, for JTA, when the in-context pool is small, TrajICI. surpasses the fine-tuning methods by effectively preventing overfitting, as it does not require additional parameter updates. However, as the pool size increases, full fine-tuning begins to achieve better results on JTA.
17.
TeE
Tunke
Toact
0:73
LotA
1.55
LoBi 24
15.
52
5.
0.2
0.8
1.0
0.6
Pate
(a) MOTSynth
(b) JTA
Figure 8: Performance change brought by different sizes of the in-context pool.
A.5 Analysis
Adaptation and Inference Cost. Our primary motivation is to eliminate the fine-tuning and back- propagation costs on edge devices. The backpropagation process requires significantly more GPU memory than inference, and our approach also mitigates the burdens of model management and unvironment-specif data collection. Therefore, in Table [3] we compare the base model with our method's 's inference cost. Our required GPU memory is significantly less than that needed for fine-tuning the base model, which aligns with our paper's primary motivation. While optimizing inference cost is not the main focus of our work, we report these figures for completeness in Table 14, We follow the experimental setup of existing ICL approaches (80), where features for in-context samples are pre-computed. Our overall inference cost is higher due to the additional Predictor module for aggregating in-context sample features and the two-stage inference via PG-ES. As mentioned in the conclusion, reducing the inference cost is a current limitation for edge-device deployment. We believe future work could explore existing techniques, such as Token Merging (6), to improve inference speed without requiring architectural modifications. The inference cost was computed on a machine with an Intel Xeon W-3235 CPU, 128GB of RAM, and an NVIDIA Titan RTX GPU, with GPU memory measured using batch size of one.
cost of the
Table 13: Comparison of adaptation cost.
Table 14: Comparison of inference time.
Model
FLOPs (GHLOPS)
Memory (MB)
Mede
FLOP, (GRLOPS)
Punero (milion)
Total Infermnee line (ms)
Social- Transmotion Fine-tuming
Social Trassntion
348
3
6.1
+TrajiCL
Tbjiis.
454
7,0 a Intricall
Table 15: Robustness evaluation against real- world errors on Image-mage using inputs from the upstream detector and tracker perception modules.
Table 16: Performance comparison of selection strategies with short observation trajectories on MOTSynth.
Midell
9 Timsesapp
Timooop
Timeeap
Model
Input
minADEso
minF-DEa
Social Trnmmmtioon
Solection
17,673
3/1/
Draial-Transooooon
2.88
3.32
Tea.CI. PC-ES Sekasiin
TraCCL STES Selectin
15,347.5
15.67992
16/W20.7
IKIVIS.
17,9229
16,4/993
Social-Transmodinn
Off-the-shelf
3.25..19.9
4.35 835
.14.28//23.95
30.495 34.3%
-32.5%
-TrajICL (Ours)
GT
2.61
2.68
-TrajICL (Ours)
Off-the-shelf
2.62....4%
2.79
Does ICL Offer Robustness to Real-World Perception Errors? While improving robustness to
trajectory noise is separate line of research [18 (2), relying on clean, ground-truth trajectories not
feasible in real-world settings. Therefore, we conducted an additional experiment using trajectories generated by off-the-shelf models (Faster R-CNN (52) for person detection and Deep-SORT (66) for tracking, pretrained on the MOTI7 dataset 46) on the JRDB Image dataset. The results are summarized Table [5] The "Input" column specifies whether trajectories are ground-truth (GT)
or predicted by off-the-shelf models. The percentage value (+X%) indicates the performance
24

---

degradation when using predicted trajectories instead of GT ones. This setup simulates fully automated pipeline that eliminates the need for manual annotation. The results demonstrate that while the baseline model's performance severely degrades noisy inputs (e.g., 30.4% increase in minFDEg), our model maintains robustness, with only a 4.1% degradation in minFDEK.
Does ICL Performance Hold with Short Observation Trajectories? We conducted an additional experiment using fewer past timesteps evaluate performance short observation trajectories. As summarized in Table T6 while absolute performance degrades with shorter inputs, our proposed selection strategies (especially PG-ES) still significantly outperform the random baseline. For instance, with only 3 timesteps, PG-ES reduces the prediction error from 25.7 (Random) to 19.3, demonstrating the effectiveness of our approach even with limited input data.
How Does the Iterative Application of PG-ES Affect Trajectory Prediction Performance? Better performance be achieved by applying the method multiple times (at least twice), shown in T7] However, as also mentioned, this increases the computational cost, since each iteration involves both sample retrieval and the feed-forward computation of the Predictor.
Table 17: Performance improvement via iterative application of PG-ES.
Iteration Times
3
4
minFDEx
17.5
16.6
16.7
16.7
What is the Relationship between Initial Prediction Error and the Effectiveness of the PG-ES Method? To better understand the conditions under which PG-ES provides the most benefit, analyze its performance gain relative to the initial prediction error. We measure the improvement of PG-ES over the STES the improvement in minADEx in Figure ? The results demonstrate PG-ES is most impactful on trajectories with high initial errors because such cases typically represent am- biguous, multimodal scenarios where single past leads to multiple plausible futures. By generating a diverse set of hypotheses, the model can then correct its initial, erroncous prediction by selecting the most viable outcome, This correction process naturally yields the most significant improvements for the most challenging predictions.
Second Step minADEzo Improvement PG-ES
2.0
1.5
1.0
minADEs
0.5
0.0
-0.5
5
First Step minADEz0
9.5
0.6
0.7
0.8
0.9
1.0
Simlarky Score
Figure 9: Impact of initial prediction error on the efficacy of PG-ES.
Figure 10: Effect of in-context similarity on per- formance
Effect of Pool Quality on Prediction Accuracy. To analyze the impact of in-context pool quality on prediction accuracy, we measured the similarity between target trajectory and the examples in its pool using the STES metric. A low maximum similarity indicates that the pool lacks relevant examples for the target, as shown in Figure TO The results show that performance degrades when the target trajectory has low similarity to the pool examples, which often corresponds to out-of-distributin (OoD) motion patterns. When we categorized target trajectories into three levels based on this similarity (low, medium, and high) in terms of minADEx. comparison, our TrajICI. framework performed
achieved respective
scores of 37.0/12.1/3 1/5.444 (low/medium/high). consistently better, achieving scores of 00.33/10.0/4.69 Notably, even for the most challenging
25

---

Past Trajectory
GT Future Trajectory Transi-Trnn
TrajICL (Ours)
Figure 11: Qualitative results MotSynth, JRDB, Wild'rrack, and SDD.
OoD inputs in the low-similarity group, our TrajICL framework shows significant performance improvement the baseline.
What is the Key Component for Effective Generalization to Real-Worke Data? Although our synthetic training data has limited motion diversity compared to real-world datasets, our model generalizes effectively to challenging benchmarks like WildTrack, JRDB, and SDD. This strong generalization is primarily due on the synthetically trained model, PG-STES dynamically selects examples relevant to the target trajectory from pool of test data. This process provides the model with in-domain knowledge of the current scene, effectively bridging the synthetic-to-real domain gap. This approach is validated in Table 3 of main paper, where using randomly selected in-context samples fails improve performance on the J'TA dataset. To further quantify the effectiveness of our selection strategy, we measured the Fréchet Inception Distance (FID) between the feature distributions of selected in-context examples and the synthetic training data. Our PG-STES method achieves an FID score of 612, stark contrast the 2, 454, 212 score from random selection. This substantially lower FID score demonstrates that our strategy is highly effective. By providing the model with the most pertinent real-world examples at inference time, PG-STES enables robust generalization and overcomes the limitations of synthetic training.
PG-STES module. During inference, rather than relying solely
A.6 More Qualitative Results
We present further qualitative comparisons of TrajICL and TrassooTon the MOTSynth, JRDB-Image WildTrack, and SDD datasets in Figure [T] Compared to the baseline, our model demonstrates closer alignment with the ground truth by incorporating finer- grained map awareness and effectively avoiding obstacles.
26


Reviews:
[{"review_id": "QEbGDm6dlk", "reviewer": "Reviewer_xGzb", "summary": "This paper presents an In-Context Learning framework for pedestrian trajectory prediction.  To address adaptability of prediction in different domains, they introduce a method without fine-tuning. A spatio-temporal similarity-based example selection and a further refine strategy are proposed. They also use synthetic dataset for training to simulate diverse outdoor environments.\nExperiments on multiple datasets are conducted to verify its effectiveness. Both in-domain and cross-domain settings are evaluated.", "questions": "1)\tThe computational complexity and efficiency are not discussed. \n\n2)\tSome implementation details to be complemented;\n\n3)\tMore discussions or explanations on the few-shot comparison;\n\n4)\tDependence of PG-ES on the quality of the trajectory predictor in the first step need be analyzed.", "limitations": "The limitation analysis is not sufficient. The computational complexity and efficiency are not discussed. The comparison with other prediction models considering adaptability seems incomplete.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1）\tIt states clearly on motivation and problems of existing methods.\n\n2）\tThe idea of using In-Context Learning for adaptability is interesting in pedestrian trajectory prediction.\n\n3）\tAblation studies on the two designed modules are provided, i.e., similarity-based example selection module and prediction-guided refinement.\n\n4）\tFigures are clearly displayed including visualization of qualitative results.\n\nWeakness:\n\n--In Equa(3), K appears for the first time and k is undefined. It can be declared at the beginning such as in the Problem Formulation section that the study focuses on multi-modal or stochastic prediction.\n\nUnclear implementation details\n--in line 194-195 “PredictionHead, which consists of K simple one-layer MLPs for multimodal prediction”. In line 244 “Predictor consists of three layers and four attention”. These two statements are inconsistent. What’s the exact prediction head’s configuration? Why use four attention modules? Maybe it can be added in supplementary material.\n\n--Different setting:\nIt comprises two sequential training phases—vanilla trajectory prediction training and in-context training. Most existing trajectory prediction methods do not use the two-stage paradigm. Though some methods consider adaptability，they adopt one-stage framework.  It is said the proposed TrajICL “offers greater efficiency”. The computational efficiency is suggested to be reported and compared.\n\n-- From Table2, when example pool size decreases from 100% (in Table 1) to 10% and 20%, prediction performances degenerate differently on different datasets. The degeneration percentage (not reported in Table 2) is greater on SDD.   I wonder its few-shot performance on the more challenging NBA SportVU Dataset. Whether TrajICL is affected by the extent of distribution divergence between training and inferring datasets. \n\n\n--it is unclear whether other trajectory prediction methods considering adaptability through transfer learning, online adaption, et al. are comparable.\n\n\t\n--The number of trainable parameters is suggested to be compared in Table 1, so it can better support the claim TrajICL is lighter and offers greater efficiency.\n\n--The PG-ES consists of two steps. Its effect will be dependent on the quality of the trajectory predictor in the first step, since future trajectories in the second step are produced by the predictor in the first step. From Table3, PG-ES improves over STES, whose improvement is greater than those from spatial and temporal components in the STES. What if low-quality or noisy future trajectories are produced by the trajectory predictor in the first step?\n\n--From Table4, VTP training has tiny improvement over without VTP training. That means, the original trajectory prediction model without VTP training is strong enough. What if the In-Context-Learning works on a weaker base?", "was_revised": true, "final_justification": "THe rebuttal has solved my concerns. I keep my rating Borderline accept.", "rebuttal": "Dear Reviewer xGzb,\n\nWe sincerely appreciate for providing the insightful feedback. We have addressed the concerns raised in the review as below.\n\n**Weakness 1.1, Question 2: Clarification of K.**\nThank you for pointing out the undefined variable. We will add an explanation for k (number of predictions) in the problem formulation section of the final manuscript.\n\n**Weakness 1.2, Question 2: Difference between PredictionHead and Predictor.**\nWe apologize for the confusion. PredictionHead and Predictor are distinct modules, as illustrated in Figure 2. The PredictionHead consists of K one-layer MLPs, while the Predictor is composed of a three-layer Transformer Encoder with four attention heads. The Multimodal Decoder in Figure 2 is the PredictionHead. We will revise the manuscript to clarify this distinction for readers.\n\n\n**Weakness 2, Weakness 6, Limitation 1, Question 3:Different Setting, Clarification of efficiency.**\nWe would like to clarify that the \"efficiency\" discussed in the paper refers to the inference phase, not the training phase. For example, our method eliminates the need for backpropagation to fine-tune the model on target domain data, resulting to reduce GPU memory after the deployment by utilizing the in-context samples.\n\nAs noted, our two-stage training paradigm does increase the overall training cost (e.g., GPU hours) compared to the base model. However, a key advantage of our approach is that after training on the synthetic MOTSynth dataset, the model does not require any additional data collection or training on the target environment.\n\nFor reference, the number of trainable parameters for Social-Transmotion is 3.1 million, while TrajICL has 4.15 million.\n\n\n**Weakness 3, Question 3: Few-shot performance on the more challenging NBA SportVU Dataset.**\nThank you for this insightful comment. We have evaluated our approach on the NBA SportVU dataset (Rebounding and Scoring subsets) using different pool sizes. The results are shown in the table below:\n\n| Model  | Rebounding Subset\t | Scoring Subset |\n| ----  | ---- | ---- |\n| Social-Transmotion |  1.03/1.57 |  1.02/1.91 |\n| TrajICL (10%) |  0.93/1.61 |  0.98/1.78 |\n| TrajICL (20%) | 0.92/1.58 | 0.97/1.75 |\n| TrajICL (100%) | **0.90**/**1.51** | **0.95**/**1.72** |\n\n(X%) denotes the pool sizes.\n\nThis table indicates that our approach performs well even in challenging scenarios with a large distribution divergence between the training and testing datasets.\n\nInterestingly, the performance drop from a 100% to a 10% pool size is relatively small (+4% degradation). This may be because, despite the high complexity of human motion in the NBA SportVU dataset, the target motions might be similar to those in the in-context pool. This allows the model to effectively use the pool information for trajectory prediction. In contrast, if a dataset like SDD has a wide variety of motions, it might be more difficult to leverage in-context information effectively during inference.\n\n**Weakness 4, Limitation 1: Comparison with adaptive trajectory prediction methods**\n\nThank you for your constructive feedback. In most existing adaptive trajectory prediction methods, the adaptation module and the prediction model architecture are tightly coupled [1,2,3]. Furthermore, they often use different training settings (e.g., training datasets and target domains), making a fair, \"apple-to-apple\" comparison difficult. These previous works also compared with baselines or commonly-used trajectory prediction models.\n\nWe have already included a comparison with the conceptual methodology from a recent adaptation paper [1] in our main experiments (Table 1, “+VPT Shallow”). This involved adapting their visual prompt tuning approach into our base model (Social-Transmotion).\n\nTo provide further context, we can review the performance of recent adaptive methods [2,3] on the same evaluation datasets, as summarized below:\n\n| Model  | SDD | NBA |\n| ----  | ---- | ---- |\n| [2] |  11.0/20.0 |  1.28/2.64 |\n| [3] |  10.1/17.1 |  ---/--- |\n| Ours | **8.40**/**14.8** | **0.92**/**1.37** |\n\nThis table shows that our approach outperforms these recent adaptive methods. However, it is important to note that significant differences in base model architecture and training datasets exist between our work and theirs.\n\n[1] Adaptive Human Trajectory Prediction via Latent Corridors. In ECCV 2024.\n\n[2]  Recurrent Aligned Network for Generalized Pedestrian Trajectory Prediction. In IEEE ITCSVT 2024.\n\n[3] MetaTra: Meta-Learning for Generalized Trajectory Prediction in Unseen Domain, In arXiv 2024.\n\n**Weakness 6, Question 4: Effect of the quality of the trajectory predictor in the first step for PG-ES**\n\nThank you for this insightful comment. We conducted an additional experiment to analyze the effectiveness of PG-ES. Specifically, we measured the percentage of performance improvement from using STES to PG-ES based on the initial prediction error (ADE).\n\nSince we cannot submit figures in this rebuttal, we have categorized the initial \"noisiness (= ADE at the first inference)\" into three (low, medium, high) and summarized the results below:\n\n|  | Low | Medium | High |\n| ----  | ---- | ---- |---- |\n| &Delta;\t |  -0.5% | -8.9% | -12.0% |\n\nA lower &Delta; value indicates a greater performance improvement from STES to PG-ES (lower ADE is better). The results confirm that PG-ES is particularly effective when the initial prediction is noisy (High means the initial prediction is distant from the ground-truth).\n\nPG-ES offers the greatest benefit for trajectories with high initial errors because these are typically cases where a similar past could lead to multiple ambiguous futures. By generating several future hypotheses and re-selecting examples that match one of them, the model effectively corrects the initial poor selection. This correction process leads to the most significant improvement where the initial prediction was wrong.\n\n**Weakness 7: Performance of  weaker base model**\n\nWe would like to clarify the training setup of our approach.\n\nIn the first VTP training stage, the encoder is trained to predict the trajectory of a target pedestrian from target and other pedestrians' trajectories. In the second in-context training stage, all modules—the encoder, predictor, and prediction head—are trained to predict the trajectory using both the target and in-context samples. Therefore, even without VTP training, the model can acquire in-context learning capabilities by backpropagating the loss through the entire network.\n\nAs you suggested, we trained a weaker base model with a reduced feature dimension size (from 128 to 64) without VTP on MOTSynth. Due to the limited time for this rebuttal, we experimented with a 4-shot model.\n\n| | ADE| FDE |\n| ----  | ---- | ---- |\n| zero-shot |  16.41 | 19.36 |\n| 4-shot |  **15.30** | **17.50** |\n\nThis table shows that the in-context capability is still acquired with a weaker base model."}, {"review_id": "vH55OX8qwI", "reviewer": "Reviewer_eEf8", "summary": "The paper proposes TrajICL, an in-context learning (ICL) framework for pedestrian trajectory prediction that eliminates fine-tuning. Key innovations include: 1. Spatio-temporal similarity-based example selection (STES) for relevant context retrieval. 2. Prediction-guided example selection (PG-ES) leveraging predicted future trajectories to refine examples. 3. Training on large-scale synthetic data (MOTSynth) to enhance generalization. Experiments across 5 benchmarks (MOTSynth, JRDB, WildTrack, SDD, JTA) show TrajICL outperforms fine-tuned baselines in cross-domain adaptation.", "questions": "1.\tThe number of baseline methods compared by the authors is relatively limited. As the authors mentioned in the related work section, there are already some existing works on Adapting Trajectory Prediction. Although these methods have certain limitations in terms of computational costs and adaptability, I believe performance comparison experiments are still necessary. I also suggest that the authors include comparisons with relevant approaches such as Domain Generalization (DG) to further demonstrate the effectiveness of ICL. Additionally, since the authors mention some existing ICL methods in computer vision, performance comparisons with those should also be conducted.\n2.\tThe paper does not seem to provide details or sensitivity analysis regarding the choice of the hyperparameter M. In addition, a runtime analysis should also be included in the paper.\n3.\tI’m also curious whether applying PG-ES multiple times would lead to further performance improvements?\n4.\tThe paper lacks a bit in terms of originality. The authors should articulate the novelty and contribution of these operations within the proposed framework.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nConcise and Accessible Writing: This paper is easy to read and follow.\nPractical Contribution: Addresses edge-device constraints by replacing fine-tuning with lightweight ICL.\nRigorous Evaluation: Extensive cross-dataset validation (5 benchmarks) with consistent improvements.\nLimitations Discussed: Notes computational trade-offs of increasing in-context examples.\n\nWeaknesses:\n\nThe experimental analysis is not sufficiently thorough.\nThe paper need more baseline comparison.", "was_revised": true, "final_justification": "After reading the rebuttal and discussion, I am raising my score to Weak Accept (WA).", "rebuttal": "Dear Reviewer  eEf8,\n\nWe sincerely appreciate for providing the insightful feedback. We have addressed the concerns raised in the review as below.\n\n**Weakness 1, Question 1.1: Comparison with adaptive trajectory prediction methods**\n\nThank you for this constructive comment. In most existing adaptive trajectory prediction methods, the adaptation module and the prediction model architecture are tightly coupled [1,2,3]. This, along with differences in experimental settings (e.g., training datasets, target domains), makes a fair, \"apple-to-apple\" comparison difficult. These previous works also compared with baselines or commonly-used trajectory prediction models.\n\nWe have already included a comparison with the conceptual methodology from a recent adaptation paper [1] in our main experiments (Table 1, \"+VPT Shallow\"). We adapted their visual prompt tuning approach to fit within our baeline model.\n\nFor further context, we can review the performance of our approach against recent adaptive methods [2,3] on the same evaluation datasets, as summarized below. We have also included experiments on the NBA SportsVU dataset [4].\n\n| Model  | SDD | NBA |\n| ----  | ---- | ---- |\n| [2] |  11.0/20.0 |  1.28/2.64 |\n| [3] |  10.1/17.1 |  ---/--- |\n| Ours | 8.40/14.8 | 0.92/1.37 |\n\n\nThis table shows that our approach outperforms these recent adaptive methods. However, it is important to note that differences in the base model architecture and training datasets exist between our work and theirs.\n\n[1] Adaptive Human Trajectory Prediction via Latent Corridors. In ECCV 2024.\n\n[2] Recurrent Aligned Network for Generalized Pedestrian Trajectory Prediction. In IEEE TCSVT 2024.\n\n[3] MetaTra: Meta-Learning for Generalized Trajectory Prediction in Unseen Domain, In arXiv 2024.\n\n[4] Learning fine-grained spatial models for dynamic sports play prediction, in IEEE ICDM 2014.\n\n**Question 1.2: Comparison with ICL methods in computer vision**\n\nSince the tasks, loss functions, and model designs of other In-Context Learning (ICL) approaches in computer vision are completely different from ours, a fair and direct comparison with these works is not possible.\n\nHowever, we can adapt their proposed in-context sample selection strategies to our trajectory prediction task. The experimental results on the MOTSynth dataset are summarized below:\n\n| Selection Method | Paper References | ADE| ADE |\n| ----  | ---- | ---- | ---- |\n| Random  | [1, 2]  | 16.6 |  21.8   |   \n| Feature Similarity  | [3,4]  | 16.0 |  19.2   |  \n| Motion Similarity   | [5] | 19.2  |  32.5   |   \n| PG-ES (Ours) | --- | **15.3** |  **17.5** |\n\nThese results verify that our selection strategy, PG-ES, achieves the best performance on the trajectory prediction task.\n\n[1] AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning, In AAAI 2025.\n\n[2] MMICL: Empowering Vision-Language Model with Multi-modal In-Context Learning, In ICLR 2024,\n\n[3] Flamingo: a Visual Language Model for Few-Shot Learning, In NeurIPS 2022.\n\n[4] VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understanding, In CVPR 2025.\n\n[5] Skeleton-in-Context: Unified Skeleton Sequence Modeling with In-Context Learning, In CVPR 2024.\n\n**Question 2: Effect of hyperparameter M, the number of in-context samples**\n\nWe would like to note that the ablation study for M, the number of in-context samples, is already included in Figure 3 of the paper. The x-axis shows the number of input in-context samples (M), and the y-axis shows the accuracy on two benchmarks (MOTSynth and JTA). The figure clearly demonstrates that increasing the number of samples improves performance.\n\n**Question 3: Performance Improvement via Iterative Application of PG-ES**\n\nThank you for this insightful comment. We conducted an additional experiment applying PG-ES multiple times. The results on the MOTSynth dataset are summarized below:\n\n| Repetition | FDE |\n| ----  | ---- | \n| 1x  |17.5   |  \n| 2x   | 16.6  |   \n| 3x |  16.7 |\n| 4x |  16.7 |\n\nThe results show that applying the method two times improves performance by 5.2% and then converges, with no further significant gains.\n\n**Question 4: Clarification of contributions**\n\nThe adaptability of a trajectory prediction model is a critical challenge for real-world applications (as noted by Reviewer xGzb's Strength 1, Reviewer od3z's Strength 3, and Reviewer 8wc2's Strength 1). To address this, we propose an in-context learning (ICL) approach that, unlike existing methods, eliminates the need to update model weights in the target environment. This is inspired by the recent advancements in Large Language Models (LLMs). This is the first work to apply ICL to pedestrian trajectory prediction, and the connection between the challenge of adaptability and the conceptual idea of LLM's ICL is a novel perspective to the field (as noted by Reviewer 8wc2's Strength 2).\n\nWe would like to re-emphasize our contributions as follows:\n\n**Effective Sampling Strategy**: As shown in Figure 3, a naive approach using randomly selected samples does not improve performance. We believe this is due to the inherent complexity of trajectory data. Our proposed sampling strategy, PG-ES, is based on spatio-temporal similarity in the input space and a two-round inference process. These ideas for sampling are novel within the context of other ICL approaches in computer vision and demonstrate clear effectiveness in trajectory prediction tasks (as shown in our responses above).\n\n**Extensive Experiments**: Existing adaptability approaches have typically been evaluated on at most three datasets (ETH-UCY, SDD, NBA) with similar data setups (e.g., surveillance cameras, pedestrian motion). In contrast, we have verified the effectiveness of our framework using seven datasets: MOTSynth, JRDB-Image, WildTrack, SDD, JRDB-World, JTA, and NBA SportsVU. This broad evaluation demonstrates the wider applicability of our framework to various domains.\n\nWe will emphasize the novelty and contributions of our work in the final manuscript."}, {"review_id": "vdmXxyqgU9", "reviewer": "Reviewer_od3z", "summary": "The paper presents TrajICL, an in-context learning framework that does not require fine-tuning for human trajectory prediction. To identify relevant in-context examples, the approach incorporates a two-stage selection mechanism based on spatio-temporal similarity and prediction-guided refinement. The model is trained on a large-scale synthetic dataset, MOTSynth, and evaluated on several cross-domain benchmarks. It achieves competitive performance compared to various fine-tuning methods, as measured by standard accuracy metrics including minADE and minFDE.", "questions": "In addition to the concerns raised in the weaknesses section, I have the following additional questions:\n\n1. What is the FPS of datasets? What is the actual time in seconds for the 9-step conditioned past and the 12-step predicted future?\n\n2. Would it be possible to include an ablation on the number of K to understand how prediction accuracy scales with smaller/larger sample sizes?\n\n3. How does the model perform when conditioned on fewer past timesteps than the default 9? Specifically, do the STES and PG-ES modules remain effective under shorter observation windows?\n\n4. There appears to be a repeated citation of [48, 49].", "limitations": "There is a lack of discussion around failure cases and limitations. For example, the paper could examine how performance degrades when STES selects suboptimal examples, or when the in-context pool has limited coverage.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The paper provides a range of experiments and ablation studies across multiple datasets, which helps assess the effectiveness of the proposed approach.\n\n2. Figures included in the paper help illustrate key concepts and contribute to clarifying the proposed methodology.\n\n3. The motivation for applying in-context learning to trajectory prediction is clear, and the challenges involved are well articulated and addressed in the introduction and related work sections.\n\n\n\nWeaknesses:\n\n1. My major concern is that while the paper discusses the size of the in-context example pool, it lacks sufficient analysis on the quality of the pool. The model’s performance could be sensitive to the diversity and representativeness of the examples, especially given the inherently stochastic nature of human trajectories. Subtle variations in motion—such as differences in speed or turning angles—can significantly affect outcomes. However, the paper primarily focuses on pool size, without examining how variation in pool quality impacts performance. This limits our understanding of how well the method generalizes to out-of-distribution behaviors or less frequent motion patterns.\n\n2. Another concern is whether the motion patterns learned from the synthetic dataset (MOTSynth) adequately capture the diversity and complexity of real-world human behavior. It remains unclear how well the synthetic data reflects the intricate and multimodal nature of pedestrian motion. If the training data lacks this level of realism, it raises questions about the model’s ability to generalize beyond what it has seen during training. The paper would be strengthened by a discussion of which components of the framework, if any, contribute to such generalization on domain-specific datasets, and by providing more analysis or evidence regarding the realism and coverage of the training dataset.\n\n3. The method relies on Social-Transmotion as the underlying architecture, but its generalizability to other trajectory prediction models is not evaluated. This raises questions about how tightly TrajICL is coupled with this specific backbone and whether the proposed in-context learning framework would transfer effectively to alternative architectures.\n\n4. Since the paper emphasizes inference-time adaptability, it would be valuable to report inference time, particularly considering the cost of retrieving and encoding multiple in-context examples.", "was_revised": true, "final_justification": "Author's rebuttal has addressed my concerns regarding pool quality and model generalization. I will keep my score at borderline accept.", "rebuttal": "Dear Reviewer od3z,\n\nWe sincerely appreciate your insightful feedback. We have addressed the concerns raised in the review as detailed below.\n\n**Weakness 1: Effect of the Pool Quality on Prediction Accuracy**\n\nTo analyze the effectiveness of the in-context pool quality, we examined the relationship between pool quality and prediction accuracy. We measured pool quality using the STES similarity metric. If the maximum similarity between a target trajectory and the trajectories in the pool is low, it suggests that the pool lacks similar examples.\n\nAs we cannot upload figures, we categorized target trajectories into three similarity levels (low, medium, high) and summarized the results in the table below (experiments on MOTSynth, ADE reported).\n\n| Model | low | medium | high |\n|---|---|---|---|\n| Social-Transmotion | 37.0 | 12.1 | 5.45 |\n| TrajICL | **30.3** | **10.0** | **4.69** |\n\nThis table indicates that when the target trajectory represents out-of-distribution behavior (e.g., unusual motion patterns), the performance is lower compared to in-distribution data. However, we would like to emphasize that even with such OoD trajectory inputs, our framework shows a significant performance improvement by using in-context examples (37.0 → 30.3).\n\n**Weakness 2: Generalization to real-world data**\n\nThank you for your insightful feedback. While our synthetic training data may have limited motion diversity, our model generalizes effectively to real-world datasets like WildTrack, JRDB, and SDD.\n\nThis strong generalization is primarily due to our PG-STES module. During inference, instead of relying solely on the synthetically trained model, PG-STES dynamically selects examples relevant to the target trajectory from a pool of test data. This process provides the model with in-domain knowledge of the current scene, effectively bridging the synthetic-to-real domain gap.\n\nThis is partially verified in Table 3, where using randomly selected in-context samples does not improve performance on the JTA dataset.\n\nFurthermore, we verified this by measuring the Fréchet Inception Distance (FID) between the feature distributions of in-context examples on real WildTrack test data and the synthetic training dataset. Our PG-STES method achieves an FID score of 612, whereas a random selection from the same pool results in a vastly higher score of 2,454,212. This dramatically lower FID score demonstrates that our selection strategy is highly effective. By providing the model with the most pertinent real-world examples at inference time, PG-STES enables robust generalization, overcoming the limitations of the initial synthetic training.\n\n**Weakness 3: Generalization Capability with Different Backbones**\n\nAs mentioned, we used Social-Transmotion as our backbone, as it is a state-of-the-art model with a generalized architecture. Our TrajICL framework is plugged in to enable in-context learning capabilities. To confirm its generalization, we replaced the backbone with ForecastMAE [1] (disabling the lane encoder for pedestrian prediction). The experimental results are summarized below:\n\n| Model | MOTSynth | WildTrack | JTA| \n| ----  | ---- | ----  | ---- | \n| ForecastMAE | 18.41/24.51  | 25.0/29.5 |0.76/1.23 | \n| ForecastMAE + TrajICL (4 shots) | **15.98**/**17.91**  |  **21.9/28.3**  | **0.64**/**0.91** | \n\nThese results verify that our framework is generalized and can be applied to different backbones.\n\n[1] Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders, In ICCV 2023.\n\n**Weakness 4: Inference time**\n\nFirst, we would like to emphasize that our main motivation is to eliminate the fine-tuning and backpropagation costs on edge devices since the backpropagation process requires much higher GPU memory compared to the inference, as well as the burden of model management and environment-specific data collection (L.4, L.31-34, L.41...). This is what we refer to as \"efficiency.\" We did not aim to accelerate inference time in the original paper. \n\nWe will revise the term \"efficiency\" in the final manuscript to avoid confusion.\nHowever, as you pointed out, it is important to analyze the inference cost and compare it with the base model.\n\n| Model | FLOPs (GFLOPS) | Params (million) | Total Inference time (ms) | \n| ----  | ---- | ----  | ---- | \n| Social-Transmotion| 3.48  | 3.10 |6.11 | \n| TrajICL | 4.54  | 4.15  |7.69 + 0.04 (retrieval) | \n\n(Inference time measure with PC equipped with NVIDIA TITAN RTX GPU, 188 GB RAM, and Intel(R) Xeon(R) Silver 4216 CPU.)\n\nWe follow the experimental setup of existing in-context learning approaches [1], where features for in-context samples are pre-computed. Our overall inference cost is higher because of the additional Predictor module for aggregating in-context sample features and the two-stage inference via PG-ES.\n\nHowever, since our primary motivation is to eliminate the training step after deployment, we also compared the fine-tuning cost of the base model with our inference cost:\n\n| Model | FLOPs (GFLOPS) |  GPU Memory (MB) | \n| ----  | ----  | ---- | \n| Social-Transmotion Fine-tuning | 10.43 | 462 | \n| TrajICL | 4.54  |288 | \n\n(GPU memory is computed with batch size = 1)\n\nOur required GPU memory is significantly less than that needed for fine-tuning the base model, which aligns with our paper's primary motivation. As mentioned in the conclusion, reducing the inference cost is a current limitation for edge-device deployment. We believe future work could explore applying existing techniques, such as Token Merging [2], to improve inference speed without requiring architectural modifications.\n\n[1] In-Context Learning Makes an Ego-evolving Scene Text Recognizer, In CVPR 2024.\n\n[2] Token Merging: Your ViT But Faster, In ICLR 2023. \n\n**Question 1: FPS of datasets**\n\nFor MOTSynth and WildTrack datasets, the FPS is 2.0. The 9-step conditioned past corresponds to 4.5 seconds, and the 12-step predicted future corresponds to 6.0 seconds.\n\nFor JRDB, JTA, and SDD datasets, the FPS is 2.5. The 9-step conditioned past corresponds to 3.6 seconds, and the 12-step predicted future corresponds to 4.8 seconds. We will incorporate this information into the final manuscript.\n\n**Question 2: ablation study of sample size**\n\nWe would like to clarify that K denotes the number of predicted trajectories for the multimodal prediction. M controls the number of in-context samples. The ablation study for M is already conducted in Figure 3. The x-axis represents the number of input in-context samples (M), and the y-axis shows the accuracy on two benchmarks (MOTSynth and JTA). It demonstrates that increasing the number of samples improves performance.\n\n**Question 3: Fewer Timesteps evaluation**\n\nWe conducted an additional experiment with fewer past timesteps during evaluation. \n\nSpecifically, we reduced the input past timesteps from 9 to 6 and 3.\n\n| Model | 9 steps | 6 steps  | 3 steps | \n| ----  | ---- | ---- | ---- | \n| Social-Transmotion | 17.6/23.0  | 20.1/27.4 | 24.3/34.1 |\n| Ours w/ Random Selection | 16.6/21.8  | 19.0/23.6 | 19.3/25.7 |\n| Ours w/ STES Selection | 15.6/19.2  | 16.9/20.7 | 17.9/22.9 |    \n| Ours w/ PG-ES Selection | **15.3**/**17.5**  | **16.0**/**18.0** | **16.4**/**19.3** |    \n\nThe table verifies that while performance degrades with fewer timesteps, the proposed selection strategies (especially PG-ES) still show significant performance improvement. For example, with 3 steps, PG-ES improves performance from 25.7 (Random) to 19.3, demonstrating the effectiveness of our approach even with limited input data.\n\n**Question 4: Repeated citation**\n\nThank you for pointing out the duplicated citation. We will correct this in the final manuscript.\n\n**Limitation 1: Failure cases and Limitation**\n\nFirst, we would like to emphasize that we have already conducted experiments on scenarios with limited pool coverage (the few-shot scenarios in Table 2).\n\nWe conducted additional experiments to analyze the effectiveness of PG-ES. Specifically, we measured the percentage of performance improvement from using STES to PG-ES based on the initial prediction error (ADE). As we cannot submit figures, we categorized the \"noisiness\" into three categories (low, medium, high) and summarized the results:\n\n|  | Low | Medium | High |\n| ----  | ---- | ---- |---- |\n| &Delta;\t |  -0.5% | -8.9% | -12.0% |\n\nA lower &Delta; value indicates a greater performance improvement from STES to PG-ES (lower ADE is better). The results confirm that PG-ES is specifically effective when the initial prediction is noisy.\n\nOne important limitation is the inference time, as noted in the last section of our conclusion (L.321). Furthermore, improving robustness to noise in trajectories is another current limitation (see Reviewer 8wc2's response to W3) and a potential future research direction."}, {"review_id": "SYYsXGJWF3", "reviewer": "Reviewer_8wc2", "summary": "The paper proposes a novel in-context learning (ICL) model for pedestrian trajectory prediction, showing improved results compared to the base model in novel environments without additonal fine-tuning.", "questions": "1. Can the authors provide an analysis of the inference speed of the proposed method, especially how it compares to the speed of the base model without the ICL modules?\n2. Can the authors show that the proposed ICL module is general to different base models, such as a more lightweight one?\n3. Can the authors explain how the proposed example selection strategies could be incorporated into real-world settings where no ground truth annotations are available?", "limitations": "Trajectory prediction, particularly in an in-context learning setting, may raise some ethical concerns.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The motivation is clear, and the paper is easy to follow.\n2. The idea of applying ICL to trajectory prediction is novel, and the results are strong and promising.\n3. Ablation studies are extensive, clearly showing the effectiveness of each component.\n\nWeaknesses:\n1. The synthetic dataset MOTSynth used to train the ICL model is an established dataset and should not be counted as a key contribution.\n2. The motivation for using ICL is to reduce the costly fine-tuning for easy deployment on edge devices; however, the base model and the addition of ICL modules all seem quite computationally expensive based on the transformer architecture, limiting their potential in real-world applications.\n3. The paper only experiments on datasets with ground-truth annotations. However, a real-world ICL system involves first collecting and annotating a few real-world examples instead of annotating a large number of samples and then selecting from them, as the latter sounds too laborious and conflicts with the motivation of ICL. I'm concerned that the proposed example selection approaches may not be informative and have enough significance in the real world.", "was_revised": true, "final_justification": "The authors addressed my concerns in the rebuttal. The additional results on noises from pseudo-labels further highlight the significance of the proposed ICL for trajectory prediction in real-world settings. I suggest accepting this work.", "rebuttal": "Dear Reviewer 8wc2,\n\nWe sincerely appreciate your insightful feedback. We have addressed the concerns raised in the review as detailed below.\n\n**Weakness 1: Using synthetic dataset as a contribution**\n\nWe agree that simply using an existing dataset like MOTSynth is not a key contribution of the paper.\n\nHowever, we want to emphasize a crucial finding: unlike previous approaches [1, 2, 3] that rely heavily on curating real-world trajectory datasets for training, our paper shows that a trajectory prediction model can be trained exclusively on synthetic data and performs well by using real-world data as in-context samples. This is a significant finding for the community, as it demonstrates a way to reduce the costly burden of real-world data curation.\n\nTo avoid confusion, we will remove this from our list of contributions.\n\n[1] Social-Transmotion: Promptable Human Trajectory Prediction. In ICLR 2024.\n\n[2] Trajectron++: Dynamically Feasible Trajectory Forecasting with Heterogeneous Data. In ECCV 2020.\n\n[3] Physical plausibility-aware trajectory prediction via locomotion embodiment. In CVPR 2025.\n\n**Weakness 2, Question 1: Inference speed analysis**\n\nFirst, we would like to emphasize that our main motivation is to eliminate the fine-tuning and backpropagation costs on edge devices since the backpropagation process requires much higher GPU memory compared to the inference, as well as the burden of model management and environment-specific data collection (L.4, L.31-34, L.41...). This is what we refer to as \"efficiency.\" We did not aim to accelerate inference time in the original paper. \n\nWe will revise the term \"efficiency\" in the final manuscript to avoid confusion.\nHowever, as you pointed out, it is important to analyze the inference cost and compare it with the base model.\n\n| Model | FLOPs (GFLOPS) | Params (million) | Total Inference time (ms) | \n| ----  | ---- | ----  | ---- | \n| Social-Transmotion| 3.48  | 3.10 |6.11 | \n| TrajICL | 4.54  | 4.15  |7.69 + 0.04 (retrieval) | \n\n(Inference time measure with PC equipped with NVIDIA TITAN RTX GPU, 188 GB RAM, and Intel(R) Xeon(R) Silver 4216 CPU.)\n\nWe follow the experimental setup of existing in-context learning approaches [1], where features for in-context samples are pre-computed. Our overall inference cost is higher because of the additional Predictor module for aggregating in-context sample features and the two-stage inference via PG-ES.\n\nHowever, since our primary motivation is to eliminate the training step after deployment, we also compared the fine-tuning cost of the base model with our inference cost:\n\n| Model | FLOPs (GFLOPS) |  GPU Memory (MB) | \n| ----  | ----  | ---- | \n| Social-Transmotion Fine-tuning | 10.43 | 462 | \n| TrajICL | 4.54  |288 | \n\n(GPU memory is computed with batch size = 1)\n\nOur required GPU memory is significantly less than that needed for fine-tuning the base model, which aligns with our paper's primary motivation. As mentioned in the conclusion, reducing the inference cost is a current limitation for edge-device deployment. We believe future work could explore applying existing techniques, such as Token Merging [2], to improve inference speed without requiring architectural modifications.\n\n[1] In-Context Learning Makes an Ego-evolving Scene Text Recognizer, In CVPR 2024.\n\n[2] Token Merging: Your ViT But Faster, In ICLR 2023. \n\n\n**Question 2: Performance of different base models**\n\nAs you suggested, we trained a lighter model with a reduced feature dimension size (from 128 to 64) without VTP on MOTSynth. Due to time constraints during the rebuttal, we experimented with a 4-shot model.\n\n| | ADE| FDE |\n| ----  | ---- | ---- |\n| zero-shot |  16.41 | 19.36 |\n| 4-shot |  **15.30** | **17.50** |\n\nThis table shows that in-context learning capabilities are still acquired with a lighter base model.\n\nAdditionally, to further confirm the generalization capability, we changed the backbone to ForecastMAE [1] (disabling the lane encoder for pedestrian prediction) and conducted experiments. The results are summarized below:\n\n| Model | MOTSynth | WildTrack | JTA| \n| ----  | ---- | ----  | ---- | \n| ForecastMAE | 18.41/24.51  | 25.0/29.5 |0.76/1.23 | \n| ForecastMAE + TrajICL (4 shots) | **15.98**/**17.91**  |  **21.9/28.3**  | **0.64**/ **0.91** | \n\nThese results verify that our framework is generalized and can be applied to different backbones.\n\n[1] Forecast-MAE: Self-supervised Pre-training for Motion Forecasting with Masked Autoencoders, In ICCV 2023.\n\n**Weakness 3, Question 3: Robustness to Real-world noises**\n\nThank you for this insightful feedback. While improving robustness to trajectory noise is a separate line of research [1, 2, 3], we agree that using clean, ground-truth trajectories is not feasible in a real-world setting. Therefore, we conducted an additional experiment using trajectories generated by off-the-shelf models (Faster R-CNN [4] for person detection and Deep-SORT [5] for tracking pretrained on the MOT17 dataset) on the JRDB Image dataset. The results are summarized below.\n\n| Model | Input | ADE &darr;\t | FDE &darr; |\n| ---- | ---- | ---- | ---- |\n| Social-Trans| GT | 2.88 |  3.32 |\n| Social-Trans | Off-the-shelf | 3.25 (+12.8%) | 4.33 (+30.4%) |\n| TrajICL (Ours) | GT | 2.61 | 2.68 |\n| TrajICL (Ours) | Off-the-shelf | 2.62 (+0.4%) | 2.79 (+4.1%) | \n\nThe \"Input\" column indicates whether the target and in-context samples are ground-truth (GT) or are predicted by off-the-shelf models. The (+X%) value shows the performance change from using GT trajectories to using predicted trajectories. This setup represents a fully automated process that doesn't require manual ground-truth annotation.\n\nThe table shows that while the zero-shot model's performance severely degrades with noisy inputs (e.g., +30.4% in FDE), our model robustly predicts future trajectories from noisy inputs, with only a 4.1% degradation in FDE.\n\n[1] RealTraj: Towards Real-World Pedestrian Trajectory Forecasting, In arXiv 2025.\n\n[2] Uncovering the missing pattern: Unified framework towards trajectory imputation and prediction, In CVPR 2023.\n\n[3] BCDiff: Bidirectional Consistent Diffusion for Instantaneous Trajectory Prediction, In NeurIPS 2023.\n\n[4] Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In NeurIPS 2025.\n\n[5] Simple Online and Realtime Tracking with a Deep Association Metric, In ICIP 2017.\n\n**Limitation: Ethical Concerns**\n\nWe would like to kindly ask the reviewer to elaborate on the ethical considerations of the trajectory prediction with in-context learning so that we will incorporate that in the final manuscript."}]

Author rebuttal:

Meta-review: This work presents an In-Context Learning framework for pedestrian trajectory prediction. All reviewers were positive about this work, e.g., though this work is clearly on motivation, the idea of applying ICL to trajectory prediction is novel, and the results are strong and promising, and easy to follow. Moreover, their concerns were addressed in the rebuttal and discussion.

Overall, I recommend accepting this paper to NeurIPS.
Decision: Accept (poster)