Title: Towards Predicting Any Human Trajectory In Context
Venue: NeurIPS 2025
Accepted: yes
Keywords: In-Context Learning

pedestrian trajectory forecasting

trajectory prediction
Abstract: Predicting accurate future trajectories of pedestrians is essential for autonomous systems but remains a challenging task due to the need for adaptability in different environments and domains. A common approach involves collecting scenario-specific data and performing fine-tuning via backpropagation. However, the need to fine-tune for each new scenario is often impractical for deployment on edge devices. To address this challenge, we introduce TrajICL, an In-Context Learning (ICL) framework for pedestrian trajectory prediction that enables adaptation without fine-tuning on the scenario-specific data at inference time without requiring weight updates. We propose a spatio-temporal similarity-based example selection (STES) method that selects relevant examples from previously observed trajectories within the same scene by identifying similar motion patterns at corresponding locations. To further refine this selection, we introduce prediction-guided example selection (PG-ES), which selects examples based on both the past trajectory and the predicted future trajectory, rather than relying solely on the past trajectory. This approach allows the model to account for long-term dynamics when selecting examples. Finally, instead of relying on small real-world datasets with limited scenario diversity, we train our model on a large-scale synthetic dataset to enhance its prediction ability by leveraging in-context examples. Extensive experiments demonstrate that TrajICL achieves remarkable adaptation across both in-domain and cross-domain scenarios, outperforming even fine-tuned approaches across multiple public benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel in-context learning (ICL) model for pedestrian trajectory prediction, showing improved results compared to the base model in novel environments without additonal fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The motivation is clear, and the paper is easy to follow.\n2. The idea of applying ICL to trajectory prediction is novel, and the results are strong and promising.\n3. Ablation studies are extensive, clearly showing the effectiveness of each component.\n\nWeaknesses:\n1. The synthetic dataset MOTSynth used to train the ICL model is an established dataset and should not be counted as a key contribution.\n2. The motivation for using ICL is to reduce the costly fine-tuning for easy deployment on edge devices; however, the base model and the addition of ICL modules all seem quite computationally expensive based on the transformer architecture, limiting their potential in real-world applications.\n3. The paper only experiments on datasets with ground-truth annotations. However, a real-world ICL system involves first collecting and annotating a few real-world examples instead of annotating a large number of samples and then selecting from them, as the latter sounds too laborious and conflicts with the motivation of ICL. I'm concerned that the proposed example selection approaches may not be informative and have enough significance in the real world.\n\nQUESTIONS:\n1. Can the authors provide an analysis of the inference speed of the proposed method, especially how it compares to the speed of the base model without the ICL modules?\n2. Can the authors show that the proposed ICL module is general to different base models, such as a more lightweight one?\n3. Can the authors explain how the proposed example selection strategies could be incorporated into real-world settings where no ground truth annotations are available?\n\nLIMITATIONS:\nTrajectory prediction, particularly in an in-context learning setting, may raise some ethical concerns.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents TrajICL, an in-context learning framework that does not require fine-tuning for human trajectory prediction. To identify relevant in-context examples, the approach incorporates a two-stage selection mechanism based on spatio-temporal similarity and prediction-guided refinement. The model is trained on a large-scale synthetic dataset, MOTSynth, and evaluated on several cross-domain benchmarks. It achieves competitive performance compared to various fine-tuning methods, as measured by standard accuracy metrics including minADE and minFDE.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper provides a range of experiments and ablation studies across multiple datasets, which helps assess the effectiveness of the proposed approach.\n\n2. Figures included in the paper help illustrate key concepts and contribute to clarifying the proposed methodology.\n\n3. The motivation for applying in-context learning to trajectory prediction is clear, and the challenges involved are well articulated and addressed in the introduction and related work sections.\n\n\n\nWeaknesses:\n\n1. My major concern is that while the paper discusses the size of the in-context example pool, it lacks sufficient analysis on the quality of the pool. The model’s performance could be sensitive to the diversity and representativeness of the examples, especially given the inherently stochastic nature of human trajectories. Subtle variations in motion—such as differences in speed or turning angles—can significantly affect outcomes. However, the paper primarily focuses on pool size, without examining how variation in pool quality impacts performance. This limits our understanding of how well the method generalizes to out-of-distribution behaviors or less frequent motion patterns.\n\n2. Another concern is whether the motion patterns learned from the synthetic dataset (MOTSynth) adequately capture the diversity and complexity of real-world human behavior. It remains unclear how well the synthetic data reflects the intricate and multimodal nature of pedestrian motion. If the training data lacks this level of realism, it raises questions about the model’s ability to generalize beyond what it has seen during training. The paper would be strengthened by a discussion of which components of the framework, if any, contribute to such generalization on domain-specific datasets, and by providing more analysis or evidence regarding the realism and coverage of the training dataset.\n\n3. The method relies on Social-Transmotion as the underlying architecture, but its generalizability to other trajectory prediction models is not evaluated. This raises questions about how tightly TrajICL is coupled with this specific backbone and whether the proposed in-context learning framework would transfer effectively to alternative architectures.\n\n4. Since the paper emphasizes inference-time adaptability, it would be valuable to report inference time, particularly considering the cost of retrieving and encoding multiple in-context examples.\n\nQUESTIONS:\nIn addition to the concerns raised in the weaknesses section, I have the following additional questions:\n\n1. What is the FPS of datasets? What is the actual time in seconds for the 9-step conditioned past and the 12-step predicted future?\n\n2. Would it be possible to include an ablation on the number of K to understand how prediction accuracy scales with smaller/larger sample sizes?\n\n3. How does the model perform when conditioned on fewer past timesteps than the default 9? Specifically, do the STES and PG-ES modules remain effective under shorter observation windows?\n\n4. There appears to be a repeated citation of [48, 49].\n\nLIMITATIONS:\nThere is a lack of discussion around failure cases and limitations. For example, the paper could examine how performance degrades when STES selects suboptimal examples, or when the in-context pool has limited coverage.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes TrajICL, an in-context learning (ICL) framework for pedestrian trajectory prediction that eliminates fine-tuning. Key innovations include: 1. Spatio-temporal similarity-based example selection (STES) for relevant context retrieval. 2. Prediction-guided example selection (PG-ES) leveraging predicted future trajectories to refine examples. 3. Training on large-scale synthetic data (MOTSynth) to enhance generalization. Experiments across 5 benchmarks (MOTSynth, JRDB, WildTrack, SDD, JTA) show TrajICL outperforms fine-tuned baselines in cross-domain adaptation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nConcise and Accessible Writing: This paper is easy to read and follow.\nPractical Contribution: Addresses edge-device constraints by replacing fine-tuning with lightweight ICL.\nRigorous Evaluation: Extensive cross-dataset validation (5 benchmarks) with consistent improvements.\nLimitations Discussed: Notes computational trade-offs of increasing in-context examples.\n\nWeaknesses:\n\nThe experimental analysis is not sufficiently thorough.\nThe paper need more baseline comparison.\n\nQUESTIONS:\n1.\tThe number of baseline methods compared by the authors is relatively limited. As the authors mentioned in the related work section, there are already some existing works on Adapting Trajectory Prediction. Although these methods have certain limitations in terms of computational costs and adaptability, I believe performance comparison experiments are still necessary. I also suggest that the authors include comparisons with relevant approaches such as Domain Generalization (DG) to further demonstrate the effectiveness of ICL. Additionally, since the authors mention some existing ICL methods in computer vision, performance comparisons with those should also be conducted.\n2.\tThe paper does not seem to provide details or sensitivity analysis regarding the choice of the hyperparameter M. In addition, a runtime analysis should also be included in the paper.\n3.\tI’m also curious whether applying PG-ES multiple times would lead to further performance improvements?\n4.\tThe paper lacks a bit in terms of originality. The authors should articulate the novelty and contribution of these operations within the proposed framework.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an In-Context Learning framework for pedestrian trajectory prediction.  To address adaptability of prediction in different domains, they introduce a method without fine-tuning. A spatio-temporal similarity-based example selection and a further refine strategy are proposed. They also use synthetic dataset for training to simulate diverse outdoor environments.\nExperiments on multiple datasets are conducted to verify its effectiveness. Both in-domain and cross-domain settings are evaluated.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1）\tIt states clearly on motivation and problems of existing methods.\n\n2）\tThe idea of using In-Context Learning for adaptability is interesting in pedestrian trajectory prediction.\n\n3）\tAblation studies on the two designed modules are provided, i.e., similarity-based example selection module and prediction-guided refinement.\n\n4）\tFigures are clearly displayed including visualization of qualitative results.\n\nWeakness:\n\n--In Equa(3), K appears for the first time and k is undefined. It can be declared at the beginning such as in the Problem Formulation section that the study focuses on multi-modal or stochastic prediction.\n\nUnclear implementation details\n--in line 194-195 “PredictionHead, which consists of K simple one-layer MLPs for multimodal prediction”. In line 244 “Predictor consists of three layers and four attention”. These two statements are inconsistent. What’s the exact prediction head’s configuration? Why use four attention modules? Maybe it can be added in supplementary material.\n\n--Different setting:\nIt comprises two sequential training phases—vanilla trajectory prediction training and in-context training. Most existing trajectory prediction methods do not use the two-stage paradigm. Though some methods consider adaptability，they adopt one-stage framework.  It is said the proposed TrajICL “offers greater efficiency”. The computational efficiency is suggested to be reported and compared.\n\n-- From Table2, when example pool size decreases from 100% (in Table 1) to 10% and 20%, prediction performances degenerate differently on different datasets. The degeneration percentage (not reported in Table 2) is greater on SDD.   I wonder its few-shot performance on the more challenging NBA SportVU Dataset. Whether TrajICL is affected by the extent of distribution divergence between training and inferring datasets. \n\n\n--it is unclear whether other trajectory prediction methods considering adaptability through transfer learning, online adaption, et al. are comparable.\n\n\t\n--The number of trainable parameters is suggested to be compared in Table 1, so it can better support the claim TrajICL is lighter and offers greater efficiency.\n\n--The PG-ES consists of two steps. Its effect will be dependent on the quality of the trajectory predictor in the first step, since future trajectories in the second step are produced by the predictor in the first step. From Table3, PG-ES improves over STES, whose improvement is greater than those from spatial and temporal components in the STES. What if low-quality or noisy future trajectories are produced by the trajectory predictor in the first step?\n\n--From Table4, VTP training has tiny improvement over without VTP training. That means, the original trajectory prediction model without VTP training is strong enough. What if the In-Context-Learning works on a weaker base?\n\nQUESTIONS:\n1)\tThe computational complexity and efficiency are not discussed. \n\n2)\tSome implementation details to be complemented;\n\n3)\tMore discussions or explanations on the few-shot comparison;\n\n4)\tDependence of PG-ES on the quality of the trajectory predictor in the first step need be analyzed.\n\nLIMITATIONS:\nThe limitation analysis is not sufficient. The computational complexity and efficiency are not discussed. The comparison with other prediction models considering adaptability seems incomplete.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}