Interactive Adjustment for Human Trajectory Prediction with Individual Feedback
Authors: Jianhua Sun
Yuxuan Li
Liang Chai
Cewu Lu
Venue: ICLR 2025 Poster (2025)
Keywords: Human Trajectory Prediction
Primary area: applications to robotics, autonomy, planning
TL;DR: For the first time, we study individual feedback, which is derived from early predictions on a target, to improve the accuracy of subsequent predictions on the same agent. We design an external module IAN to model it and prove its effectiveness.

Abstract: Human trajectory prediction is fundamental for autonomous driving and service robot. The research community has studied various important aspects of this task and made remarkable progress recently. However, there is an essential perspective which is not well exploited in previous research all along, namely individual feedback. Individual feedback exists in the sequential nature of trajectory prediction, where earlier predictions of a target can be verified over time by his ground-truth trajectories to obtain feedback which provides valuable experience for subsequent predictions on the same agent. In this paper, we show such feedback can reveal the strengths and weaknesses of the model's predictions on a specific target and heuristically guide to deliver better predictions on him. We present an interactive adjustment network to effectively model and leverage the feedback. This network first exploits the feedback from previous predictions to dynamically generate an adjuster which then interactively makes appropriate adjustments to current predictions for more accurate ones. We raise a novel displacement expectation loss to train this interactive architecture. Through experiments on representative prediction methods and widely-used benchmarks, we demonstrate the great value of individual feedback and the superior effectiveness of proposed interactive adjustment network.

Published as a conference paper at ICLR 2025
INTERACTIVE ADJUSTMENT FOR HUMAN TRAJECTORY PREDICTION WITH INDIVIDUAL FEEDBACK
Jianhua Sun; Yuxuan Li; Liang Chai, Cewu Lu Shanghai Jiao Tong University
ABSTRACT
Human trajectory prediction is fundamental for autonomous driving and service robot. The research community has studied various important aspects of this task and made remarkable progress recently. However, there is an essential perspective which not well exploited in previous research all along, namely individual feed- back. Individual feedback exists in the sequential nature of trajectory prediction, where earlier predictions of a target can be verified over time by his ground-truth trajectories to obtain feedback which provides valuable experience for subsequent predictions on the same agent. In this paper, we show such feedback can reveal the strengths and weaknesses of the model's predictions heuristically guide to deliver better predictions on him. We present an interactive adjustment network effectively model and leverage the feedhack. This net- work first exploits the feedback from previous predictions to dynamically gener- ate an adjuster which then interactively makes appropriate adjustments current predictions for more accurate ones. We raise a novel displacement expectation loss to train this interactive architecture. Through experiments on representative prediction methods and widely-used benchmarks, we demonstrate the great value individual feedback and the superior effectiveness of proposed interactive ad- justment network.
specific target and
INTRODUCTION
Human trajectory prediction is a task to forecast the future movements of pedestrians according to the observations from the past. Over the past years, researchers have studied this topic from numerous aspects such as multi-modal prediction (Li et al. 2017 Gupta et al., 2018), human social interactions AAlai et al., 2016; Xu al. 2022a) and scene context restrictions (Sadeghian al., 2019 Liang et al. 2019). and have achieved remarkable progress. Beyond the above points, when reflecting on the sequential nature of trajectory prediction, i.e. an agent's presence in a scene is typically long sequence and thus series of consecutive predictions is performed over time, we believe here lies another essential information that is not well exploited in previous research all along, namely individual feedback.
Individual feedback refers to the information derived from the differences between the model's previous predictions and the ground-truth trajectory, and can provide valuable experience for subse- quent predictions on the same agent. Particularly as illustrated Fig. [ when model is continu- ously making predictions ago, could already be verified by the agent's ground-truth trajectory which has become available through the progression of time. Such verification offers the individual feedback. Since the feed- back includes references to the strengths and weaknesses of the model's predictions on the agent, this information is properly utilized, it is able to heuristically guide to deliver more accurate predic- tions on this agent, and thereby brings overall improvements. However, none of the existing studies have paid much attention to exploring such information.
single agent, its previous predictions, e.g. those from several seconds
"Equal Contribution.
1 "orresponniig Author.

---

Published as a conference paper at ICLR 2025
Observation
Preditiin
Gevand Truth
Proposal
M
Adjuster
Prediction
Time
Feedbark
(a)
(b)
Figure 1: (a) Illustration of the individual feedback. Previous predictions of target can be verified by ground truths through the progression time, offering valuable experience on the strengths and weaknesses of the prediction model. (b) A brief schematic of the interactive adjustment process. M refers to the prediction model.
Still, is non-trivial effectively learn and leverage the feedback to adjust the original predictions into better ones. First, the feedback information cannot be simply integrated into the prediction model and learned end-to-end, since the calculation individual feedback requires the output the prediction model. Therefore, the feedback adjustment network should be developed as external module to the prediction model. But this leads to a second problem as an external module cannot directly be aware of the restraints given by the prediction model when adjusting, e.g. the adjusted prediction may still violate social rules even though the prediction model has already learnt the social restrictions.
In this paper, we design the interactive adjustment network, a...a. IAN, to present a novel scheme on feedback modeling and utilization. IAN is external module for the aforementioned first is- sue. As the trained prediction model is continuously performing predictions on a target individual, IAN first embeds the feedback from previous predictions and corresponding ground truths, and then aggregates all the feedback together to dynamically generate an adjuster specifically for the target. Thereby, the adjuster can adjust the model's current predictions into several proposals regarding future trajectories according to the integrated feedback information. Although these proposals can be directly decoded into trajectories final predictions, this faces the aforementioned second issue. To this end, we generate the final predictions by querying the prediction model for trajectory candi- dates and further filtering candidates to figure out those with high confidence in mandidate-proposss coherency as final results. In this way, the adjusted final predictions are not only optimized by the feedback but are also in line with the prediction model. A brief schematic of the whole interac- tive process is shown in Fig. b,, where the adjuster gets feedback information from the prediction model (black arrows) and the prediction model in turn leverages adjusted proposals from the adjuster generate the final results for improved accuracy (orange arrows).
Considering the final results are provided by the prediction model rather than IAN itself, conven- tional loss functions for trajectory prediction such L2 are not available since the gradient will be back- pack-propagated to IAN. To train this interactive architecture, we further raise a novel displace- ment expectation loss. Observing the confidence evaluated by the filter between proposal and candidate trajectory reveals the probability to select this candidate for the proposal, the expectation of displacement between the candidates and the ground-truit -tuun trajectory for proposal indicates the envor between the proposal and the ground truths. By optimizing with our proposed loss, IAN can be trained end-to-end to learn the feedback and leverage to generate accurate proposals.
As external module, IAN can be easily adopted to various prediction models. We conduct ex- haustive experiments on three widely-used trajectory prediction benchmarks (Pellegrini et al. 2009 Leal-Taixé et al. 2014 Zhou et al. 2012 [inouk23] 2016) with 6 representative prediction mod-
els (Gupta et al. 2018 Shi et al. [202 Pang et al. 2202 Xu et al.] 2022a Shi et al.) 2023) Bae et al. 2024). including sfate-of-the- art (Bac et al. 2024). The results demonstrate the great value of individual feedback, the superior effectiveness of IAN and the significant performance boost on trajectory prediction.
2

---

Published as a conference paper at ICLR 2025
We summarize the contributions of paper follows.
We study individual feedback, which reveals the prediction model's performance in pre- dicting a specific agent. Individual feedback cxists in the sequential nature of trajectory prediction tasks, and can be derived from the differences between model'ss previous pre- dictions and the ground-trutt -ouun trajectory of the specific agent. To the best of our knowledge, we the first to adopt such agent-specific information into trajectory prediction tasks.
We analyze the properties of individual feedback and design an interactive adjustment net- work (IAN) to properly leverage individual feedback. The proposed IAN is fully external module and can be easily adopted to other prediction models only with small computational overhead.
Our experiments show that IAN significantly boosts the performances of multiple base prediction models on various datasets, proving the effectiveness of individual feedback.
2 RELATED WORKS
Human trajectory prediction (Hirakawa et al., 2018
Lee et al. 2017
Gupta et al. [2018] Shi al., 2022 Shi et al.
2021 Sun et al. 2021 Xu et al. 2022a| Sun et al. 2022] Bae et al.
[023] Sun
forecast the future movements of traffic agents given past observations. It has numerous important applications
al. 2023b a Dong et al., 2023 Li et al.) 2024
Yang et al. 2024
is proposed
such as autonomous vehicles and robots (Hirakawa et al. 2018 Rudenko et al. 2019) Due to the fact that there is no single correct future, some works (Lee et al. 2017] Gupta et al. 2018) have paid their attention to multi-modal prediction, which aims at generating multiple possible future trajectories given single observation to cover the uncertainty in the future. Based on the multi- modal setting, many recent works focus on exploiting various additional information apart from the target's past trajectory to aid in the prediction. For example, approaches like Sophie (Sadeghian ct al. 2019). Trajectron+++ (Salzmann et al., 2020) and SingularTrajectory (Bae et al.
(2024) take
scene context into consideration during prediction. while others like Social GAN (Gupta et al. 2018). SGCN (Shi et al. (02I). GroupNet (Xu et al., (022a) and SocialCircle (Wong et al. 2024] design various methods model the social interactions. Further, some recent works (Meng al., 2022 Thakkar et al.) 2024) tried to capture scene-specific patterns to adapt the predictions better fit the current scenario. However, there another type of agent-specific information that has been neglected by previous works, namely individual feedback. It emerges from subsequent predictions of the model on the same agent, and offers experience from the past to improve the current prediction.
3 FORMULATION OF INDIVIDUAL FEEDBACK IN TRAJECTORY PREDICTION
In this section, we first briefly describe the task setting of trajectory prediction and then introduce the problem formulation of using individual feedback to improve the prediction performance. A list of key notations and their meanings are provided in Tab. 3 of the Technical Appendix for better reference.
Trajectory Prediction In the conventional setting of trajectory prediction, a prediction model M takes observation sequence O with length Tobs of agent input and then predicts series of trajectories Y with length Tpred for him. If the agent can be observed since timestep Ta, the model's prediction of this agent, V, at timestep T 2 Tobs + Ta can be formulated
= ++=m((O)) = 8 1,2,
k)
(1)
where Y is one of the predictions at timestep T and k is the number of required predictions. Yr is used denote the agent's ground truth trajectory timestep T. For simplicity, we omit T2 by assuming Te = 1 in the following context.
2 Tobs + Tpred, the qualities of previous predic- tions ( (Wrees..., Wo-aaraa of agent be verified by the ground-truth trajectories
Individual Feedback
When T
3

---

Published as a conference paper at ICLR 2025
Obsevvation
Prodiction
Ground Truth
P
1
Observation Erooder
Fontideece Network
JM
Prafistion Enosder
Mau Fesling
Adjuntr
Coufience Network
Candadate Filtring
Current Uncoder
Indiciduul
Candidate Predictions
Train Path
B
Illustration of the overall IAN structure including both training and testing stages. Train/test path refers to steps that are only present during training/testing. The 'Confidence Net- work's in the figure are in fact the same network, duplicate its block representation for better clarity. is acquired according to Eq. The proposed displacement expectation loss illustrated in the upper-right part of the figure.
Figure 2:
{Y Tor YT--peed Tpred We consider the similarity and disparity between the two bring individ- ual feedback F, revealing the performance, including the strengths and weaknesses, of the model's predictions this agent. This feedback information can be used to adjust the current predictions Yr to better results Y with a function F, i.e. Y7 F(Yr, F), w.r.t. d( Yr) s d(īr, Yr), where d refers to the evaluation metrics for trajectory prediction.
4 INTERACTIVE ADJUSTMENT NETWORK
4.1 OVERVIEW
As motioned in Sec. when T' 2 Tobe Tpreds we can learn pieces of feedback from previous predictions and corresponding ground truths with feedback embedding module f by
F = -
(Ye, Xi) Tahe StsT
Tpred
Considering each piece of feedback information F; is related to the corresponding observed trajec- tory Ot, raise feedback aggregation operation g that first integrates the observation feature into each piece of feedback and then aggregates all of them together as whole. The full process can be written
F GF (I( Y., = (((F), O,)) Tobe StsT Tpred
where Gr(-) denotes the feedback generator and F is the individual feedback which is further used as set of (dynamic) parameters of the adjuster Ap-
In this manner, the adjuster is aware of individual feedback information, and be used adjust the current predictions Yr with the observation Of and generate series of proposals S regarding the future trajectories,
S = (51, s 55 AF (Eeurr(O:: Yr)
where Fourr refers to encoders for the current observation and predictions, and is the number of required predictions the multi-modal setting. The proposals are then used to filter the candidates queried from the prediction model to get the final results
Y + (M,
(5)
I(Y ( Yr-re are ground-truth trajectories of previous timesteps, which can be already observed the current time step We do NOT involve ground-truth trajectories of the current prediction Yr as input.
4

---

Published as a conference paper at ICLR 2025
where @ refers to the proposal query & filtering process.
Eucsók
Fig. 2 and Fig. B illustrate the chitecture our proposed approach. And in the following Sec. 4.2 Sec. 3.3 Sec. we respectively discuss the feedback generator, the adjuster and the query & filtering pro- cess in detail. Then Sec. 4.5 we explain how to train IAN with a novel displacement expectation loss. The implementation details are provided the technical appendices.
Prefrtion Eauader
Mat Pieling
MLP
Groand Truil
Excader
Atamiann
Observation
Feedback Embedding
Prediction
Cencatenation
Figure 3: Illustration of feedback embedding process and observation integration. Here Tohs S 215 T Tpred-
4.2 FEEDBACK GENERATOR
According to Eq. and 3 the feedback generator GF learns the whole individual feedback F for the current predictions Yr from series of previous observations {Ot). predictions {We), ). and cor- responding ground truths (Yr in Tobs t T Tpred with two modules, feedback embedding f and feedback aggregation g.
Feedback Embedding To obtain each piece of feedback Fú, first encode each trajectory Ý e V. and Y, into deep representations fłद and R. with prediction encoder Epred and ground-truth encoder gg
(6) Then, considering the attention mechanism is a common approach that helps models to learn the correlation between data, we introduce an attention gate to discovery the key similarity and disparity between features the ground truth RG and each prediction ft;
RR Rt Expred b=) - 1.2,..,, .k:
Rg = Egg (Y,)
Refe
R Vd
& softmax
R
In this manner, the result oj indicates the verification a prediction Y by the ground-truth Y,. is together get piece of feedback Fe with max pooling, considering is an effective and efficient symmetric operation.
the dimension of RG Finally, aggregate {8 .... k}
Fi maxpool ({8))
Feedback Aggregation With pieces of feedback embedding F,, we then aggregate them together for the whole individual feedback F. Considering the prediction model predicts differently on dis- tinct observations, the feedback will also vary among the observations. To this end, we first integrate the observation features into each feedback embedding,
F = mlp (Fabs (Ot)FF.)
where Eobs is the observation encoder, and [. denotes the concatenation operation. After that, all pieces of observation-awan aware feedback embedding Fü are fed to an aggregation function for F. We max pooling in our implementation for its symmetric property, formally,
F
maxpool (LFF (Tabs < t - -Tpred))
(10)
4.3 ADJUSTER FOR PROPOSAL
In Eq. adjuster is derived from the individual feedback F and generates series of proposals S given the current observation Or and predictions 7z. Inspired by previous works (Jia et al. 2016 Tian et al. 2020). we dynamically generate an mlp adjuster by regarding the feedback as its parameters, denoting as Ap. That is Ar() mlp(:: F). This mechanism enables the feedback information to directly influence the input of the adjuster, i.e. Yr and Or, and offers adjusted proposals for future predictions as output. The encoders for Or and E share the same architecture with Fats and Fppred-
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Candidate Filtering
Input: Confidence Matrix M R Candidate Predictions {Yzlj 1, .... 17
Output: Set 7 consisting of filtered candidates;
Initialize: W7 (): Index p.
while M o [- -0)x kxy do
P.q arg max M;
P-4
Add Y to Y7;
Set Mp.: -0, M. = -0;;
end while
return 77:
4.4 PROPOSAL QUERY & FILTERING
So far, we have adjusted the current predictions and acquired set proposals S {syli 1, J.2,...,k k}. these trajectories may contradict the prediction model since the adjustment process is not aware of the features learnt by the model. In this part, we introduce how to use such proposals to query the prediction model for prediction candidates and further filter them to get the qualified final results as 5
Although these proposals can be directly decoded into exact trajectories as final results,
Given the proposals S, we first query the prediction model .M for y candidate predictions (ÝA 1,2,.... n7 with 77 2 (we use 7j = 200 in our experiments), then a confidence matrix can be calculated
M [m:/i /i = 1,2, .. . E RR m; = (4),)) 15= 1,2, .. .n) -
(11)
where d(-) is a confidence network, and m² E M is the confidence of the i-th proposal being coherent with the j-th candidate prediction. We then run a greedy algorithm (shown in Alg. I filter & candidates as the final results in V7. As ls is usually small value, egg. 20, the computational overhead of this loop is negligible.
4.5 TRAINING
Training Set Collection We first distinguish the training/test set of the prediction model M and IAN with IP and A respectively. A special point about A is that it requires not only the observations and the ground truths, but also the predictions from M according to Eq. 3 The former ones are available in while the latter one is not. Therefore, these predictions should first be collected to build the training set of A. Intuitively, they can be acquired by using M to infer on the training set of P. Yet this practice flawed. Since M has already been specifically optimized on the training set of IP but has never seen the test set of F. there is considerable gap between the distribution of M's predictions on the training and test set of IP. This further leads to inconsistency between the distribution of training data and test data of A.
To tackle this problem, we draw on the idea of K-fold cross validation. Specifically, the training set of P is first split into K folds. By considering each fold of training data as pscudo test set and the rest K 1 folds together as a new training set, K sub-datasets are created. Then we train and test M on each sub-dataset, producing K prediction models and K groups of pseudo test predictions. When K is large enough, the prediction models trained on the sub-datasets will be similar as that trained on the full training set (i.e. the training of P), and thus the distribution of these predictions on the pseudo test sets will be consistent to that on the real test set produced by the model trained on the full training set. In this way, the K groups of pseudo test set predictions can be reasonably used as the predictions for the training set of A. We use K = 5 in our experiments.
Displacement Expectation Loss IAN cannot be directly optimized by the error of the predictions not given by IAN and thus the gradient cannot be back
due to the fact that the predictions propagated to IAN. To tackle this issue, we raise a novel displacement expectation loss for training and its application process is shown in Fig.

---

Published as a conference paper at ICLR 2025
During the training process of IAN, first acquire S with & proposals for training sample
cording to Eq. 4 Afterwards, we take N predictions from the prediction model and calulate the confidence matrix M &x following Eq. We use N 200 in experiments. Meanwhile, we calculate the displacement error between all N predictions and the ground-truth future trajectory Yr
e=
le le, d (1 YT) j-= 1,2, ,N] e R
(12)
where d(-) is the ADE distance.
Observing the confidence score reveals the probability select a candidate for a certain proposal, with both the confidence score matrix M and the error vector we calculate the expected error of each proposal as
{softmar (m.) e/i 1,2, ,.2,
(13)
which can reveal both the quality of the proposals and the effectiveness of the filter network. By opti- mizing the expectation error, IAN can gradually learn from the feedback to generate both reasonable proposals as well as high-quality
prrppooaa filter.
Considering the mudti-modal nature of trajectory prediction, simultaneously optimizing all the expectations will have negative impact and cause mode collapse on the proposals, therefore, we refer to the winner-takes all optimization technique and define our loss function as
L min
(14)
5 EXPERIMENTS
5.1 EXPERIMENT SETTINGS
Benchmarks We conduct experiments the following three widely-used benchmarks. ETH Pel- legrini et al. /UCY (Lixe-Taixé et al. 2014) Dataset is one of the most commonly used bench- marks. We follow Alahi et al. (2016) for the leave-one-out evaluation and observatioo//preiicion horizon. Grand Central Station Dataser (GCS) (Zhou et al. 2012) contains trajectories extracted from 30-min video recorded the Grand Central Station. 8 steps (3.7 seconds) are for observation and 12 steps (4.8 seconds) are for prediction following Gupta et al. (218)). We split the first 80% of the dataset for training, and the rest 20% for test. NBA Sports VU Dataset (NBA) (limouk23 2016) contains trajectories of all ten players in real NBA games. We adopt the traditional setting of steps (2.0 seconds) for observation and 10 steps (4.0 seconds) for prediction. We select 50k samples in total from the 201 5-1016 season with a split of 65%, 10%, 25% as training, validation and testing data following Li et al. 2020)). It is worth noting that since we only consider data samples with at least one piece of feedback available and ignore the rest, i.e. those appear in the scene for less than Tobs + Tpred timesteps, the reported baseline performances are different from their original value.
We use the common metrics ADE/FDE for evaluation. In the multi-modal prediction setting, both metries calculated the minimum over all k trajectories. We set & to 20 following the common setting (Gupta et al. 2018).
Metrics
Prediction Models We conduct experiments on the following five representative prediction mod- els. Social GAN (Gupta et al.] 2018), GAN based prediction framework modeling the social interactions with a pooling mechanism. SGCN (Shi et al. 2021). a graph convolutional network that learns motion tendency with a temporal graph and social interactions with a directed spatial graph. LB-EBM (Pang et al.) 202I). a probabilistic model with cost function defined in the latent space to account for the movement history and social context. GroupNer (Xu et al. 2022a), an en- coding framework that models social interactions with multi-scale hypergraphs. We use GroupNet on CVAE with their official implementation (sjtuxcx 2022). TUTR (Shi et al. (023), a transformer encoder-decoder architecture that unifies the trajectory prediction components, social interactions, and multimodal trajectory prediction. SingularTrajectory (Bae et al. 2024). a diffusion- based uni- versal trajectory prediction framework designed to bridge the performance gap across five tasks. We use official models for testing if available, otherwise we train models according to the official implementations.
7

---

Published as a conference paper at ICLR 2025
Obsavatioo
Ground
Original Prediction
Adjusted Prediction
Figure 4: Qualitative analysis of the predicted trajectories before and after IAN is applied. The plotted predictions are the with minimum among total of 20 predictions.
Table I: Performance (ADE/FDE) of representative prediction models before and after IAN is ap- plied on three benchmarks (1:ETH/UCY; 2:GCS; 3:NBA). All the baseline performances are evalu- ated models trained with corresponding official implementations. Since ignore test samples with no feedback, the reported baseline results are different from the original values. ST stands for SingularTrajectory.
Method
Social GAN
SGCN
LB-EBM
GroupNet
TUTR
Baseline
0.2270.39
0.28 7
0.2070.36
0.2170.35
0.2370.39
0.2070.30
W/N
0.20 0.34
0.26 0.44
0.18 0.31
0.19 0.30
0.22 0.36
0.19 0.29
Impr.
9.1% 12.8%
7.1% 8.33%
10.0% 13.9%
9.5% 14.3%
4.3% 7 7.7%%
5,0% 73.3%
Baseline
4.47/7.40
4.1076.51
3.1975.24
166574.05
2.81 74.40
3.05 74.60
2
w/ TAN
4.28/7.04
3.79 /5.4
2.98 / 4.72
2.56 / 3.83
2.72 / 4.25
2.96 / 4.45
Impr.
4..9%%
7.6% 7888%
6.6% 7 99.9%
3.4% 7 5.4%
3.2% 7.44%
3.0% 7.33%
Baseline
1.53/2.24
1.56
1.40 2.08
1.17
1.23 1.93
1.24 1.60
IAN
/2.1/
1.44/2.9
1.94
1.13/
Impr.
3.88%
7.7% 71.0%
4.3% 6.7%
3,4% 4.99%
3.3% 74.7%
733.1%
5.2 MAIN RESULTS
We first quantitatively analyze the effectiveness of our approach. Results in Tab. show that sub- stantial performance improvements are achieved for all the prediction models on three benchmarks after applying IAN to model the individual feedback information. Particularly, improvements up to 10.0% for ADE/FDE are achieved on strong baselines such LB-EBM and GroupNet. In some cases, the improvements brought by IAN is about 3%. Such improvements on these benchmarks, as similar improvements were achieved by recent of-e-of-theear prediction models (Shi al. 2023 Bae al. 2024) over their respective predecessors. We find such phe- nomenon intuitive as performances on these benchmarks advance towards the limits.
still significant
We further compare the adjusted predictions against the original ones with visualization in Fig, These cases demonstrate that IAN can successfully adjust predictions to more accurate ones in terms of the moving direction (e.g. b, c), acuteness of turning (e.g. d), and velocity (e.g. e, f), etc.
8

---

Published as a conference paper at ICLR 2025
History
Obsarvation
Ground Truth
Original Prediction
Adjusted Prediction
(a)
(b)
(c)
(d)
Figure 5: Examples of how IAN learns from feedback. (a. b): positive feedback, and (c, d): negative feedback. Figures in the first second row shows the corresponding feedback from some while ago. And the third row illustrates the predictions after adjustment.
illustrations of current predictions before adjustment. The
5.3 ABLATION STUDY AND ANALYSIS
Learning from Individual Feedback To better demonstrate how the individual feedback con- tributes to the adjustment, we show visualizations in Fig. 5 that IAN learns from either positive (a, b) or negative (c, d) feedback adjust the current predictions into more accurate ones.
Fig. a illustrates a jittery trajectory. The original predictions fail to focus on the true direction of the agent's 's movements and largely mislead by the jitters. Luckily, there is piece of feedback from similar situation that an accurate result is achieved with more diverse predictions. Based on experience, IAN similarly increases the diversity of the current predictions and decreases the error. Fig. b is another challenging case of double-turning, Leveraging the feedback of success- fully predicting the other turning after the first one, IAN increases the tendency of predictions with opposite-direction turns in this similar scenario.
Fig. 5 and d, the feedback records previous situation with similar observation and predictions the current one, which finally turns failure according the ground truth. Learning from this negative feedback, i.e. insufficient diversity in e and lack sharp turning in d, IAN adjusts the original predictions to compensate the weaknesses of the original predictions and generates much better results.
Contribution Analysis Compared with the baseline prediction models, our approach introduces additional historical information. To demonstrate that the improvement is induced by the feedback rather than additional information, we further compare with two ablative approaches which use the same historical information ours. i) Baseline w/ fullobs: The baseline prediction model using all the observation since the predicted target appears instead using the observation with a fixed length. ii) IAN w/o feedback: An IAN architecture where the individual feedback F is replaced by an embedding of the predicted target's full history trajectory, extracted by an L.TM.. Results are shown in Tab. [
Although intuitively incorporating additional historical information would lead to performance im- provements, simply adding additional observation directly to the baseline models (i.e. Baseline w/

---

Published as a conference paper at ICLR 2025
Table 2: Comparison of ADE/DDE different ablative approaches against IAN. ST stands for SingularTrajectory.
Baseline w/ fullobs
IAN
Direct
M
Dataset
Baseline
IAN
w/o feedback
Decoding
ETH/UCY
0.30/0.36
0.3170.56
0.19/ /.33
0.23/0.38
0.18 / 0..1
LB-EBM
GCS
5.29
4.06 6.70
3.117 5.03
3.12
2.98 4.72
NBA
1.4072.08
1.7872.72
1.367 2.00
1.472.011
1.347 1.94
ETH/UCY
0.21/0.33
0.27 /0.49
0.20 0.33
0.21/0.33
0.19 / 0.30
GroupNet
GCS
2.65 4.00
2.60 3.95
3.25
2.56 / 3.83
NBA
1.17/
1.49 1.92
1.16 1.59
1.44/2.00
1.13 / 1.56
ETH/UCY
0.20/0.30
0.2170.32
0.2070.30
0.22/0.33
0.197 0.1970.29 0.29
ST
GCS
3.05 /4.60
3.15/4.71
3.03/ /.556
3.17
2.96 / 4.45
NBA
1.2471.60
1.27/1.74
1.237 1.58
1.2971.60
1.207 1.55
fullobs) actually resulted in performance decline. This indicates that the baseline networks do successfully handle long observation with arbitrary length so that they may not exploit useful infor- mation but get additional noise. In comparison, the IAN architecture can effectively obtain steady boost.
Compared with IAN w/o feedback, the original IAN achieves much better performance. This demonstrates that the modeling of feedback is effective to bring more improvements upon just using additional historical information.
Interactive Architecture As is described in Sec. 4.4
proposed IAN obtains the final results by querying the prediction model with proposals instead of directly decoding them. We argue that directly decoding the proposals into trajectories is flawed, since the adjustment process is not aware of the features learnt by the prediction model. In Tab. P we compare the results of direct decoding against IAN. While IAN achieves significant improvements, the direct decoding approach faces certain performance decrease many of these experiments.
5.4 INFERENCE TIME
As an external module for a trajectory prediction model, IAN does not operate on the prediction model itself but rather after it. In other words, the prediction model and IAN are consecutive modules in pipeline and can run simultaneously. Therefore, using IAN will not slow down the prediction model produce predictions. Please refer to Sec. E. the Technical Appendices for more details.
Under our test enviromments with single RTX3090, IAN takes average of 0.02 seconds to produce the adjusted predictions for an agent. a reference, LB-EBM, GroupNet, TUTR and SingularTrajectory take 0.03, 0.04, 0.05 and 0.02 seconds respectively to make the predictions. Our approach can conduct inference at a high frequency while bringing substantial improvements.
CONCLUSION
In this paper, we study the individual feedback to reveal the prediction model's performance for specific agent, which has not been studied in previous research. An interactive adjustment network (IAN) is then proposed to learn and leverage valuable experience from the past feedback to aid in the present prediction with small computational overhead. IAN analyzes and aggregates the feed- back information from previous predictions on the target and uses it to make adjustments to current predictions in an interactive manner with the prediction model. Moreover, novel displacement expectation loss is proposed to train the IAN. Exhaustive experiments demonstrate the effectiveness of our approach multiple prediction models across various benchmarks.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 961-971, 2016.
Inhwan Bae, Jin Park, and Hae-Gon Jeon. Learning pedestrian group representations for multi- modal trajectory prediction. In Proceedings of the European Conference on Computer Vision, 2022.
Inhwan Bac, Young-Jar Park, and Hae-Gon Jeon. Singulartrajectry:: Universal trajectory predictor using diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
Yuning Chai, Benjamin Sapp, Mayank Bansal, and Dragomir Anguelov. Multipath: Multiple prob- abilistic anchor trajectory hypotheses for behavior prediction. arXiv preprint 2019.
Yonghao Dong, Le Wang. Sanping Zhou, and Gang Hua. Sparse instance conditioned multimodal trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9763-9772, 2023.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: So- cially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2255-22544 2018.
Tsubasa Hirakawa, Takayoshi Yamashita, Toru Tamaki, and Hironobu Fujiyoshi. Survey on vision- based path prediction. In International Conference on Distributed, Ambient, and Pervasive Inter- actions, pp. 18-64. Springer, 2018.
Boris Ivanovic, James Harrison, and Marco Pavone. Expanding the deployment envelope of behavior prediction via adaptive meta-learning. In 2023 IEEE International Conference on Robotics and Automation (ICRA). pp. 7786-7793. IEEE, 2023.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue V Gool. Dynamic filter networks. Advances in neural information processing systems, 29, 2016.
Laura Leal-Taixe Michele Fenzi, Alina Kuznetsova, Bodo Rosenhahn, and Silvio Savarese. Learn- ing image-based motion context for multiple people tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3542-3549, 2014.
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B. Choy, Philip H. S. Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. Advances in neural information processing systems, 33:19783-19794, 2020.
Rongqing Li, Changsheng Li, Dongchun Ren, Guangyi Chen, Ye Yuan, and Guoren Wang. Bediff: Bidirectional consistent diffusion for instantancous trajectory prediction. Advances in Neural Information Processing Systems, 36, 2024.
Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiaogang Wang. Scene graph generation from objects, phrases and region captions. The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Junwei Liang, Lu Jiang. Juan Carlos Niebles, Alexander G Hauptmann, and Li Fei-Fei. Peeking into the future: Predicting future person activities and locations in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5725-5734, 2019.
linouk23.
Nba
player movements.
Bttps://gitann com wwtp://ow/..nnnn....mmmmm
NRA-Plane PllayerMMovemen 2016.

---

Published as a conference paper at ICLR 2025
Mancheng Meng, Ziyan Wu, Terrence Chen, Xiran Cai, Xiang Zhou, Fan Yang, and Ding- gang Shen. Forecasting human trajectory from scene history. In S. Koyejo, Mo- hamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural In- formation Processing Systems, volume 35, pp. 24920-24933. Curran Associates, Inc., 2022. URL https::/pprocee:eeee w.w. neurips cc/ httpppperr.fllspppppp///000 Paper Conference. pdf
Bo Pang. Tianyang Zhao, Xu Xie, and Ying Nian Wu. Trajectory prediction with latent belief energy-based model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pat- tern Recognition, pp. 11814-11824, 2021.
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In 2009 IEEE 12th International Conference on Computer Vision, pp. 261-268. IEEE, 2009.
Andrey Rudenko, Luigi Palmieri, Michael Herman, Kris M. Kitani, Dariu M. Gavrila, and Kai O. Arras. Human motion trajectory prediction: A survey. 2019. doi: 10448550ARRXIV.1966001/
URL
wttp:///ggabb///////////// 06113
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical con- straints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1349-1358, 2019.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasee feassbbee trajectory forecasting with heterogeneous data. In European Conference on Computer Vision, pp. 683-700. Springer, 2020.
Liushuai Shi, Le Wang, Chengjiang Long, Sanping Zhou, Mo Zhou, Zhenxing Niu, and Gang Hua. Sgen: Sparse graph convolution network for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8994-9003, 2021.
Liushuai Shi, Le Wang, Sanping Zhou, and Gang Hua. Trajectory unified transformer for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9675-9684, 2023.
sjtuxex.
2022.
Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu. Three steps to multimodal trajectory predic- tion: Modality clustering, classification and synthesis. In Proceedings of the IEEE/CVF Interna- 2021.
tional Conference on Computer Vision, pp.
Jianhua Sun, Yuxuan Li, Liang Chai, Hao-Shu Fang, Yong-Lu Li, and Cewu Lu. Human trajec- tory prediction with momentary observation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6467 6476, 2022.
Jianhua Sun, Yuxuan Li, Liang Chai, and Cewu Lu. Modality exploration, retrieval and adaptation for trajectory prediction. JEEE Transactions on Pattern Analysis and Machine Intelligence, 45 (12):15051-1566) 2023a.
Jianhua Sun, Yuxuan Li, Liang Chai, and Cewu Lu. Stimulus verification a universal and effective sampler in modai human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22014-22023, 2023b.
Neerja Thakkar, Karttikeya Mangalam, Andrea Bajesy, and Jitendra Malik. Adaptive human tra- jectory prediction via latent corridors. In European Conference Computer Vision (ECCV), 2024.
Zhi Tian, Chunhua Shen, and Hao Chen. Conditional convolutions for instance segmentation. In European Conference on Computer Vision, pp. 282-298. Springer, 2020.
Waymo. Waymo open dataset motion prediction challenge. challeegges20224/moo http:///222mmmmooomoooo
https: waymo. wwm/ open./
ct on/ 2024.
12

---

Published as a conference paper at ICLR 2025
Conghao Wong, Beihao Xia, Ziqian Zou, Yulong Wang, and Xinge You. Socialcircle: Learning the ungle-based social interaction representation for pedestrian trajectory prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19005-19015, 2024.
Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. Groupnet: Multiscale hyper- graph neural networks for trajectory prediction with relational reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6498-6507 2022a.
Yi Xu, Lichen Wang, Yizhou Wang, and Yun Fu. Adaptive trajectory prediction via transferable gnn, Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp.
2022b.
Biao Yang, Fucheng Fan, Rongrong Ni, Hai Wang, Ammar Jafaripoumimchahi, and Hongyu Hu. A multi-task learning network with collision-aware graph transformer for traffic-agents trajectory prediction. IEEE Transactions on Intelligent Transportation Systems, 2024.
Bolei Zhou, Xiaogang Wang, and Xiaoou Tang. Understanding collective crowd behaviors: Learn- ing mixture model of dynamic teasstrraanagnnts.. In 2012 IEEE Conference on Computer Vision and Puttern Recognition, pp. 2871-2878. IEEE, 2012.
13

---

Published as a conference paper at ICLR 2025
A NOTATIONS
As our approach involves more notations than average, we provide table of key notations used in paper and their meanings in Tab. 3 for better clarity.
Table 3: Notations and their respective meanings used in our paper.
Notation
Mcaning
Toba
The observation horizon, a constant value.
Tpred
The prediction horizon, a constant value.
The timestep when the prediction target appears (assumed equal to I for simplicity).
Ta
i.e. target is observable from the start, in our paper.
T
The timestep of 'current' prediction.
M
Prediction model, NOT IAN.
Observation sequence with length Tobs:
O.
subscript denotes the timestep of its last element.
The target's 's ground truth future trajectory with length Tpred- subscript denotes the timestep right before first element.
Y.
The target's future trajectory predicted by M, subscript denotes the timestep right before its first element.
Y,
Y
A set containing Y.
k
The number required predictions under multi-moda trajectory prediction setting.
K
The number of splits used for training set collection, see Sec. 4.5
B IAN PIPELINE
In order to better formulate the IAN's pipeline at both training and testing stages, illustrate the pipeline via pseudo-codes in Alg. 2 and
Algorithm 2 IAN Training
Input: Training data Attainn = { (Hę. Yg. Yr, 04
U
VT > Ta + Tobe + Tpred - 1
aEagents
with HIg ( Y7, O2)[Ta Tobs 1 S t T
Tpred] j.
Y7 M(O Fz = M(O7). Y8 = k, |
N;
Output: Trained feedback generator GF, adjuster A, and confidence network Initialize: Gr, A, d;
for all Hy: 1/9 Y7, Og.) Atrain do
Calculate F F (Hq.) according to Eq.
10
mll Calculate S = {31, $2, s&F according to Eq.
Parameterize with (Sec. 4.3 Ar(.)
F);
Calculate confidence matrix according to Eq.
Calculate displacement expectation loss L according to Eq 12 14
paok-pro and optimize GF, A, d; end for return Gp, A, o:
C IMPLEMENTATION DETAILS
In our implementation, we use LSTM networks for all the encoders in IAN, and the output sizes of these encoders are all 64. Each proposal generated by the adjuster has dimension of 32. The adjuster a four-layer mlp with embedding sizes (128, 256, 512, 640, 640) and individual feed- back F serves the biases of its last two layers for low computational cost and a good convergence.
14

---

Published as a conference paper at ICLR 2025
Algorithm 3 IAN Inference
Input:
Va Tr 07) with 2 Ta + Tobs + 7pred 1. Hig
H = (37, Yr" OT) + Tobs - -1st t T - Tpred
Y7 M(())) 87 M(O)), Y/ k. | 17.
Output: Filtered trajectories 77
Calculate F = F (H)) according to Eq
10
Parameterize A with F (Sec. 4.3 AF
mlp : F);
Calculate $ = {81, $2, Skj according to Eq.
4
Calculate confidence matrix M according Eq.
return Sennddatt_Filterinnee 14) (Alg.
Accordingly, the mlp used during feedback aggregation has 3 layers and an output size of 1280. The confidence network first encodes the input trajectory into deep feature, then concatenate it with the proposal. The concatenated feature is fed to triple-layer mip with input size of 96 and outputs a single number as the confidence score. We use = N = 200 for all prediction models except TUTR, where set 7) N L (notation L indicates the number of 'general motion modes' in the TUTR paper) for ETH/UCY, and use 7 = N = 80 for the other datasets. During collection of the training set, the original training set is split into K epochs using Adam optimizer. In this paper, we primarily focus on proposing the idea of individual feedback and introducing how the IAN architecture can effectively leverage individual feedback to improve trajectory prediction. Therefore, we do not delve into adopting more complex structures for each module of IAN, e.g. using a heavier encoder instead of LSTM.
folds. The network is trained for 30
D ADDITIONAL RESULTS
D.1 DETAILED PERFORMANCES
In Tab. we give the detailed performances and widths of 95% confidence intervals the prediction models with/without IAN on all datasets. Improvements are achieved across all the sub- datasets, further proving IAN's effectiveness. The confidence intervals are acquired using results from 5 separate runs.
D.2 ALTERNATE METRICS
To more comprchensively evaluate the performance of IAN, we additionally report the miss rates (Waymo] 2024) of base model predictions with and without IAN on ETH/UCY dataset in Tab. We also report the brier-ADE/FDE following TUTR (Shi et al. 2023 in Tab, 6 These results demonstrate the consistent improvements provided by the IAN across various metrics.
D.3 MORE ABLATIONS
Number of Candidate Predictions Sec. 4.4 we use the proposals filter y candidate predic- tions to get the final results. We investigate the influence of different values of in Tab. 7 (left). When 7) is relatively small, the candidates are usually not sufficient to satisfy all the proposals with high confidence scores. Then 7) grows larger, the performance improves fast since more candi- dates that are highly coherent with the proposals are available. When r) is large enough, all proposals may have already been satisfied with high confidence scores and the performance changes little varies.
N in Displacement Expectation Loss In the proposed displacement expectation loss, the larger N is, the closer the calculated expectation is to the ideal one, and thereby this tends to bring better performance. We show the influence of different values of N in Tab. (right).
15

---

Published as a conference paper at ICLR 2025
Table 4: Detailed performances of IAN and widths of confidence intervals with various prediction models (1:EHH; 2:HOTEL; 3:UNIV; 4:ZARAI; 5:ZARA2; 6:GCS; 7:NBA). ST stands for Singu- lar Trajecctor..
Method
SGAN
SGCN
LB-EBM
GroupNet
TUTR
Baseline
0.293 0.444
0.441 / 0.669
0.327 /0.567
0.285 0.393
0.414 0.628
0.287 0.346
0.006 0.0009
0.010/0.015
0.007 /0.010
0.003 0.007
0.008 0.011
0.003 0.006
w/ TAN
0.26970.385
1.40370.586
0.2727 0.436
0.231 0.268
0.39170.583
0.28470.333 0.333
0.005 0.007
0.008 0.013
0.007 00011
0.004 / 0.007
0.009 0.011
0.003 / 0.006
Baseline
0.172
0.27
0.09470.133
0.1467 0.226
0.1 0.154
0.160
0.001 0.002
0.002
0.001 0.001
0.001 0.002
0.001 0.001
0.001 0.002
TAN
0.147 0.2331
0.16870.222
0.087 7.08770.120
0.13270.182
0.10970.131
0.10670.115
0.001 0.0002
0.001
0.001 0.001
0.001 0.001
0.001 /00002
0.001 / 0.001
Baseline
0.27570.51
0.338 / 0.602
0.258 0,497
0.259 0.259/0.497 0.497
0.27170.472
0.273/0.463
0.004 0.005
1.004/0.007
0.002 0.004
0.003 0.004
0.003 0.005
0.002 0.004
IAN
0.266 / 0.493
0.314/ 0.548
0.424
0.235 0.438
0.258
0.461
0.261 0.457
0.003 0.004
0.002 / 0.004
0.002 / 0.002
0.002 0.003
0.003 0.004
0.002 / 0.003
Baseline
0.211 7.3994
7.5907
0.198 00.385
0.201 0.334
0.182
0.187 0.333
0.002 0.003
0.002 0.003
0.001 0.002
0.001 0.003
0.002 0.002
0.001 0.002
w/ IAN
0.200 / 0.364
0.281 0.519
0.187 0.358
0.200 0.363
0.178 0.331
0.183 0.327
0.002 0.003
0.002 / 0.002
0.001 0,002
0.001 0.003
0.001 0.002
0.001 / 0.003
Baseline
0.152 0.300
0.165 / 0.320
0.106 / 00211
0.136/0.277
0.162
0.119/0.11 0.197
0.001 0.002
0.002 0.002
0.001 00.01
0.001 0.002
0.001 0.000
0.001 0.002
w/
TAN
0.1317
0.159/ 0.359 0.303
0.1007 0.191
0.126/ 0.242
0.1517 0.320
0.110/ 0.199
0.001 / 0.002
0.001 / 0.002
0.002 / 0,002
0.001 0.002
0.001 / 0.001
0.001 / 0.002
Baseline
4.47 4.47 7.440
4.10/6.51
3.19 5.24
2.6574.80
2.81/4.40
3.0574.66
0008
0.05
0.06 00.1
0.04 0.08
0.03/
0.03 / 0.07
w/ TAN
1.2877.00
5.994
2.987 4,72
2.86
2.7274.25
2.96 4.44
0.05 0.07
0.05 0.08
0.05 0.06
0.00
0.04
0.03/0.00
Baseline
1.53/
1.46
1,40 2.08
1.17/1.64
1,23/1.9 1.93
1.24/
0.01 /0.02
0.01 0.02
0.01 0.02
0.01 /0.03
0.02 0.02/0.03
0.01 0.02
w/ TAN
1.172.11
1.442.19
1.34/ 1.94
I.33
1.191188 1.84
1,207
0.01 0.002
0.01 0.00
0.01 0.002
0.01 /0.02
0.01 /0.02
0.01 0.02
Table 5: Performance (miss rate) of prediction models before and after IAN is applied. ST stands for
M
Method
ETH
HOTEL
UNIV
ZARAI
ZARA2
Social-
Baseline
0.0909
0.0289
0.0611
0.0236
0.0505
GAN
w/ IAN
0.0909
0.0120
0.0473
0.0124
0.0372
Baseline
0.1212
0.0216
0.1249
0.0933
0.0844
SGCN
w/ IAN
0.0909
0.0144
0.0993
0.0833
0.0755
Bascline
0.1320
0.0062
0.0594
0.0185
0.0350
LB-EBM
w/ IAN
0.0644
0.0000
0.0405
0.0116
0.0216
Baseline
0.0303
0,0072
0.0609
0.0087
0.0263
GroupNet
IAN
0.0303
0.0000
0.0391
0.0062
0.0193
Baseline
0.0595
0.0040
0.0452
0.0417
0.0260
TUTR
w/ IAN
0.0536
0.0040
0.0386
0.0243
0.0242
Baseline
0.0606
0.0000
0.0241
0.0050
0.0133
ST
w/ IAN
0.0606
0,0000
0.0231
0.0037
0.0128
Table Performance (Drier-ADE/FDE) of TUTR before and after IAN is applied.
M
Method
ETH
HOTEL
UNIV
ZARAT
ZARA2
Baseline
118/1.39
0.64 0.68
0.98 / 1.19
1.01
0.73/ 0.91
TUTR
w/ IAN
1.14/1.33
0.63 / 0.66
0.96 1.16
0.99/1.1 1.15
0.72/ / 0.89
16

---

Published as a conference paper at ICLR 2025
Table 7: [Left] Comparison between different maximum number of candidates ETH/UCY.
[Right] Comparison between different N on ETH/UCY, ST stands for SingulaTTraecctor
M
100
200
N
100
200
ADE
0.183
0.176
0.175
ADE
0.183
0.181
0.175
LB-EBM
FDE
0.324
0.308
0.306
FDE
0.325
0.320
0.306
ADE 0.201
0.193
0.192
ADE
0.200
0.196
0.192
GroupNet
FDE
0.312
0.296
0.296
FDE
0.324
0.313
0.296
ADE
0.192
0.190
0.189
ADE
0.192
0.190
0.189
ST
FDE
0.298
0.294
0.293
FDE
0.299
0.295
0.293
Demnding
(al)
0bD
622
(b2)
622
Figure 6: Comparisons of predictions acquired by direct proposal decoding and IAN. The observed trajectory is mark with solid lines whereas predictions are marked with dashed ones. The social agents are marked with dots with various colors, and the direction that the dots get bigger indicate the moving direction an agent.
D.4 IAN DIRECT PROPOSAL DECODING
To qualitatively demonstrate the weakness of direct proposal decoding, i.e. the predictions of direct decoding cannot follow the features learnt by the prediction model and thereby they may be flawed, compare the predictions generated by direct decoding against those acquired by querying in We use social-awant approach GroupNet as the prediction model.
Fig. 6
clear that predictions given by direct proposal decoding are not aware of the social behaviors and face a serious problem of social impossibility (marked with black rectangles), although the prediction model has already learnt the social features. For example, in Fig. Gla the prediction at the bottom is highly probable to collide with the orange agent. Similarly, the top prediction Fig. 6bb1 moves directly through an neighbor- ing agent who stands still. A more sophisticated example Fig. 6 shows that the predictions from direct proposal decoding exhibit little consideration to these agents, leading to numerous predictions with high chance of collision. In comparison, this issue is very well addressed by adopting ap- proach. Particularly in Fig. c2, our results show clear tendencies avoiding the green oncoming agents.
For each of the three cases, all of the 20 predictions are plotted.
17

---

Published as a conference paper at ICLR 2025
Otsevation
Octgina Prodktionn
Adjunded Predictio
Figure 7: Consecutive visualizations of feedback pieces and adjustments the same agent over time. [Top] In each of the first 12 frames show the predictions Y. (red), ground truth Y, (green), observation O, (blue), such information will be used in the future by IAN to adjust future predictions. Note that during this period of time, IAN have no access any feedback information and does take effect. [Bottom] In frames 13-24 we show IAN continuously improving the prediction accuracy by adjusting. Note that IAN at frame (133 < < 24) only have access to feedback information from frames j: E bill < j I i 12}, and the ground truth for frame i is not leaked to the IAN.
D.5 MORE QUALITATIVE RESULTS
In Fig. 7 we we give 24 consecutive frames of predictions on the same agent. In the first Tpred 12 frames we show the observation (dark bluc), predicted trajectories (red), as well as the ground truth (green). IAN does not take effect during these 12 frames since ground truth trajectories available during this period. Note that the predicted trajectories are constantly longer than the ground truth during these 12 frames, indicating that the predictions tend to have higher speed.
From the 13th frame, we show the observation (dark blue), best prediction before adjustment (red), best prediction after adjustment (light blue), and the ground truth (green). Note that the i-th frame (13 < i < 24) only has ground truths from frame ji (1 S jr < i Tpred) available for calculating the individual feedback F. These frames with IAN operational clearly show that the adjusted pre- dictions tend to be slower than those prior to adjustments, indicating IAN's capability of acquiring information from the individual feedback i.e. predictions tends be faster than the ground truth. Further, the adjusted trajectory also show trends of gradual improvement.
D.6 DISPLACEMENT EXPECTATION Loss: ANALYSIS
In this paper, we introduced a novel displacement expectation loss to train the IAN. The loss is devised based on the winner-takes-all optimization technique, which is widely used multi-modal trajectory prediction studies (Chai et al. 2019 Xu et al. 2022a Shi et al. 2023). Further, the value of the displacement expectation (Eq. 13 is bounded by the minimum and maximum displacement errors of the predictions from the base prediction model. Therefore, ideally, the displacement pectation will converge to the minimum displacement error during optimization. We additionally provide example training curves in Fig. to demonstrate its converging property.
18

---

Published as a conference paper at ICLR 2025
4/4
G1
622
(a1)
(22)
Figure 8: Example training curves of displacement expectation loss from our experiments. (al&a2) Training of IAN under GroupNet + ZARAI repeated twice. (b1&b2) Training of IAN under LB- EBM + GCS repeated twice.
E MORE DISCUSSIONS
E. INFERENCE TIME
In Sec 5.4 of our paper, we discussed about the base prediction model M and IAN running simul- taneously in a pipeline as consecutive models. Here we provide an illustration such pipeline in Fig. 9
Prediction Model
20ms
Figure 9: Illustration of the prediction model M and IAN as pipeline. Each prediction task is identified with unique color. IAN and M can run at the same time without interfering each other, and thereby using IAN will not slow down M to produce new predictions. The additional runtime for getting better predictions with IAN will not accumulate and is always 20ms.
E.2 DIFFERENCE WITH CERCHER-ENFORIIGG
As teacher-enforcing technique also involves the ground truth, here we clarify that our approach is essentially difffrrnnt from that, to avoid misunderstanding.
Teacher-enforcing is a training technique that feeds ground truths back into the RNN after each step during the training phase. In comparison, 1) approach is external module beyond the base prediction model (teacher-enforcing is a training technique for the base model) and 2) approach works by adjusting the predictions of the base model in the test phase (teacher- enforcing works only in the training phase). Therefore, our approach is essentially different from teacher-enforeing.
E.3 DIFFERENCE WITH PRIOR ADAPTATION WORKS
Prior to IAN, there have been other researches that use the idea of adaptation for better results (Xu al.] 2022b [vanovic al.) 2023), which involves using data from the test domain/scene to Train/fine-tun the prediction model. In comparison, data from the test domain/scene is not used to train IAN. Another major difference between IAN and these approaches is that IAN uses agent- specific information, i.e. individual feedback, and adjusts the predictions for each agent respectively. On the other hand, typical adaptation approaches aggregate the trajectories of all the agents in the
19

---

Published as a conference paper at ICLR 2025
test domain/scene a whole and use it modify the predictions for all the agents in the test do- main/scene.
E.4 DIFFERENCE WITH TEST-TIME TRAINING/ONLINE LEARNING
While IAN operates by adjusting the prediction model's outputs, none of the network (both the base prediction model and the IAN) parameters are updated during test-time. Therefore, IAN is entirely different from test-time training/fine-tnninn
addition, test-time training in fact not compatible with our aim, iee. individual feedback, due to its individual-speccfic nature. Such trait indicates that when actually deployed, distinct model for each of the individuals present in the scene needs to be stored and updated, which can be extremely costly for online deployment embodied agents such as autonomous vehicles and robots.
approaches.
E.5 LIMITATION AND FUTURE WORKS
One scenario where predictions are adjusted in worse way occurs when an agent whose past trajectory is smooth and almost straight suddenly changes direction. Before the change in direction, the base model performs better with more concentrated predictions (e.g. pattern that all predictions are near straight lines with minor changes in direction). Under such circumstances, the adjuster may notice this pattern and therefore adjusts scattered predictions to concentrate more, which will harm the prediction accuracy when the change in direction occurs, examples are shown in Fig. [0] The figure shows some examples this case. are actively addressing this issue as our future work.
History
Observation
Trath
Original Prodiction
Adjusted Prediction
JAB
(al)
(a2)
(b1)
(h2)
Figure 10: Examples the adjuster worsening predictions.
20


Reviews:
[{"review_id": "TPVgaSBWMf", "reviewer": "Reviewer_ykSs", "summary": "In this work, an online learning methodology is proposed for trajectory prediction based on the insight that ground truth observations checking past predictions (named feedback) can be collected as time goes on. To leverage this insight, the authors propose an interactive adjustment network (IAN) and evaluate its efficacy using a variety of trajectory prediction algorithms on a set of pedestrian and athlete motion datasets.", "questions": "The most important question: How does this approach compare to prior works on adaptive trajectory prediction? Why is IAN better or more preferable to use by practitioners?\n\nHow does this approach perform on modern large-scale datasets with multiple types of agents and varying dynamics, e.g., Waymo Open Motion Dataset, nuPlan? These datasets also have less saturated metrics which should make it easier for the IAN's performance improvements to stand out.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The idea is sound and sensible.\n\nThe writing is clear and it is easy to follow the core idea being proposed.\n\nThere are a wide variety of baseline approaches used to demonstrate the core idea.", "weaknesses": "In contrast to the claims in the paper, this work is not the first to study feedback in trajectory predictions. Accordingly, the primary weakness of this work is a lack of discussion and comparisons to prior adaptive prediction work that investigates this same idea. Notable examples (which focus on both temporal adaptation, equivalent to individual feedback, and geographic adaptation) include:\n* Y. Xu, L. Wang, Y. Wang, and Y. Fu, \"Adaptive trajectory prediction\nvia transferable GNN,\" in IEEE Conf. on Computer Vision and Pattern\nRecognition, 2022.\n* B. Ivanovic, J. Harrison, and M. Pavone, \"Expanding the deployment envelope of behavior prediction via adaptive meta-learning,\" in IEEE Conf. on Robotics and Automation, 2023.\n\nThe experimental setup relies primarily on small-scale pedestrian-only datasets. Further, these datasets are quite old (ETH/UCY are 10-15 years old, GCS is more than 10 years old, the NBA dataset is almost 10 years old), meaning performance has largely saturated on them and it is difficult to tell if the proposed method significantly improves upon baselines.\n\nAs stated in Section 5.4, a prediction model and the IAN are _consecutive_ modules, meaning they have to run one after the other (and not simultaneously as stated in Line 523). This means that there is a 20ms overall runtime increase as a result of adding the IAN.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CeM00oSJyy", "reviewer": "Reviewer_dXFb", "summary": "The paper proposes an Interactive Adjustment Network (IAN) that leverages individual feedback from previous predictions to improve human trajectory prediction models. The idea is to use the differences between earlier predictions and actual trajectories (ground truths) to adjust future predictions for the same agent. The authors claim that their method can be applied as an external module to various existing prediction models and that it significantly boosts performance on datasets.", "questions": "1. How do you justify the assumption that ground truth trajectories are available in real-time for computing feedback? In what practical scenarios would this be feasible?\n2. How does your method differ from existing online learning approaches that adapt models based on prediction errors?\n3. How does the computational cost of IAN affect inference time in real-world applications (even a small delay in prediction can go a long way)?\n4. Are there experiments to isolate the effects of each component of your model? If so, please provide detailed results.\nStatistical Significance: Are the reported improvements statistically significant? Can you include variance measures or confidence intervals?\n5. How does your model handle agents for whom no feedback data is available?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The concept of utilizing individual feedback from previous predictions is interesting.\n- The proposed method is designed to be model-agnostic and can be applied to various trajectory prediction frameworks.\n- Experiments are conducted on multiple datasets and with several baseline models.", "weaknesses": "Major issues:\n- The paper's biggest assumption is the fact is that IAN relies on having immediate access to the ground truth trajectories of agents to compute feedback. In real-world applications, example autonomous driving or robotics, such ground truth data is usually never available in real-time. \n- The concept of using feedback from previous predictions is not entirely new. Adaptive models and online learning techniques have long incorporated past errors to improve future predictions. The bigger issue is the lack of adequate differentiation of IAN from existing methods i.e., a thorough literature review that situates its contributions within the broader context.\n- While the authors introduced a new loss function, there is no analysis of the properties of the displacement expectation loss or proofs of convergence and stability.\n- There is a lot of poorly defined notation, making it difficult to understand the proposed method fully. It would make sense to move the table 3 from the appendix (which has the notation!) to the main paper. \n\nThere are some issues with the presentation. \n\nWriting: \n- Section 3, Trajectory Prediction (just before equation 1): \\hat{Y} of \"him\"\n- Section 4.5, \"At the beginning of this paragraph\" \n- Section 5.4, Last sentence: Our approach can \"inference\" at a high frequency\n- Personal preference: please use \"Figure X represents ...\" (not abbreviation) instead of \"Fig. X represents...\" because your captions have the former. \n\nImages:\n- The image labels for Figure 2 are too small to read -- had to Zoom 150% to read it\n- Figure 2, it is hard to tell the difference between Train Path and Test Path, the arrows look very similar, please consider a different color. \n- Figure 4, 5, it is hard to see lighter colors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sYpIvSrnWQ", "reviewer": "Reviewer_ncs6", "summary": "In this paper, an individual feedback framework (termed IAN) for Trajectory Prediction is proposed that dynamically adjusts the confidence score conditioned on prediction consistency between prediction and GT trajectories. Through addtional feedback generator during training, a consistency score could be derived through attention in supervising the prediction condifence. Comprehensive experiment results demonstrate the effectiveness of IAN module when collaborating with a series of state-of-the-art predictors.", "questions": "1. How is the feedback being connected (with A_F for example) in Eq3-4 and Eq9-10, and being differentiated between training and testing stage?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Novel design for trajectory prediction pipeline considering closed-loop decoding (or individual feedback). Through proposed feedback generater and adjuster model in IAN, the multi-modal prediction garners extra feedback consistency supervisions in prediction confidence.\n\n2. Comprehensive experimental evaluations. 1) Broad improvements are reported across various datasets when integrating IAN with several SOTA motion prediction frameworks; 2) Clear comparison with other decoding strategies.", "weaknesses": "1. Unclear writings in methodology: It is partly clear to understand the feedback mechanism by Figure 2, However, by simply go through the content the reviewer could hardly understand the feedback generation process between training and testing. Hence, it is better to provide additional Algorithm part for clearer understanding. \n\n2. Additional evaluation for the feedback. In Tab1-2, minADE/minFDE is a direct metric in mearusing the best-case similarity for multi-modal predicted trajectory. However, it is unclear whether the confidence are well-performed by the feedback enhancement. Hence, several extra results measured by Brier-minADE/FDE, MissRate, or mAP seems needed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IVebY1gEEr", "reviewer": "Reviewer_Bpfi", "summary": "This paper introduces the concept of individual feedback in trajectory prediction problems by designing an Interaction Adjustment Network (IAN). The IAN, comprising a feedback generator, an adjuster, and a filtering process, is designed to be an external module that can be seamlessly integrated with other trajectory prediction models. To ensure its adaptability, the authors have designed a displacement loss function to train the IAN. Experiments have shown the IAN's efficacy on widely adopted benchmarks for trajectory prediction.", "questions": "However, this paper provides a promising way to improve the trajectory prediction method by incorporating the agent-specific past trajectories and groundtruths, here are some comments that need to be addressed\n1. Since the authors introduced the individual feedback in the paper and provided the intuition, it would be good to see any theoretical justification for why it is important.\n2. It would be good to see if the authors can address that incorporating the agent-specific previous trajectories may not lead to overfitting the base-model prediction. \n3. Another important concern is that, since the IAN is heavily dependent on the base-model prediction if there are accumulated errors in the base-model prediction, then the proposal generated from the IAN would also be spurious. What kind of risk assessment should be incorporated into the IAN?\n4. It would be good to see if there is more explanation on the architecture of IAN, including the feature generation network, what kind of encoder architecture is used, and the rationale behind it.\n5. Another thing that would be good is for the authors to include at least some description of the process happening in the figures to help the reader understand the figures.\n6. In the conclusion, the authors should include the limitations of their work and possible future directions. It would also be good to see if authors can report the failure cases.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Following are the strengths of this paper:\n1. The introduction of individual feedback to enhance trajectory prediction by integrating agent-specific past trajectories with groundtruth is promising.\n2. Evaluation of the proposal IAN network on trajectory prediction benchmarks highlights their claim.\n3. As an external module, IAN can be integrated with various trajectory prediction methods.\n4. The presentation of results is good.", "weaknesses": "However, this paper provides a promising way to improve the trajectory prediction method by incorporating the agent-specific past trajectories and groundtruths, here are some comments that need to be addressed\n1. Since the authors introduced the individual feedback in the paper and provided the intuition, it would be good to see any theoretical justification for why it is important.\n2. It would be good to see if the authors can address that incorporating the agent-specific previous trajectories may not lead to overfitting the base-model prediction. \n3. Another important concern is that, since the IAN is heavily dependent on the base-model prediction if there are accumulated errors in the base-model prediction, then the proposal generated from the IAN would also be spurious. What kind of risk assessment should be incorporated into the IAN?\n4. It would be good to see if there is more explanation on the architecture of IAN, including the feature generation network, what kind of encoder architecture is used, and the rationale behind it.\n5. Another thing that would be good is for the authors to include at least some description of the process happening in the figures to help the reader understand the figures.\n6. In the conclusion, the authors should include the limitations of their work and possible future directions. It would also be good to see if authors can report the failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)