Interactive Adjustment for Human Trajectory Prediction with Individual Feedback

Authors: Jianhua Sun, Yuxuan Li, Liang Chai, Cewu Lu
Primary area: applications to robotics, autonomy, planning
Keywords: Human Trajectory Prediction

Abstract:
Human trajectory prediction is fundamental for autonomous driving and service robot. The research community has studied various important aspects of this task and made remarkable progress recently. However, there is an essential perspective which is not well exploited in previous research all along, namely individual feedback. Individual feedback exists in the sequential nature of trajectory prediction, where earlier predictions of a target can be verified over time by his ground-truth trajectories to obtain feedback which provides valuable experience for subsequent predictions on the same agent. In this paper, we show such feedback can reveal the strengths and weaknesses of the model's predictions on a specific target and heuristically guide to deliver better predictions on him. We present an interactive adjustment network to effectively model and leverage the feedback. This network first exploits the feedback from previous predictions to dynamically generate an adjuster which then interactively makes appropriate adjustments to current predictions for more accurate ones. We raise a novel displacement expectation loss to train this interactive architecture. Through experiments on representative prediction methods and widely-used benchmarks, we demonstrate the great value of individual feedback and the superior effectiveness of proposed interactive adjustment network.

Decision: accept (poster)
Reviewer scores: 3, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730509926071

thread_type:
official_review

<thread_object>
creator: reviewer_bpfi

<content>
summary:
This paper introduces the concept of individual feedback in trajectory prediction problems by designing an Interaction Adjustment Network (IAN). The IAN, comprising a feedback generator, an adjuster, and a filtering process, is designed to be an external module that can be seamlessly integrated with other trajectory prediction models. To ensure its adaptability, the authors have designed a displacement loss function to train the IAN. Experiments have shown the IAN's efficacy on widely adopted benchmarks for trajectory prediction.

strengths:
Following are the strengths of this paper:
1. The introduction of individual feedback to enhance trajectory prediction by integrating agent-specific past trajectories with groundtruth is promising.
2. Evaluation of the proposal IAN network on trajectory prediction benchmarks highlights their claim.
3. As an external module, IAN can be integrated with various trajectory prediction methods.
4. The presentation of results is good.

weaknesses:
However, this paper provides a promising way to improve the trajectory prediction method by incorporating the agent-specific past trajectories and groundtruths, here are some comments that need to be addressed
1. Since the authors introduced the individual feedback in the paper and provided the intuition, it would be good to see any theoretical justification for why it is important.
2. It would be good to see if the authors can address that incorporating the agent-specific previous trajectories may not lead to overfitting the base-model prediction. 
3. Another important concern is that, since the IAN is heavily dependent on the base-model prediction if there are accumulated errors in the base-model prediction, then the proposal generated from the IAN would also be spurious. What kind of risk assessment should be incorporated into the IAN?
4. It would be good to see if there is more explanation on the architecture of IAN, including the feature generation network, what kind of encoder architecture is used, and the rationale behind it.
5. Another thing that would be good is for the authors to include at least some description of the process happening in the figures to help the reader understand the figures.
6. In the conclusion, the authors should include the limitations of their work and possible future directions. It would also be good to see if authors can report the failure cases.

questions:
However, this paper provides a promising way to improve the trajectory prediction method by incorporating the agent-specific past trajectories and groundtruths, here are some comments that need to be addressed
1. Since the authors introduced the individual feedback in the paper and provided the intuition, it would be good to see any theoretical justification for why it is important.
2. It would be good to see if the authors can address that incorporating the agent-specific previous trajectories may not lead to overfitting the base-model prediction. 
3. Another important concern is that, since the IAN is heavily dependent on the base-model prediction if there are accumulated errors in the base-model prediction, then the proposal generated from the IAN would also be spurious. What kind of risk assessment should be incorporated into the IAN?
4. It would be good to see if there is more explanation on the architecture of IAN, including the feature generation network, what kind of encoder architecture is used, and the rationale behind it.
5. Another thing that would be good is for the authors to include at least some description of the process happening in the figures to help the reader understand the figures.
6. In the conclusion, the authors should include the limitations of their work and possible future directions. It would also be good to see if authors can report the failure cases.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730542937946

thread_type:
official_review

<thread_object>
creator: reviewer_ncs6

<content>
summary:
In this paper, an individual feedback framework (termed IAN) for Trajectory Prediction is proposed that dynamically adjusts the confidence score conditioned on prediction consistency between prediction and GT trajectories. Through addtional feedback generator during training, a consistency score could be derived through attention in supervising the prediction condifence. Comprehensive experiment results demonstrate the effectiveness of IAN module when collaborating with a series of state-of-the-art predictors.

strengths:
1. Novel design for trajectory prediction pipeline considering closed-loop decoding (or individual feedback). Through proposed feedback generater and adjuster model in IAN, the multi-modal prediction garners extra feedback consistency supervisions in prediction confidence.

2. Comprehensive experimental evaluations. 1) Broad improvements are reported across various datasets when integrating IAN with several SOTA motion prediction frameworks; 2) Clear comparison with other decoding strategies.

weaknesses:
1. Unclear writings in methodology: It is partly clear to understand the feedback mechanism by Figure 2, However, by simply go through the content the reviewer could hardly understand the feedback generation process between training and testing. Hence, it is better to provide additional Algorithm part for clearer understanding. 

2. Additional evaluation for the feedback. In Tab1-2, minADE/minFDE is a direct metric in mearusing the best-case similarity for multi-modal predicted trajectory. However, it is unclear whether the confidence are well-performed by the feedback enhancement. Hence, several extra results measured by Brier-minADE/FDE, MissRate, or mAP seems needed.

questions:
1. How is the feedback being connected (with A_F for example) in Eq3-4 and Eq9-10, and being differentiated between training and testing stage?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730577256575

thread_type:
official_review

<thread_object>
creator: reviewer_dxfb

<content>
summary:
The paper proposes an Interactive Adjustment Network (IAN) that leverages individual feedback from previous predictions to improve human trajectory prediction models. The idea is to use the differences between earlier predictions and actual trajectories (ground truths) to adjust future predictions for the same agent. The authors claim that their method can be applied as an external module to various existing prediction models and that it significantly boosts performance on datasets.

strengths:
- The concept of utilizing individual feedback from previous predictions is interesting.
- The proposed method is designed to be model-agnostic and can be applied to various trajectory prediction frameworks.
- Experiments are conducted on multiple datasets and with several baseline models.

weaknesses:
Major issues:
- The paper's biggest assumption is the fact is that IAN relies on having immediate access to the ground truth trajectories of agents to compute feedback. In real-world applications, example autonomous driving or robotics, such ground truth data is usually never available in real-time. 
- The concept of using feedback from previous predictions is not entirely new. Adaptive models and online learning techniques have long incorporated past errors to improve future predictions. The bigger issue is the lack of adequate differentiation of IAN from existing methods i.e., a thorough literature review that situates its contributions within the broader context.
- While the authors introduced a new loss function, there is no analysis of the properties of the displacement expectation loss or proofs of convergence and stability.
- There is a lot of poorly defined notation, making it difficult to understand the proposed method fully. It would make sense to move the table 3 from the appendix (which has the notation!) to the main paper. 

There are some issues with the presentation. 

Writing: 
- Section 3, Trajectory Prediction (just before equation 1): \hat{Y} of "him"
- Section 4.5, "At the beginning of this paragraph" 
- Section 5.4, Last sentence: Our approach can "inference" at a high frequency
- Personal preference: please use "Figure X represents ..." (not abbreviation) instead of "Fig. X represents..." because your captions have the former. 

Images:
- The image labels for Figure 2 are too small to read -- had to Zoom 150% to read it
- Figure 2, it is hard to tell the difference between Train Path and Test Path, the arrows look very similar, please consider a different color. 
- Figure 4, 5, it is hard to see lighter colors.

questions:
1. How do you justify the assumption that ground truth trajectories are available in real-time for computing feedback? In what practical scenarios would this be feasible?
2. How does your method differ from existing online learning approaches that adapt models based on prediction errors?
3. How does the computational cost of IAN affect inference time in real-world applications (even a small delay in prediction can go a long way)?
4. Are there experiments to isolate the effects of each component of your model? If so, please provide detailed results.
Statistical Significance: Are the reported improvements statistically significant? Can you include variance measures or confidence intervals?
5. How does your model handle agents for whom no feedback data is available?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659626063

thread_type:
official_review

<thread_object>
creator: reviewer_ykss

<content>
summary:
In this work, an online learning methodology is proposed for trajectory prediction based on the insight that ground truth observations checking past predictions (named feedback) can be collected as time goes on. To leverage this insight, the authors propose an interactive adjustment network (IAN) and evaluate its efficacy using a variety of trajectory prediction algorithms on a set of pedestrian and athlete motion datasets.

strengths:
The idea is sound and sensible.

The writing is clear and it is easy to follow the core idea being proposed.

There are a wide variety of baseline approaches used to demonstrate the core idea.

weaknesses:
In contrast to the claims in the paper, this work is not the first to study feedback in trajectory predictions. Accordingly, the primary weakness of this work is a lack of discussion and comparisons to prior adaptive prediction work that investigates this same idea. Notable examples (which focus on both temporal adaptation, equivalent to individual feedback, and geographic adaptation) include:
* Y. Xu, L. Wang, Y. Wang, and Y. Fu, "Adaptive trajectory prediction
via transferable GNN," in IEEE Conf. on Computer Vision and Pattern
Recognition, 2022.
* B. Ivanovic, J. Harrison, and M. Pavone, "Expanding the deployment envelope of behavior prediction via adaptive meta-learning," in IEEE Conf. on Robotics and Automation, 2023.

The experimental setup relies primarily on small-scale pedestrian-only datasets. Further, these datasets are quite old (ETH/UCY are 10-15 years old, GCS is more than 10 years old, the NBA dataset is almost 10 years old), meaning performance has largely saturated on them and it is difficult to tell if the proposed method significantly improves upon baselines.

As stated in Section 5.4, a prediction model and the IAN are _consecutive_ modules, meaning they have to run one after the other (and not simultaneously as stated in Line 523). This means that there is a 20ms overall runtime increase as a result of adding the IAN.

questions:
The most important question: How does this approach compare to prior works on adaptive trajectory prediction? Why is IAN better or more preferable to use by practitioners?

How does this approach perform on modern large-scale datasets with multiple types of agents and varying dynamics, e.g., Waymo Open Motion Dataset, nuPlan? These datasets also have less saturated metrics which should make it easier for the IAN's performance improvements to stand out.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the human trajectory prediction via the perspective of individual feedback. Specifically, authors propose interactive adjustment network to model and leverage the feedback. Experiments look promising on several benchmarks. The major points to be summarized are as follows:

Positive: 

+ easy to follow / good motivation
+ comprehensive experiments

Negative:

- Lack of some theoretical justification / missing technical details
- Elaboration on the key module, IAN
- Additional ablations / evaluation
- Unclear writing / typos

After rebuttal, authors did a good job to address most of the concerns. Most of the reviewers (4), out of 5 in total, are convincing that the manuscript is polished to great extent. All key issues (e.g. W1 by Reviewer dXFb on the assumption of ground truth, novelty compared to [1,2] by Reviewer ykSs, addiontal experiments Waymo, etc.) have been well addressed. In AC's view, the overall good merits over-weigh the downside of the manuscript. Please incorporate all the comments and elevate the paper for camera-ready version.