Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This manuscript addresses the difficulty in interpreting what information a model learns from human trajectory prediction datasets. The authors propose a feature attribution method for trajectory prediction based on Shapley values. The authors also show that for commonly used datasets such as ETH/UCY, SDD, and nuScenes, models Trajectron++ and PECNet do not learn interaction information. However, on datasets featuring more interaction, such as SportVU, these models are able to learn social interactions.

main_review: The manuscript has good clarity and explains its contributions well.
The analysis of the evaluation aligns with the claim that models are not learning interactions from ETH/UCY, nuScenes, and SDD, but are learning interactions from SportVU.
The evaluation of HTP models is an interesting research area that has not received nearly as much attention as works producing new HTP models using the same traditional evaluation metrics as others (i.e., minADE and minFDE).

When dropping a subset of agents to determine the influence on model performance, the subset is replaced with static agents that are assumed not to influence the remaining agents. However in practice, static, non-interacting obstacles are still considered by humans during navigation due to collision avoidance.
To my understanding, the modified version of baseline Shapley values does not completely ensure that there is no interaction.

In Section 3.3, the authors note that random agents should not contribute to the future of the agent-of-interest, but again, they still play a role in the movement of the agent-of-interest due to collision avoidance.
Yet, in the results reported in Figure 4, random agents contribute at times negatives Shapley values (e.g., Traj++Edge on ETH/UCY, nuScenes, and SDD).
Also, what is the interpretation of the negative contributions of the random agent and interaction (specifically for Traj++Edge on ETH/UCY)?

In Figure 4, the Shapley values reported differ for each dataset. ETH/UCY is missing PECNet, nuScenes is missing STGCNN and PECNet, SDD is missing STGCNN, and SportVU is missing STGCNN.
Also, the Shapley values for ETH/UCY, nuScenes, and SDD are reported based on NLL, but not on min-ADE with the exception of subfigures (d) and (f).

Regarding the interactions in ETH/UCY and nuScenes, there are numerous scenes in which no interaction occurs between agents (especially in nuScenes).
It would be helpful to see the contribution of social interactions on the subset of scenes in each dataset that have meaningful social interactions present in them.

It is surprising to see that the same models show significantly higher contributions of interaction in SportVU than in ETH/UCY, which has hundreds of scenes under "students001" and "students003" and feature extremely high levels of interaction between many pedestrians.

The authors note that the contributions of past trajectory and neighbors can be compared for an individual model (e.g., for Traj++Edge), but do not explain how the contributions can be compared between models (e.g., between PECNet and Traj++).
According to Figure 4(f), the contribution of past trajectory to PECNet is 3 to 4 times larger than the contributions of past trajectory to Traj++ and Traj++Edge.
What does this mean?

The authors note that the interaction encoder of Traj++Edge is stronger, so it has higher contributions of history and interaction on SportVU, but for ETH/UCY, the contribution of interaction is less than Traj++ and is negative.

The PECNet paper reports results on ETH/UCY, but the results have not been included into Table 1.
The results of PECNet and Traj++ differ from the results reported in their respective papers. The authors should make a note of why they differ and also clarify on the missing data.

I have some concern about the results reported in Table 1 for models without interaction. The ideal way to report this result is to train on a modified dataset in which every scene with $n$ agents is split up into $n$ subscenes with one agent each, thus ensuring that there is no possibility of agent interaction.

It would be helpful to see the local analysis on SportVU also done for interactive and non-interactive scenes in ETH/UCY, nuScenes, and SDD to showcase the lack of interaction learning.

summary_of_the_review: Although I believe that this manuscript is well-written, tackles an interesting problem, and has an evaluation that is consistent with its claims, I do not think there is enough evidence to back the strong claim that SOTA HTP models are unable to learn almost any interactions for well-studied datasets that have cases of high interaction, but are able to learn interactions for a dataset with more interaction.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes assessing the contribution of each feature by computing a version of Shapley values based on the existing trajectory predictor formulation. They apply this measure to the state-of-the-art trajectory prediction methods and found that many standard benchmarks don’t require reasonings of agent interactions.
main_review: This paper is presented clearly and a better feature attribution method can help us understand blackbox predictors and make sure if the presented model learns the interactions we want. I only have a few questions.

Novelty. Changing latent codes or features and computing their influence on performance is not entirely new. It has been used to understand blackbox vision/language models. The authors need to compare with prior work and show why this proposed measure is novel rather than applying it to trajectory prediction. 

Extending to predictors that include map info. The way to drop features and aggregate the Shapley value is defined purely based on the agent interaction graph. How does this extend to other approaches that use information such as map features?

Choice of behavior for computing the Shapley values. The contribution of the other agents does not necessarily influence the accuracy of the predicted accuracy, it may reduce the uncertainty of behaviors too. How does this paper handle the influence not directly reflected in accuracy?

-----
Post-rebuttal update: 
The authors answered my questions and address other reviewers' questions. While the proposed approach directly extends Shapley value for trajectory prediction, the analysis is valuable for the community. So, I’m raising my score.
summary_of_the_review: The paper clearly presents a variant of Shapley value for trajectory prediction tasks. The proposed measure helps identify if a model uses a feature for prediction. My main question is about the novelty of the proposed approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on analysis on whether the existing trajectory prediction methods indeed capture the interactions between multiple agents and uses two example methods on multiple benchmark datasets to obtain supportive results for their claims. More specifically, the contributions are three folds:
1. This paper addresses feature attribution for trajectory prediction to gain insights about the actual cues contemporary methods use to make predictions. The authors designed a variant of Shapley values that is applicable to a large set of trajectory prediction models.
2. This paper quantifies feature attributions both locally and globally and studies the robustness of given models.
3. This paper claims that existing models do not use interaction features, which is contrary to the statements made by the authors of the original papers.
main_review: Strengths:
1. The paper is generally well written and easy to follow. The authors provided sufficient background information and a relatively comprehensive literature review. The major motivation of this work is clearly stated.
2. Evaluation and analysis of the interaction modeling of trajectory prediction models is an interesting task and could contribute to figuring out the most useful approaches/representations to capture mutual interactions.
2. In order to support the authors' claims, they conducted many experiments and the results seem supportive based on the assumption that the quantifiable metric they proposed is reasonable.

Weaknesses:
1. Since all the experimental results are based on the Shapley value, it would be better to provide more explanation and evidence for its applicability in the area of trajectory prediction. Also, is there any limitation/caveat when using Shapley value as the metric for this purpose?
2. Three closely related papers are listed below, which explicitly model the interactions/relations between interactive agents. It would be better to include them in the literature review. In particular, it would be better to conduct some experiments using the model proposed in [1], which makes the statement even more convincing.
[1] Neural relational inference for interacting systems, ICML 2018.
[2] EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning, NeurIPS 2020.
[3] Dynamic Neural Relational Inference, CVPR 2020.

summary_of_the_review: Overall, I think that the contributions of this paper are interesting and could raise more discussion and exploration on the interaction modeling in multi-agent systems. I have a positive feeling about this paper. My major concern is the suitability of the proposed Shapley value metric in the task of trajectory prediction. Hope to see more explanations in the rebuttal.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This work discusses the features that attribute to the prediction results in trajectory prediction models. They apply a Shapley values to attribute the features and results, attempting to answer a question if the social models are really social or not. The meaning of a model being social in this case is that the model can use neighbour features in order to predict the trajectory. The work defines a measurement metric called the social interaction score, in which it measures how well the interaction between agents attributes to the prediction of the other agent trajectory. 

main_review: I believe this work is valuable overall for the trajectory prediction community. They present the problem well and the general framework of trajectory prediction[history encoder, interaction encoder and decoder] is workable and applied to most of the current works. 
Question: The choice of using static, non-interacting agents seems feasible but what about other options? Can the authors elaborate on this and add a discussion for other choices and how these choices could alternate the results? For example, one might suggest using the slowest agent. 

It seems from the Shapley values that the random agents doesn't contribute much to the results because of the lack of interaction as shown in the work and this is a good finding. 

The only thing I’m skeptical about is the definition of the social interaction score. Though it seems justified by the presented argument, it needs a kind of empirical verification. Training from scratch the trajectory prediction models with no interactions (aka single agents only, no neighbours) should produce similar (or better) results to the ones with the neighbours. This will prove the feasibility of the social score and the whole findings of the papers. I can understand that the training can take a while, but providing results on two models with at least a subset of the ETH/UCY (I’d avoid the ETH and use Zara1, Zara2, Hotel as ETH is highly unbalanced) dataset will be a strong addition. 

=== Post Rebuttal Comment ===
I believe that the authors have answered the concerns raised by the reviewers and my concerns. The new experiments on Trajectron++ proved the theoretical analysis. Overall, in the past years since 2016 plenty of deep models were introduced to solve the trajectory prediction problem with emphasis on the "Social Interaction" between the agents. This paper proves via analysis and empirical results that such "Social Interaction" does not exist or have a minimal effect. Thus, I find this work significant in this area as it will lead future research into figuring out a true form of pedestrian’s interaction. Based on this, I’m raising my score.

summary_of_the_review: The paper is valuable to the community of trajectory prediction models. It directly discusses if the “social” interaction exists or not in previously reported models. Some questions were raised regarding the choice of the static, non-interacting agent and the feasibility of the presented social interaction score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Different from attributing the trajectory prediction power with the success of modeling agent-agent interactions, this paper presents a new perspective that the past trajectory of the target is the only feature used for predicting its future for state-of-the-art trajectory prediction methods on standard benchmark datasets.  To analyze the contribution of different clues, this paper applies to compute the Shapley values. The proposed method modifies the value of the adjacency matrix as zero to drop features and analyze. Besides, a local-global contribution aggregation method is proposed for the problem of variable features in different scenarios. 

This paper analyzes many state-of-the-art trajectory prediction models (Trajectorn++, PECNet, and Social-STGCNN) on many datasets (ETH-UCY, SDD, nuScenes, and SportVU). The experimental results on SportVU show the trajectory prediction models have the capability to learn the interactions on the datasets where interactions are more important.

main_review: This paper proposes to analyze the contribution of different clues with the Shapley values and obtain an interesting conclusion that the past trajectory of the target is the only feature used for predicting its future for state-of-the-art trajectory prediction methods on standard benchmark datasets. 

Strengths:
1) I think this conclusion makes sense. Some casual-inference based trajectory prediction methods [1] also found the interactions are biased in the current trajectory prediction systems and datasets. This paper proposes to use the Shapley values to analyze the contribution of different clues, which provides a quantitative evaluation of these biased interactions. 
2) The contribution of this method is clear. The discovery of the uselessness of interactions can help other researchers to develop better trajectory prediction methods. The experimental results on SportVU are also an important discovery to show the trajectory prediction models have the capability to learn the interactions.  I guess it is because the interactions in the training and testing dataset on SportVU are not biased, while the interactions in ETH-UCY or SDD are based.  
3）The paper is well written, which clearly shows the motivation, conclusion, and technical details. Compared with original Shapley values, this method modifies the value of the adjacency matrix as zero to drop features and proposes a local-global contribution aggregation strategy for variable features in different scenarios.

Weakness:
1) Some methods[1] also found the biased interactions and analyzed this in the perspective of causal inference (Effect of treatment on the treated). I think the differences between the two methods are clear, and there are the links of this method to causality in the discussion  Yet, it is suggested to provide the discussion with this paper, which also thinks the interactions are biased and analyzes this with the Effect of Treatment on the Treated. 
2) In my opinion, some conclusions should be further discussed. For example, why SportVU is ok to learn interactions but not SDD?  I guess it is because of the level of interaction bias in training and testing environments.  SportVU only contains the scene of NBA for both training and testing. However, in other datasets, the scenes are changed, which is similar to the out-of-distribution problem. This setting may be more appropriate for self-driving. It is suggested to conduct a new experiment, where fewer interactions biases in training and testing environments. For example, all training and testing trajectories are from the ETH scene, but not from hotel, zara or univ. 
3) Some technical details are suggested to provide. The performance of Traj++ on SDD shows a dramatic improvement than PECNet. Can you provide the details of the training and testing of Traj++ on SDD?

[1]Chen G, Li J, Lu J, et al. Human Trajectory Prediction via Counterfactual Analysis[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 9824-9833.


summary_of_the_review: Overall, I think it is a good paper with some minor problems that presents a new perspective that the past trajectory of the target is the only feature used for predicting its future for state-of-the-art trajectory prediction methods on standard benchmark datasets. On the balance of positive and negative points, I think it is a borderline paper and tend to accept.

Post Rebuttal： Thanks for the authors' feedback. The rebuttal answers most of my questions, which raises my score. It is suggested to further analyze why the interactions have limited effect in trajectory prediction. It is unintuitive but makes sense. There are some potential reasons in my views: 1) There are biases in interactions? 2) The multi-modality generation breaks the effect of interactions? 3) Is this finding generalized to rule-based methods, such as Social Force? I will further raise my score if these questions can be addressed.
review_title: null

Meta-review: The manuscript brings up an important issue: that current methods and datasets don't generally highlight interactions when it comes to trajectory prediction. This is despite the fact that it would seem that current methods incorporate agent interactions and that datasets appear to require reasoning about agent interactions. This qualitative and quantitative observation should lead to better datasets in the future as well as more refined metrics pushing the field forward. Reviewers were in agreement that this is a strong submission. The authors responded with substantive new experiments that cleared up any lingering issues.
Decision: Accept (Poster)