You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Prediction
Authors: Osama Makansi
Julius Von Kügelgen
Francesco Locatello
Peter Vincent Gehler
Dominik Janzing
Thomas Brox
Bernhard Schölkopf
Venue: ICLR 2022 Poster (2022)
Keywords: Feature Attribution
Shapley values
Trajectory Prediction
Causality

Abstract: Predicting the future trajectory of a moving agent can be easy when the past trajectory continues smoothly but is challenging when complex interactions with other agents are involved. Recent deep learning approaches for trajectory prediction show promising performance and partially attribute this to successful reasoning about agent-agent interactions.  However, it remains unclear which features such black-box models actually learn to use for making predictions. This paper proposes a procedure that quantifies the contributions of different cues to model performance based on a variant of Shapley values. Applying this procedure to state-of-the-art trajectory prediction methods on standard benchmark datasets shows that they are, in fact, unable to reason about interactions. Instead, the past trajectory of the target is the only feature used for predicting its future. For a task with richer social interaction patterns, on the other hand, the tested models do pick up such interactions to a certain extent, as quantified by our feature attribution method. We discuss the limits of the proposed method and its links to causality.

Published as a conference paper at ICLR 2022
YOU MOSTLY WALK ALONE: ANALYZING FEATURE AT- TRIBUTION IN TRAJECTORY PREDICTION
Osama Makansit2, Julius von Kügelgen 3,4 Francesco Locatello', Peter Gehler', Dominik Janzing',
Thomas Brox 1.2 and Bernhard Schölkopf"
Amazon, University of Freiburg, Max Planck Institute for Intelligent Systems Tübingen, UUniversity of Cambridge
ABSTRACT
Predicting the future trajectory of a moving agent be easy when the past trajec- tory continues smoothly but is challenging when complex interactions with other agents are involved. Recent deep learning approaches for trajectory prediction show promising performance and partially attribute this to successful reasoning about agent-agent interactions. However, remains unclear which features such black-box models actually learn to use for making predictions. This paper pro- poses a procedure that quantifies the contributions of different cues to model per- formance based on a variant of Shapley values. Applying this procedure to state- of-the-art art trajectory prediction methods on standard benchmark datasets shows that they are, in fact, unable reason about interactions. Instead, the past trajec- tory of the target is the only feature used for predicting its future. For a task with richer social interaction patterns, up such interactions to a certain extent, as quantified by our feature attribution method. We discuss the limits of the proposed method and its links causality.
the other hand, the tested models do pick
INTRODUCTION
Predicting the future trajectory of moving agent is significant problem relevant to domains such as autonomous driving (Weisswange et al., 2021; Xu et al., 2014), robot navigation (Chen al., 2018), surveillance systems (Morris & Trivedi, 2008). Accurate trajectory prediction requires successful integration of multiple sources of information with varying hignal-t-noiss ratios: whereas the target agent's 's past trajectory is quite informative most of time, interactions with other agents are typically sparse and short in duration, but can be crucial when they occur.
Graph neural networks (N. Kipf & Welling, 2017) and transformers (Vaswani al., 2017) have led to recent progress w.r.t. average prediction error (Mohamed et al., 2020; Yu et al., 2020; Salzmann et al., 2020; Mangalam et al., 2020), but the predicted distributions over future trajectories remain far from perfect, particularly in unusual cases. Moreover, the black-box nature of state-of-the-ar models (typ- ically consisting of various recurrent, convolutional, graph, and attention layers) makes it increas- ingly difficult to understand what information model actually learns to to make its predictions.
To identify shortcomings existing approaches and directions for further progress, in the present work, propose tool for explainable trajectory prediction. In more detail, we use Shapley val- ues (Shapley, 1953; Lundherg & Lee, 2017) to develop a feature attribution method tailored specifi- cally to models for multi-modal trajectory prediction. This allows to quantify the contribution of each input variable to the model's performance (as measured, c.g., by the negative log likelihood) both locally (for particular agent and time point) and globally (across an entire trajectory, scene, or dataset). Moreover, we propose an aggregation scheme to summarize the contributions of all neighboring agents into a single social interaction score that captures how well a model able to use information from interactions between agents.
"ork done during internship at Amazon. Contact email: makansio @es.uni-iaee w.s.uni-frebbbug freiburg.dd "Equal contribution.

---

Published as a conference paper at ICLR 2022
A
1
Our proposed method allows for comparing different trajectory prediction models in terms of the extent which they use social interactions for making predictions (left: none, middle: weak, right: strong). The target agent, whose future trajectory is to be predicted, is shown in red, and modelled interactions are represented by arrows with width indicating interaction strength.
Figure 1:
By applying our analysis to representative state-of-the-art models, Trajectron++ (Salzmann et al., 2020) and PECNet (Mangalam et al., 2020), we find chat-contrary to claims made in the respective works-the predictions of these models on the common benchmark datasets ETH-UCY (Pellegrini al., 2009; Leal-Taixé al., 2014), SDD (Robicquet 2020) are actually not based on interaction information. We, therefore, also analyze these models' behavior on an additional sports dataset SportVU (Yue et al., 2014) for which we expect a larger role of interaction. There, models do learn to use social interaction, and the proposed feature attribution method can quantify (i) when these interaction signals are active and (ii) how relevant they are, see Fig. 1 for an overview.
al., 2016), and nuScenes (Caesar et al.,
Overall, our analysis shows that established trajectory prediction datasets are suboptimal for bench- marking the learning of interactions among agents, but existing approaches do have the capability to learn such interactions on more appropriate datasets. We highlight the following contributions:
We address, for the first time, feature attribution for trajectory prediction to gain insights about the actual cues contemporary methods use to make predictions. We achieve this by designing a variant of Shapley values that is applicable to a large set of trajectory prediction models ($ 3).
With this method, we quantify feature attributions both locally (per scenario; 3.1) and globally (over the whole dataset; 3.2) and study the robustness of given models ($ 3.3).
Our study uncovers that, on popular benchmarks, existing models do not use interaction features. However, when those features have strong causal influence on the target's future trajectory, these models start to reason about them (S 4.4).
2 BACKGROUND & RELATED WORK
Our primary goal is to better understand the main challenges faced by existing trajectory prediction models. First, we formalize the problem setting ($ 2.1) and review prior approaches ($ 2.2), well as explainability tools ($ 2.3), which we will use to accomplish this goal in $ 3.
2.1 THE TRAJECTORY PREDICTION PROBLEM SETTING
Let {X}, X7'), e E denote a set of time series corresponding to the trajectories of n agents that potentially interact with each other, see Fig. 2 (a) for an illustration. All X are assumed to take values in Rd that is, the state of agent at time a vector xt Rdd encoding, e.g., its 2D position, velocity, and acceleration. We refer to an observed temporal evolution æt कं. = (x) 1 .... zsp) of a sin- gle agent as a trajectory of length and to the collection of trajectories x 11.f (x1.r- .... "irr) of all n agents as scene. Assume that we have access to a training dataset consisting of M such scenes.!
The trajectory prediction task then consists of predicting the future trajectory (X.+ xi At) of a given target agent and at given time t up to a time horizon At given the observed past a of both the target agent itself and of all the other agents, where h is the length of the history that is taken into account. Formally, we thus aim at learning the distributions PPX (41+1++ 111m t-ha) for any E Z and i E N == {1....,?) n) with given At and h.
lin
Since any given agent (e.g.. a particular person) typically only appears in single scene in the training dataset, solving the trajectory prediction task requires generalizing across scenes and agent
In practice scenes may differ both in length and in the number of present agents; for the latter, we define the maximum number of agents present in given scene across the dataset and add "dummy" agents to scenes with fewer agents, $ 4.1 for further details.
2

---

Published as a conference paper at ICLR 2022
X
Fs
X
Fl
F2
F E
i
X
FN
F
(a) Scene
(b) History Encoder
(c) Interaction Encoder
(d) Decoder
Figure 2: General framework of existing trajectory prediction methods: Given (a) a scene with dynamic agents, (b) the history of every agent is encoded as F and (c) edges other agents are learned by an interaction encoder and aggregated to form the overall edge features E*. Finally, (d) both the history and interaction features are passed to decoder to predict the future trajectory.
identities. In other words, necessary to learn how agent's future trajectory, in general, depends on its own past and on the past behavior of other neighboring agents.
2.2 EXISTING APPROACUESS FOR TRAJECTORY PREDICTION
Fig. 2 (b)-(d) shows a general framework that unifies most existing approaches for trajectory pre- diction. It consists of three main modules:
(b) history encoder v that learns an embedding F (X( of the history of each agent;
(c) an interaction encoder & that incorporates information from neighhoring agents by learning an embedding for the edge between two agents E E(Xi_ xi-n and
(d) a decoder that combines both history and interaction features to generate the predicted future trajectory &f #1+1++A1 := 0 O(F EE) of the target agent, where E is the result of aggregating all edge embeddings (see below).
Existing approaches mostly differ in the choices of these modules. To handle the temporal evolution of the agent state, LSTMs (Hochreiter & Schmidhuber, 1997) are widely used for 1a and 9, i.e., encode the history and decode the future trajectory, respectively (Alahi al., 2016; Zhang al., 2019). Since the future is highly uncertain, stochastic decoders are typically used to sample multiple trajectories, lat +:
H.), e.g.. using GANs (Gupta et al., 2018) or conditional VAEs (Lee al., 2017: Salzmann et al., 2020). Alternatiely. Makansi et al. (2019) proposed multi-head network that directly predicts the parameters of mixture model over the future trajectories.
To handle interactions between moving agents (Fig. 2c), existing works model scene graph in which nodes correspond to the state embeddings F of the n agents and edges are specified by an adjacency matrix A with A iff. agents and j are considered neighbors (based, e.g., on their relative distance). Given this formulation, local social pooling layers have been used to encode and aggregate the relevant information from nearby agents within specific radius (Alahi et al., 2016; Gupta et al., 2018). More recently, Mangalam et al. (2020) proposed a non-local social pooling layer, which uses attention and is more robust to false neighbor identification, while Salzmann al. (2020) modeled the scene as a directed graph to represent a more general set of scenes and interaction types, e.g., asymmetric influence. In different line of work, Mohamed et al. (2020) proposed undi- rected spatio-temporal graph convolutional neural networks to encode social interactions, and Yu al. (2020) additionally incorporated transformers based on self-attention to learn better embed- dings. In general, the edge encodings {E*), } for agent are then aggregated over the set of its neigh- bors (i.e., those j with A = 1) to yield the interaction features E e.g., by averaging as follows:
E Ejss AVEU
ΣjØ A
Although some of the above works show promising results on common benchmarks, it is unclear what information these methods actually use predict the future. In the present work, focus mainly on this aspect and propose an evaluation procedure that quantifies the contribution of different features the predicted trajectory, both locally and globally.
3

---

Published as a conference paper at ICLR 2022
2.3 EXPLAINABILITY AND FEATURE ATTRIBUTION
An important step toward interpretability in deep learning are feature attribution methods which aim quantifying the extent to which a given input feature is responsible for the behavior (e.g.. prediction, uncertainty, or performance) of a model. Among the leading approaches in this field concept from cooperative game theory termed Shapley values (Shapley, 1953), which fairly distributes the payout of a game among a set of n players. 2 Within machine learning, Shapley values can be used for feature attribution by mapping an input x = (I1,...,, In) to a game in which players are the individual features z; and the payout is the model behavior f: Xg x X A₂ - R on that example (Lundberg & Lee, 2017). Formally, one defines a set function V : 2 - R with N : {I,...,71} n. whose output (S)) for subset S C N corresponds to running the model on modified version of the input r for which features not in S are "dropped" or replaced (see below for details). The contribution of Iı, quantified by its Shapley value o(x;), is then given by the difference vSS {i}) - v(5) between including and not including i, averaged over all subsets S:
d(z,) Σ
( -60))-1(1 -(S)) SENVE) n((1s))
1
(())-
(2)
There are different variants of Shapley values based mainly on the following two design choices:
ii) What model behavior (prediction, uncertainty, performance) is to be attributed? (choice of f)
(ii) What is the reference baseline, that is, how are features dropped? (choice of v)
A common choice for (i) is the output (i.e., prediction) of the model. Alternatively, Janzing et al. (2020a) defined f as the uncertainty associated with the model prediction, thus quantifying to what extent each feature value contributes to or reduces uncertainty. In the present work, we will focus on attributing model performance. As for (ii) the set function 2, Lundberg & Lee (2017) originally proposed to replace the dropped features 2 N with samples from the conditional data distribu- tion given the non-dropped features: vSS) = E[s(as, XN = æs]: they then approximate this with the marginal distribution, v(S) = [[s, XN(s)). using the simplifying assumption of feature independence. Janzing al. (2020b) argued from causal perspective that the marginal distribution is, in fact, the right distribution to sample from since the process of dropping features naturally corresponds to an interventional distribution, rather than an observational (conditional) one. In an alternative termed baseline Shapley values, the dropped features are replaced with those of a predefined baseline x': v(S)= = fxr, 2' sys)). see Sundararajan & Najmi (2020) for details.
EXPLAINABLE TRAJECTORY PREDICTION
In the present work, we are interested in better understanding what information is used by tra- jectory prediction models to perform well, that is, quantifying the contribution of each input fea- ture to the performance of given model (rather than to its actual output). Therefore, we de- fine the behavior f to be attributed (i.e., choice (i) in $ 2.3) as the error of the output prediction f = L(# #(+1:ttch where L denotes any loss function (see $ 4.3 for common choices).
3.1 WHICH SHAPLEY VALUE VARIANT: HOW TO DROP FEATURES?
Next, we need to decide how to define the set function (choice (ii) in $ 2.3), i.c., how to drop features. Since this choice can be highly context-dependent, we need to ask: what is the most rea- sonable treatment for dropped features in the context of trajectory prediction? Intuitively, dropping a subset of agents should have the same effect as if they were not part of the scene to begin with: ideally, we would like to consider the behavior difference between agents being present or not.
As discussed in & 2.3, Shapley values are commonly computed by replacing the dropped features with random samples from the dataset or with a specific baseline value x'. For the latter, choosing the bascline value x' is often non-trivial: for instance, Dabkowski & Gal (2017) use the feature- mean as baseline, while Ancona et al. (2019) set the baseline to zero. However, for either choice the dropped features still contribute some signal (e.g., causing gray-ish or black pixels, respectively,
Shhpley values are the only method that satisfy a certain set of desirable properties or axioms, see (Lund- herg & Lee, 2017) for details. For other feature attribution methods, we refer Appendix A.
4

---

Published as a conference paper at ICLR 2022
in the context of images) that may affect the feature attribution scores. The same caveat applies to using random samples which in addition to the computational overhead also makes this variant unattractive for our needs.
For trajectory prediction, we thus opt for a modified version of baseline Shapley values where we choose the baseline to be static, non-interacting agent. This means that the contribution of the past trajectory of a target agent is quantified relative to a static non-moving trajectory. and the contribu- tions of neighboring agents are quantified relative to the same scene after removing those agents. Owing to the flexible formulation of the interaction encoder as a graph (see $ 2.2 and Fig. 2c), dropping a neighboring agent from the scene is equivalent to cutting the edge between the target agent and that neighbor, and thus easy to implement. Formally, we achieve this by modifying the adjaceney matrix of the graph used in (1) as follows:
A 00
Tj otherwise
A
This formulation applies to most existing frameworks using directed (Salzmann et al., 2020) or undi- rected graphs (Alahi et al., 2016; Gupta et al., 2018; Mangalam et al., 2020; Mohamed et al., 2020).
3.2 CONTRIBUTION AGGREGATION: THE SOCIAL INTERACTION SCORE
With the above formulation, quantifying the contribution of features (i.e, the past trajectory of the agent-of-interest and the interaction with other agents) in a specific scenario is possible. On the other hand, aggregating these Shapley values over multiple scenarios is important draw conclusions about the behavior of the model on a specific dataset. Formally, given the Shapley values of a feature for specific scenario o(z;), we aim at quantifying the contribution of the feature over the whole dataset Φ;. The common methodology is to average the obtained Shapley values for every feature over multiple scenarios. This is problematic when features vary over scenarios. For instance, the neighbors of the agent-of- inteeteest may change over time and are different over scenes, thus aggregating their contribution over the whole dataset yields the wrong statistics.
To address this problem, we split the features into two types: permanent features P and temporary features T. The past trajectory of the agent-of-interest is an example of the former type, while the neighboring agents examples of the latter. For every permanent feature, use the common average aggregation to estimate the overall contribution of that feature o; = E[o(x,]]. For the temporary features, we first aggregate them locally (within the same scenario) and then globally. For the global aggregation, we use the average operator. Since interactions with neighbors are usually sparse (i.e., only few neighbors have true causal influence on the target), we choose the max operator for the local aggregation. In other words, we quantify the contribution of the most influential neighboring agent, which we call the social interaction score:
E [maax (#(;)) JET
A larger non-negative score indicates that at least of the neighbors has a significant positive contribution to the predicted future while a zero score indicates that none of the neighbors are used.
3.3 ROBUSTNESS ANALYSIS
Beside the contribution analysis of existing features (such as neighboring agents) via our formulation of the Shapley values, we propose to use the same methodology to study the robustness of pretrained models when random agents are added to the scene. Thanks to the structure of the interaction encoder (modeled as a graph), adding a random neighbor is equivalent to adding a new incoming edge in Fig. (c). The random agent can be drawn from the same scene different time or from another scene. Random agents should not contribute to the future of the agent-of-interes. thus their Shapley values given robust model should be close to zero; the dummy property of Shapley values (Sundararajan & Najmi, 2020). In other words, this analysis tests the model ability in distinguishing between real and random neighbors.
In the context of pedestrian navigation, random agent may still play role for avoiding collisions. Using some rules for the random agent (e.g., not in close proximity the target agent) may alleviate such concerns.
5

---

Published as a conference paper at ICLR 2022
X
X
X
X
X
X
X
(a) Trajectron
(b) Trajectron+-Fdge
Figure 3: The interaction encoder of Trajectron++ (a) and our Trajectron++Edge (b). Instead of learning a single edge to all neighbors the same type (i.e., to all pedestrians), we allow the model learn separate edges to existing neighbors and add dummy agents to handle the variable number of neighbors (dashed circle). Here, red denotes the target agent and neighbors are colored according their type (e.g., green for pedestrians, blue for vehicles).
EXPERIMENTS
Given the proposed method, we analyze the role of interaction both locally and globally for different state-of-the-art models in trajectory prediction.
4.1 MODELS
Our analysis is based on three recent state-of-the-art models: Social-STGCNN (Mohamed et al., 2020), Trajectron++ (Salzmann et al., 2020) and PECNet (Mangalam et al., 2020). We refer to 2.2 for details about the implementation choices of the key modules (history/interaction encoders and decoder). For PECNet, we compare the three provided pretrained models (Mangalam et al., 2020).
Alternatively, we propose new variant of the Trajectron++ (named Trajectron++Edge) in which we make both the history and interaction encoders stronger. For the history encoder (LSTTM), we make it one layer deeper. To handle variable number of neighboring agents, they propose to aggregate all neighbors from the same type (e.g., all pedestrians) into one node and learn a single edge to the target agent; see Fig. 3 (a). Alternatively, to add "dummy" agents to the scene until we reach predefined maximum number; see Fig. 3 (b). Notably, giving the network the freedom to learn separate edges between any two agents is more powerful. For instance, two neighboring pedestrians could have different effects on the future the target agent. Using evaluation procedure, contribution of both features (history and interaction).
propose to learn separate edges to all neighboring agents and
compare the two variants by quantifying the
4.2 DATASETS
ETH-UCY one of the most common benchmarks for trajectory prediction. The dataset a combination of the ETH (Pellegrini al., 2009) and UCY (Leal-Taixé et al., 2014) datasets with 5 different scenes and 1, 536 pedestrians. We use the standard 5-fold cross validation for our analysis.
Stanford Drone Dataset (SDD) (Robicquet et al., 2016) is large dataset with 20 different scenes covering multiple areas at Stanford university. It consists of over 11k agents of different types (e.g, pedestrians, bikers, skaters, cars, etc.). We follow the standard train/test split used in previous works (Mangalam et al., 2020; Gupta et al., 2018).
nuScenes (Caesar al., 2020) is one of the largest autonomous driving datasets with than 1, 000 scenes collected from Boston and Singapore, where each 20 seconds long. The dataset also has semantic maps with 11 different layers such drivable areas and pedestrian crossings. We also use the standard training/testing splits (Salzmann et al., 2020).
Vp (Yue et al., 2014) is tracking dataset for games recorded from multiple seasons of the NBA. Every scene has two teams of 5 players and the ball. We pre-process the dataset to remove short scenes and randomly select 7,000 scenes for training and 100 scenes for testing.
4.3 EVALUATION METRICS
min-ADE is the minimum over all predicted trajectories of the average displacement error (L2 distance) to the ground truth. Similarly, min-FDE is the minimum final displacement error, i.c., the minimum 1.2 distance over all predicted trajectories to the ground truth end point at time l. At.

---

Published as a conference paper at ICLR 2022
Ragee Kiudaes
Inge
Thep DDgg
1.5
85-
BMSt
interaction Shapley
random
interaction
randors
pest
Interaction
random
nuScenes: Shapley
SDD: Shapley
KOCH KCONG
390
Thes
Trag- ldge
interaction
candomm
past
(d) SDD:
Shapley valses based on mit- ADD
(c) SpertVU: Shapley values based on NEL
SporVVU Shupley values based on min ADE
Figure 4: Shapley values of Trajectron++ variants and PECNet on the three common benchmarks for future trajectory (a-d) and the VU dataset (e.f). Notably, the contribution of social interac- tions (neighbors) is almost zero on the common benchmarks, while neighbors do play an important role when predicting the future trajectory the SportVU dataset.
The previous two metrics are biased since they report only the minimum error over many predictions and are commonly used for testing multi-modality of the prediction. To assess all predicted trajec- tories for those methods predicting multimodal probabilistic distribution of the future, we report also the NLL of the ground truth given the predicted distribution.
For the Shapley values, we choose the NLL of the model as the output of the set function for both variants of Trajectron++ and Social-STGCNN. On the other hand, PECNet is non-probabilistic approach and can only output multiple trajectories by decoding multiple samples drawn from the latent space. Therefore, choose the min-ADE of the model when evaluating PECNet models.
4.4 RESULTS
Analysis common benchmarks. As discussed in 3, report the contribution for the past trajectory (since it is permanent feature) and the neighboring agent with the maximum contribution (our social interaction score). Additionally, added to the scene (robustness). From the results shown in Fig. 4 on ETH-UCY (a), nuScenes (b) and SDD (c-d), we observe the following: (I) the past trajectory has the largest contribution on the predicted future, (2) the contribution of the neighbors (our social interaction score) is insignificant and close zero, and (3) the contribution of random neighbor is almost identical existing neighbors. Notably, for those datasets with heterogeneous agents (e.g, pedestrians, vehicles, bikers, etc.), we report the average results over different types of agents. From last two observations, we conclude that recent methods unable to exploit information from neighboring agents to predict the future on these common benchmarks.
also compare the contribution of random agent
Analysis on the VU dataset. We conduct the same analysis on dataset with rich patterns of interactions and show the results in Fig. (c, f). Although the past trajectory still has the largest contribution, neighboring agents start to play an important role by contributing to the predicted future. Moreover, there significant difference between the maximum contribution neighbors (our social interaction score) and the contribution of random neighbor.
Our analysis is not only important to explain and test models but also to compare them. As can be seen in Fig, (e), the contributions of both the past trajectory and the neighbors of our Trajec- tron++Edge variant are significantly larger than the original model. Referring to Š 4.1, both the history and interaction encoders stronger in our variant which explains the increase in their tributions. On the other hand, both variants are robust to random neighbors added to the game. In Fig. 4 (f), we compare PECNet and both variants of Trajectron++ on the SportVU dataset where

---

Published as a conference paper at ICLR 2022
Table 1: Average errors (min-ADE/ min-FDE/ / NLL) on trajectory prediction benchmarks. For ev- ery model, we show the emrors with and without (w/o) interaction, as well as their relative difference. Method Dataset
ETH-UCY
SDD
nuSeenes
SportVU
Social-STGCNN
0.45/ / 0.76/-1005
w/o interaction
0.45/ 0.76/ -0.95
Diff
0.0 / 0.0 /-0.1
PECNet
9.29 / 15.93/
7.957 / 17.38/
w/o interaction
9.28 / 15.93
9.78 / 17.38/
Diff
0.0 0.0 /
-1.83/ 0.0 /
Traj++
0.30/ 0.51 -0.33
1.35/ 2.05 1.76
0.49/0.77
0.52
4.86 5.39
w/o interaction
0.31/ 0.52/ -0.33
1.35 / 2.07 / 1.75
0.49/0.77
0.50
6.62/ 8.78
Diff
-0.01 / -0.01/ 0.0
0.0 / /0.02/ /.001
0.0 /0.0 / 0.02
1,76/-3.67
/ -0,78
Traj++Edge
0.307 0.497
1.397 2.147 1.98
0.26/0.3//1126
4.78/
5,22
w/o interaction
0.30/ 0.50 -0.41
1.39/ 2.16 1.98
0,26/0.44 /-1.26
7.49/
11.18/7.19
Diff
0.0 -0.01 -0.02
0.0 /00022 / 0.0
0.0 / -0.01 -0.02
2.71/-5.96/ -
observe that our Trajectron+++ has the largest social interaction score while PECNet is largely based on the past.
Similarly to our social interaction score, Tab. I shows the relative difference in the average error when dropping all neighbors.' Here we also observe that social interaction is only important on the SUortVU dataset. Instead of dropping all neighbors during inference, we drop them during training and report the difference in Appendix C.
Local analysis. In Fig. 5, we show the results of our Trajectron++Edge on two interesting scenar- ios. Here, every player (and the ball) is represented as circle and colored according to the team. We plot the past trajectories of all players (solid lines), the ball (yellow) and the future trajectory of the target player (dashed line). We overlay the predicted distribution of the model (blue heatmap) on top of the scene. Additionally, we show the Shapley values of the scenario (right) for the past (red), neighboring players and the ball. For the first scenario (a), we clearly see that the past is the main feature contributing to the predicted future where all other players have insignificant contributions. On the other hand, for the second scenario where interaction with other players is critical to make the correct prediction (b), we can identify that the three main players (counter attacking) and the ball contribute positively to the predicted future, while the past has relatively small influence on the pre- diction. Interestingly, agent 6 moves to the right (opposite of the true future of the target) and thus has a negative contribution on the prediction of the target agent. Overall, our method is able to identify and visualize which feature has large influence on the target agent future for specific local events.
5 DISCUSSION
Links to Granger causality. Although the relevance of features for predicting the target should not be confused with their causal impact, the two are tightly linked. According to the seminal work of Granger (1969), time series X causally influences a target Y, whenever the past of X helps better predict Y. from its own past and that of all features other than X. This reasoning is justified-and follows from the causal Markov condition and causal faithfulness, see (Peters et al., 2017, Thm. 10.3)- whenever the following three assumptions are satisfied: (i) every feature influ- ences others only with nonzero time lag (i.e., no instantaneous influence); (ii) there are latent (unobserved) common causes of the observed features (causal sufficiency); and (iii) the prediction is optimal in the sense that it employs the full statistical information from the past of the features under consideration. Although these assumptions are hard to satisfy exactly in practice, Granger causality has been widely applied since decades due its simplicity. Since our attribution analysis based on comparing model performance (as opposed to predictions) with and without a certain feature, it is loosely related to Granger causality.The main differences are that we (a) perform a post-hoc analysis on single model, rather than training separate ones with and without the feature of interest; and (b) average over subsets of the remaining features, instead of always including them.
"Our numbers for PECNet on SDD are obtained by retraining their model using their source code. On Traj+t, our numbers are obtained after fixing an issue with the derivative computation and is reported to the corresponding authors. Moreover, the reported results on SDD for PFCNet and Traj++ are not directly compa- rable as the errors in different domains (in pixels for PF:Nett and in meters for Traj++).
8

---

Published as a conference paper at ICLR 2022
O
Cattegary
Home
Gless
Ball
Ot
(a) No social interaction is needed
072
539
032
3.0
4.55
Interaction with other agents critical
Figure 5: Qualitative analysis of variant Trajectron++Edge for scenario where the role of interaction is minimal (a) and scenario where the role is important (b). For both scenarios, we show the scene with players, the predicted distribution of the future after fixed At in blue (left), and the Shapley values of all players, the ball and the target's past trajectory (right). In red the target agent, the ball in yellow, and the teams' other players in blue/green. Solid lines the observed past trajectory and the dashed line is the ground truth future.
Interpretation and limitations. While assumptions (i) and (ii) above describe the general scenar- ios for which our attribution method may allow causal interpretation, assumption (iii) points to an important limitation. Since a model simply may not have learned to pick up and make use of cannot conclude from small Shupley values that causal influence exists. If, however, we know priori that a causal influence does exist and the corresponding Shapley value (close to) zero, then this points to failure of the model. Similarly, if there is a causal influence and the model can pick this up, we will see it in the Shapley value. For the trajectory prediction task, where only past information is used for prediction, positive Shapley values thus suggest the existence of causal influence, subject to (i) and (ii).
certain influences,
Exclusion vs randomization of features. As opposed to delecting causal influence, note that quantifying strength via the degree of predictive improvement is not always justified, contrary to what is often believed. Janzing et al. (2013, Sec. 3.3 and Example 7) argued, based modification of paradox described by Ay & Polani (2008), that assessing causal strength by excluding the feature for prediction is flawed and showed that it should be randomized instead. However, experiments in Appendix show only a minor difference between exclusion and randomization our case.
CONCLUSION
We addressed feature attribution for trajectory prediction by analyzing to what extent the available cues are actually used by existing methods to improve predictive performance. To this end, we proposed a variant of Shapley values for quantifying feature attribution, both locally and globally, and for studying the robustness of given models. Subject the assumptions and caveats discussed in $ 5, our attribution method can be interpreted as computationally efficient way of (approxi- mately) quantifying causal influence in the context of trajectory prediction, both for indicating the strength of causal influence of dataset and for benchmarking how well particular model uses such influence for its prediction. Using our method, we reveal that existing methods, contrary to their claims, do not rely interaction features when trained popular data sets, but that such information is used on other data sets such as VU where interactions play a larger role, see Appendix D for further discussion of our finding.

---

Published as a conference paper at ICLR 2022
7 ACKNOWLEDGMENTS
We would like to thank Lenon Minorics and Atalanti Mastakouri for the useful discussions and feedback. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S18039B, and by the Machine Learning Cluster of Excel- lence, EXC number 2064/1 Project number 390727645. We also thank Amazon Web Services for providing enough resources to conduct the experiments.
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961-971, 2016.
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a poly- nomial time algorithm for Shapley value approximation. In Proceedings of the International Conference on Machine Learning (ICML), pp. 272-281, 2019.
Nihat Ay and Daniel Polani. Information flows in causal networks. Advances in Complex Systems, 11(1):1741,, 2008.
Alexander Binder, Scbastian Bach, Gregoire Montavon, Klaus-Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for deep neural network architectures. In Information Science and Applications (ICISA), pp. 913-922, 2016.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11618-11628, 2020.
Guangyi Chen, Junlong Li, Jiwen Lu, and Jie Zhou, Human trajectory prediction via counterfactual analysis. In ICCV, 2021.
Zhixian Chen, Chao Song, Yuanyuan Yang, Baoliang Zhao, Ying Hu, Shoubin Liu, and Jianwei Zhang. Robot navigation based on human trajectory prediction and multiple travel modes. Applied Sciences, 8(11), 2018
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Proceedings of the International Conference on Neural Information Processing Systems (NIPS), pp. 6970-6979, 2017.
Colin Graber and Alexander G. Schwing. Dynamic neural relational inference. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
C. W. J. Granger. Investigating causal relations by econometric models and cross-spectral methods. Econometrica, 37(3):24-33,, July 1969.
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social gan: So- cially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2055-226666 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
Dominik Janzing, David Balduzzi, Moritz Wennsse and Bernhard Schölkopf. Quantifying causal influences. Annals of Statistics, 41(5):2324-2358, 2013.
Dominik Janzing, Patrick Blöbaum, and Lenon Minorics. Quantifying causal contribution via struc- ture preserving interventions. CoRR, abs/2007.0071 0000000000144 2020a.
Dominik Janzing, Lenon Minorics, and Patrick Blöbaum. Feature relevance quantification in plainable Al: A causal problem. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 2907-2916, 2020b.
10

---

Published as a conference paper at ICLR 2022
Thomas Kipf, Ethan Fetaya, tional inference for interacting systems. In Proceedings of the 35th International Conference on Machine Learning. pp. 1688-26977, 2018.
Wang, Max Welling, and Richard Zemel. Neural rela-
Laura Leal-Taixé, Michele Fenzi, Alina Kuznetsova, Bodo Rosenhahn, and Silvio Savarese. Learn- ing an image-based motion context for multiple people tracking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3542-3549, 2014.
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher Choy, Philip H. S. Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 2165 2174, 2017.
Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. In Advances in Neural Information Processing Systems, pp. 19783-19794, 2020.
Sindy Löwe, David Madras, Richard Zemel, and Max Welling. Amortized causal discovery: Learn- ing to infer causal graphs from time-series data. arXiv preprint, 2020.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Pro- ceedings of the International Conference on Neural Information Processing Systems (NIPS), pp- 4768 4777, 2017.
Osama Makansi, Eddy Ilg, Özgün Çiçek, and Thomas Brox. Overcoming limitations of mixture den- sity networks: A sampling and fitting framework for multimodal future prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7137-7146, 2019.
Osama Makansi, Özgünn Cicek, Yassine Marrakchi, and Thomas Brox. On exposing the challenging long tail in future prediction of traffic actors. In IEEE International Conference Computer Vision (ICCV), 2021.
Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned tra- jectory prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 759-776, 2020.
Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian Claudel. Social-stgenn: A social spatio-tempord graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14412-14420, 2020.
Brendan Tran Morris and Mohan Manubhai Trivedi. A survey of vision-based trajectory learning and analysis for surveillance. IEEE Transactions on Circuits and Systems for Video Technology, pp. 1111-1127, 2008.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In Proceedings of the International Conference Learning Representations (ICLR), 2017.
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In Proceedings of the International Confer- ence on Computer Vision (ICCV), pp. 261-268, 2009.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of Causal Inference Founda- lions and Leurning Algorithms. MIT Press, 2017.
Alexandre Robiequet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social eti- quette: Human trajectory understanding in crowded scenes. In Proceedings of the European Conference Computer Vision (ECCV), pp. 549-565, 2016.

---

Published as a conference paper at ICLR 2022
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically-feasibee trajectory forecasting with heterogeneous data. In Proceedings of the Eu- ropean Conference Computer Vision (ECCV), pp. 683-700, 2020.
Lloyd S. Shapley. The value of n-person games. Annals of Math. Studies, 28:307-317, 1953.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the International Conference on Machine Learning (ICML), pp. 3145-3153 2017.
Mukund Sundararajan and Amir Najmi. The many Shapley values for model explanation. In Pro- ceedings of the International Conference Machine Learning (ICML), pp. 9269-9278, 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the International Conference on Machine Learning (ICML), pp. 3319-3328, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, un- definedukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the Inter- national Conference on Neural Information Processing Systems (NIPS), pp. 6000-6010, 2017.
Thomas H. Weisswange, Sven Rebhan, Bram Bolder, Nico A. Steinhardt, Frank Joublin, Jens Schmuedderich, and Christian Goerick. Intelligent traffic flow assist: Optimized highway driving using conditional behavior prediction. In IEEÉ Intelligent Transportation Systems Magazine, pp- 20-38, 2021.
Wenda Xu, Jia Pan, Junqing Wei, and John M. Dolan. Motion planning under uncertainty for on- road autonomous driving. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). pp. 2507-2512, 2014.
Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks pedestrian trajectory prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 507-523, 2020.
Yisong Yue, Patrick Lucey, Peter Carr, Alina Bialkowski, and lain Matthews. Learning fine-grained spatial models for dynamic sports play prediction. In Proceedings of the IEEE International Conference on Data Mining, pp. 670-679, 2014.
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 818-833, 2014.
Pu Zhang, Wanli Ouyang, Pengfei Zhang, Jianru Xue, and Nanning Zheng. Sr-lstm: State refinement for 1stm towards pedestrian trajectory prediction. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12077-12086, 2019.
12

---

Published as a conference paper at ICLR 2022
Ours Marginal
2.5
2.0
1.5
1.0
0.5
0.0
past
interaction
random
Figure 6: Comparison between the baseline Shapley values (used in our paper) and the marginal Shapley values for our Trajectron++Edde on the VU dataset.
A OTHER FEATURE ATTRIBUTION METHODS
Other feature attribution methods can generally be categorized into two streams: perturbation-basee and gradient-hand based approaches. The former perturb some parts of the input and use the change the output measure of feature relevance (Zeiler & Fergus, 2014). Such methods applied perturbation and violate the sensitivity axiom (Sundararajan et al., 2017). Gradient-based approaches estimate the feature attributions by investigating the local gradients of the model. To address the saturation problem of gradients, DeepLIFT (Shrikumar et al., 2017) and LRP (Binder al., 2016) approximate gradients with discrete differences, which results in violating the imple- mentation invariance axiom (Sundararajan et al., 2017). Sundararajan et al. (2017) proposed the Integrated Gradients (IG) estimate the feature attribution relevant to baseline by integrating over the gradient along the path between the given input and the baseline, which satisfy all required axioms and can be seen as a variant of the Shapley values (Sundararajan Najmi, 2020).
sensitive to the
B EXCLUSION VS RANDOMIZATION OF FEATURES
In Fig. 6, compare the obtained Shapley values using our method and the ones obtained by running the marginal Shapley values. For the latter, we replace the feature by random feature drawn from the marginal distribution. replace it by random agent from another game and average over multiple replacements. Clearly, we draw the same conclusions (as 4.4) using the marginal Shapley values while our method more computationally efficient since we do not need to marginalize over many samples.
other words, instead of removing a neighboring agent, we
C RETRAINING WITHOUT INTERACTION
To further support our findings, we conduct additional experiment where we train one of the SOTA models from scratch on a dataset without interaction by removing neighbors to assess models need interaction or not. We selected the Traj++ model, retrained it on the ETH-UCY dataset, and compared the results to the same model trained with interactions. Tab. 2 shows that both models (with and without interaction) yield the same error, which is consistent with our findings.
D FURTHER DISCUSSION
Here provide complementary explanations for finding that existing models behave dif- ferently between the popular benchmarks (e.g., ETH-UCY) and the SportVU dataset: (i) a large difference in the frequency of interactions, and (ii) large difference in the nature of interactions. As
13

---

Published as a conference paper at ICLR 2022
Table 2: Average errors (min-ADE/ min-FDE) for the Traj++ model ETH-UCY. We observe al-
most no difference between training with/without interaction with the ETH scene to be an exception where training without interaction yields lower errors.
Method Scene
ETH
HOTEL
UNIV
ZARA1
ZARA2
AVG
Traj++
0.61 / 1.02
0.19
0.30/0.50
0.24/0.42
0.18/0..
0.30 0.51
Retrained interaction
0.58 0.96
/0.27
0.30/ 0.54
0.25/0.42
0.17/0.31
0.30 0.50
Diff
0.03 0.03/0.06
0.0
0.0/0.0
-0.01/0.0
0.01/000
0.0
Histogram of SportVU
175
150
125
50
25
0
0 25
50
75
100
125
150
175
kalman error
Figure 7: Histogram the errors of the linear Kalman filter on the SportVU dataset. The fraction of challenging scenarios (i.e., with large kalman errors) are much larger than the ETH-UCY, Figure of (Makansi et al., 2021).
for (i), the concurrent work (Makansi al., 2021 hiighliits the long-tailed distribution of common pedestrian benehmarks (c.g., ETH-UCY) meaning that most scenarios can be predicted well using a simple linear Kalman filter (so-called easy scenarios) while more challenging scenarios (where interaction is needed) are rare; this is less pronounced the sports dataset where the proportion of hard scenes (requiring interaction modelling) is larger, see Fig. 7. As for (ii), in existing benchmark settings, people tend to walk toward their destinations mostly avoiding interactions. except for the purpose of avoiding obstacles and collisions, whereas interaction is an integral part of the activity in sporty scenario such playing game of basketball.
Concurrently to our work, Chen et al. (2021 hhighlight the importance of the bias between training and testing environments and introduced counterfactual analysis in which they intervene on the causal graph by cutting the edge between the predicted trajectory and the environment. To do so, they utilize several choices for the counterfactual trajectory such as the mean trajectory or a random trajectory. Our method, on the other hand, remove totally the influence of a feature (neighboring trajectory) by removing its edges in the interaction graph.
E INTERACTION GRAPH LEARNING
An interesting and related line of works is to explicitly learn an interaction graph part of the tra- jectory prediction task. Kipf et al. (2018) introduce the NRI framework, a variational auto-encoder, where the learned latent code represents the underlying interaction graph of the system. Several recent works build upon the NRI framework by allowing the interaction graph to change over time (i.e.. dynamic relations) (Li et al., 2020; Graber & Schwing, 2020). Löwe al. (2020) leverage the shared knowledge across different samples and scenes and learn the causal relations between time series (e.g., moving agents). We adapted the latter work into the state-of-the the-arrt framework for tra-
14

---

Published as a conference paper at ICLR 2022
Table 3: Average emoos (min-ADE/ min-FDE) of different variants of the Traj++NRI framwork with different priors on the learned interaction graph (0.5, 0.9, 0.1).
Method \Setup
With Interaction
Without Interaction
Traj++
0.30 0.54
0.30 0.54
Traj++NRI (0.5)
0.32 0.58
0.30 0.53
Traj++NRI (0.9)
0.31 0.55
0.30 0.55
Traj++NRI (0.1)
0.32 0.58
0.30 0.54
jectory prediction (Traj++ (Salzmann et al., 2020)) by replacing the edge encoder the framework with learned interaction graph (referred as Traj++NRI) and train the model on the ETH-UCC//niv scene. Tab. shows an analysis the Traj++ model with its original edge encoder and the learned interaction graph from Löwe al. (2020) for different prior graphs. We observe that learning interaction graph did not lead to a better model and dropping the edges during inference yielded similar/better performance. This supports our hypothesis that common benchmarks, social inter- action features are not learned.
F SCENE-AGENT INTERACTION
Although our work targets mainly the analysis between agents, extending this analysis to the in- teraction between dynamic agents and the scene (e.g., semantic map) valuable. To this end, one option would be to learn a compact set of high-level map features (e.g., via a CNN encoder) and to add this an additional single node to the interaction graph. Our approach could then be applied to quantify also the contribution of map information to model performance.
15


Reviews:
[{"review_id": "p2_V2juA-q", "reviewer": "Reviewer_jvES", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes to analyze the contribution of different clues with the Shapley values and obtain an interesting conclusion that the past trajectory of the target is the only feature used for predicting its future for state-of-the-art trajectory prediction methods on standard benchmark datasets. \n\nStrengths:\n1) I think this conclusion makes sense. Some casual-inference based trajectory prediction methods [1] also found the interactions are biased in the current trajectory prediction systems and datasets. This paper proposes to use the Shapley values to analyze the contribution of different clues, which provides a quantitative evaluation of these biased interactions. \n2) The contribution of this method is clear. The discovery of the uselessness of interactions can help other researchers to develop better trajectory prediction methods. The experimental results on SportVU are also an important discovery to show the trajectory prediction models have the capability to learn the interactions.  I guess it is because the interactions in the training and testing dataset on SportVU are not biased, while the interactions in ETH-UCY or SDD are based.  \n3）The paper is well written, which clearly shows the motivation, conclusion, and technical details. Compared with original Shapley values, this method modifies the value of the adjacency matrix as zero to drop features and proposes a local-global contribution aggregation strategy for variable features in different scenarios.\n\nWeakness:\n1) Some methods[1] also found the biased interactions and analyzed this in the perspective of causal inference (Effect of treatment on the treated). I think the differences between the two methods are clear, and there are the links of this method to causality in the discussion  Yet, it is suggested to provide the discussion with this paper, which also thinks the interactions are biased and analyzes this with the Effect of Treatment on the Treated. \n2) In my opinion, some conclusions should be further discussed. For example, why SportVU is ok to learn interactions but not SDD?  I guess it is because of the level of interaction bias in training and testing environments.  SportVU only contains the scene of NBA for both training and testing. However, in other datasets, the scenes are changed, which is similar to the out-of-distribution problem. This setting may be more appropriate for self-driving. It is suggested to conduct a new experiment, where fewer interactions biases in training and testing environments. For example, all training and testing trajectories are from the ETH scene, but not from hotel, zara or univ. \n3) Some technical details are suggested to provide. The performance of Traj++ on SDD shows a dramatic improvement than PECNet. Can you provide the details of the training and testing of Traj++ on SDD?\n\n[1]Chen G, Li J, Lu J, et al. Human Trajectory Prediction via Counterfactual Analysis[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 9824-9833.\n\n", "summary_of_the_paper": "Different from attributing the trajectory prediction power with the success of modeling agent-agent interactions, this paper presents a new perspective that the past trajectory of the target is the only feature used for predicting its future for state-of-the-art trajectory prediction methods on standard benchmark datasets.  To analyze the contribution of different clues, this paper applies to compute the Shapley values. The proposed method modifies the value of the adjacency matrix as zero to drop features and analyze. Besides, a local-global contribution aggregation method is proposed for the problem of variable features in different scenarios. \n\nThis paper analyzes many state-of-the-art trajectory prediction models (Trajectorn++, PECNet, and Social-STGCNN) on many datasets (ETH-UCY, SDD, nuScenes, and SportVU). The experimental results on SportVU show the trajectory prediction models have the capability to learn the interactions on the datasets where interactions are more important.\n", "summary_of_the_review": "Overall, I think it is a good paper with some minor problems that presents a new perspective that the past trajectory of the target is the only feature used for predicting its future for state-of-the-art trajectory prediction methods on standard benchmark datasets. On the balance of positive and negative points, I think it is a borderline paper and tend to accept.\n\nPost Rebuttal： Thanks for the authors' feedback. The rebuttal answers most of my questions, which raises my score. It is suggested to further analyze why the interactions have limited effect in trajectory prediction. It is unintuitive but makes sense. There are some potential reasons in my views: 1) There are biases in interactions? 2) The multi-modality generation breaks the effect of interactions? 3) Is this finding generalized to rule-based methods, such as Social Force? I will further raise my score if these questions can be addressed.\n\n ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VXzjaWwwqIK", "reviewer": "Reviewer_hYNN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The manuscript has good clarity and explains its contributions well.\nThe analysis of the evaluation aligns with the claim that models are not learning interactions from ETH/UCY, nuScenes, and SDD, but are learning interactions from SportVU.\nThe evaluation of HTP models is an interesting research area that has not received nearly as much attention as works producing new HTP models using the same traditional evaluation metrics as others (i.e., minADE and minFDE).\n\nWhen dropping a subset of agents to determine the influence on model performance, the subset is replaced with static agents that are assumed not to influence the remaining agents. However in practice, static, non-interacting obstacles are still considered by humans during navigation due to collision avoidance.\nTo my understanding, the modified version of baseline Shapley values does not completely ensure that there is no interaction.\n\nIn Section 3.3, the authors note that random agents should not contribute to the future of the agent-of-interest, but again, they still play a role in the movement of the agent-of-interest due to collision avoidance.\nYet, in the results reported in Figure 4, random agents contribute at times negatives Shapley values (e.g., Traj++Edge on ETH/UCY, nuScenes, and SDD).\nAlso, what is the interpretation of the negative contributions of the random agent and interaction (specifically for Traj++Edge on ETH/UCY)?\n\nIn Figure 4, the Shapley values reported differ for each dataset. ETH/UCY is missing PECNet, nuScenes is missing STGCNN and PECNet, SDD is missing STGCNN, and SportVU is missing STGCNN.\nAlso, the Shapley values for ETH/UCY, nuScenes, and SDD are reported based on NLL, but not on min-ADE with the exception of subfigures (d) and (f).\n\nRegarding the interactions in ETH/UCY and nuScenes, there are numerous scenes in which no interaction occurs between agents (especially in nuScenes).\nIt would be helpful to see the contribution of social interactions on the subset of scenes in each dataset that have meaningful social interactions present in them.\n\nIt is surprising to see that the same models show significantly higher contributions of interaction in SportVU than in ETH/UCY, which has hundreds of scenes under \"students001\" and \"students003\" and feature extremely high levels of interaction between many pedestrians.\n\nThe authors note that the contributions of past trajectory and neighbors can be compared for an individual model (e.g., for Traj++Edge), but do not explain how the contributions can be compared between models (e.g., between PECNet and Traj++).\nAccording to Figure 4(f), the contribution of past trajectory to PECNet is 3 to 4 times larger than the contributions of past trajectory to Traj++ and Traj++Edge.\nWhat does this mean?\n\nThe authors note that the interaction encoder of Traj++Edge is stronger, so it has higher contributions of history and interaction on SportVU, but for ETH/UCY, the contribution of interaction is less than Traj++ and is negative.\n\nThe PECNet paper reports results on ETH/UCY, but the results have not been included into Table 1.\nThe results of PECNet and Traj++ differ from the results reported in their respective papers. The authors should make a note of why they differ and also clarify on the missing data.\n\nI have some concern about the results reported in Table 1 for models without interaction. The ideal way to report this result is to train on a modified dataset in which every scene with $n$ agents is split up into $n$ subscenes with one agent each, thus ensuring that there is no possibility of agent interaction.\n\nIt would be helpful to see the local analysis on SportVU also done for interactive and non-interactive scenes in ETH/UCY, nuScenes, and SDD to showcase the lack of interaction learning.\n", "summary_of_the_paper": "This manuscript addresses the difficulty in interpreting what information a model learns from human trajectory prediction datasets. The authors propose a feature attribution method for trajectory prediction based on Shapley values. The authors also show that for commonly used datasets such as ETH/UCY, SDD, and nuScenes, models Trajectron++ and PECNet do not learn interaction information. However, on datasets featuring more interaction, such as SportVU, these models are able to learn social interactions.\n", "summary_of_the_review": "Although I believe that this manuscript is well-written, tackles an interesting problem, and has an evaluation that is consistent with its claims, I do not think there is enough evidence to back the strong claim that SOTA HTP models are unable to learn almost any interactions for well-studied datasets that have cases of high interaction, but are able to learn interactions for a dataset with more interaction.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RV5DSrdvTiW", "reviewer": "Reviewer_dgLJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is generally well written and easy to follow. The authors provided sufficient background information and a relatively comprehensive literature review. The major motivation of this work is clearly stated.\n2. Evaluation and analysis of the interaction modeling of trajectory prediction models is an interesting task and could contribute to figuring out the most useful approaches/representations to capture mutual interactions.\n2. In order to support the authors' claims, they conducted many experiments and the results seem supportive based on the assumption that the quantifiable metric they proposed is reasonable.\n\nWeaknesses:\n1. Since all the experimental results are based on the Shapley value, it would be better to provide more explanation and evidence for its applicability in the area of trajectory prediction. Also, is there any limitation/caveat when using Shapley value as the metric for this purpose?\n2. Three closely related papers are listed below, which explicitly model the interactions/relations between interactive agents. It would be better to include them in the literature review. In particular, it would be better to conduct some experiments using the model proposed in [1], which makes the statement even more convincing.\n[1] Neural relational inference for interacting systems, ICML 2018.\n[2] EvolveGraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning, NeurIPS 2020.\n[3] Dynamic Neural Relational Inference, CVPR 2020.\n", "summary_of_the_paper": "This paper focuses on analysis on whether the existing trajectory prediction methods indeed capture the interactions between multiple agents and uses two example methods on multiple benchmark datasets to obtain supportive results for their claims. More specifically, the contributions are three folds:\n1. This paper addresses feature attribution for trajectory prediction to gain insights about the actual cues contemporary methods use to make predictions. The authors designed a variant of Shapley values that is applicable to a large set of trajectory prediction models.\n2. This paper quantifies feature attributions both locally and globally and studies the robustness of given models.\n3. This paper claims that existing models do not use interaction features, which is contrary to the statements made by the authors of the original papers.", "summary_of_the_review": "Overall, I think that the contributions of this paper are interesting and could raise more discussion and exploration on the interaction modeling in multi-agent systems. I have a positive feeling about this paper. My major concern is the suitability of the proposed Shapley value metric in the task of trajectory prediction. Hope to see more explanations in the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NrRUigyalm2", "reviewer": "Reviewer_9atf", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe this work is valuable overall for the trajectory prediction community. They present the problem well and the general framework of trajectory prediction[history encoder, interaction encoder and decoder] is workable and applied to most of the current works. \nQuestion: The choice of using static, non-interacting agents seems feasible but what about other options? Can the authors elaborate on this and add a discussion for other choices and how these choices could alternate the results? For example, one might suggest using the slowest agent. \n\nIt seems from the Shapley values that the random agents doesn't contribute much to the results because of the lack of interaction as shown in the work and this is a good finding. \n\nThe only thing I’m skeptical about is the definition of the social interaction score. Though it seems justified by the presented argument, it needs a kind of empirical verification. Training from scratch the trajectory prediction models with no interactions (aka single agents only, no neighbours) should produce similar (or better) results to the ones with the neighbours. This will prove the feasibility of the social score and the whole findings of the papers. I can understand that the training can take a while, but providing results on two models with at least a subset of the ETH/UCY (I’d avoid the ETH and use Zara1, Zara2, Hotel as ETH is highly unbalanced) dataset will be a strong addition. \n\n=== Post Rebuttal Comment ===\nI believe that the authors have answered the concerns raised by the reviewers and my concerns. The new experiments on Trajectron++ proved the theoretical analysis. Overall, in the past years since 2016 plenty of deep models were introduced to solve the trajectory prediction problem with emphasis on the \"Social Interaction\" between the agents. This paper proves via analysis and empirical results that such \"Social Interaction\" does not exist or have a minimal effect. Thus, I find this work significant in this area as it will lead future research into figuring out a true form of pedestrian’s interaction. Based on this, I’m raising my score.\n", "summary_of_the_paper": "This work discusses the features that attribute to the prediction results in trajectory prediction models. They apply a Shapley values to attribute the features and results, attempting to answer a question if the social models are really social or not. The meaning of a model being social in this case is that the model can use neighbour features in order to predict the trajectory. The work defines a measurement metric called the social interaction score, in which it measures how well the interaction between agents attributes to the prediction of the other agent trajectory. \n", "summary_of_the_review": "The paper is valuable to the community of trajectory prediction models. It directly discusses if the “social” interaction exists or not in previously reported models. Some questions were raised regarding the choice of the static, non-interacting agent and the feasibility of the presented social interaction score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "FJPYTzWQT5S", "reviewer": "Reviewer_ef1N", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is presented clearly and a better feature attribution method can help us understand blackbox predictors and make sure if the presented model learns the interactions we want. I only have a few questions.\n\nNovelty. Changing latent codes or features and computing their influence on performance is not entirely new. It has been used to understand blackbox vision/language models. The authors need to compare with prior work and show why this proposed measure is novel rather than applying it to trajectory prediction. \n\nExtending to predictors that include map info. The way to drop features and aggregate the Shapley value is defined purely based on the agent interaction graph. How does this extend to other approaches that use information such as map features?\n\nChoice of behavior for computing the Shapley values. The contribution of the other agents does not necessarily influence the accuracy of the predicted accuracy, it may reduce the uncertainty of behaviors too. How does this paper handle the influence not directly reflected in accuracy?\n\n-----\nPost-rebuttal update: \nThe authors answered my questions and address other reviewers' questions. While the proposed approach directly extends Shapley value for trajectory prediction, the analysis is valuable for the community. So, I’m raising my score.", "summary_of_the_paper": "This paper proposes assessing the contribution of each feature by computing a version of Shapley values based on the existing trajectory predictor formulation. They apply this measure to the state-of-the-art trajectory prediction methods and found that many standard benchmarks don’t require reasonings of agent interactions.", "summary_of_the_review": "The paper clearly presents a variant of Shapley value for trajectory prediction tasks. The proposed measure helps identify if a model uses a feature for prediction. My main question is about the novelty of the proposed approach.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript brings up an important issue: that current methods and datasets don't generally highlight interactions when it comes to trajectory prediction. This is despite the fact that it would seem that current methods incorporate agent interactions and that datasets appear to require reasoning about agent interactions. This qualitative and quantitative observation should lead to better datasets in the future as well as more refined metrics pushing the field forward. Reviewers were in agreement that this is a strong submission. The authors responded with substantive new experiments that cleared up any lingering issues.
Decision: Accept (Poster)