You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Prediction

1 INTRODUCTION . Predicting the future trajectory of a moving agent is a significant problem relevant to domains such as autonomous driving ( Weisswange et al. , 2021 ; Xu et al. , 2014 ) , robot navigation ( Chen et al. , 2018 ) , or surveillance systems ( Morris & Trivedi , 2008 ) . Accurate trajectory prediction requires successful integration of multiple sources of information with varying signal-to-noise ratios : whereas the target agent ’ s past trajectory is quite informative most of time , interactions with other agents are typically sparse and short in duration , but can be crucial when they occur . Graph neural networks ( N. Kipf & Welling , 2017 ) and transformers ( Vaswani et al. , 2017 ) have led to recent progress w.r.t . average prediction error ( Mohamed et al. , 2020 ; Yu et al. , 2020 ; Salzmann et al. , 2020 ; Mangalam et al. , 2020 ) , but the predicted distributions over future trajectories remain far from perfect , particularly in unusual cases . Moreover , the black-box nature of state-of-the-art models ( typically consisting of various recurrent , convolutional , graph , and attention layers ) makes it increasingly difficult to understand what information a model actually learns to use to make its predictions . To identify shortcomings of existing approaches and directions for further progress , in the present work , we propose a tool for explainable trajectory prediction . In more detail , we use Shapley values ( Shapley , 1953 ; Lundberg & Lee , 2017 ) to develop a feature attribution method tailored specifically to models for multi-modal trajectory prediction . This allows us to quantify the contribution of each input variable to the model ’ s performance ( as measured , e.g. , by the negative log likelihood ) both locally ( for a particular agent and time point ) and globally ( across an entire trajectory , scene , or dataset ) . Moreover , we propose an aggregation scheme to summarize the contributions of all neighboring agents into a single social interaction score that captures how well a model is able to use information from interactions between agents . By applying our analysis to representative state-of-the-art models , Trajectron++ ( Salzmann et al. , 2020 ) and PECNet ( Mangalam et al. , 2020 ) , we find that—contrary to claims made in the respective works—the predictions of these models on the common benchmark datasets ETH-UCY ( Pellegrini et al. , 2009 ; Leal-Taixé et al. , 2014 ) , SDD ( Robicquet et al. , 2016 ) , and nuScenes ( Caesar et al. , 2020 ) are actually not based on interaction information . We , therefore , also analyze these models ’ behavior on an additional sports dataset SportVU ( Yue et al. , 2014 ) for which we expect a larger role of interaction . There , models do learn to use social interaction , and the proposed feature attribution method can quantify ( i ) when these interaction signals are active and ( ii ) how relevant they are , see Fig . 1 for an overview . Overall , our analysis shows that established trajectory prediction datasets are suboptimal for benchmarking the learning of interactions among agents , but existing approaches do have the capability to learn such interactions on more appropriate datasets . We highlight the following contributions : • We address , for the first time , feature attribution for trajectory prediction to gain insights about the actual cues contemporary methods use to make predictions . We achieve this by designing a variant of Shapley values that is applicable to a large set of trajectory prediction models ( § 3 ) . • With this method , we quantify feature attributions both locally ( per scenario ; § 3.1 ) and globally ( over the whole dataset ; § 3.2 ) and study the robustness of given models ( § 3.3 ) . • Our study uncovers that , on popular benchmarks , existing models do not use interaction features . However , when those features have a strong causal influence on the target ’ s future trajectory , these models start to reason about them ( § 4.4 ) . 2 BACKGROUND & RELATED WORK . Our primary goal is to better understand the main challenges faced by existing trajectory prediction models . First , we formalize the problem setting ( § 2.1 ) and review prior approaches ( § 2.2 ) , as well as explainability tools ( § 2.3 ) , which we will use to accomplish this goal in § 3 . 2.1 THE TRAJECTORY PREDICTION PROBLEM SETTING . Let { X1t , ... , Xnt } , t ∈ Z denote a set of time series corresponding to the trajectories of n agents that potentially interact with each other ; see Fig . 2 ( a ) for an illustration . All Xit are assumed to take values in Rd , that is , the state of agent i at time t is a vector xit ∈ Rd encoding , e.g. , its 2D position , velocity , and acceleration . We refer to an observed temporal evolution xi1 : T = ( x i 1 , ... , x i T ) of a single agent as a trajectory of length T and to the collection of trajectories x1 : n1 : T = ( x 1 1 : T , ... , x n 1 : T ) of all n agents as a scene . Assume that we have access to a training dataset consisting of M such scenes.1 The trajectory prediction task then consists of predicting the future trajectory ( Xit+1 , ... , X i t+∆t ) of a given target agent i and at a given time t up to a time horizon ∆t given the observed past x1 : nt−h : t of both the target agent itself and of all the other agents , where h is the length of the history that is taken into account . Formally , we thus aim at learning the distributions P ( Xit+1 : t+∆t|X1 : nt−h : t ) for any t ∈ Z and i ∈ N : = { 1 , ... , n } with given ∆t and h. Since any given agent ( e.g. , a particular person ) typically only appears in a single scene in the training dataset , solving the trajectory prediction task requires generalizing across scenes and agent identities . In other words , it is necessary to learn how an agent ’ s future trajectory , in general , depends on its own past and on the past behavior of other neighboring agents . 1In practice , scenes may differ both in length and in the number of present agents ; for the latter , we define n as the maximum number of agents present in a given scene across the dataset and add “ dummy ” agents to scenes with fewer agents , see § 4.1 for further details . 2.2 EXISTING APPROACHES FOR TRAJECTORY PREDICTION . Fig . 2 ( b ) – ( d ) shows a general framework that unifies most existing approaches for trajectory prediction . It consists of three main modules : ( b ) a history encoder ψ that learns an embedding F i = ψ ( Xit−h : t ) of the history of each agent ; ( c ) an interaction encoder ξ that incorporates information from neighboring agents by learning an embedding for the edge between two agents Eij = ξ ( Xit−h : t , X j t−h : t ) ; and ( d ) a decoder ϑ that combines both history and interaction features to generate the predicted future trajectory x̂it+1 : t+∆t : = ϑ ( F i , Ēi ) of the target agent , where Ēi is the result of aggregating all edge embeddings ( see below ) . Existing approaches mostly differ in the choices of these modules . To handle the temporal evolution of the agent state , LSTMs ( Hochreiter & Schmidhuber , 1997 ) are widely used for ψ and ϑ , i.e. , to encode the history and decode the future trajectory , respectively ( Alahi et al. , 2016 ; Zhang et al. , 2019 ) . Since the future is highly uncertain , stochastic decoders are typically used to sample multiple trajectories , x̂it+1 : t+∆t ∼ ϑ ( . ) , e.g. , using GANs ( Gupta et al. , 2018 ) or conditional VAEs ( Lee et al. , 2017 ; Salzmann et al. , 2020 ) . Alternatively , Makansi et al . ( 2019 ) proposed a multi-head network that directly predicts the parameters of a mixture model over the future trajectories . To handle interactions between moving agents ( Fig . 2c ) , existing works model a scene as a graph in which nodes correspond to the state embeddings F i of the n agents and edges are specified by an adjacency matrix A with Aij = 1 iff . agents i and j are considered neighbors ( based , e.g. , on their relative distance ) . Given this formulation , local social pooling layers have been used to encode and aggregate the relevant information from nearby agents within a specific radius ( Alahi et al. , 2016 ; Gupta et al. , 2018 ) . More recently , Mangalam et al . ( 2020 ) proposed a non-local social pooling layer , which uses attention and is more robust to false neighbor identification , while Salzmann et al . ( 2020 ) modeled the scene as a directed graph to represent a more general set of scenes and interaction types , e.g. , asymmetric influence . In a different line of work , Mohamed et al . ( 2020 ) proposed undirected spatio-temporal graph convolutional neural networks to encode social interactions , and Yu et al . ( 2020 ) additionally incorporated transformers based on self-attentions to learn better embeddings . In general , the edge encodings { Eij } j for agent i are then aggregated over the set of its neighbors ( i.e. , those j with Aij = 1 ) to yield the interaction features Ēi , e.g. , by averaging as follows : Ēi = ∑ j 6=iA ijEij∑ j 6=iA ij . ( 1 ) Although some of the above works show promising results on common benchmarks , it is unclear what information these methods actually use to predict the future . In the present work , we focus mainly on this aspect and propose an evaluation procedure that quantifies the contribution of different features to the predicted trajectory , both locally and globally . 2.3 EXPLAINABILITY AND FEATURE ATTRIBUTION . An important step toward interpretability in deep learning are feature attribution methods which aim at quantifying the extent to which a given input feature is responsible for the behavior ( e.g. , prediction , uncertainty , or performance ) of a model . Among the leading approaches in this field is a concept from cooperative game theory termed Shapley values ( Shapley , 1953 ) , which fairly distributes the payout of a game among a set of n players.2 Within machine learning , Shapley values can be used for feature attribution by mapping an input x = ( x1 , ... , xn ) to a game in which players are the individual features xi and the payout is the model behavior f : X1 × ... × Xn → R on that example ( Lundberg & Lee , 2017 ) . Formally , one defines a set function ν : 2N → R with N : = { 1 , ... , n } whose output ν ( S ) for a subset S ⊂ N corresponds to running the model on a modified version of the input x for which features not in S are “ dropped ” or replaced ( see below for details ) . The contribution of xi , as quantified by its Shapley value φ ( xi ) , is then given by the difference ν ( S ∪ { i } ) − ν ( S ) between including and not including i , averaged over all subsets S : φ ( xi ) = ∑ S⊆N\ { i } 1 n ( n−1 |S| ) ( ν ( S ∪ { i } ) − ν ( S ) ) . ( 2 ) There are different variants of Shapley values based mainly on the following two design choices : ( i ) What model behavior ( prediction , uncertainty , performance ) is to be attributed ? ( choice of f ) ( ii ) What is the reference baseline , that is , how are features dropped ? ( choice of ν ) A common choice for ( i ) is the output ( i.e. , prediction ) of the model . Alternatively , Janzing et al . ( 2020a ) defined f as the uncertainty associated with the model prediction , thus quantifying to what extent each feature value contributes to or reduces uncertainty . In the present work , we will focus on attributing model performance . As for ( ii ) the set function ν , Lundberg & Lee ( 2017 ) originally proposed to replace the dropped features xN\S with samples from the conditional data distribution given the non-dropped features : ν ( S ) = E [ f ( xS , XN\S ) |XS = xS ] ; they then approximate this with the marginal distribution , ν ( S ) = E [ f ( xS , XN\S ) ] , using the simplifying assumption of feature independence . Janzing et al . ( 2020b ) argued from a causal perspective that the marginal distribution is , in fact , the right distribution to sample from since the process of dropping features naturally corresponds to an interventional distribution , rather than an observational ( conditional ) one . In an alternative termed baseline Shapley values , the dropped features are replaced with those of a predefined baseline x′ : ν ( S ) = f ( xS , x′N\S ) , see Sundararajan & Najmi ( 2020 ) for details .

Summary: This paper focuses on analysis on whether the existing trajectory prediction methods indeed capture the interactions between multiple agents and uses two example methods on multiple benchmark datasets to obtain supportive results for their claims. More specifically, the contributions are three folds: 1. This paper addresses feature attribution for trajectory prediction to gain insights about the actual cues contemporary methods use to make predictions. The authors designed a variant of Shapley values that is applicable to a large set of trajectory prediction models. 2. This paper quantifies feature attributions both locally and globally and studies the robustness of given models. 3. This paper claims that existing models do not use interaction features, which is contrary to the statements made by the authors of the original papers.