You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Prediction

1 INTRODUCTION . Predicting the future trajectory of a moving agent is a significant problem relevant to domains such as autonomous driving ( Weisswange et al. , 2021 ; Xu et al. , 2014 ) , robot navigation ( Chen et al. , 2018 ) , or surveillance systems ( Morris & Trivedi , 2008 ) . Accurate trajectory prediction requires successful integration of multiple sources of information with varying signal-to-noise ratios : whereas the target agent ’ s past trajectory is quite informative most of time , interactions with other agents are typically sparse and short in duration , but can be crucial when they occur . Graph neural networks ( N. Kipf & Welling , 2017 ) and transformers ( Vaswani et al. , 2017 ) have led to recent progress w.r.t . average prediction error ( Mohamed et al. , 2020 ; Yu et al. , 2020 ; Salzmann et al. , 2020 ; Mangalam et al. , 2020 ) , but the predicted distributions over future trajectories remain far from perfect , particularly in unusual cases . Moreover , the black-box nature of state-of-the-art models ( typically consisting of various recurrent , convolutional , graph , and attention layers ) makes it increasingly difficult to understand what information a model actually learns to use to make its predictions . To identify shortcomings of existing approaches and directions for further progress , in the present work , we propose a tool for explainable trajectory prediction . In more detail , we use Shapley values ( Shapley , 1953 ; Lundberg & Lee , 2017 ) to develop a feature attribution method tailored specifically to models for multi-modal trajectory prediction . This allows us to quantify the contribution of each input variable to the model ’ s performance ( as measured , e.g. , by the negative log likelihood ) both locally ( for a particular agent and time point ) and globally ( across an entire trajectory , scene , or dataset ) . Moreover , we propose an aggregation scheme to summarize the contributions of all neighboring agents into a single social interaction score that captures how well a model is able to use information from interactions between agents . By applying our analysis to representative state-of-the-art models , Trajectron++ ( Salzmann et al. , 2020 ) and PECNet ( Mangalam et al. , 2020 ) , we find that—contrary to claims made in the respective works—the predictions of these models on the common benchmark datasets ETH-UCY ( Pellegrini et al. , 2009 ; Leal-Taixé et al. , 2014 ) , SDD ( Robicquet et al. , 2016 ) , and nuScenes ( Caesar et al. , 2020 ) are actually not based on interaction information . We , therefore , also analyze these models ’ behavior on an additional sports dataset SportVU ( Yue et al. , 2014 ) for which we expect a larger role of interaction . There , models do learn to use social interaction , and the proposed feature attribution method can quantify ( i ) when these interaction signals are active and ( ii ) how relevant they are , see Fig . 1 for an overview . Overall , our analysis shows that established trajectory prediction datasets are suboptimal for benchmarking the learning of interactions among agents , but existing approaches do have the capability to learn such interactions on more appropriate datasets . We highlight the following contributions : • We address , for the first time , feature attribution for trajectory prediction to gain insights about the actual cues contemporary methods use to make predictions . We achieve this by designing a variant of Shapley values that is applicable to a large set of trajectory prediction models ( § 3 ) . • With this method , we quantify feature attributions both locally ( per scenario ; § 3.1 ) and globally ( over the whole dataset ; § 3.2 ) and study the robustness of given models ( § 3.3 ) . • Our study uncovers that , on popular benchmarks , existing models do not use interaction features . However , when those features have a strong causal influence on the target ’ s future trajectory , these models start to reason about them ( § 4.4 ) . 2 BACKGROUND & RELATED WORK . Our primary goal is to better understand the main challenges faced by existing trajectory prediction models . First , we formalize the problem setting ( § 2.1 ) and review prior approaches ( § 2.2 ) , as well as explainability tools ( § 2.3 ) , which we will use to accomplish this goal in § 3 . 2.1 THE TRAJECTORY PREDICTION PROBLEM SETTING . Let { X1t , ... , Xnt } , t ∈ Z denote a set of time series corresponding to the trajectories of n agents that potentially interact with each other ; see Fig . 2 ( a ) for an illustration . All Xit are assumed to take values in Rd , that is , the state of agent i at time t is a vector xit ∈ Rd encoding , e.g. , its 2D position , velocity , and acceleration . We refer to an observed temporal evolution xi1 : T = ( x i 1 , ... , x i T ) of a single agent as a trajectory of length T and to the collection of trajectories x1 : n1 : T = ( x 1 1 : T , ... , x n 1 : T ) of all n agents as a scene . Assume that we have access to a training dataset consisting of M such scenes.1 The trajectory prediction task then consists of predicting the future trajectory ( Xit+1 , ... , X i t+∆t ) of a given target agent i and at a given time t up to a time horizon ∆t given the observed past x1 : nt−h : t of both the target agent itself and of all the other agents , where h is the length of the history that is taken into account . Formally , we thus aim at learning the distributions P ( Xit+1 : t+∆t|X1 : nt−h : t ) for any t ∈ Z and i ∈ N : = { 1 , ... , n } with given ∆t and h. Since any given agent ( e.g. , a particular person ) typically only appears in a single scene in the training dataset , solving the trajectory prediction task requires generalizing across scenes and agent identities . In other words , it is necessary to learn how an agent ’ s future trajectory , in general , depends on its own past and on the past behavior of other neighboring agents . 1In practice , scenes may differ both in length and in the number of present agents ; for the latter , we define n as the maximum number of agents present in a given scene across the dataset and add “ dummy ” agents to scenes with fewer agents , see § 4.1 for further details . 2.2 EXISTING APPROACHES FOR TRAJECTORY PREDICTION . Fig . 2 ( b ) – ( d ) shows a general framework that unifies most existing approaches for trajectory prediction . It consists of three main modules : ( b ) a history encoder ψ that learns an embedding F i = ψ ( Xit−h : t ) of the history of each agent ; ( c ) an interaction encoder ξ that incorporates information from neighboring agents by learning an embedding for the edge between two agents Eij = ξ ( Xit−h : t , X j t−h : t ) ; and ( d ) a decoder ϑ that combines both history and interaction features to generate the predicted future trajectory x̂it+1 : t+∆t : = ϑ ( F i , Ēi ) of the target agent , where Ēi is the result of aggregating all edge embeddings ( see below ) . Existing approaches mostly differ in the choices of these modules . To handle the temporal evolution of the agent state , LSTMs ( Hochreiter & Schmidhuber , 1997 ) are widely used for ψ and ϑ , i.e. , to encode the history and decode the future trajectory , respectively ( Alahi et al. , 2016 ; Zhang et al. , 2019 ) . Since the future is highly uncertain , stochastic decoders are typically used to sample multiple trajectories , x̂it+1 : t+∆t ∼ ϑ ( . ) , e.g. , using GANs ( Gupta et al. , 2018 ) or conditional VAEs ( Lee et al. , 2017 ; Salzmann et al. , 2020 ) . Alternatively , Makansi et al . ( 2019 ) proposed a multi-head network that directly predicts the parameters of a mixture model over the future trajectories . To handle interactions between moving agents ( Fig . 2c ) , existing works model a scene as a graph in which nodes correspond to the state embeddings F i of the n agents and edges are specified by an adjacency matrix A with Aij = 1 iff . agents i and j are considered neighbors ( based , e.g. , on their relative distance ) . Given this formulation , local social pooling layers have been used to encode and aggregate the relevant information from nearby agents within a specific radius ( Alahi et al. , 2016 ; Gupta et al. , 2018 ) . More recently , Mangalam et al . ( 2020 ) proposed a non-local social pooling layer , which uses attention and is more robust to false neighbor identification , while Salzmann et al . ( 2020 ) modeled the scene as a directed graph to represent a more general set of scenes and interaction types , e.g. , asymmetric influence . In a different line of work , Mohamed et al . ( 2020 ) proposed undirected spatio-temporal graph convolutional neural networks to encode social interactions , and Yu et al . ( 2020 ) additionally incorporated transformers based on self-attentions to learn better embeddings . In general , the edge encodings { Eij } j for agent i are then aggregated over the set of its neighbors ( i.e. , those j with Aij = 1 ) to yield the interaction features Ēi , e.g. , by averaging as follows : Ēi = ∑ j 6=iA ijEij∑ j 6=iA ij . ( 1 ) Although some of the above works show promising results on common benchmarks , it is unclear what information these methods actually use to predict the future . In the present work , we focus mainly on this aspect and propose an evaluation procedure that quantifies the contribution of different features to the predicted trajectory , both locally and globally . 2.3 EXPLAINABILITY AND FEATURE ATTRIBUTION . An important step toward interpretability in deep learning are feature attribution methods which aim at quantifying the extent to which a given input feature is responsible for the behavior ( e.g. , prediction , uncertainty , or performance ) of a model . Among the leading approaches in this field is a concept from cooperative game theory termed Shapley values ( Shapley , 1953 ) , which fairly distributes the payout of a game among a set of n players.2 Within machine learning , Shapley values can be used for feature attribution by mapping an input x = ( x1 , ... , xn ) to a game in which players are the individual features xi and the payout is the model behavior f : X1 × ... × Xn → R on that example ( Lundberg & Lee , 2017 ) . Formally , one defines a set function ν : 2N → R with N : = { 1 , ... , n } whose output ν ( S ) for a subset S ⊂ N corresponds to running the model on a modified version of the input x for which features not in S are “ dropped ” or replaced ( see below for details ) . The contribution of xi , as quantified by its Shapley value φ ( xi ) , is then given by the difference ν ( S ∪ { i } ) − ν ( S ) between including and not including i , averaged over all subsets S : φ ( xi ) = ∑ S⊆N\ { i } 1 n ( n−1 |S| ) ( ν ( S ∪ { i } ) − ν ( S ) ) . ( 2 ) There are different variants of Shapley values based mainly on the following two design choices : ( i ) What model behavior ( prediction , uncertainty , performance ) is to be attributed ? ( choice of f ) ( ii ) What is the reference baseline , that is , how are features dropped ? ( choice of ν ) A common choice for ( i ) is the output ( i.e. , prediction ) of the model . Alternatively , Janzing et al . ( 2020a ) defined f as the uncertainty associated with the model prediction , thus quantifying to what extent each feature value contributes to or reduces uncertainty . In the present work , we will focus on attributing model performance . As for ( ii ) the set function ν , Lundberg & Lee ( 2017 ) originally proposed to replace the dropped features xN\S with samples from the conditional data distribution given the non-dropped features : ν ( S ) = E [ f ( xS , XN\S ) |XS = xS ] ; they then approximate this with the marginal distribution , ν ( S ) = E [ f ( xS , XN\S ) ] , using the simplifying assumption of feature independence . Janzing et al . ( 2020b ) argued from a causal perspective that the marginal distribution is , in fact , the right distribution to sample from since the process of dropping features naturally corresponds to an interventional distribution , rather than an observational ( conditional ) one . In an alternative termed baseline Shapley values , the dropped features are replaced with those of a predefined baseline x′ : ν ( S ) = f ( xS , x′N\S ) , see Sundararajan & Najmi ( 2020 ) for details .

Summary: This manuscript addresses the difficulty in interpreting what information a model learns from human trajectory prediction datasets. The authors propose a feature attribution method for trajectory prediction based on Shapley values. The authors also show that for commonly used datasets such as ETH/UCY, SDD, and nuScenes, models Trajectron++ and PECNet do not learn interaction information. However, on datasets featuring more interaction, such as SportVU, these models are able to learn social interactions.