Collaborative Uncertainty in Multi-Agent Trajectory Forecasting

1 Introduction . A multi-agent trajectory forecasting system aims to predict future trajectories of multiple agents based on their observed trajectories and surroundings [ 1 , 2 ] . Precise trajectory prediction provides essential information for decision making and safety in numerous intelligent systems , including autonomous vehicles [ 3 , 4 , 5 , 6 ] , drones [ 7 ] , and industrial robotics [ 8 , 9 ] . The rapid development of deep learning has enabled a number of deep-learning-based algorithms to handle multi-agent trajectory forecasting [ 3 , 4 , 5 , 6 , 10 , 11 , 12 , 13 , 14 , 15 ] . These methods exhibit state-of-the-art performances , with some having been integrated into real-world systems . However , deep-learning-based forecasting is not always reliable or interpretable [ 16 , 17 , 18 ] . In circumstances when noises from the environment are overwhelmingly distracting , or when the situation has never been encountered before , a deep-learning-based algorithm could provide baffling predictions , which might cause terrible tragedies . A fundamental challenge is to know when we could rely on those deep-learning-based forecasting algorithms . To tackle this problem , one solution is to report the uncertainty of each prediction . Finding ways to best conceptualize and measure the prediction uncertainty of deep-learning-based algorithms becomes an imperative , which motivates this work . ∗The corresponding author is Siheng Chen . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There are two main types of uncertainty to model in deep-learning-based algorithms [ 19 ] : 1 ) aleatoric uncertainty , regarding information aside from statistic models , which data can not explain ; 2 ) epistemic uncertainty , the uncertainty inside a model , when the model lacks knowledge of the system/process being modeled ( e.g. , due to limited training data ) . As it is most effective to model aleatoric uncertainty in big data regimes such as those common to deep learning with image data [ 17 ] , this work focuses on aleatoric uncertainty . In the following passage , we use the term “ uncertainty ” to represent aleatoric uncertainty . [ 16 ] uses the predictive variance to approximate uncertainty in the Bayesian deep learning model , which has been widely adapted in many works [ 20 , 21 , 22 , 11 ] for uncertainty modeling in multi-agent trajectory forecasting . However , the predictive variance of a single agent alone may not suffice to reflect the complete landscape of uncertainty , especially when agent-wise interaction is present . Recent works that attempt to exploit the interaction among agents have impressively boosted the prediction precision , which further highlights the need to better measure uncertainty in multi-agent trajectory forecasting . We seek to build a more sophisticated and robust measurement for capturing the previously neglected uncertainty brought by correlated predictions . In this paper , we coin a concept individual uncertainty ( IU ) to describe the uncertainty that can be approximated by the predictive variance of a single agent . Relatively , we propose a new concept , collaborative uncertainty ( CU ) to estimate the uncertainty resulting from the usage of interaction modules in prediction models . We further introduce an original probabilistic CU-based framework to measure both individual and collaborative uncertainty in the multi-agent trajectory forecasting task . We apply this framework to two special cases : multivariate Gaussian distribution and multivariate Laplace distribution . In each case , our CU-based framework allows our model to simultaneously learn the mappings that are from input data to 1 ) accurate prediction , 2 ) individual uncertainty , and 3 ) collaborative uncertainty ; see Figure 1 for model illustration . Extensive experiments demonstrate that CU modeling yields significantly larger performance gains in prediction models equipped with interaction modules ( See Figure 4 ) , confirming that CU is highly related to the existence of the interaction modeling procedure , and adding CU modeling benefits accurate predictions . The contributions of this work are summarized as follows : ● We propose , analyze , and visualize a novel concept , collaborative uncertainty ( CU ) , to model the uncertainty brought by the interaction modules in multi-agent trajectory forecasting . ● We design a general CU-based framework to empower a prediction model to generate a probabilistic output , where the mean is the future trajectory and the covariance reflects the corresponding uncertainty . Under this framework , we show two special cases based on multivariate Gaussian and Laplace distributions respectively . ● We conduct extensive experiments to validate the CU-empowered prediction model on both synthetic datasets and two large-scale real-world datasets . On self-generated synthetic datasets , we validate the proposed method is able to closely reconstruct the ground-truth distribution . On the public benchmarks , the CU-empowered prediction model consistently outperforms the corresponding one without CU . Specially , by leveraging the proposed CU , VectorNet improves by 57 cm regarding Final Displacement Error ( FDE ) on nuScenes dataset ! 2 Related Works . Aleatoric uncertainty modeling in deep learning . Recent efforts are rising as to improve the measurement of aleatoric uncertainty in deep learning models . One seminal work is [ 17 ] . It proposes a unified Bayesian deep learning framework to explicitly represent aleatoric uncertainty using predictive variance for generic regression and classification tasks . Many existing works [ 23 , 24 , 25 , 26 , 27 , 28 ] follow this idea and formulate uncertainty as learned loss attenuation . For example , to make predictive- variance-based aleatoric uncertainty measurements more efficient , [ 24 ] adds data augmentation during the test-time . But , these works only pay attention to individual uncertainty . Other recent works attend to the uncertainty measurement for correlated predictive distributions . For example , [ 29 ] and [ 30 ] measure spatially correlated uncertainty in a generative model for respectively image reconstruction and pixel-wise classification , and [ 31 ] captures joint uncertainty as discrete variables in the field of reinforcement learning . Despite these three works , our work is the first to conceptualize and measure collaborative uncertainty in the multi-agent trajectory forecasting task . To the best of our knowledge , there are only two papers [ 32 , 33 ] close to our track . [ 32 ] and [ 33 ] model the joint uncertainty in the pose estimation task and multi-agent trajectory forecasting task respectively . However , they present several limitations : 1 ) They only examined the circumstance where the model ’ s output follows Gaussian distribution ; 2 ) They did not provide a theoretical conceptualization or definition for the uncertainty due to correlated predictive distributions , and they did not analyze the causes of such uncertainty . These are essential problems to tackle . In this work , we not only formulate a general framework that works for both Gaussian and Laplace distributions , but we also theoretically conceptualize collaborative uncertainty and analyze its causes . Multi-agent trajectory forecasting . This task takes the observed trajectories from multiple agents as the inputs , and outputs the predicted trajectory for each agent . Like many other sequence prediction tasks , this task used to use a recurrent architecture to process the inputs [ 34 , 35 , 36 ] . Later , however , the graph neural networks become a more common approach as they can significantly assist trajectory forecasting by capturing the interactions among agents [ 3 , 4 , 5 , 6 , 12 , 13 , 14 , 15 ] . For safety reasons , it is necessary to report the uncertainty of each predicted trajectory . Works to date about uncertainty measurement [ 11 , 21 , 22 , 37 , 38 ] have appropriately modeled the interaction among multi-agent trajectories for boosting performances , but they overlook the uncertainty resulting from the correlation in predicted trajectories . We seek to fill this gap by introducing and modeling collaborative uncertainty . 3 Methodology . 3.1 Problem Formulation . Considerm agents in a data sample , and let X= { x1 , x2 , ... , xm } , Y= { y1 , y2 , ... , ym } be the past observed and the future trajectories of all agents , where xi ∈R2T− and yi ∈R2T+ are the past observed and the future trajectories of the i-th agent . Each xi/yi consists of two-dimensional coordinates at different timestamps of T−/T+ . We assume that a training dataset D consists of N individual and identically distributed data samples { ( Xi , Yi ) } Ni=1 . For predicting future trajectories of multiple agents and modeling the uncertainty over the predictions , we seek to use a probabilistic framework to model the predictive distribution p ( Y ∣X ) of multiple agents ’ future trajectories based on the training dataset D. Previous works in uncertainty modeling [ 16 , 17 , 39 ] use Gaussian distribution to approximate p ( Y ∣X ) . The assumption behind this approach is that p ( yi∣xi ) is independent for every i ∈ { 1,2,3 , ... , m } . Mathematically , they set the covariance matrix of p ( Y ∣X ) as a diagonal matrix . This assumption is valid for the regression task that uses the model shown in Figure 2 ( a ) . We refer the uncertainty under the independence assumption as individual uncertainty in this paper . However , Figure 2 ( b ) considers a prediction model that includes interaction modeling among multiple agents : yi is no longer dependent solely on xi , but also on other agents xj where j ≠ i in the scene . We call the uncertainty brought by this interaction collaborative uncertainty . The existence of collaborative uncertainty turns p ( Y ∣X ) from the individual distribution into the joint distribution of multiple agents . Contrary to existing methods , we consider collaborative uncertainty and model p ( Y ∣X ) more accurately by making the covariance matrix a full matrix without imposing any restrictions on its form . In the following subsection , we will introduce an approach to modeling both individual uncertainty and collaborative uncertainty using a unified CU-based framework . 3.2 General Formulation of Collaborative Uncertainty . In this work , to model collaborative uncertainty , we abandon the independence assumption held by previous works [ 16 , 17 , 39 ] , setting p ( Y ∣X ) as a joint multivariate distribution , whose mean is µ ∈ Rm×2T+ and covariance Σ ∈ Rm×m×2T+ . Element µi , t is the expected position of the i-th agent at timestamp t. As the diagonal elements of Σ are considered individual uncertainty [ 16 , 17 , 39 ] , we further let off-diagonal elements describe collaborative uncertainty . Diagonal element Σi , i , t models the variance of the i-th agent at timestamp t ; off-diagonal element Σi , j , t models the covariance between the i-th and j-th agents at timestamp t. Therefore , we can simultaneously obtain individual and collaborative uncertainty by estimating the covariance Σ of p ( Y ∣X ) . Accordingly , we propose a CU-based comprehensive uncertainty estimation framework ( see Figure 3 ) with the following steps : Step 1 : Choose a probability density function , p ( Y ∣X ; µ , Σ , Φ ) , for the predictive distribution , which includes a mean µ∈Rm×2T+ used to approximate the future trajectories , a covariance Σ∈Rm×m×2T+ used to quantify individual uncertainty and collaborative uncertainty , and some auxiliary parameters Φ used to describe the predictive distribution . Further , we set covariance matrix Σt , which represents the covariance matrix at timestamp t , as a full matrix instead of an identity or diagonal matrix . Step 2 : Design a prediction model , F [ µw ( X ) , Σw ( X ) , Φw ( X ) ] , where µw ( X ) , Σw ( X ) and Φw ( X ) are three neural networks , which approximate values of mean µ , covariance Σ and auxiliary parameters Φ respectively . Note that w only indicates the parameters of these neural networks are trainable , and does not mean they share same parameters . Step 3 : Derive a loss function from p ( Y ∣X ; µ , Σ , Φ ) via maximum likelihood estimation : L ( w ) = − N ∑ i=1 log p ( Yi ∣Xi ; µw ( Xi ) , Σw ( Xi ) , Φw ( Xi ) ) minimized to update trainable parameters in µw ( ⋅ ) , Σw ( ⋅ ) and Φw ( ⋅ ) .

Summary: The paper presents a novel concept — collaborative uncertainty — in the context of multi-agent trajectory forecasting. Unlike many previous works that only model the individual uncertainty of joint future trajectory distribution and use a diagonal covariance matrix, this paper considers both individual and collaborative uncertainty of multi-agent trajectories and uses a full covariance matrix with off-diagonal components representing collaborative uncertainty between agents. The paper argues that such a design could model the GT multi-agent trajectory distribution more accurately. The paper performs synthetic experiments showcasing the method's ability to model joint trajectory distribution better than models that only consider individual uncertainty. Experiments on real-world trajectory forecasting benchmarks also demonstrate that incorporating collaborative uncertainty improves the performance of deterministic baselines.