Collaborative Uncertainty in Multi-Agent Trajectory Forecasting

1 Introduction . A multi-agent trajectory forecasting system aims to predict future trajectories of multiple agents based on their observed trajectories and surroundings [ 1 , 2 ] . Precise trajectory prediction provides essential information for decision making and safety in numerous intelligent systems , including autonomous vehicles [ 3 , 4 , 5 , 6 ] , drones [ 7 ] , and industrial robotics [ 8 , 9 ] . The rapid development of deep learning has enabled a number of deep-learning-based algorithms to handle multi-agent trajectory forecasting [ 3 , 4 , 5 , 6 , 10 , 11 , 12 , 13 , 14 , 15 ] . These methods exhibit state-of-the-art performances , with some having been integrated into real-world systems . However , deep-learning-based forecasting is not always reliable or interpretable [ 16 , 17 , 18 ] . In circumstances when noises from the environment are overwhelmingly distracting , or when the situation has never been encountered before , a deep-learning-based algorithm could provide baffling predictions , which might cause terrible tragedies . A fundamental challenge is to know when we could rely on those deep-learning-based forecasting algorithms . To tackle this problem , one solution is to report the uncertainty of each prediction . Finding ways to best conceptualize and measure the prediction uncertainty of deep-learning-based algorithms becomes an imperative , which motivates this work . ∗The corresponding author is Siheng Chen . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . There are two main types of uncertainty to model in deep-learning-based algorithms [ 19 ] : 1 ) aleatoric uncertainty , regarding information aside from statistic models , which data can not explain ; 2 ) epistemic uncertainty , the uncertainty inside a model , when the model lacks knowledge of the system/process being modeled ( e.g. , due to limited training data ) . As it is most effective to model aleatoric uncertainty in big data regimes such as those common to deep learning with image data [ 17 ] , this work focuses on aleatoric uncertainty . In the following passage , we use the term “ uncertainty ” to represent aleatoric uncertainty . [ 16 ] uses the predictive variance to approximate uncertainty in the Bayesian deep learning model , which has been widely adapted in many works [ 20 , 21 , 22 , 11 ] for uncertainty modeling in multi-agent trajectory forecasting . However , the predictive variance of a single agent alone may not suffice to reflect the complete landscape of uncertainty , especially when agent-wise interaction is present . Recent works that attempt to exploit the interaction among agents have impressively boosted the prediction precision , which further highlights the need to better measure uncertainty in multi-agent trajectory forecasting . We seek to build a more sophisticated and robust measurement for capturing the previously neglected uncertainty brought by correlated predictions . In this paper , we coin a concept individual uncertainty ( IU ) to describe the uncertainty that can be approximated by the predictive variance of a single agent . Relatively , we propose a new concept , collaborative uncertainty ( CU ) to estimate the uncertainty resulting from the usage of interaction modules in prediction models . We further introduce an original probabilistic CU-based framework to measure both individual and collaborative uncertainty in the multi-agent trajectory forecasting task . We apply this framework to two special cases : multivariate Gaussian distribution and multivariate Laplace distribution . In each case , our CU-based framework allows our model to simultaneously learn the mappings that are from input data to 1 ) accurate prediction , 2 ) individual uncertainty , and 3 ) collaborative uncertainty ; see Figure 1 for model illustration . Extensive experiments demonstrate that CU modeling yields significantly larger performance gains in prediction models equipped with interaction modules ( See Figure 4 ) , confirming that CU is highly related to the existence of the interaction modeling procedure , and adding CU modeling benefits accurate predictions . The contributions of this work are summarized as follows : ● We propose , analyze , and visualize a novel concept , collaborative uncertainty ( CU ) , to model the uncertainty brought by the interaction modules in multi-agent trajectory forecasting . ● We design a general CU-based framework to empower a prediction model to generate a probabilistic output , where the mean is the future trajectory and the covariance reflects the corresponding uncertainty . Under this framework , we show two special cases based on multivariate Gaussian and Laplace distributions respectively . ● We conduct extensive experiments to validate the CU-empowered prediction model on both synthetic datasets and two large-scale real-world datasets . On self-generated synthetic datasets , we validate the proposed method is able to closely reconstruct the ground-truth distribution . On the public benchmarks , the CU-empowered prediction model consistently outperforms the corresponding one without CU . Specially , by leveraging the proposed CU , VectorNet improves by 57 cm regarding Final Displacement Error ( FDE ) on nuScenes dataset ! 2 Related Works . Aleatoric uncertainty modeling in deep learning . Recent efforts are rising as to improve the measurement of aleatoric uncertainty in deep learning models . One seminal work is [ 17 ] . It proposes a unified Bayesian deep learning framework to explicitly represent aleatoric uncertainty using predictive variance for generic regression and classification tasks . Many existing works [ 23 , 24 , 25 , 26 , 27 , 28 ] follow this idea and formulate uncertainty as learned loss attenuation . For example , to make predictive- variance-based aleatoric uncertainty measurements more efficient , [ 24 ] adds data augmentation during the test-time . But , these works only pay attention to individual uncertainty . Other recent works attend to the uncertainty measurement for correlated predictive distributions . For example , [ 29 ] and [ 30 ] measure spatially correlated uncertainty in a generative model for respectively image reconstruction and pixel-wise classification , and [ 31 ] captures joint uncertainty as discrete variables in the field of reinforcement learning . Despite these three works , our work is the first to conceptualize and measure collaborative uncertainty in the multi-agent trajectory forecasting task . To the best of our knowledge , there are only two papers [ 32 , 33 ] close to our track . [ 32 ] and [ 33 ] model the joint uncertainty in the pose estimation task and multi-agent trajectory forecasting task respectively . However , they present several limitations : 1 ) They only examined the circumstance where the model ’ s output follows Gaussian distribution ; 2 ) They did not provide a theoretical conceptualization or definition for the uncertainty due to correlated predictive distributions , and they did not analyze the causes of such uncertainty . These are essential problems to tackle . In this work , we not only formulate a general framework that works for both Gaussian and Laplace distributions , but we also theoretically conceptualize collaborative uncertainty and analyze its causes . Multi-agent trajectory forecasting . This task takes the observed trajectories from multiple agents as the inputs , and outputs the predicted trajectory for each agent . Like many other sequence prediction tasks , this task used to use a recurrent architecture to process the inputs [ 34 , 35 , 36 ] . Later , however , the graph neural networks become a more common approach as they can significantly assist trajectory forecasting by capturing the interactions among agents [ 3 , 4 , 5 , 6 , 12 , 13 , 14 , 15 ] . For safety reasons , it is necessary to report the uncertainty of each predicted trajectory . Works to date about uncertainty measurement [ 11 , 21 , 22 , 37 , 38 ] have appropriately modeled the interaction among multi-agent trajectories for boosting performances , but they overlook the uncertainty resulting from the correlation in predicted trajectories . We seek to fill this gap by introducing and modeling collaborative uncertainty . 3 Methodology . 3.1 Problem Formulation . Considerm agents in a data sample , and let X= { x1 , x2 , ... , xm } , Y= { y1 , y2 , ... , ym } be the past observed and the future trajectories of all agents , where xi ∈R2T− and yi ∈R2T+ are the past observed and the future trajectories of the i-th agent . Each xi/yi consists of two-dimensional coordinates at different timestamps of T−/T+ . We assume that a training dataset D consists of N individual and identically distributed data samples { ( Xi , Yi ) } Ni=1 . For predicting future trajectories of multiple agents and modeling the uncertainty over the predictions , we seek to use a probabilistic framework to model the predictive distribution p ( Y ∣X ) of multiple agents ’ future trajectories based on the training dataset D. Previous works in uncertainty modeling [ 16 , 17 , 39 ] use Gaussian distribution to approximate p ( Y ∣X ) . The assumption behind this approach is that p ( yi∣xi ) is independent for every i ∈ { 1,2,3 , ... , m } . Mathematically , they set the covariance matrix of p ( Y ∣X ) as a diagonal matrix . This assumption is valid for the regression task that uses the model shown in Figure 2 ( a ) . We refer the uncertainty under the independence assumption as individual uncertainty in this paper . However , Figure 2 ( b ) considers a prediction model that includes interaction modeling among multiple agents : yi is no longer dependent solely on xi , but also on other agents xj where j ≠ i in the scene . We call the uncertainty brought by this interaction collaborative uncertainty . The existence of collaborative uncertainty turns p ( Y ∣X ) from the individual distribution into the joint distribution of multiple agents . Contrary to existing methods , we consider collaborative uncertainty and model p ( Y ∣X ) more accurately by making the covariance matrix a full matrix without imposing any restrictions on its form . In the following subsection , we will introduce an approach to modeling both individual uncertainty and collaborative uncertainty using a unified CU-based framework . 3.2 General Formulation of Collaborative Uncertainty . In this work , to model collaborative uncertainty , we abandon the independence assumption held by previous works [ 16 , 17 , 39 ] , setting p ( Y ∣X ) as a joint multivariate distribution , whose mean is µ ∈ Rm×2T+ and covariance Σ ∈ Rm×m×2T+ . Element µi , t is the expected position of the i-th agent at timestamp t. As the diagonal elements of Σ are considered individual uncertainty [ 16 , 17 , 39 ] , we further let off-diagonal elements describe collaborative uncertainty . Diagonal element Σi , i , t models the variance of the i-th agent at timestamp t ; off-diagonal element Σi , j , t models the covariance between the i-th and j-th agents at timestamp t. Therefore , we can simultaneously obtain individual and collaborative uncertainty by estimating the covariance Σ of p ( Y ∣X ) . Accordingly , we propose a CU-based comprehensive uncertainty estimation framework ( see Figure 3 ) with the following steps : Step 1 : Choose a probability density function , p ( Y ∣X ; µ , Σ , Φ ) , for the predictive distribution , which includes a mean µ∈Rm×2T+ used to approximate the future trajectories , a covariance Σ∈Rm×m×2T+ used to quantify individual uncertainty and collaborative uncertainty , and some auxiliary parameters Φ used to describe the predictive distribution . Further , we set covariance matrix Σt , which represents the covariance matrix at timestamp t , as a full matrix instead of an identity or diagonal matrix . Step 2 : Design a prediction model , F [ µw ( X ) , Σw ( X ) , Φw ( X ) ] , where µw ( X ) , Σw ( X ) and Φw ( X ) are three neural networks , which approximate values of mean µ , covariance Σ and auxiliary parameters Φ respectively . Note that w only indicates the parameters of these neural networks are trainable , and does not mean they share same parameters . Step 3 : Derive a loss function from p ( Y ∣X ; µ , Σ , Φ ) via maximum likelihood estimation : L ( w ) = − N ∑ i=1 log p ( Yi ∣Xi ; µw ( Xi ) , Σw ( Xi ) , Φw ( Xi ) ) minimized to update trainable parameters in µw ( ⋅ ) , Σw ( ⋅ ) and Φw ( ⋅ ) .

Summary: ** Update 8/19 **   Thanks to the authors for providing detailed responses to comments. Based on their clarifications and the other reviews, I have updated my score.   **   The authors propose a collaborative uncertainty (CU) module to model the uncertainty between agents for multi-agent trajectory prediction. It is evidenced that when the CU module is incorporated into two recently published models without environment information, there is an improvement in the predictive accuracy. The authors also provide a visualization for CU, which shows that in situations where there should be no collaborative uncertainty, CU does not produce unintuitive results.