Collaborative Uncertainty in Multi-Agent Trajectory Forecasting
Year: 2022

Abstract:
Uncertainty modeling is critical in trajectory forecasting systems for both interpretation and safety reasons. To better predict the future trajectories of multiple agents, recent works have introduced interaction modules to capture interactions among agents. This approach leads to correlations among the predicted trajectories. However, the uncertainty brought by such correlations is neglected. To fill this gap, we propose a novel concept, collaborative uncertainty (CU), which models the uncertainty resulting from the interaction module. We build a general CU-based framework to make a prediction model learn the future trajectory and the corresponding uncertainty. The CU-based framework is integrated as a plugin module to current state-of-the-art (SOTA) systems and deployed in two special cases based on multivariate Gaussian and Laplace distributions. In each case, we conduct extensive experiments on two synthetic datasets and two public, large-scale benchmarks of trajectory forecasting. The results are promising: 1) The results of synthetic datasets show that CU-based framework allows the model to appropriately approximate the ground-truth distribution. 2) The results of trajectory forecasting benchmarks demonstrate that the CU-based framework steadily helps SOTA systems improve their performances. Specially, the proposed CU-based framework helps VectorNet improve by 57 cm regarding Final Displacement Error on nuScenes dataset. 3) The visualization results of CU illustrate that the value of CU is highly related to the amount of the interactive information among agents.

heading: 1 Introduction
text: A multi-agent trajectory forecasting system aims to predict future trajectories of multiple agents based on their observed trajectories and surroundings [1, 2]. Precise trajectory prediction provides essential information for decision making and safety in numerous intelligent systems, including autonomous vehicles [3, 4, 5, 6], drones [7], and industrial robotics [8, 9].
The rapid development of deep learning has enabled a number of deep-learning-based algorithms to handle multi-agent trajectory forecasting [3, 4, 5, 6, 10, 11, 12, 13, 14, 15]. These methods exhibit state-of-the-art performances, with some having been integrated into real-world systems. However, deep-learning-based forecasting is not always reliable or interpretable [16, 17, 18]. In circumstances when noises from the environment are overwhelmingly distracting, or when the situation has never been encountered before, a deep-learning-based algorithm could provide baffling predictions, which might cause terrible tragedies. A fundamental challenge is to know when we could rely on those deep-learning-based forecasting algorithms. To tackle this problem, one solution is to report the uncertainty of each prediction. Finding ways to best conceptualize and measure the prediction uncertainty of deep-learning-based algorithms becomes an imperative, which motivates this work.
∗The corresponding author is Siheng Chen.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
There are two main types of uncertainty to model in deep-learning-based algorithms [19]: 1) aleatoric uncertainty, regarding information aside from statistic models, which data cannot explain; 2) epistemic uncertainty, the uncertainty inside a model, when the model lacks knowledge of the system/process being modeled (e.g., due to limited training data). As it is most effective to model aleatoric uncertainty in big data regimes such as those common to deep learning with image data [17], this work focuses on aleatoric uncertainty. In the following passage, we use the term “uncertainty” to represent aleatoric uncertainty. [16] uses the predictive variance to approximate uncertainty in the Bayesian deep learning model, which has been widely adapted in many works [20, 21, 22, 11] for uncertainty modeling in multi-agent trajectory forecasting. However, the predictive variance of a single agent alone may not suffice to reflect the complete landscape of uncertainty, especially when agent-wise interaction is present. Recent works that attempt to exploit the interaction among agents have impressively boosted the prediction precision, which further highlights the need to better measure uncertainty in multi-agent trajectory forecasting. We seek to build a more sophisticated and robust measurement for capturing the previously neglected uncertainty brought by correlated predictions.
In this paper, we coin a concept individual uncertainty (IU) to describe the uncertainty that can be approximated by the predictive variance of a single agent. Relatively, we propose a new concept, collaborative uncertainty (CU) to estimate the uncertainty resulting from the usage of interaction modules in prediction models. We further introduce an original probabilistic CU-based framework to measure both individual and collaborative uncertainty in the multi-agent trajectory forecasting task. We apply this framework to two special cases: multivariate Gaussian distribution and multivariate Laplace distribution. In each case, our CU-based framework allows our model to simultaneously learn the mappings that are from input data to 1) accurate prediction, 2) individual uncertainty, and 3) collaborative uncertainty; see Figure 1 for model illustration. Extensive experiments demonstrate that CU modeling yields significantly larger performance gains in prediction models equipped with interaction modules (See Figure 4), confirming that CU is highly related to the existence of the interaction modeling procedure, and adding CU modeling benefits accurate predictions.
The contributions of this work are summarized as follows:
● We propose, analyze, and visualize a novel concept, collaborative uncertainty (CU), to model the uncertainty brought by the interaction modules in multi-agent trajectory forecasting.
● We design a general CU-based framework to empower a prediction model to generate a probabilistic output, where the mean is the future trajectory and the covariance reflects the corresponding uncertainty. Under this framework, we show two special cases based on multivariate Gaussian and Laplace distributions respectively.
● We conduct extensive experiments to validate the CU-empowered prediction model on both synthetic datasets and two large-scale real-world datasets. On self-generated synthetic datasets, we validate the proposed method is able to closely reconstruct the ground-truth distribution. On the public benchmarks, the CU-empowered prediction model consistently outperforms the corresponding one without CU. Specially, by leveraging the proposed CU, VectorNet improves by 57 cm regarding Final Displacement Error (FDE) on nuScenes dataset!

heading: 2 Related Works
text: Aleatoric uncertainty modeling in deep learning. Recent efforts are rising as to improve the measurement of aleatoric uncertainty in deep learning models. One seminal work is [17]. It proposes a unified Bayesian deep learning framework to explicitly represent aleatoric uncertainty using predictive variance for generic regression and classification tasks. Many existing works [23, 24, 25, 26, 27, 28] follow this idea and formulate uncertainty as learned loss attenuation. For example, to make predictive-
variance-based aleatoric uncertainty measurements more efficient, [24] adds data augmentation during the test-time. But, these works only pay attention to individual uncertainty.
Other recent works attend to the uncertainty measurement for correlated predictive distributions. For example, [29] and [30] measure spatially correlated uncertainty in a generative model for respectively image reconstruction and pixel-wise classification, and [31] captures joint uncertainty as discrete variables in the field of reinforcement learning. Despite these three works, our work is the first to conceptualize and measure collaborative uncertainty in the multi-agent trajectory forecasting task. To the best of our knowledge, there are only two papers [32, 33] close to our track. [32] and [33] model the joint uncertainty in the pose estimation task and multi-agent trajectory forecasting task respectively. However, they present several limitations: 1) They only examined the circumstance where the model’s output follows Gaussian distribution; 2) They did not provide a theoretical conceptualization or definition for the uncertainty due to correlated predictive distributions, and they did not analyze the causes of such uncertainty. These are essential problems to tackle. In this work, we not only formulate a general framework that works for both Gaussian and Laplace distributions, but we also theoretically conceptualize collaborative uncertainty and analyze its causes.
Multi-agent trajectory forecasting. This task takes the observed trajectories from multiple agents as the inputs, and outputs the predicted trajectory for each agent. Like many other sequence prediction tasks, this task used to use a recurrent architecture to process the inputs [34, 35, 36]. Later, however, the graph neural networks become a more common approach as they can significantly assist trajectory forecasting by capturing the interactions among agents [3, 4, 5, 6, 12, 13, 14, 15]. For safety reasons, it is necessary to report the uncertainty of each predicted trajectory. Works to date about uncertainty measurement [11, 21, 22, 37, 38] have appropriately modeled the interaction among multi-agent trajectories for boosting performances, but they overlook the uncertainty resulting from the correlation in predicted trajectories. We seek to fill this gap by introducing and modeling collaborative uncertainty.

heading: 3 Methodology

heading: 3.1 Problem Formulation
text: Considerm agents in a data sample, and let X={x1,x2, ...,xm}, Y={y1,y2,...,ym} be the past observed and the future trajectories of all agents, where xi ∈R2T− and yi ∈R2T+ are the past observed and the future trajectories of the i-th agent. Each xi/yi consists of two-dimensional coordinates at different timestamps of T−/T+. We assume that a training dataset D consists of N individual and identically distributed data samples {(Xi,Yi)}Ni=1. For predicting future trajectories of multiple agents and modeling the uncertainty over the predictions, we seek to use a probabilistic framework to model the predictive distribution p(Y ∣X) of multiple agents’ future trajectories based on the training dataset D. Previous works in uncertainty modeling [16, 17, 39] use Gaussian distribution to approximate p(Y ∣X).
The assumption behind this approach is that p(yi∣xi) is independent for every i ∈ {1,2,3, ...,m}. Mathematically, they set the covariance matrix of p(Y ∣X) as a diagonal matrix. This assumption is valid for the regression task that uses the model shown in Figure 2(a). We refer the uncertainty under the independence assumption as individual uncertainty in this paper. However, Figure 2(b) considers a prediction model that includes interaction modeling among multiple agents: yi is no longer dependent solely on xi, but also on other agents xj where j ≠ i in the scene. We call the
uncertainty brought by this interaction collaborative uncertainty. The existence of collaborative uncertainty turns p(Y ∣X) from the individual distribution into the joint distribution of multiple agents. Contrary to existing methods, we consider collaborative uncertainty and model p(Y ∣X) more accurately by making the covariance matrix a full matrix without imposing any restrictions on its form. In the following subsection, we will introduce an approach to modeling both individual uncertainty and collaborative uncertainty using a unified CU-based framework.

heading: 3.2 General Formulation of Collaborative Uncertainty
text: In this work, to model collaborative uncertainty, we abandon the independence assumption held by previous works [16, 17, 39], setting p(Y ∣X) as a joint multivariate distribution, whose mean is µ ∈ Rm×2T+ and covariance Σ ∈ Rm×m×2T+ . Element µi,t is the expected position of the i-th agent at timestamp t. As the diagonal elements of Σ are considered individual uncertainty [16, 17, 39], we further let off-diagonal elements describe collaborative uncertainty. Diagonal element Σi,i,t models the variance of the i-th agent at timestamp t; off-diagonal element Σi,j,t models the covariance between the i-th and j-th agents at timestamp t. Therefore, we can simultaneously obtain individual and collaborative uncertainty by estimating the covariance Σ of p(Y ∣X). Accordingly, we propose a CU-based comprehensive uncertainty estimation framework (see Figure 3) with the following steps: Step 1: Choose a probability density function, p(Y ∣X;µ,Σ,Φ), for the predictive distribution, which includes a mean µ∈Rm×2T+ used to approximate the future trajectories, a covariance Σ∈Rm×m×2T+ used to quantify individual uncertainty and collaborative uncertainty, and some auxiliary parameters Φ used to describe the predictive distribution. Further, we set covariance matrix Σt, which represents the covariance matrix at timestamp t, as a full matrix instead of an identity or diagonal matrix. Step 2: Design a prediction model, F[µw(X),Σw(X),Φw(X)], where µw(X), Σw(X) and Φw(X) are three neural networks, which approximate values of mean µ, covariance Σ and auxiliary parameters Φ respectively. Note that w only indicates the parameters of these neural networks are trainable, and does not mean they share same parameters. Step 3: Derive a loss function from p(Y ∣X;µ,Σ,Φ) via maximum likelihood estimation: L(w) = − N
∑ i=1 log p(Yi ∣Xi;µw(Xi),Σw(Xi),Φw(Xi)) minimized to update trainable parameters in µw(⋅), Σw(⋅) and Φw(⋅).

heading: 3.3 Two Special Cases
text: In multi-agent trajectory forecasting, based on Laplace and Gaussian distributions, the `1- and `2- based loss functions are commonly adopted to train prediction models [3, 4, 15, 40]. Here we apply the probabilistic framework proposed in Section 3.2 to model the individual and collaborative uncertainty based on multivariate Gaussian distribution and multivariate Laplace distribution respectively, which leads to two novel loss functions. Mathematically, the essential difference between our proposed loss functions and previous loss functions derived from Gaussian distribution and Laplace distribution for modeling individual uncertainty is that they have different assumptions about the covariance matrix; see a summary in Table 1. We regard the covariance as a full matrix.

heading: 3.3.1 Gaussian collaborative uncertainty
text: We start by the multivariate Gaussian distribution, as it has a simpler probability density function than the multivariate Laplace distribution.
Probability density function. We follow the framework proposed in Section 3.2 and choose the probability density function as the multivariate Gaussian distribution:
p(Y ∣X;µ,Σ,Φ) = (2π)−m2 ⋅ det[Σ]− 12 ⋅ e− 12 (Y−µ)Σ −1(Y−µ)T , (1)
where det[Σ] represents the determinant of covariance Σ. Model design. Based on (1), we can approximate the value of mean µ via a neural network µw(⋅). When using the same way to approximate the value of covariance Σ, however, we face two challenges: 1) each covariance matrix Σt in covariance Σ needs to be inverted, which could lead to numerical instability; 2) it is computationally expensive and numerically unstable to compute the determinant of each covariance matrix Σt in covariance Σ directly given a large amount of trainable parameters. For the first challenge, we use a neural network Σ−1w (⋅) to directly approximate the inverse of covariance Σ. For the second challenge, similar to [29] and [32], we apply the square-root-free Cholesky decomposition to each Σ−1tw in Σ −1 w (X): Σ−1w (X) = Lw(X)Dw(X)LTw(X), where Lw(X) is a lower unit triangular matrix and Dw(X) is a diagonal matrix. Then, the determinant of the inverse of covariance Σ−1 is obtained by m
∏ j=1 djj , where djj is the j-th diagonal element in Dw(X).
We can thus get the parameterized form of (1) as: p(Y∣X;w) = (2π)−m2 ( m
∏ j=1 djj)
1 2e− qw(Y,X) 2 , where
qw(Y,X)=[Y−µw(X)]Σ−1w (X)[Y−µw(X)]T . As there are no auxiliary parameters in the parameterized form of (1), we can get the prediction model F[µw(X),Σ−1w (X)], whose framework is illustrated in Figure 3. Once Σ−1w (X) is fixed and given, individual and collaborative uncertainty are computed through the inversion.
Loss function. According to the square-root-free Cholesky decomposition and the parameterized form of (1), the Gaussian collaborative uncertainty loss function is then:
LGau−cu(w) = 1
2
1
N
N ∑ i=1 [qw(Yi,Xi) − m ∑ j=1 log(dijj)]. (2)
We update the trainable parameters in µw(⋅) and Σ−1w (⋅) through minimizing (2). Note that qw(⋅, ⋅) is related to µw(⋅) and Σ−1w (⋅), and dijj is related to Σ−1w (⋅).

heading: 3.3.2 Laplace collaborative uncertainty
text: In multi-agent trajectory forecasting, previous methods [3, 4, 15] have found that the `1-based loss function derived from Laplace distribution usually leads to better prediction performances than the `2-based loss function from Gaussian distribution, because the former is more robust to outliers. It is thus important to consider multivariate Laplace distribution.
Probability density function. We follow the framework proposed in Section 3.2 and choose the probability density function as the multivariate Laplace distribution:
p(Y ∣X;µ,Σ,Φ) = 2det[Σ] − 12 (2π)m2 λ ⋅ K(m2 −1)(
√ 2 λ (Y−µ)Σ−1(Y−µ)T )
( √
λ 2 (Y−µ)Σ−1(Y−µ)T )m2
, (3)
where det[Σ] denotes the determinant of covariance Σ, and K(m2 −1)(⋅) denotes the modified Bessel function of the second kind with order (m
2 − 1).
Model design. Similar to Section 3.3.1, we employ two neural networks µw(⋅) and Σ−1w (⋅) to approximate the values of µ and Σ−1 respectively, and represent Σ−1w (X) via its square-root-free Cholesky decomposition we used in the Gaussian collaborative uncertainty. Since the modified Bessel function is intractable for a neural network to work with, different from Section 3.3.1, we should simplify (3).
Inspired by [41], we simplify (3) by utilizing the multivariate Gaussian distribution to approximate the multivariate Laplace Distribution. We reformulate a multivariate Laplace distribution by introducing auxiliary variables. Let z ∈ R+ be a random variable with the probability density function: p(z∣X;w) = 1
λ e− z λ , then we can get: p(Y ∣z,X;w) = det[Σ −1 w(X)] 1 2 (2πz)m2 e −qw(Y,X)2z , where
qw(Y,X) = [Y−µw(X)]Σ−1w (X)[Y−µw(X)]T . Further, if the value of z is given, p(Y ∣z,X;w) is a multivariate Gaussian distribution. In this work, instead of drawing a value for z from the exponential distribution, we use a neural network Φw(⋅) to directly output a value for z. The intuition is that, in the training process of the prediction model, the value of p(Y ∣X;w) is the conditional expectation of z given X and Y, which makes p(Y ∣z,X;w) a function of z whose domain is R+. Thus, there should exist an appropriate z∗ ∈ R+ to make: p(Y ∣X;w) = p(Y ∣z∗,X;w) (see proof in the appendix). To find such a z∗, we use Φw(X), which can employ its learning ability. Then, we can get the parameterized form of p(Y ∣X;w) as: p(Y ∣X;w) = det[Σ −1 w (X)] 1 2
(2πΦw(X)) m 2 e− qw(Y,X) 2Φw(X) .
Finally, we can get the prediction model F[µw(X),Σ−1w (X),Φw(X)], whose framework is illustrated in Figure 3. Individual and collaborative uncertainty are indirectly learned by the Σ−1w (X). Loss function. On the basis of the square-root-free Cholesky decomposition and the parameterized form of p(Y ∣X;w), the Laplace collaborative uncertainty loss function is then:
LLap−cu(w) = 1
2
1
N
N ∑ i=1 [qw(Y i,Xi) Φw(Xi) +m log Φw(Xi) − m ∑ j=1 log(dijj)]. (4)
where dijj is the j-th diagonal element in Dw(Xi). The parameters of µw(⋅), Σ−1w (⋅) and Φw(⋅) are updated by minimizing (4). And qw(⋅, ⋅) is related to µw(⋅) and Σ−1w (⋅), and dijj belongs to Σ−1w (⋅).

heading: 3.4 Discussion
text: After presenting how to quantify collaborative uncertainty in Section 3.2 and Section 3.3, here we discuss the nature of collaborative uncertainty.
As mentioned in Section 3.2, we can divide a prediction model for multi-agent trajectory forecasting into two types: individual models and collaborative models. An individual model predicts the future trajectory and the corresponding uncertainty for each agent independently; a collaborative model leverages an interaction module to explicitly capture the interactions among multiple agents, which makes all the predicted trajectories correlated. Moreover, this interaction modeling procedure can bring extra uncertainty to the model; in other words, we consider that the interaction modeling in a prediction model leads to collaborative uncertainty (CU).
To validate this, we can empirically compare how much impact of collaborative uncertainty modeling would bring to an individual model and a collaborative model. In our experiment, we use two cutting-edge multi-agent trajectory forecasting models, LaneGCN [3] and VectorNet [4].2 In
2To focus on agents’ behaviors, we remove the map information and the related modules in this experiment.
LaneGCN/VectorNet, an agent-to-agent (A2A) module/a global-interaction-graph (GIG) module explicitly models the interactions among multiple agents. As illustrated in Figure 4, when we remove the A2A/GIG module from LaneGCN/VectorNet, collaborative uncertainty modeling brings much less gain to LaneGCN/VectorNet; see Section 4.2.3 for more details of this experiment. This reflects that the cause of collaborative uncertainty mainly comes from the interaction modeling.

heading: 4 Experiments
text: We first use two self-generated synthetic datasets with a limited number of agents as the toy version of the real world problem. We use the simplified datasets to test our method’s ability of capturing distribution information of the input data that obeys a certain type of multivariate distribution. We then conduct extensive experiments on two published benchmarks to prove the value of our proposed method in solving real world problems. We introduce the experiments in Sec. 4.1 and Sec. 4.2.

heading: 4.1 Toy Problem
text: We define a toy problem to validate the capability of the proposed framework for accurate probability distribution estimation. The toy problem requires models to take the mutually correlated trajectories sampled from a given distribution as the input and output the mean and the covariance of the distribution.
As far as we know, in real-world datasets, we only have the ground truth for the predicted trajectory, which is the mean of the distribution while have no access to the ground truth of the uncertainty, which is the covariance matrix of the distribution. Therefore, we generate two synthetic datasets with the ground truth for both mean and covariance matrix given two distributions respectively.
Datasets. We generate two synthetic datasets for ternary Gaussian and ternary Laplace distribution respectively. Each dataset contains training, validation and test sets, which have 36000, 7000 and 7000 instances respectively. Each instance includes the trajectories of three agents, which consist of the two-dimensional point coordinates of the three agents at 50 different timestamp. In each instance, the trajectories of three agents are sampled from a ternary joint Gaussian/Laplace distribution. Generation details are provided in the appendix.
Implementation details. The network architecture used in the experiment contains an encoder and two decoders, all of which are four-layer MLPs. The neural network outputs the predicted mean and covariance matrix of the given ternary Gaussian/Laplace distribution. Although the ground truth covariance matrix is known for synthetic datasets, they are not used in training. We train the network using the previous uncertainty modeling method and our proposed method on each synthetic dataset.
Metric. Here, we adopt three metrics for evaluation: the `2 distances between the estimated mean and the ground truth mean, the `1 distances between the estimated covariance matrix and the ground truth covariance matrix and the KL divergence between the ground truth distribution and the estimated distribution. We provide metrics computing details in the appendix.
Evaluation results. The test set results of the synthetic dataset are shown in Table 2 and Figure 5. Since the previous uncertainty modeling method only models IU (individual uncertainty) but our method models both IU and CU (collborative uncertainty), our method allows the model to estimate a more accurate mean and covariance matrix on a given distribution, which leads to a much lower KL divergence between the ground truth distribution and the estimated distribution.

heading: 4.2 Real World Problem
text: The experiments results above show the superiority of our method for simplified problems. Here we further validate its ability on two real-world large-scale autonomous driving datasets in single future prediction. We also conduct ablation studies in this subsection to present the effectiveness of the method design.

heading: 4.2.1 Experiment setup
text: Datasets. Argoverse [42] and nuScenes [47] are two widely used multi-agent trajectory forecasting benchmarks. Argoverse has over 30K scenarios collected in Pittsburgh and Miami. Each scenario is a sequence of frames sampled at 10 Hz. The sequences are split as training, validation and test sets, which have 205942, 39472 and 78143 sequences respectively. nuScenes collects 1000 scenes in Boston and Singapore. Each scene is annotated at 2 Hz and is 20s long. The prediction instances are split as training, validation and test sets, which have 32186, 8560 and 9041 instances respectively.
Table 2: Comparison with the prior uncertainty modeling method on synthetic datasets under the two different assumptions of distribution. µw(X): the estimated mean. µgt: the ground truth mean. Σw(X): the estimated covariance matrix. Σgt: the ground truth covariance matrix. KL: the KL divergence DKL(pg(X)∣∣pe(X)), where pe(X) is the estimated distribution and pg(X) is the ground truth distribution.
GAUSSIAN LAPLACE IU ONLY IU + CU IU ONLY IU + CU
`2 OF µ 0.68 0.49 0.42 0.34 `1 OF Σ 1.98 1.01 1.96 1.13
KL 6.68 0.40 12.6 1.65
Table 3: Comparison with SOTA methods on Argoverse test set. CU boosts performances in single future prediction.
METHOD ADE FDE
ARGO BASELINE (NN) [42] 3.45 7.88 ARGO BASELINE [42] 2.96 6.81 UULM-MRM [43, 44] 1.90 4.19 VECTORNET [4] 1.81 4.01 TNT [45] 1.78 3.91 JEAN [46] 1.74 4.24 LANEGCN [3] 1.71 3.78 OURS: LANEGCN (OUR IMPLEMENTATION) 1.76 3.84 USING LGau−cu 1.73 3.83 USING LLap−cu 1.70 3.74
(a) Gaussian Synthetic Dataset
(b) Laplace Synthetic Dataset
Figure 5: Sample visualization on synthetic dataset. Our proposed CU-based framework allows the model to learn the mean and covariance matrix of ground truth distribution accurately. Each input instance X is the numerical sum of a ground truth mean µgt and a random variable with the information of the ground truth covariance matrix Σgt. Please find more details in Appendix.
For Argoverse, we forecast future trajectories for 3s based on the observed trajectories of 2s. For nuScenes, we forecast future trajectories for 6s based on the observed trajectories of 2s. Metrics. We adopt two extensively used multi-agent trajectory forecasting metrics, Average Displacement Error (ADE) and Final Displacement Error (FDE). ADE is defined as the average of pointwise `2 distances between the predicted trajectory and ground truth. FDE is defined as the `2 distance between the final points of the prediction and ground truth (ADE and FDE are measured in meters). Implementation details. The proposed model is implemented follow the basis of LaneGCN [3] and VectorNet [4], two cutting-edge multi-agent trajectory forecasting models. We implement the encoder of LaneGCN/VectorNet as the trajectory encoder in our proposed model. We further use the four-layer multilayer perceptrons (MLPs) to respectively implement three decoders µw(⋅), Σw(⋅) and Φw(⋅). Note that all experiments are based on our own implementation of LaneGCN/VectorNet, which may not perform as well as the official LaneGCN/VectorNet.

heading: 4.2.2 Results
text: Evaluation results on benchmark datasets. In this subsection, we implement our proposed framework based on LaneGCN [3] as it is the SOTA model in multi-agent trajectory forecasting. We compare our proposed methods in the Argoverse trajectory forecasting benchmark with two official baselines, including Argoverse Baseline [42] and Argoverse Baseline (NN) [42], and five SOTA methods of this benchmark: LaneGCN [3], TNT [45], Jean [46], VectorNet [4] and uulm-mrm [43, 44]. Table 3 shows that although our implementation of LaneGCN appears less good than the official implementation of LaneGCN, our implementation of LaneGCN with the individual and collaborative uncertainty notably outperforms all of the other competing methods in both ADE and FDE. Therefore, our proposed collaborative uncertainty modeling enhances the SOTA prediction models.
Visualization of collaborative uncertainty.3 To visually understand which factor might influence the value of collaborative uncertainty in multi-agent trajectory forecasting, we present the visualization results generated by our model in Figure 6. We visualize 3 scenarios, and each scenario includes 3 agents (orange, blue and green lines) trajectories (solid lines are the past trajectories and dashed lines are the future trajectories) and their corresponding collaborative uncertainty values changing over the last 30 frames (the heatmap, where σij denotes the collaborative uncertainty of agent i and agent j). These results show that the value of collaborative uncertainty is highly related to the amount of the interactive information among agents.

heading: 4.2.3 Ablation study
text: We study: 1) how different approaches of uncertainty modeling would affect the prediction model; 2) how the interaction module would influence the effectiveness of collaborative uncertainty modeling. In this part, we adopt LaneGCN/VectorNet as our trajectory encoder for proving that our proposed method can be used as a plug-in module to improve the performances of existing models in multi-agent trajectory forecasting. To focus on the agents’ behaviors, we ignore map information and map-related modules in LaneGCN/VectorNet; we only use the agent encoder of LaneGCN/VectorNet, which extracts intermediate feature from agents’ past trajectories and actor-to-actor (A2A) module/global-interaction-graph (GIG) module that exploits the interactions among agents. In this part, the experiments are conducted on the validate sets of Argoverse and nuScenes benchmarks.
Different approaches of uncertainty modeling. We consider two approaches of uncertainty modeling: assuming the covariance matrix is a diagonal matrix (DIA) to model individual uncertainty only, and assuming the covariance matrix is a full matrix (FULL) to model both individual and collaborative uncertainty. Based on the results in Table 4, we see that 1) modeling individual and collaborative uncertainty together (FULL) is consistently better than modeling individual uncertainty only (DIA); and 2) our proposed Laplace CU-based framework enables LaneGCN and VectorNet to achieve the best performances on ADE & FDE metrics on both Argoverse and nuScenes benchmarks. These results reflect that our proposed collaborative uncertainty modeling can work as a plugin module to significantly improve the prediction performance of existing models.
Effects of interaction module. A2A/GIG module is the interaction module in LaneGCN/VectorNet. We study the effects of the interaction module in collaborative uncertainty modeling via the ablation study on A2A/GIG module. ∆ADE/FDE is the difference of ADE/FDE values between versions with and without collaborative uncertainty. Higher values reflect bigger gains brought by modeling collaborative uncertainty. From Table 4, gains from modeling collaborative uncertainty in a collaborative model (with A2A/GIG) are much greater than in an individual model (without A2A/GIG). Figure 4 visualizes the ∆FDE on nuScenes benchmark; see discussions in Section 3.4.

heading: 5 Conclusions
text: This work proposes a novel probabilistic collaborative uncertainty (CU)-based framework for multiagent trajectory forecasting. Its key novelty is conceptualizing and modeling CU introduced by interaction modeling. The experimental results in this work demonstrate the ability of the proposed CU-based framework to boost the performance of SOTA prediction systems, especially collaborativemodel-based systems. The proposed framework potentially lead to more reliable self-driving systems.
This work shows the promise of our method for single-future trajectory prediction, which predicts the single best trajectory for agents. With growing research interests in predicting multiple potential trajectories for agents (i.e. multi-future trajectory prediction), we leave the generalization of our method to multi-future trajectory prediction as an important direction for future research.

heading: Acknowledgement
text: This work was supported in part by the National Key R&D Program of China under Grant 2021YFB1714800, the National Natural Science Foundation of China under Grant 6217010074, 62173034, 62088101, the Science and Technology Commission of Shanghai Municipal under Grant 21511100900, the DiDi GAIA Research Collaboration Plan, and the CAAI-Huawei MINDSPORE Academic Award Fund.
3These visualization results are based on Gaussian CU. Please see more visualization results in the appendix.

Review summaries:
** Update 8/19 **   Thanks to the authors for providing detailed responses to comments. Based on their clarifications and the other reviews, I have updated my score.   **   The authors propose a collaborative uncertainty (CU) module to model the uncertainty between agents for multi-agent trajectory prediction. It is evidenced that when the CU module is incorporated into two recently published models without environment information, there is an improvement in the predictive accuracy. The authors also provide a visualization for CU, which shows that in situations where there should be no collaborative uncertainty, CU does not produce unintuitive results. 

The work introduces the idea of estimating joint uncertainties between all actors in a motion forecasting setting, instead of the established way of doing so by actor individually. The concept is simple enough to be applied to a variety of different existing motion forecasting systems and the authors demonstrate that the additional loss leads to better prediction accuracy after training.  Overall I liked this paper a lot. It may be a simple idea but it is powerful, as the authors demonstrate in their results. The paper is also very clearly written and the experiments are sufficient.

The paper presents a novel concept — collaborative uncertainty — in the context of multi-agent trajectory forecasting. Unlike many previous works that only model the individual uncertainty of joint future trajectory distribution and use a diagonal covariance matrix, this paper considers both individual and collaborative uncertainty of multi-agent trajectories and uses a full covariance matrix with off-diagonal components representing collaborative uncertainty between agents. The paper argues that such a design could model the GT multi-agent trajectory distribution more accurately. The paper performs synthetic experiments showcasing the method's ability to model joint trajectory distribution better than models that only consider individual uncertainty. Experiments on real-world trajectory forecasting benchmarks also demonstrate that incorporating collaborative uncertainty improves the performance of deterministic baselines.