Collaborative Uncertainty in Multi-Agent Trajectory Forecasting
Authors: Bohan Tang
Yiqi Zhong
Ulrich Neumann
Gang Wang
Siheng Chen
Ya Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multi-agent Trajectory Forecasting
Uncertainty Analysis
Interaction Modeling
Deep Learning

Abstract: Uncertainty modeling is critical in trajectory-forecasting systems for both interpretation and safety reasons. To better predict the future trajectories of multiple agents, recent works have introduced interaction modules to capture interactions among agents. This approach leads to correlations among the predicted trajectories. However, the uncertainty brought by such correlations is neglected. To fill this gap, we propose a novel concept, collaborative uncertainty (CU), which models the uncertainty resulting from the interaction module. We build a general CU-based framework to make a prediction model learn the future trajectory and the corresponding uncertainty. The CU-based framework is integrated as a plugin module to current state-of-the-art (SOTA) systems and deployed in two special cases based on multivariate Gaussian and Laplace distributions. In each case, we conduct extensive experiments on two synthetic datasets and two public, large-scale benchmarks of trajectory forecasting. The results are promising: 1) The results of synthetic datasets show that CU-based framework allows the model to nicely rebuild the ground-truth distribution. 2) The results of trajectory forecasting benchmarks demonstrate that the CU-based framework steadily helps SOTA systems improve their performances. Specially, the proposed CU-based framework helps VectorNet improve by 57 cm regarding Final Displacement Error on nuScenes dataset. 3) The visualization results of CU illustrate that the value of CU is highly related to the amount of the interactive information among agents.

Collaborative Uncertainty in Multi-Agent Trajectory Forecasting
Bohan Tang'
Yiqi Zhong
UIrich Neumann"
Gang Wang" Ya Zhang'
Siheng Chen'
"Shanghai Jiao Tong University
University of Southern California BBeijing Institute of Technology (sihenge, ya_zhang) @sju.. edu. cn 2 Lyiqizhon, meeuann]]Oueee edu "gangsang@bit. edu. cn
*tangbohan@al umni sjtu. edu. . cn
Abstract
Uncertainty modeling is critical in trajectory forecasting systems for both inter- pretation and safety reasons. To better predict the future trajectories of multiple agents, recent works have introduced interaction modules to capture interactions among agents. This approach leads to correlations among the predicted trajec- tories. However, the uncertainty brought by such correlations is neglected. To fill this gap, propose novel concept, collaborative uncertainty (CU), which models the uncertainty resulting from the interaction module. We build a gen- eral CU-based framework to make a prediction model learn the future trajectory and the corresponding uncertainty. The CU-based framework is integrated as a plugin module current state-of sfate-of-the-ar (SOTA) systems and deployed in two special cases based on multivariate Gaussian and Laplace distributions. In each case, we conduct extensive experiments on two synthetic datasets and two public, large-scale benchmarks of trajectory forecasting, The results are promising: 1) The results of synthetic datasets show that CU-based framework allows the model appropriately approximate the ground-truth distribution. 2) The results of trajectory forecasting benchmarks demonstrate that the CU-based framework steadily helps SOTA systems improve their performances. Specially, the proposed CU-based framework helps VectorNet improve by 57 cm regarding Final Displacement Error nuScenes dataset. 3) The visualization results of CU illustrate that the value of CU is highly related to the amount of the interactive information among agents.
Introduction
multi-agent trajectory forecasting system aims predict future trajectories of multiple agents based on their observed trajectories and surroundings 21. Precise trajectory prediction provides essential information for decision making and safety numerous intelligent systems, including autonomous
vehicles
3 4 5 6), drones (2]. and industrial robotics g
The rapid development of deep learning has enabled number of Seeep-learning-bee algorithms to handle multi-agent trajectory forecasting 3
4
5
6
10
12
13 [4] [5). These methods exhibit
state-of-the-art performances, with some having been integrated into real-world systems. However, deep-learning-basce based forecasting is not always reliable or interpretable [16] 17 [8]. In circumstances when noises from the environment are overwhelmingly distracting. when the situation has never been encountered before, a deep-lcarning-bor based algorithm could provide baifling predictions, which might cause terrible tragedies. A fundamental challenge to know when we could rely on those deep-learning based forecasting algorithms. To tackle this problem, one solution is to report the uncertainty of each prediction. Finding ways to best conceptualize and measure the prediction uncertainty of deep-learning- based algorithms becomes an imperative, which motivates this work.
'The corresponding author is Siheng Chen.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

()) typical encoder in mult mutr-agent trajectory forecasting systems
(8) Previous decoder: Y
[c) Ours:Y +dur Collborative Unceralney. dy
h;
bu
Deoeds
hn
Pudd
Decod
Xg
Ouipe
brann
bg
Desada
19.-99)
Figure 1: Uncertainty modeling in multi-agent trajectory forecasting, (a) a typical pipeline of an encoder in multi-agent trajectory forecasting systems. (b) and (c) illustrate the decoder pipeline of previous methods and our method respectively. Previous methods output the predicted trajectory Y and individual uncertainty d; and our method additionally outputs collaborative uncertainty Jij.
There are two main types of uncertainty to model in seep-learning-baie -aaedd algorithms 19:
1) aleatoric uncertainty, regarding information aside from statistic models, which data cannot explain; 2) epistemic uncertainty, the uncertainty inside a model, when the model lacks knowledge of the system/procees being modeled (e.g., due to limited training data). As it most effective model aleatoric uncertainty in big data regimes such as those common to deep learning with image data [1.] this work focuses on aleatoric uncertainty. In the following passage, we use the term "uncertainty" represent aleatoric uncertainty. [16] uses the predictive variance to approximate uncertainty in the Bayesian deep learning model, which has been widely adapted in many works [20] 21 [2 [ for uncertainty modeling in multi-agent trajectory forecasting. However, the predictive variance of a single agent alone may suffice to reflect the complete landscape of uncertainty, especially when agent-wise interaction is present. Recent works that attempt to exploit the interaction among agents have impressively boosted the prediction precision, which further highlights the need better measure uncertainty in multi-agent trajectory forecasting. We seek to build a more sophisticated and robust measurement for capturing the previously neglected uncertainty brought by correlated predictions.
In this paper, we coin a concept individual uncertainty (IU) to describe the uncertainty that can be approximated by the predictive variance of single agent. Relatively, we propose a new concept, collaborative uncertainty (CU) to estimate the uncertainty resulting from the usage of interaction modules in prediction models. We further introduce an original probabilistic CU-based framework to measure both individual and collaborative uncertainty in the multi-agent trajectory forecasting task. We apply this framework two special cases: multivariate Gaussian distribution and multivariate Laplace distribution. In each case, our CU-based framework allows our model to simultaneously learn the mappings that are from input data to 1) accurate prediction, 2) individual uncertainty, and 3) collaborative uncertainty; see Figure for model illustration. Extensive experiments demonstrate that CU modeling yields significantly larger performance gains in prediction models equipped with interaction modules (See Figure 4. confirming that CU is highly related to the existence of the interaction modeling procedure, and adding CU modeling benefits accurate predictions.
The contributions of this work are summarized as follows:
We propose, analyze, and visualize novel concept, collaborative uncertainty (CU), model the uncertainty brought by the interaction modules in multi-agent trajectory forecasting.
We design general CU-based framework to empower prediction model to generate proba- bilistic output, where the mean is the future trajectory and the covariance reflects the corresponding uncertainty. Under this framework, we show two special cases based on multivariate Gaussian and Laplace distributions respectively.
We conduct extensive experiments to validate the CU-empowered prediction model on both synthetic datasets and two large-scale real-world datasets. On self-generated synthetic datasets, we validate the proposed method is able to closely reconstruct the ground-truth distribution. On the public benchmarks, the CU-empowered prediction model consistently outperforms the corresponding one without CU. Specially, by leveraging the proposed CU, VectorNet improves by 57 cm regarding Final Displacement Error (FDE) on nuScenes dataset!
2 Related Works
Aleatoric uncertainty modeling in deep learning. Recent efforts are rising as to improve the
measurement of aleatoric uncertainty in deep learning models. One seminal work is [7]. proposes a
unified Bayesian deep learning framework to explicitly represent aleatoric uncertainty using predictive
variance for generic regression and classification tasks. Many existing works 23 25 26 27 28
follow this idea and formulate uncertainty learned loss attenuation. For example, to make predictive-

---

variance- aleatoric uncertainty measurements more efficient, [24] adds data augmentation during the test-time. But, these works only pay attention to individual uncertainty.
Other recent works attend to the uncertainty measurement for correlated predictive distributions. For example, (29 and [30] measure spatially correlated uncertainty in generative model for respectively image reconstruction and pixel-wise classification, and [31] captures joint uncertainty as discrete variables in the field of reinforcement learning. Despite these three works, our work is the first to conceptualize and measure collaborative uncertainty in the multi-agent trajectory forecasting task, To the best of our knowledge, there are only two papers [32 331 close to our track. [32 and [33]
model the joint uncertainty in the pose estimation task and multi-agent trajectory forecasting task respectively. However, they present several limitations: 1) They only examined the circumstance where the model's output follows Gaussian distribution; 2) They did not provide theoretical conceptualization or definition for the uncertainty due to correlated predictive distributions, and they did not analyze the causes of such uncertainty. These are essential problems to tackle. In this work, we not only formulate general framework that works for both Gaussian and Laplace distributions, but we also theoretically conceptualize collaborative uncertainty and analyze its causes.
Multi-agent trajectory forecasting. This task takes the observed trajectories from multiple agents as the inputs, and outputs the predicted trajectory for each agent. Like many other sequence prediction
tasks, this task used to use recurrent architecture to process the inputs [34 35 36 Later, however, the graph neural networks become a more common approach as they can sigmificantly assist trajectory forecasting by capturing the interactions among agents reasons, it is necessary to report the uncertainty of each predicted trajectory. Works to date about 37 38 have appropriately modeled the interaction among multi-agent trajectories for boosting performances, but they overlook the uncertainty resulting from the correlation in predicted trajectories. We seek to fill this gap by introducing and modeling collaborative uncertainty.
6
[5]. For safety
uncertainty measurement [[] 21 [2
3 Methodology
3.1
Problem Formulation
Consider m agents in data sample, and let X= Xm)) Y (= Yy...2.... be the past observed and the future trajectories of all agents, where x, 2T and y, 2T. are the past observed and the future trajectories of the i-th agent. Each x./y; consists of two-dimensional coordinates at different timestamps of T./T. We assume that a training dataset D consists of N individual and identically distributed data samples {(', Y For predicting future trajectories of multiple agents and modeling the uncertainty over the predictions, we seek to use probabilistic framework to model the predictive distribution p(Y X) of multiple agents' future trajectories based on the training dataset D. use Gaussian distribution to approximate p(Y X).
Previous works in uncertainty modeling [6 17 39
X1
h
Y1
X1
X2
h2
Y2
X2
h
y2
X3
hg
Yz
X3
hg
Y3
(a) Individual Model
(b) Collaborative Model
Figure 2: Graphical model for deep learning networks in the three-agent trajectory forecasting setting: (a) represents the model that predicts the trajectory of each agent independently; (b) shows the model that explicitly captures the interaction among multiple agents. x; is the observed trajectory of the i-th agent; h, and y, are its corresponding hidden feature and future trajectory respectively.
The assumption behind this approach is that ((.) is independent for every i 21,2,3..... m). Mathematically, they set the covariance matrix of p(Y (X) as a diagonal matrix. This assumption valid for the regression task that uses the model shown in Figure 2(a) We refer the uncertainty under the independence assumption as individual uncertainly in this paper. However, Figure 2(b) considers prediction model that includes interaction modeling among multiple agents: y; is no longer dependent solely on x,, but also other agents x, where j # in the scene. We call the
3

---

uncertainty brought by this interaction collaborative uncertainty. The existence of collaborative uncertainty turns p( X) from the individual distribution into the joint distribution of multiple agents.
Contrary to existing methods, we consider collaborative uncertainty and model p(Y more accurately by making the covariance matrix full matrix without imposing any restrictions its form. In the following subsection, we will introduce an approach to modeling both individual uncertainty and collaborative uncertainty using unified CU-based framework.
Prediction Model
Decoder
Pix (X)
V1
Y2
(X)
Trajectory
Encoder
Decoder
(x)
(X)
Loss
ys
Decoder
(x)
Figure 3: Proposed uncertainty estimation framework. The encoder may contain a module that exploits agent-wise interaction. Decoders output the mean ww (X). covariance Ew (X) containing individual and collaborative uncertainty, and auxiliary parameters w (X). The outputs formulate the training loss with the ground truth Y. (-) only used in Laplace collaborative uncertainty.
3.2 General Formulation of Collaborative Uncertainty
In this work, to model collaborative uncertainty, we abandon the independence assumption held by previous works [16 17 (3]. setting X) as joint multivariate distribution, whose mean is e Rmx22,, and covariance E IR R=xmxx24 Element fi,t is the expected position of the i-th agent at timestamp t. As the diagonal elements of are considered individual uncertainty [6 39 we further let off-diagonal elements describe collaborative uncertainty. Diagonal element E,,., models the variance of the i-th agent timestamp off-diagonal element Σi,,t models the covariance between the i-th and j-th agents at timestamp f. Therefore, we can simultaneously obtain individual and collaborative uncertainty by estimating the covariance Σ p CU-based comprehensive uncertainty estimation framework (see Figure with the following steps:
X). Accordingly, we propose a
Step 1: Choose a probability density function, p(Y p;, pu, Σ, 4). for the predictive distribution, which includes a mean ju RToT.. used to approximate the future trajectories, covariance Σ RRmom22T.. used to quantify individual uncertainty and collaborative uncertainty, and some auxiliary parameters used to describe the predictive distribution. Further, we set covariance matrix Σ,, which represents the covariance matrix at timestamp
full matrix instead of identity diagonal matrix.
Step 2: Design prediction model, F[" Ew (X), &w (X)], where (X), Ew (X) and Pw (X) are three neural networks, which approximate values of mean J4, covariance Σ and auxiliary parameters t respectively. Note that only indicates the parameters of these neural networks trainable, and does not mean they share same parameters.
Step 3: Derive loss function from p( Y Xx,,,,, Σ) via maximum likelihood estimation: L(w)
logp(Y' XX: nw (X'), Enw (X'), T (X')) minimized to update trainable parameters in Pw
E. (.) and (.).
3.3
Two Special Cases
Table Two special cases with various assumptions about covariance Σ. DIA: the diagonal matrix (individual uncertainty only). FULL: the full matrix (both individual and collaborative uncertainty).
Loss FUNCTION OF TWO SPECIAL CASES GAUSSIAn DISTRIBUTION
ASSUMPTION
LAPLACE DISTRIBUTION
DIA:
022
1 (x)
L0 (x.,la +logo
011
012
0lsn
FULL:
021
022
HY, X)- T--Eten Elog(ds)) Jul
log (x) -

---

In multi-agent trajectory forecasting, based on Laplace and Gaussian distributions, the é1- and é2- based loss functions are commonly adopted train prediction models B 4 40 Here we apply the probabilistic framework proposed in Section 3.2 to model the individual and collaborative uncertainty based on multivariate Gaussian distribution and multivariate Laplace distribution respectively, which leads to two novel loss functions. Mathematically, the essential difference between our proposed loss functions and previous loss functions derived from Gaussian distribution and Laplace distribution for modeling individual uncertainty is that they have different assumptions about the covariance matrix; see a summary Table
We regard the covariance as full matrix.
3.3.1 Gaussian collaborative uncertainty
We start by the multivariate Gaussian distribution, it has simpler probability density function than the multivariate Laplace distribution.
Probability density function. We follow the framework proposed in Section 3.2 and choose the probability density function as the multivariate Gaussian distribution:
X ui, E, +) (2+)- det[[E] 2 3 '(Y=#)²
where [ represents the determinant of covariance Σ.
Model design. Based on [ we can approximate the value of mean μ via neural network Hou (-). When using the same way to approximate the value of covariance Σ, however, we face two challenges: 1) each covariance matrix E, covariance Σ needs to be inverted, which could lead to numerical instability; 2) it is computationally expensive and numerically unstable to compute the determinant of each covariance matrix E, in covariance Σ directly given a large amount of trainable parameters.
For the first challenge, we use a neural network (-) to directly approximate the inverse of covariance Σ. For the second challenge, similar to 22 and (32). we apply the square-root-free Cholesky decomposition to each Se in [x((): D (X) = La (x) (X) (X), where Lo (X) is lower unit triangular matrix and (X) diagonal matrix. Then, the determinant of the
inverse of covariance E-1 is obtained by Π where djj is the j-th diagonal element in Dwv
j-1
We can thus get the parameterized form of (1) as: p(Y[Xuu) = (2T) - (II F=1
djj) be
xx0
where
Tw (Y (Y,))=(YYxua (X)EE (X]
As there are no auxiliary parameters in the parameterized form of we can get the prediction
model [[(mo( Enj (X)], whose framework is illustrated in Figure Once E (X) is fixed and given, individual and collaborative uncertainty are computed through the inversion.
Loss function. According to the square-root-free Cholesky decomposition and the parameterized form of T the Gaussian collaborative uncertainty loss function is then:
1 1 N LGau-eu (w) = N 1=1
m
Σ
Lgg (Y',,'))))
Σ log (d
(2)
We update the trainable parameters in (.) and Eggl (.) through minimizing 2 Note that goo (-.-) is related to n (-) and (.). (-), and d' is related to En (.).
3.3.2 Laplace collaborative uncertainty
In multi-agent trajectory forecasting, previous methods 15 have found that the 4 based loss function derived from Laplace distribution usually Icads to better prediction performances than the ba-based loss function from Gaussian distribution, because the former is more robust to outliers. It is thus important to consider multivariate Laplace distribution.
Probability density function. We follow the framework proposed in Section 3.2 and choose the probability density function as the multivariate Laplace distribution:
2det E K(T- / 1W)
p Y X: X,
(3)
(2T) 5
( (Y-))=-(YY))))
where dct[-] denotes the determinant of covariance Σ, and KG ( denotes the modified Bessel function of the second kind with order (m - l).
5

---

Model design. Similar to Section 3.3.1 we employ two neural networks (w and En (-) to approximate the values μ and Σ respectively, and represent "n'(( via its square-root-fie Cholesky decomposition we used in the Gaussian collaborative uncertainty. Since the modified Bessel function is intractable for neural network to work with, different from Section
3.3.T
we should
simplify (3).
Inspired by [41]. we simplity (3) by utilizing the multivariate Gaussian distribution approxi- mate the multivariate Laplace Distribution. We reformulate a multivariate Laplace distribution by introducing auxiliary variables. Let z E IR be random variable with the probability den-
sity function: p(z) w) = te= &, then we can get: p( (; (X0)
2w(Y..)
where Further, if the value of z is given, p(Y Iz, X; qw (Y) -M (X) -M]
20
(2mm)
is multivariate Gaussian distribution. In this work, instead of drawing a value for z from the exponential distribution, we a neural network (-) to directly output value for z. The intuition that, in the training process of the prediction model, the value of p( ( X w) is the conditional expectation of z given X and Y, which makes p(Y |z, x;w) function of whose domain is R* Thus, there should exist an appropriate z" R to make: p(Y Y (X; p Izz ;X; w) (see proof in the appendix). To find such a zz, we use P (X), which can employ its learning ability. Then,
can get the parameterized form of p(Y X; w) as: p( Y X; w) des[z,," (X)1 (w
20w(X)
(X))
Finally, we can get the prediction model Flu (X), 2 (. whose framework is illus- trated in Figure 3 Individual and collaborative uncertainty are indirectly learned by the E (X).
Loss function. On the basis of the square-root-free Cholesky decomposition and the parameterized form of p(Y w), the Laplace collaborative uncertainty loss function is then:
1
Jw (Y', m log P (X') (x'(
m
Lap-een w)
Σ
log log(dd;))) (d,
(4)
2N
where d', is the j-th diagonal element Dw (X'). The parameters of (.) End (.) and (.) are updated by minimizing (4).
And (-:-) is related to pio (.) and Ew (.), and di belongs to End (-).
3.4 Discussion
After presenting how to quantify collaborative uncertainty in Section 3.2 and Section 3.3 here we discuss the nature of collaborative uncertainty.
As mentioned in Section 3.2 we can divide a prediction model for multi-agent trajectory forecasting into two types: individual models and collaborative models. An individual model predicts the future trajectory and the correspouding uncertainty for each agent independently; a collaborative model leverages interaction module explicitly capture the interactions among multiple agents, which makes all the predicted trajectories correlated. Moreover, this interaction modeling procedure can bring extra uncertainty to the model; in other words, we consider that the interaction modeling in prediction model leads to collaborative uncertainty (CU).
0.24 = With A2A Interaction WWithout A2A Interactionn
0.6
With GIG Interaction
WWithuut GIG Interaction
0.18
0.45
AFDE
AFDE
0.3
x15
X19
0.15
X5
x14
Gaussian
Laplace
Gousslan
Laplace
Figure CU modeling gains more when a prediction model includes interaction modeling. Blue and red bars reflect the gains from CU modeling for a prediction model w/o an interaction module respectively. AFDE the difference of FDE values between versions w/o CU modeling.
To validate this, we can empirically compare how much impact of collaborative uncertainty mod- eling would bring to an individual model and a collaborative model. In our experiment, we use two cutting-edes multi-agent trajectory forecasting models, LaneGCN 3 and VectorNet 4 In
"To focus agents' behaviors, we remove the map information and the related modules in this experiment.

---

LaneGCN/VectorNet, an agent-to-agen agent (A2A) module/a global-interann interaction- graph (GIG) module explicitly models the interactions among multiple agents. As illustrated in Figure when we remove the A2A/GIG module from LaneGCN/VectorNet collaborative uncertainty modeling brings much less gain to LaneGCN/VectorNett see Section (42.3 for more details of this experiment. This reflects that the cause of collaborative uncertainty mainly comes from the interaction modeling.
4 Experiments
We first use two self-generated synthetic datasets with limited number of agents as the toy version of the real world problem. We use the simplified datasets to test our method's s ability of capturing distribution information of the input data that obeys certain type of multivariate distribution. We then conduct extensive experiments on two published benchmarks to prove the value of our proposed method in solving real world problems. We introduce the experiments in Sec. 4.1 and Sec. 4.2
4.1
Toy Problem
We define toy problem to validate the capability of the proposed framework for accurate probability distribution estimation. The toy problem requires models take the mutually correlated trajectories sampled from a given distribution as the input and output the mean and the covariance of the distribution.
As far as we know, in real-world datasets, we only have the ground truth for the predicted trajectory, which is the mean of the distribution while have no access to the ground truth of the uncertainty, which is the covariance matrix of the distribution. Therefore, we generate two synthetic datasets with the ground truth for both mean and covariance matrix given two distributions respectively.
Datasets. We generate two synthetic datasets for ternary Gaussian and ternary Laplace distribution respectively. Each dataset contains training, validation and test sets, which have 36000, 7000 and 7000 instances respectively. Each instance includes the trajectories of three agents, which consist of the two-dimensional point coordinates of the three agents at 50 different timestamp. In each instance, the trajectories of three agents are sampled from ternary joint Gaussian/Laplaae distribution. Generation details are provided in the appendix.
Implementation details. The network architecture used in the experiment contains an encoder and two decoders, all of which are four-layer MIPs. 'he neural network outputs the predicted mean and covariance matrix of the given ternary Gaussian/L Laplace distribution. Although the ground truth covariance matrix is known for synthetic datasets, they are not used in training. We train the network using the previous uncertainty modeling method and our proposed method on each synthetic dataset.
Metric. Here, we adopt three metrics for evaluation: the &2 distances between the estimated mean and the ground truth mean, the & distances between the estimated covariance matrix and the ground truth covariance matrix and the KL divergence between the ground truth distribution and the estimated distribution. We provide metrics computing details in the appendix.
Evaluation results. The test set results of the synthetic dataset are shown in Table 2 and Figure 5 Since the previous uncertainty modeling method only models IU (individual uncertainty) but method models both IU and CU (collborative uncertainty), our method allows the model to estimate more accurate mean and covariance matrix on a given distribution, which leads a much lower KL divergence between the ground truth distribution and the estimated distribution.
4.2
Real World Problem
The experiments results above show the superiority of our method for simplified problems. Here we further validate its ability on two real-world large-scale autonomous driving datasets in single future prediction. We also conduct ablation studies in this subsection to present the effectiveness of the method design.
4.2.1 Experiment setup
Datasets. Argoverse 42 and nuScenes 47 are two widely used multi-agent trajectory forecasting benchmarks. Argoverse has over 30K scenarios collected in Pittsburgh and Miami. Each scenario a sequence of frames sampled at 10 Hz. The sequences are split as training, validation and test sets, which have 205942, 39472 and 78143 sequences respectively. nuScenes collects 1000 scenes in Boston and Singapore. Each scene annotated at Hz and is 20s long. The prediction instances are split as training, validation and test sets, which have 32186, 8560 and 9041 instances respectively.
7

---

Table 2: Comparison with the prior uncertainty modeling method on synthetic datasets under the two different assumptions of distribution. Mo (X); the estimated mean. Pyti the ground truth mean. (X): the estimated covariance matrix. Eye: the ground truth covariance matrix. KL: the KL divergence D (X)ll.()))) where Pe(X) is the estimated distribution and Py (X) is
Table 3: Comparison with SOTA methods on Argoverse test CU boosts performances in single future prediction.
METHOD
ADE
FDE
ARGO BASELINE (NN)
3.45
7.88
ARGO BASELINE 42
2.96
6.81
UULM-MRM
1.90
4.19
VECTORNET
1.81
4.01
the ground truth distribution.
TNT 45
1.78
3.91
JEAN 46
1.74
4.24
GAUSSIAN
LAPLACR
LANEGCN
1.71
3.78
ONLY
ONLY
OURS:
21 OF
0.68
0.49
0.42
0.34
LANEGCN (OUR IMPLEMENTATION!
1.76
3.84
4 OF Σ
1.98
1.01
1.96
1.13
USING Lan-ccu
1.73
3.83
KL
6.68
0.40
12.6
1.65
USING LLap-cu
1.70
3.74
Apend
Aprit
15
Gaussian Synthetic Dataset
(b) Laplace Synthetic Dataset
Figure 5: Sample visualization synthetic dataset. Our proposed CU-based framework allows the model to learn the mean and covariance matrix of ground truth distribution accurately. Each input instance the numerical sum of ground truth mean Pgt and random variable with the information of the ground truth covariance matrix Egt. Please find more details in Appendix.
For Argoverse, we forecast future trajectories for based on the observed trajectories of 2s. For nuScenes, we forecast future trajectories for 6s based on the observed trajectories of 2s.
Metrics. We adopt two extensively used multi-agent trajectory forecasting metrics, Average Displace- ment Error (ADE) and Final Displacement Error (FDE). ADE defined as the average of pointwise l2 distances between the predicted trajectory and ground truth. FDE is defined as the & distance between the final points of the prediction and ground truth (ADE and FDE measured in meters).
Implementation details. The proposed model is implemented follow the basis of LaneGCN [ and VectorNet [4) two cutting-cdge multi-agent trajectory forecasting models. We implement the encoder of LaneGCN/VectonNet the trajectory encoder in our proposed model. We further the four-layer multilayer perceptrons (MLPs) to respectively implement three decoders wod (.), En (.) implementation of LaneGCN/VectorNett.
and & (t). Note that all experiments based which may not perform as well as the official LaneGCN/VectorNet.

---

Table 4: Ablation covariance Σ with chosen probability density function (PDF) and Interaction module (INT.). DIA denotes the diagonal matrix (individual uncertainty). FULL denotes the full matrix (individual and collaborative uncertainty). On Argoverse and nuScenes, both LaneGCN and VectorNet with individual and collaborative uncertainty surpasses the ones with individual uncertainty only. Collaborative uncertainty makes a larger impact for the model with interaction module.
ADE
FDE
METHOD
DATASET
PDF TYPE
INT.
AADE
AFDE
DIA
FULL
DIA
FULL
1.69
1.67
0.02
3.88
3.85
0.03
GAUSSIAN
1.45
1.42
0.03
3.19
3.14
0.05
BRGVVERSB
I.67
1.67
0.00
3.82
3.82
0.00
LAPLACE
1.43
1.41
0.02
3.16
3.11
0.05
LANEGCN
4.60
4.58
0.02
11.02
11.01
0.01
V
4.47
4.39
0.08
10.59
10.44
0.15
NUSCENES
4.53
4.52
0.01
10.93
10.92
0.01
LAPLACE
4.34
4.25
0.09
10.34
10.15
0.19
1.82
1.78
0.04
4.16
4.08
0.08
GAUSSIAN
J
1.63
1.57
0.06
3.60
3.46
0.14
1.78
1.76
0.02
4.06
4.02
0.04
LAPLACE
V
1.56
1.52
0.04
3.42
3.34
0.08
VECTORNET
4.25
4.23
0.02
10.35
10.29
0.06
GAUSSIAN
V
4.07
3.99
0.08
9.86
9.57
0.29
NUSEES
4.19
4.18
0.01
10.25
10.21
0.04
4.02
3.81
0.21
9.79
9.22
0.57
4.2.2 Results
Evaluation results on benchmark datasets. In this subsection, we implement our proposed frame- work based on LaneGCN 3 as it is the SOTA model in multi-agent trajectory forecasting. We compare proposed methods in the Argoverse trajectory forecasting benchmark with two official baselines, including Argoverse Baseline [42 and Argoverse Baseline (NN) (42), and five SOTA meth- ods of this benchmark: LaneGCN 3]. TNT 45 Jean [46]. VectorNet 4 and uulm-mmn mrm 43 44) Table 3 shows that although our implementation implementation of LaneGCN, implementation of LaneGCN with the individual and collaborative uncertainty notably outperforms all of the other competing methods in both ADE and FDE. Therefore, proposed collaborative uncertainty modeling enhances the SOTA prediction models.
LaneGCN appears less good than the official
Time
Time
Time
<12
Ro
fgeet Apert
Agee0
Apwr?
(a) Scenario
(b) Scenario II
(c) Scenario III
Figure 6: Visualization of CU on Argoverse dataset. (a) In scenario I, AgentO and Agentl are driving the same road, which is next to the road where Agent2 is driving. This type of scenario might generate complicated interactive information making dol, and 0 show non-monotonic change over time. (b) In scenario II, AgentO and Agentl moving towards each other, thus increases over time. Agent2 is parking at an intersection waiting for green light, as little new interactive information between Agent2 and the other
agents would be generated before the red light turns green, Z02 and 012 decrease over time. (c) In scenario III. AgentO, Agent and Agent2 are located in completely different areas heading to different directions. dol dog and 012 are close to zero.

---

Visualization of collaborative uncertainty To visually understand which factor might influence the value of collaborative uncertainty multi- trajectory forecasting, we present the visualization results generated by our model in Figure 6 We visualize 3 scenarios, and each scenario includes 3 agents (orange, blue and green lines) trajectories (solid lines are the past trajectories and dashed lines are the future trajectories) and their corresponding collaborative uncertainty values changing over the last 30 frames (the heatmap, where ozj denotes the collaborative uncertainty of agent i and agent j). These results show that the value of collaborative uncertainty is highly related to the amount of the interactive information among agents.
4.2.3 Ablation study
We study: 1) how different approaches of uncertainty modeling would affect the prediction model; 2) how the interaction module would influence the effectiveness of collaborative uncertainty mod- cling. In this part, we adopt LaneGCN/VectorNee as our trajectory encoder for proving that our proposed method can be used as plug-in module to improve the performances of cxisting models in multi-agent trajectory forecasting. To focus on the agents' behaviors, we ignore map information and map-related modules in LaneGCNNVectorNett we only use the agent encoder of LaneGCN/Vec- torNet, which extracts intermediate feature from agents' past trajectories and actor-to-acton (A2A) module/global-inee iataaaaliniatttonnngaia -ggaph (GIG) module that exploits the interactions among agents. In this part, the experiments are conducted on the validate sets of Argoverse and nuScenes benchmarks.
Different approaches of uncertainty modeling. We consider two approaches of uncertainty mod- eling: assuming the covariance matrix a diagonal matrix (DIA) to model individual uncertainty only, and assuming the covariance matrix is a full matrix (FULL) to model both individual and collaborative uncertainty. Based on the results in Table 4 we see that 1) modeling individual and collaborative uncertainty together (FULL) is consistently better than modeling individual uncertainty only (DIA); and 2) our proposed Laplace CU-based framework enables LaneGCN and VectorNet to ADE & FDE metrics both Argoverse and nuScenes benchmarks. These results reflect that our proposed collaborative uncertainty modeling can work as a plugin module to significantly improve the prediction performance of existing models.
achieve the best performances
Effects of interaction module. A2A/GIG module is the interaction module in LaneGCN/VectorNer We study the effects of the interaction module in collaborative uncertainty modeling via the ablation study on A2A/GIG module. AADE/FDE is the diffcrence of ADE/FDE values between versions with and without collaborative uncertainty. Higher values reflect bigger gains brought by modeling collaborative uncertainty. From Table 4, gains from modeling collaborative uncertainty in a collab- orative model (with A2A/GIG) are much greater than in individual model (without A2A/GIG). Figure visualizes the AFDE on nuScenes benchmark; see discussions in Section 3.4
Conclusions
This work proposes a novel probabilistic collaborative uncertainty (CU)-based framework for multi- agent trajectory forecasting. Its key novelty is conceptualizing and modeling CU introduced by interaction modeling. The experimental results in this work demonstrate the ability of the proposed CU-based framework boost the performance of SOTA prediction systems, especially collaborative- model-based systems. The proposed framework potentially lead to more reliable self-driving systems.
This work shows the promise of our method for single-future trajectory prediction, which predicts the single best trajectory for agents. With growing research interests in predicting multiple potential trajectories for agents (i.e. multi-futur trajectory prediction), we leave the generalization of our method to multi- future trajectory prediction as an important direction for future research.
Acknowledgement
This work was supported in part by the National Key R&D Program of China under Grant 202IYFB1714800, the National Natural Science Foundation of China under Grant 6217010074, 62173034, 62088101 101, the Science and Technology Commission of Shanghai Municipal under Grant 21511100900, the DiDi GAIA Research Collaboration Plan, and the CAAl-Huawei MINDSPORE Academic Award Fund.
'These visualization results based Gaussian CU, Please see more visualization results in the appendix.
10

---

References
[1] Patrick Stahl, Birsen Donmez, and Greg A. Jamieson. Anticipation in driving: The role of experience in the efficacy of pre-event conflict cues. IEEE Trans. Hum. Mach. Syst., 44(5):603 613, 2014.
[2] Patrick Stahl, Birsen Donmez, and Greg A Jamieson. Supporting anticipation in driving through attentional and interpretational in-vehicle displays. Accident Analysis and Prevention, 91 2016.
[3] Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lanc graph representations for motion forecasting. In ECCV, 2020.
[4] Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguclov, Congeong Li, and Cordelia Schmid. Vectomet: Encoding hd maps and agent dynamies from vectorized representation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[5] Maosheng Ye, Tongyi Cao, and Qifeng Chen. Tpen: Temporal point cloud networks for motion forecasting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11318- 11187- June 2021.
[6] Thomas Gilles, Stefano Sabatini, Dumitry Tsishkou, Bogdan Stanciulescu, and Fabien Moutarde. Home: Heatmap output for future motion estimation, 2021.
[7] Xiao, Jianyu Zhao, Yunhua He, and Shui Yu. Trajectory prediction UAV smart city using recurrent neural networks. In 2019 IEEE International Conference on Communications, ICC 2019, Shanghai, China, May 20-24, 2019, pages 1-6, IEEE, 2019.
[8] Nikolay Jetchev and Mare Toussaint. Trajectory prediction: learning to map situations to robot trajectories. In Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L. Littman, editors, Proceedings of the 26th Annual International Conference on Machine Learning. ICML 2009, Montreal, Quebec. Canada, June 14-18, 2009, volume 382 of ACM International Conference Proceeding Series, pages 449-456. ACM, 2009.
[9] Christoph Rösmann, Malte Oeljeklaus, Frank Hoffmann, and Torsten Bertram. Online trajectory prediction and planning for social robot navigation. In IEEE International Conference on Advanced Intelligent Mechatronics, AIM 2017, Munich, Germany, July 3-7, 2017, pages 1255-1260. IEEE, 2017.
[10] Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, and Ying Nian Wu. Multi-agent tensor fusion for contextual trajectory prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), June 2019.
[11] Chiho Choi and Behzad Dariush. Looking to relations for future trajectory forecast. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
[12] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone, Trajectron++: Dynamically- feasible trajectory forecasting with heterogeneous data, arXiv preprint 20Xiv:2011.33093 2020.
[13] Wenyuan Zeng, Ming Liang, Renjie Liao, and Raquel Urtasun. Lanerena: Distributed representations for graph-centric motion forecasting, arXiv preprint arXiv:2 101606630,6,,, 2021.
[14] Jiachen Li, Fan Yang. Masayoshi Tomizuka, and Chiho Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. Proceedings of the Neural Information Processing Systems (NeurIPS), 2020.
[15] Vincet Kosaraju, Amir Sadeghian, Roberto Martín Martin,, lan D. Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 841,, 2019, Vancouver, BC, Canada, pages 137-146, 2019.
[16] Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University of Cambridge, 2016.
[17] Alex Kendall and Yarin Gal. What uncertainties we need bayesian deep learning for computer vision? In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 30, pages 5574-5584. Curran Associates. Inc., 2017.
[18] Xujiang Zhao, Feng Chen, Shu Hu, and Jin- Hee Cho. Uncertainty aware semi- cemi-supervised learning on graph data, 2020.

---

[19] Armen Der Kiureghian and Ove Ditlevsen. Aleatory epistemic? does it matter? Structural safety. 31(2):105-12,, 2009.
[20] Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Long-term on- board prediction of people in traffic scenes under uncertainty. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 4194 4202. IEEE Computer Society, 2018.
[21] Ajay Jain, Sergio Casas, Renjie Liao, Yuwen Xiong, Song Feng, Sean Segal, and Raquel Urtasun. Discrete residual flow for probabilistic pedestrian behavior prediction. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30- November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 407-419. PMLR, 2019.
[22] Joey Hong. Benjamin Sapp, and James Philbin. Rules of the road: Predicting driving behavior with convolutional model of semantic interactions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 1454-8462. Computer Vision Foundation JEEE, 2019.
[23] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In 2018 IEEE Conference Computer Vision and Pottern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7482 7491. IEEE Computer Society, 2018.
[24] Murat Seckin Ayhan and Philipp Berens. Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks. 2018.
[25] Stefan Depeweg, Jose-Miguel Hernaddz-LLohe Lohato, Finale Doshi-Veler and Steffen Udluft. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In Jeanifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1184-1193, Stockholmsmissann, Stockholm Sweden, 10-15 Jul 2018 PMLR.
[26] Di Feng, Lars Rosenbaum, and Klaus Dietmayer. Towards safe autonomous driving: Capture uncertainty in the deep neural network for lidar 3d vehicle detection. In 2018 2/ International Conference on Intelligent Transportation Systems (ITSC), pages 3266-3273. IEEE, 2018
[27] Guotai Wang, Wenqi Li, Michael Aertsen, Jan Deprest, Sébastien Ourselin, and Tom Vercauteren. Aleatoric uncertainty estimation with tist-time augmentation for medical image segmentation with convolutional neural networks. Neurocomputing. 338:34 45, 2019.
[28] Lingkai Kong, Jimeng Sun, and Chao Zhang. SDE-net: Equipping deep neural networks with uncertainty estimates. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5405-5415. PMLR, 13-18 Jul 2020.
[29] G. Dorta, S. Vicente, L. Agapito, N. D. F. Campbell, and I. Simpson. Structured uncertainty prediction networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5477-5485 2018.
[30] Miguel Monteiro, Loic Le Folgoc, Daniel Caelho de Castro, Nick Pawlowski, Bernardo Marques, Kon- stantinos Kamnitsas, Mark van der Wilk, and Ben Glocker. Stochastic segmentation networks: Modelling spatially correlated alcatoric uncertainty. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tier Lin. editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[31] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models, 2021.
[32] Nitesh B. Gundavarapu, Divyansh Srivastava, Rahul Mitra, Abhishek Sharma, and Arjun Jain. Structured aleatoric uncertainty in human pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019, pages 50-53. Computer Vision Foundation IEEE, 2019.
[33] Roger Girgis, Florian Golemo, Felipe Codevilla, Jim Aldon D'Souza, Martin Weiss, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Autobots: Latent variable sequential transformers, 2021.
12

---

[34] Alexandre Alahi, Kratarth Goel. Vignesh Ramanathan, Alexandre Robicquet, Fei-Fei Li, and Silvio Savarese. Social LSTM: human trajectory prediction in crowded spaces. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 961-971 IEEE Computer Society, 2016.
[35] Irtiza Hasan, Francesco Setti, Theodore Tsesmelis. Alessio Del Bue, Fabio Galasso, and Marco Cristani. MX-LSTM, mixing tracklets and vislets to jointly forecast trajectories and head poses. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 6067-6076. IEEE Computer Society, 2018.
[36] Junwei Liang, Lu Jiang, Juan Carlos Niebles, Alexander G. Hauptmann, and Li Fei-Fei. Peeking into the future: Predicting future person activities and locations in videos. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 5725-5734. Computer Vision Foundation IEFF, 2019.
[37] Namhoon Lec, Wongun Choi, Paul Vernaa,, Christopher B. Choy, Philip H. S. Torr, and Manmohan Chandraker. DESIRE: distant future prediction dynamic scenes with interacting agents. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 2 21-26, 2017, pages 2165-2174. IEEE Computer Society, 2017.
[38] Panna Felsen, Patrick Lucey, and Sujoy Ganguly. Where will they go? predicting fine-grained adversarial multi-agent motion using conditional variational autoencoders. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision ECCV 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XI, volume 11215 of Lecture Notes Computer Science, pages 761-776. Springer, 2018
[39] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q Weinberger, editors, Proceedingy of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050-1059, New York, New York, USA, 20 22 Jun 2016. PMLR.
[40] Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It not the journey but the destination: Endpoint conditioned trajectory prediction. 2020.
[41] T. Eltoft, Taesu Kim, and Te- Won Lec. On the multivariate laplace distribution. IEEE Signal Processing Letters, 13(5):000- 300,303,, 2006.
[42] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Luccy, Deva Ramanan, and James Hays. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[43] Henggang Cui, Vladan Radosavljevic, Fang-Chich Chou, Tsung-Han Lin, Thi Nguyen, Tzu-Kuo Huang, Jeff Schneider, and Nemanja Djurie. Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019. pages 2090-2099 IEEE, 2019.
[44] Nemanja Djuric, Vladan Radosavljevic, Henggang Cui, Thi Nguyen, Fang-Chieh Chou, Tsung- Han Lin, and Jeff Schneider. Motion prediction of traffic actors for autonomous driving using deep convolutional networks. CoRR, abs/1808 1088.08819 2018
[45] Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun. Benjamin Sapp. Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, Congcong Li, and Dragomir Anguelov. TNT: target-driven trajectory prediction. CoRR, abs/2008.08299 2020.
[46] Jean Mercat, Thomas Gilles, Nicole EI Zoghby, Guillaume Sandou, Dominique Beauvois, and Guillermo Pita Gil. Multi-head attention for multi-modal joint vehicle motion forecasting. In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 August 31, 2020, pages 0638-9644. IEEE, 2020.
[47] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:I903.111 19027 2019.
13


Reviews:
[{"review_id": "gk4tLtlM0-", "reviewer": "Reviewer_9D3B", "summary": "** Update 8/19 ** \n\nThanks to the authors for providing detailed responses to comments. Based on their clarifications and the other reviews, I have updated my score. \n\n** \n\nThe authors propose a collaborative uncertainty (CU) module to model the uncertainty between agents for multi-agent trajectory prediction.\nIt is evidenced that when the CU module is incorporated into two recently published models without environment information, there is an improvement in the predictive accuracy.\nThe authors also provide a visualization for CU, which shows that in situations where there should be no collaborative uncertainty, CU does not produce unintuitive results.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The references are appropriate, and the exposition has good clarity.\nPlease also cite these works:\n\tYe, Maosheng, Tongyi Cao, and Qifeng Chen. \"TPCN: Temporal Point Cloud Networks for Motion Forecasting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\tGilles, Thomas, et al. \"HOME: Heatmap Output for future Motion Estimation.\" arXiv preprint arXiv:2105.10968 (2021).\n\nQuantitative Experiments:\nThe quantitative experiments include comparisons to the recent works and leverage well-studied evaluation metrics.\nHowever, I have several concerns.\n1.\tThe authors do not report the number of predicted trajectories used for computing ADE and FDE in any of their results tables. I believe it is K=1.\n2.\tIt is very common for the compared models to rely on the min ADE/FDE across 6 predicted trajectories per test case instead of just 1. This manuscript only reports the results for K=1. As such, it is unclear whether the improvement from the CU module is observed in these critical metrics (min ADE and min FDE for K=6).\n3.\tTable 3 reports the ADE/FDE for Jean with K=1 from the public Argoverse leaderboard, which conflicts with the Jean results reported in the LaneGCN and VectorNet papers.\n4.\tThe results reported for TNT conflict with the public Argoverse leaderboard.\n5.\tIt is unclear whether the CU module would improve the performance of the current SOTA: HOME, LaneRCNN, and TPCN, which all outperform LaneGCN according to K=1. These should be included in Table 3 and the application of CU should be discussed w.r.t. these works as well.\n\nThe toy problem shows simple cases in which the proposed collaborative uncertainty module performs well.\nSince the random variable epsilon is effectively noise and independent between synthetic agents, it appears that the synthetic agent trajectories are completely non-interacting (e.g., there are no collisions). Yet, according to Figure 5, there is collaborative uncertainty. Shouldn't this result match Pairs 5-8 in Figure 6?\nIt is also unclear how this toy problem relates to the trajectories of real interacting agents.\n\nQualitative results are not shown for scenarios with more than 3 interacting agents in the naive toy problem or more than 2 interacting agents in the Argoverse dataset.\nIt is therefore difficult to evaluate the estimated CU for important situations with many interacting agents.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nFor improvement suggestions, please refer to the full review.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "LY9v2amD1D5", "reviewer": "Reviewer_q6Rk", "summary": "The work introduces the idea of estimating joint uncertainties between all actors in a motion forecasting setting, instead of the established way of doing so by actor individually. The concept is simple enough to be applied to a variety of different existing motion forecasting systems and the authors demonstrate that the additional loss leads to better prediction accuracy after training.\n\nOverall I liked this paper a lot. It may be a simple idea but it is powerful, as the authors demonstrate in their results. The paper is also very clearly written and the experiments are sufficient.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** \n\nEstimating single-agent uncertainty is quite common but joint uncertainty or \"collaborative uncertainty\" is less well-studied. In model-based RL, works like [1] try to capture joint uncertainty as discrete variables but this does not address the uncertainty over the motion of individual actors. The only work that comes to mind, that has attempted something like this in motion forecasting is the recent [2]. In this work, the model captures both a per-actor uncertainty at every step and the collaborative uncertainty, which is expressed as discrete modes that can condition the future rollouts. \nI still think this work is sufficiently novel since it's a more general (theoretical) framework but it may be good to briefly mention this recent work in the related works section.\n\n**Quality:** \n\nThe submission is technically very sound. Well done!\n\n**Clarity:** \n\nThe submission is clearly written and well-organized. I liked the little toy example before the main application to Argoverse and nuScenes. I wasn't able to follow all the math in section 3.3.1 but I hope another reviewer can help point out if this makes sense or not. The rest of the paper is clear.\nTogether with the supplementary video, Fig.6 is great but on its own, it's harder to parse. Maybe a link in the caption of the figure to the video would be great for the final version of the paper.\nI also liked the detailed depictions of the networks and explanation of training details in the appendix, making this easy to reproduce.\n\n**Significance:** \n\nI think this method is straightforward and easy to adapt into a multitude of motion forecasting systems, leading to better predictive results and more explainability.\n\n**Nitpicks and Questions:** \n\n1. Starting in the abstract, you write multiple times that something approximates something \"nicely\" and I think that's a bit too colloquial for a paper\n2. line 15: rebuild -> approximate?\n3. line 1: no dash between trajectory and forecasting\n4. contribution 2 in the abstract mentions improving predictive performance but up until then, only safety+explainability was mentioned\n5. Last sentence of abstract sounds like \"our predicted output is highly related to the input\"\n6. Fig. 1 is great but Fig. 2 is kind of redundant wrt Fig. 1. Maybe it'd be possible to redesign Fig. 1 to be a bit bigger and bolder like Fig. 2 and then cut Fig. 2?\n7. What's the complexity to compute the loss in eq 4 over the variant with only the diagonal covariance matrix?\n8. To read+review, I printed the paper in black/white and Fig. 5a is utterly unreadable that way. But even looking at the color image in the PDF, I can hardly make out what's happening there.\n9. Section 4.2.1 is great and more papers should do a quick recap like this.\n10. How do you explain the vastly different $\\Delta ADE$s across the same model and the same dataset in Tab. 4?\n11. Why do you think the Laplacian model leads so consistently to better ADE results in Tab.4?\n12. line 285 \"less good\" -> \"worse\"\n13. line 301 \"plug\" -> \"plugin\"? \"plug & play\"?\n\n**References:**\n\n- [1] Hafner, Danijar, et al. \"Mastering atari with discrete world models.\" arXiv preprint arXiv:2010.02193 (2020).\n- [2] Girgis, Roger, et al. \"Latent Variable Nested Set Transformers & AutoBots.\" arXiv preprint arXiv:2104.00563 (2021).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think you missed the part of the authors' instructions that said to discuss limitations and societal impact of this method. I have to slightly lower my overall score of the paper to be fair to other submissions that discussed these in detail.\n\nFor societal impact, I'd mention that this doesn't have any impact beyond other trajectory forecasting models and that the big possible positive consequence could be having more reliable self-driving cars.\n\nFor limitations, I'd mention for example that it might not be practical to calculate the predicted CU between all actors in a scene in crowded scenes. Specifically, in a dense urban environment with many pedestrians, this might be tricky and maybe some heuristic might be required here. Also, I'd like to see more of the safety aspect implemented, i.e. the abstract mentioned how this was great for making systems more explainable but then this wasn't really executed. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "KtRVqkkx6eq", "reviewer": "Reviewer_vRWU", "summary": "The paper presents a novel concept — collaborative uncertainty — in the context of multi-agent trajectory forecasting. Unlike many previous works that only model the individual uncertainty of joint future trajectory distribution and use a diagonal covariance matrix, this paper considers both individual and collaborative uncertainty of multi-agent trajectories and uses a full covariance matrix with off-diagonal components representing collaborative uncertainty between agents. The paper argues that such a design could model the GT multi-agent trajectory distribution more accurately. The paper performs synthetic experiments showcasing the method's ability to model joint trajectory distribution better than models that only consider individual uncertainty. Experiments on real-world trajectory forecasting benchmarks also demonstrate that incorporating collaborative uncertainty improves the performance of deterministic baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength:**\n\n- The proposed concept of collaborative uncertainty is novel in the context of multi-agent trajectory forecasting. Although there are works that model the joint distribution of multi-agent trajectories, little work has addressed the uncertainty (covariance) stemming from the interaction of different agents. Collaborative uncertainty could become an important tool for measuring interactions between agents and improve the interpretability of trajectory forecasting models. I believe this is also a new direction other people could build on.\n- The experiments demonstrated the effectiveness of collaborative uncertainty modeling as it improves modeling synthetic data distributions as well as improving the performance of existing deterministic trajectory forecasting methods. It is also nice to see that collaborative uncertainty mainly improves models with interaction modeling, confirming that the performance gain is coming from better uncertainty modeling in agent interaction.\n- The paper is generally well-written and well-motivated and super easy to understand.\n- The paper also presents detailed derivations of the non-trivial case of Laplace distribution and addresses several practical problems of how to approximate the distribution efficiently and provides proof to justify predicting the auxiliary parameters z.\n\n**Weakness:**\n\n- As admitted by the authors, the biggest limitation of this paper is that collaborative uncertainty has only been applied to deterministic trajectory forecasting models. It is unclear how difficult it is to apply it to generative models such as VAEs and GANs. It is also unclear how effective the method would be for those generative models as these models already have stochasticity and uncertainty built in by latent variables. Given the popularity and effectiveness of generative trajectory forecasting methods, many of them also consider stochastic interaction modeling, the contributions of this paper diminish when not applying to those generative models.\n- The method is technically a generative model, and the collaborative uncertainty seems to be able to deal with stochasticity in agent interaction. Thus, it would be good to generate multiple samples and compare with other generative models on both Argoverse and nuScenes datasets for metrics like minADE_k and minFDE_k. The current FDE_1 and ADE_1 results are not very competitive on the nuScenes leaderboard, and many methods with better FDE_1 exist, such as [a]. It would be better to apply collaborative uncertainty to these methods.\n- The visualization in Fig 6 only shows simple interaction scenarios when there are only two cars and many scenarios with little or no interaction. It would be more interesting to show the interaction of multiple agents (≥ 3) and how their collaborative uncertainty changes over time. I also don't understand why in Pair 1, the collaborative uncertainty becomes lower as time goes by. One would expect the collaborative uncertainty goes higher as agents are closer and have more chances to interact.\n\n[a] Trajectory Prediction for Autonomous Driving based on Multi-Head Attention with Joint Agent-Map Representation. Messaoud et al. 2020.\n\n**Minor Comments:**\n\nTypos:\n\n- line 105:  change \",\" to \".\" or use \"and\" after \",\"\n- line 257: \"shows\" → \"show\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- The paper discussed its limitations, i.e., currently only applied to deterministic trajectory forecasting models.\n- The paper didn't discuss its societal impact, as deemed not applicable by the authors.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend accepting this paper.

The work addresses the issue of estimating joint uncertainties between actors in a motion forecasting setting.
Experiments are provided on some commonly used benchmarks for motion forecasting.
Reviewers generally felt the paper was clearly written and the experiments were sufficient.

The AC recommends acceptance.
Decision: Accept (Poster)