Exploring Social Posterior Collapse in Variational Autoencoder for Interaction Modeling
Authors: Chen Tang
Wei Zhan
Masayoshi Tomizuka
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-agent interaction modeling
variational autoencoder
applications

Abstract: Multi-agent behavior modeling and trajectory forecasting are crucial for the safe navigation of autonomous agents in interactive scenarios. Variational Autoencoder (VAE) has been widely applied in multi-agent interaction modeling to generate diverse behavior and learn a low-dimensional representation for interacting systems. However, existing literature did not formally discuss if a VAE-based model can properly encode interaction into its latent space. In this work, we argue that one of the typical formulations of VAEs in multi-agent modeling suffers from an issue we refer to as social posterior collapse, i.e., the model is prone to ignoring historical social context when predicting the future trajectory of an agent. It could cause significant prediction errors and poor generalization performance. We analyze the reason behind this under-explored phenomenon and propose several measures to tackle it. Afterward, we implement the proposed framework and experiment on real-world datasets for multi-agent trajectory prediction. In particular, we propose a novel sparse graph attention message-passing (sparse-GAMP) layer, which helps us detect social posterior collapse in our experiments. In the experiments, we verify that social posterior collapse indeed occurs. Also, the proposed measures are effective in alleviating the issue. As a result, the model attains better generalization performance when historical social context is informative for prediction. 

Exploring Social Posterior Collapse in Variational Autoencoder for Interaction Modeling
Chen Tang
Wei Zhan
Department of Mechanical Engineering University of California Berkeley Berkeley, CA then_targeberkeleyyoo edu
Department of Mechanical Engineering University of California Berkeley Berkeley, CA wzhan@berkeley, edu
Masayoshi Tomizuka
Department of Mechanical Engineering University California Berkeley
Berkeley, CA tomizuka@berkeley. eddu edu
Abstract
Multi-agent behavior modeling and trajectory forecasting are crucial for the safe navigation of autonomous agents in interactive scenarios. Variational Autoencoder (VAR) has been widely applied multi-agent interaction modeling to generate diverse behavior and learn low-dimensional representation for interacting systems. However, existing literature did not formally discuss if VAE-based model can properly encode interaction into its latent space. In this work, argue that one of the typical formulations VAEs in multi-agent modeling suffers from an issue we refer to social posterior collapse, i.e. the model is prone to ignoring historical social context when predicting the future trajectory an agent. could cause significant prediction emrors and poor generalization performance. We analyze the reason behind this under-explored phenomenon and propose several measures to tackle it. Afterward, real-world datasets for multi-agent trajectory prediction. In particular, we propose novel sparse graph attention peessage-passing (sparse-GAMP) layer, which helps us detect social posterior collapse in our experiments. verify that social posterior collapse indeed occurs. Also, the proposed measures are effective in alleviating the issue. As result, the model attains better generalization performance when historical social context is informative for prediction.
implement the proposed framework and experiment on
the experiments, we
Introduction
Modeling the behavior of intelligent agents is essential subject for autonomous systems. Safe operations of autonomous agents require accurate prediction of other agents' future motions. In particular, social interaction among agents should be modeled, so that downstream planning and decision-making modules can safely navigate the robots through interactive scenarios. Generative latent variable models are popular modeling options for their ability generate diverse and naturalistic 2
3 4 In this work, we focus on one category of generative models, Variational Autoencoder ІАΣ) 5 which has been widely used in multi-agent behavior modeling and trajectory
behavior M1
prediction 4 6 9 is desirable as it learns low-dimensional representation of the original
high-dimensionnll data.
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

However, VAEs do not necessarily learn good representation of the data [TO]. For instance. prior works sequence modeling have found out that the model tends to ignore the latent variables the decoder is overly powerful [1] 12 1 (e.g., an autoregressive decoder). It leads us to wonder whether VAF-based model can always learn a good representation for a multi-agent interacting system. It is a rather general question as researchers may look for different properties of the latent space, for instance, interpretability [5) and multi-modality 6. In this work, we focus on a fundamental aspect of this general question: Does the latent space always properly model interaction? Formally, given a latent variable model of an interacting system, where a latent variable governs each agent's s behavior, we wonder if the VAE learns to encode social context into the latent variables.
raises such concern since VAE handles two distinct tasks in training and testing. At the training stage, it learns to reconstruct a datum instead of generating one, For multi-agent interaction modeling, the sample for reconstruction learn to model the interaction among agents and jointly reconstruct the trajectories. However, there no mechanism to prevent the model from separately reconstructing the trajectories. In fact, could even be more efficient at the early stage of training when the model has not learned an informative embedding of social context. We then end up with a model that models each agent's s behavior separately without social context, which might suffer from over-estimated variance and large prediction error. More importantly, since the joint behavior of the agents is a consequence of their interactions, ignoring the causes may lead to poor generalization ability [15] [6].. We find that typical formulation of VAE for multi-agent interaction is indeed prone to ignoring historical social context (i.e., interactions over the observed time horizon). We refer to this phenomenon social posterior collapse. The issue has never been discussed in the literature. Considering those potential defects, we think it is necessary to study such crucial and fundamental issue.
set of trajectories of all the agents. Ideally, the model should
In this work, we first abstract the VAE formulation from a wide range of existing literature 4 Then we analyze social posterior collapse under this formulation and propose several measures to alleviate the issue. Afterward, we study the issue under real-world settings with a realization of the abstract formulation we design. In particular, we propose a novel sparse graph attention message- peessgg-ppassing Apprre-GAAP) layer and incorporate it into the model, which helps us detect and analyze social posterior collapse. Our experiments show that social posterior collapse occurs in real-world prediction tasks and that the proposed measures can effectively alleviate the issue. We also evaluate how social posterior collapse affects the model performance these tasks. The results suggest that the model without social posterior collapse can attain better generalization performance if historical social context is informative for prediction.
2 Social Posterior Collapse in Variational Autoencoder Interaction Models
2.1 A Latent Variable Model for Interaction Modeling
Given an interacting system with n agents, an interaction-aware trajectory prediction model takes all the agents' observed historical trajectories, denoted by {x,) ) input and predicts the future trajectories of all the agents or a subset of enquired agents. We denote the collection of future trajectories by {y:ja In this work, we focus the latent variable model illustrated in Fig. [ which is abstracted from existing literature on VAES for multi-agent behavior modeling [ 4
6
g
We model interaction by introducing a set of variables (T. which aggregates each agent's 's state and its observation of other agents. Formally, each T, is modeled as a deterministic function of (x:)"=+ i.e.. T, ft (xx.)__)) Afterward, the agents make decisions based the aggregated information over the predicted horizon. Latent variables {2:}7 are introduced to model the inherent uncertainty in each agent's 's behavior. It should be noticed that interaction over the predicted horizon is not modeled explicitly in this formulation. Although it can be achieved by exchanging information between agents recurrently (e.g., social pooling in (17). it is a common practice avoid explicit modeling of future interaction in consideration of computational cost and memory [ g 18]
In this work, train the model as a VAE, where encoder q (z|x, y) is introduced approx- imate the posterior distribution p (z) x, y) for efficient sampling at the training stage"] The per- formance of variational inference is optimized if the KL-divergence between the posteriors, i.e.
DR lg y) lpp y)], is small [10]. To derive a better approximation, we can incorporate
'The vectors x, y and collect the corresponding variables for all agents.
2

---

21
X1
T1
Y11
Z2
X2
T2
Y/2
:
Zn
X
Tn
Yn
Figure 1: A latent variable model for interaction modeling. The dash edges from T; 2; apply only in the Conditional Variational Autoencoder (CVAF) setting.
inductive bias based on the characteristics of the true posterior into the encoder function. We introduce the following simple proposition that guides our model design. Its proof can be found in Appx. A Proposition For any i 1,2, ..., n. and j 1, .... n, 1) If + i, z, and y, are conditionally independent given and y:; 2) If F i, 2, and 2, are conditionally independent given x; 3) 2, and Xj are conditionally independent given T,,
Following the proposition, we decompose the posterior distribution as II p(z: Tú The decomposition suggests two insights. First, the encoder does not need aggregate future context information when inferring the posterior distribution of z, for each agent. Second, the historical context variable T; is all need for the historical information of the agent The encoder and decoder can share the same function to encode historical information.
2.2 Social Posterior Collapse
We then design VAE model reflecting the characteristics of the true posterior discovered in Sec. 2.1 To simplify the problem. we further make the assumption of homogeneous agents. The model has three basic building blocks: 1) A function modeling the historical context, i.c., T, fe (x, x); 2) function decoding the distribution the future trajectory y; given T, and 2,, i.e., Pa (y: S.yTT, 2:): 3) A function approximating the posterior of z, conditioned on T, and y;, i.e. qo (z, T, y;). They build up the encoder and decoder of the VAE model as follows:
n
48.0 (z/x,)) = II
gv (z, i=1
Pa. (y|x, 2) = II Pe (x,, 2.). (1)
i=1
We consider a continuous latent space and model do and Pa as diagonal Gaussian distributions. The model trained by maximizing the evidence lower bound (ELBO):
max Exx,y-D (Fzz-o,,00llx.. [logPo.0 po. (y x, 2)] 3D (1o. (z|x,y) [[p(z)]] 6,
(2)
However, we find a critical issue of this naive formulation during experiments, which is the social posterior collapse phenomenon mentioned before. With the help of the sparse graph attention mechanism introduced in Sec. 3 we find that the model prone ignoring historical social context when reconstructing the future trajectory of one agent. Equivalently, the generative model collapses the one with all the variables of other agents marginalized out.
We think the reason behind is similar to, but different from, the well-known phenomenon in VAE training--postrroor collapse [10]. Due to the KL regularization term in ELBO, the variational posterior distribution is likely to collapse towards the prior. is particularly likely to occur at the early stage of training when the latent space is still uninformative [19]. In our case, the posterior of z, collapses into qa vi (z, (,)y)))
y:). ). It does minimize the KL-divergence: if both qe, (2, y:) and qe.v (z, xx yi) exactly approximate the true posteriors, then conditioning on more context information increases the expected value of KL. regularization:
ED [DEL ip (z, y.) [[]( [[,,; x,) > T(z = Ep DD (2, [ppz))]]

---

However, we argue that an additional factor contributes to the social posterior collapse problem. which makes a unique phenomenon for interaction modeling. At the training stage, the goal is to reconstruct the future trajectories y. The trajectory itself contains all the information needed for reconstruction. The history of the agent provides complementary information such as the current state and static environmental information. There is no explicit regulation in the current framework to prevent the model from extracting information solely from x; and y;. In fact, it is more efficient coding scheme when the model has not learned informative representation of interaction. Techniques such as KL annealing 13 [9] rely on various scheduling schemes of B to prevent KL vanishing, which has been shown effective in mitigating the posterior collapse problem. However, reducing 3 could merely privilege the model to gather more information from Yı. which is consistent with the reconstruction objective. Therefore, we need to explore alternative solutions to tackle the social posterior collapse problem deliberately.
We start with changing the model into a conditional generative one 120 21 Additional edges {T; + z; )= are added into the original graph, which are annotated with dash lines in Fig. 1 We follow the practice in [20] and formulate the model as Conditional Variational Autoencoder (CVAE). It straightforward to verify that the conclusion Proposition 2. still applies. We still model the encoder and decoder as in Eqn. ① The CVAE framework introduces an additional module -a a function approximating the conditional prior pn (Z,)T,), which becomes po,n) (z, (x) after incorporating fo (x). The objective then becomes:
[(0, v. 2, n) = Ex.y-. [Eurgo, [logPo.o (y|x, 2)] - 3Dxz.. [go.o (z)x, [2)o,
Compared to Eqn. 2). we no longer penalize the encoder for aggregating context information but only restrict the information encoded from future trajectories. Therefore, the encoder does not need to ignore context information to fulfill the information bottleneck. However, the model still lacks a mechanism to encourage context information encoding deliberately. To achieve the goal. we propose to incorporate an auxiliary prediction task into the training scheme. Concretely, we introduce another module y; = 9 (T; 2.). Composing IC with fo gives us another decoder y he,c (x,z) z). The difference is that the latent variables are always sampled from Pr (z, T,). The auxiliary task is training this trajectory decoder which shares the same context encoder and conditional prior with the CVAE. Because the auxiliary model does not have access to the ground-truth future trajectory, it needs to utilize context information for accurate prediction. Consequently, it encourages the model to encode context information into T and We use mean squared error (MSE) loss the objective function. The overall objective minimizes a weighted sum of the two objective functions.
The training scheme looks similar to the one in [20), where they trained a Gaussian stochastic neural network (GSNN) together with the CVAE model. However, ours different from theirs in some major aspects. The GSNN model shares the same decoder with the CVAE model. The primary motivation of incorporating another learning task is to optimize the generation procedure during training directly. In contrast, our auxiliary prediction model has separate trajectory decoder. It is because we do not want the auxiliary task to interfere with the decoding procedure of the CVAE model. We find that sharing the decoder leads to less diversity in trajectory generation, which is not desirable.
3 Sparse Graph Attention Message-Passing Layer
Before jumping introducing the specific model develop for real-world prediction tasks, would like to present a novel sparse graph attention message-passing (sparse-GA.MP) layer, which helps detect and analyze the social posterior collapse phenomenon.
3.1 Sparse Graph Attention Mechanism
Our sparse- GAMMP layer incorporates a-entmax 22 as the graph attention mechanism. a-entmax is sparse transformation that unifies softmax and sparsemax [23] Sparse activation functions have drawn growing attention recently because they can induce sparse and interpretable outputs. In the context of VAËs, they have been used to sparsify discrete latent space for efficient marginalization (4 and tractable multimodal sampling (25). our case, we mainly use o-entmax to induce a sparse and interpretable attention map within the encoder for diagnosing social posterior collapse.

---

We are particularly interested in the 1.5-entmax variant, which is smooth and can be exactly computed.
It is also easy to implement on GPUs using existing libraries (e.g., Py PyTorch (26]). Concretely, the 1. 1.-eetmax function maps d-dimensional input s E Rd into p A [ > 0, ||p|| = }
as p - [s/2 =1] where t is a unique threshold value computed using s. Upon the theoretical results in [22]. we derive an insightful proposition which makes 15--entaxx merited option in our framework. The proof of the proposition can be found in Appx. B
Proposition Let Sid] < .... 8/1) denote the sorted coordinates of Define the top-p mean, unnormalized variance, and induced threshold for p {l,...d d]
Ma(p)
1 Σ S Sa(p) P 5=1
- Σ ($y Ma(p)) Ta(p)
M2(P)
/
1-8((p)
Sa(p) < 1, otherwise.
100,
5=1
Let s' E Rd+1 satisfy 5/ for i & d, and define p =1.5 1.5-entmax(s) and p' 1:.0=1.5mmmmee 1.5-entmax(s').
1, then p/: = P: for i = 1, -,,... d and p = 0; 2) 1f p: > 0 for i == 1,...d, d, then p' = P. for = 1,2,... d and P' = 0 iff sa & (
Then we have: If *441 < {u
2
The first statement of the proposition provides sufficient condition for augmenting input vector without affecting its original attention values. It useful when applying 1.5-entmax to graph attention. Unlike typical neural network models, graph neural networks (GNN) operate graphs whose sizes vary over different samples. Meanwhile, nodes within the same graph may have different numbers of incoming edges. Therefore, the 1.5-entmax function has inputs of varying dimensions even within the same batch of training samples, making it inefficient to compute using available primitives. The proposition suggests simple solution to this problem. Given { with Sj E R4, we can compute a dummy value as minje j (1,...,m) (I....dd.. šj - 2. We can augment the input vectors with this dummy value to transform them into matrix in Rmxd" where d' is the largest value of dl. The dummy elements will not affect the attention computation, and existing primitives based on matrix computation can be directly used.
im
The second statement implies that the activated coordinates determine a unique threshold value for augmented coordinates. This property is beneficial when modeling interactions with a large number of agents (e.g.. dense traffic scenes). It ensures that the effects of interacting agents will not be diluted by the irrelevant agents, which could potentially improve the robustness of the model (27]. In this work. we mainly utilize the interpretability of the sparse graph attention, but we will investigate its application in generalization in future work.
3.2
Sparse-GAMP
To obtain the Sparse-GAME layer, we combine the sparse graph attention with a message-passing
GNN 28 [29]. Given directed graph G = (V,E) with vertices E V and edges (v, v') E E. we define the sparse-GAAe GAMP layer as a composition of a to-eet message- message-passing step v + e and edge-to-node posssage-passing step -
+ e: h(d) = fe ([h, hú, m(n)))
e u- h, = Σ w(a)) where w, G-entmax ((Drs))eeN) iEN;
In our prediction model, will use this sparse-GAMP layer model the function for historical social context encoding, i.e., T, = So x).
4 Social-CVAE
In this section, we design a realization of the abstract framework studied Sec. 2 for real-world trajectory prediction tasks. We choose to design the model with GNNs, which enable flexible graph representation data. Several based approaches achieved the state-of-the-art performance on trajectory prediction task 30 31 32 The resulting model is depicted in Fig. 2 which we refer to as social-CVAF. During training, we first encode the historical and future trajectories of all the agents using Gated Recurrent Unit (GRU) [3 network. For vehicle trajectory prediction tasks,
5

---

Pnjjectories
Future
GRU Encoder
Pastertor Encoder
Encoder
Pasterior Encoder
Posterior
Meon Varionce
Encoder
Posterior Encoder
Past
Context
GRU Encoder
Encoder
Conditional
GRU Fncoder
Fnooder
Meon Varionce
GRU Encoder
Prior Encoder
GRU Decoder
Environment
GRU Decoder
Puture
Trajectories
Decoder
Lanelet
GRU Encoder
Sampled Mode Latent Vartabies
Context
GRU Decoder
Encoder
GRU Decoder
Future Trajectories
Sparse GAMP
GRU Decoder
Sampled Node Lotent Variobles
Figure 2: Social-CVAE architecture. We adopt graph representation and use Sparse-GAMi to encode context information. The modules within the orange box are the auxiliary decoders.
we incorporate the map information in a manner similar to 34 Different from 34 where road
boundaries modeled nodes in the scene graph, adopt the representation in 35 map is denoted as graph consisting of lanelet nodes, i.e., drivable road segments. For each lanelet, represented by its left and right boundaries composed by two sequences of points. We use another GRU network to encode them. Although not utilized in this work, the lanelet representation allows us combine routing information in a natural manner as in 32. We will investigate
where the
future work.
To encode the historical context information, we construct a graph using the embeddings of historical trajectories and lanelets if available. Each pair of agent nodes has bidirectional edge connecting each other. For each lanelet node, we add edges connecting it to all the agent nodes. If the map information not available, we add self-edges for all the agents enable direct self-encoding channels. One sparse-GAMP layer is applied to encode historical context for all the agents. If social posterior collapse occurs, the agent-to-agent edges, except for self-edges, attention weights thanks to the sparse attention. Therefore, we can use the percentage of unattended agent-to-agent edges as metric detect social posterior collapse. It is more objective metric than looking into the quantitative magnitude of attention weights. We can assert that an agent node does not contribute to the output its attention is strictly zero. However, we need be more careful when comparing the importance of two agents based on non-zero attention weights [36 37 For the same reason, we do consider techniques such multiple message- pasge-gggnnnggg layers 34 31] or separating self-edges from aggregation [8). Even though they can potentially boost the performance, might become inconclusive to analyze the model based on the attention map. After computing the historical context, we use a MI.P to model the conditional Gaussian prior Pr, (z. T.). To model the variational posterior, we concatenate the future trajectory embedding with each agent's historical context and use another MLP to output the posterior mean and variance. For the decoder, another GRU to decode the future trajectory for each agent separately. The auxiliary decoder shares the same structure but always samples the latent variables from the conditional prior regardless of training or testing.
more likely to receive zero
Experiments
In this section, we report the experimental results on two trajectory prediction tasks. The main purpose not achieving ofae--of-the-ar performance but to study the social posterior collapse problem. We compare social-CVA, CVAE with two variants: 1) A model without the auxiliary task; 2) A model without the auxiliary task or the conditional prior. They correspond to the vanilla CVAE and VAE formulations discussed in Sec We are curious about whether the proposed measures can alleviate social posterior collapse. We will briefly introduce the experiment settings and present the main results. Please refer to Appx. Fl more details, visualization, and additional experiments.

---

5.1 Vehicle Trajectory Prediction
The first task is the vehicle trajectory prediction problem, where we are asked to predict the future trajectory of one target vehicle given the historical trajectories of itself and other surrounding vehicles. We train the models on two datasets, INTERACTION Dataset (38] and Argoverse Motion Forecasting dataset [39] To evaluate the prediction performance, we use the standard minimum Average Displacement Error (minADE) and minimum Final Displacement Error (minFDE) over K sampled trajectories as metries. And we follow [39] to define minADE as ADE of the trajectory with minimum FDE. Additionally, we define a unique metric, Agent Ratio (AR), to study social posterior collapse:
AR = EiE ( #0)
n n-
where the agent j refers to the predicted target vehicle and w(jj) is the attention weight assigned to the edge from the agent i to the target vehicle. It equals the percentage of surrounding vehicles which receive non-zero attention. A value close to zero implies that the model ignores the majority of the surrounding vehicles, which is a sign of social posterior collapse.
INTERACTION Dataset. The INTERACTION dataset provides three categories of driving scenar- ios: intersection (IS), roundabout (RA), and highway merging (HM). For each experiment, we ran five trials to account for the randomness in initialization. We then evaluated them on the validation sets and computed the mean and standard deviation of the evaluated metries over all the trials. The best models were selected for testing on the regular (R) and generalization (G) tracks of the INTERPRET Challenge The results are summarized in Table [ the VAE variants on IS and HM. It is consistent with our argument that merely changing the model to a conditional one insufficient.
The CVAE variants have AR values similar to
In contrast, our social-CVAB model consistently attains high AR values. However, the CVAE variant achieves similar prediction performance as ours on the validation sets of IS and RA, even if the CVAE variant suffers from the social posterior collapse problem. It is because the driving behavior within intersections and roundabouts depends highly on locations. When the map is less informative (c.g.. validation set in HM) or novel scenario is encountered (e.g., generalization track), our social-CVAE model, which does not have social posterior collapse outperforms the other variants. Notably, we compare it with another instance that attains a AR value close to zero even with the auxiliary task. Their test performances, especially on the generalization track, are pretty different. shows that is mainly the historical social context that improves the prediction performance. Even if the auxiliary task also contributes, the improvement is not significant, especially in novel scenes, unless it prevents social posterior collapse.
Argoverse Dataset. Similar to the INTERACTION dataset, we trained five models with random initialization for each case and evaluated them on the validation sets. The best social-CVAE instance was selected for submission to the Argoverse Motion Forecasting Challenge. Although achieving ffute-ofthh--se performance is not our objective, we still report the testing result in Appx. Faand compare it with the other models on the leaderboard in order to provide the audience a complete picture of the model. Here, we mainly focus on the validation results in Table 2 The conclusion consistent. The difference is that the social-CVAE model outperforms the others even on the validation set. It is because the validation set of the Argoverse dataset is collected in different areas of the cities, which is more analogous to the generalization track of the INTERPRET Challenge.
5.2 Pedestrian Trajectory Prediction
The second task is the pedestrian trajectory prediction problem, where we are asked to forecast the future trajectories of all the pedestrians in each scenario. We trained the models on the well- established ETH/UCY dataset, which combines two datasets, ETH (40) and UCY [41). Following prior works 2 environmental information. All the prediction metrics are computed with K 20 samples. Similar to the vehicle case, we ran five trials for each experiment. The results are summarized in Table where we report the testing results for each group and their average over all the groups. The
7 91 we adopt the leave-one-out evaluation protocol and do not use any
"The challenge adopts different group of evaluation metrics, please check their website for the formal detinitions: http://challenge wtw.://onn...ttttrciiooooooooooo interaction datanstddatttttat.a wtt.aaattt..ccaaa........ooooaad
7

---

Table 1: INTERACTION Dataset Validation and Test Results
K =1
Scene
Model
AR(%)
minFDE
minADE
minFDD
minADE
VAE
1.85 + 4.02
1.32 + 0.03
0.42 + 0.01
0.73 + 0.01
0.26 + 0.01
CVAE
0.18 + 0.36
1.27 + 0.02
0.41 + 0.01
0.68 + 0.02
0.24 0.01
Ours
15.8 + 10.4
1.27 + 0.02
0.41 + 0.01
0.70 0.02
0.25 0.01
VAB
0.05
1.34 = 0.02
0.02
0.76 = 0.01
0.26 0.01
RA
CVAE
13.4 14.9
1.32 = 0.01
0.42
0.73 = 0.02
0
Ours
19.5 + 15.4
1.29 0.03
0.42 + 0.01
0.72 + 0.01
0.26 0.01
VAE
0.87 0.08
0.29
0.42 0.04
HM
CVAE
5.42 10.2
0.83 + 0.03
0.28 0.03
0.40 = 0.05
0.16
Ours
15.5 8.13
0.65 + 0.09
0.22 - 0.02
0.32 - 0.04
0.13 1
K 6
K 50
Track
Model
AR(%)
ADE
FDE
MoN
ADE
FDE
MoN
VAE
0.35 + 5.37
0.5685
1.7573
0.2238
0,5712
1.7709
0.1036
CVAE
0.07 + 1.43
0.5323
1.6425
0.2144
0.5410
1.6725
0.0983
R
Ours*
0.88 2.49
0.5157
1.5823
0.2195
0.5158
1.5823
0.1106
Ours
24.8 + 20.2
0.4665
1.4174
0.2011
0.4662
1.4187
0.1050
VAE
0.02 + 1.27
1.3428
3.8542
0.8193
1.3436
3.8564
0.5181
CVAE
0.05 2.200
1.4517
4.2179
0.8743
1.3824
3.9972
0.5676
G
Ours*
0.37 +
1.1615
3.3927
0.6811
1.1617
3.3939
0.4218
Ours 14.7
0.9205
2.7049
0.6075
0.9186
2.6969
0.4891
Ours refers to an instance of social-CVAE that still suffers from the social posterior collapse problem. The results are presented in the format of mean + std. On the validation set, the mean and std are computed multiple trials. On the test set, the mean and std of AR are computed over all the samples.
Table 2: Argoverse Dataset Validation Results
K =
K = 6
Model
AR(%)
minFDE
minADE
minFDE
minADE
VAE
1.27 + 1.00
0.17
1.86 - 0.07
2.33 = 0.06
1.30 + 0.04
CVAE
0.40 + 0.31
3.85 + 0.01
1.73
0.01
2.09 0.02
1.19 0.01
Ours
28.5 + 15.7
3.52 + 0.03
1.59 + 0.02
1.98 0.02
1.15 0.01
The results are presented in the format of mean + std computed over multiple trials.
comparison between our model and other methods can be found in Appx. F In Table we see that changing CVAE formulation leads significant boost in terms of AR, implying that the model gives more attention to surrounding agents. However, neither changing to CVAE nor the auxiliary task improves prediction performance. It because historical social context is less informative for pedestrian trajectory prediction. The ETH/UCY dataset is collected in unconstrained environments with few objects. Also, compared to vehicles, pedestrians have fewer physical constraints in 2D motion, resulting in shorter temporal dependency in their movement. It is consistent with the results model that achieves the current state-of-the-art
[9) where the authors proposed a transformer-basee performance on ETH/UCY. The visualized attention maps in 9 show that social context in nearby timesteps is more important their prediction model. contrast, the historical trajectories of other agents always receive low attention weights. Therefore, even if the auxiliary prediction task encourages our model to encode historical social context, it does not lead either larger AR value or better prediction performance. It suggests that the simplified latent variable model studied in this work might not sufficient to model pedestrian motion. Explicit future interaction should be considered, and an autoregressive decoder as the one in [9] should be used to account for it.

---

6 Discussion
Limitations. As the first study on social posterior collapse, cannot explore every aspect of this subject. Many possible factors could contribute to this phenomenon. Our experimental results can only speak for the particular model we develop and the tasks we study. The occurrence of social posterior collapse and its effect on the overall performance may vary across different problems and different model architectures. For instance, finding from the pedestrian trajectory prediction task is pretty different from its vehicle counterpart. The format of the graph representation, especially how the environmental information is incorporated, also matters. Also, the diagnosis based on the AR value could be inconclusive under different graph representations. In Appx. E.2 we show that the AR values of the three models become similar after removing the lanelet nodes for the highway merging subset of the INTERACTION collapse does not occur. Our social-CVAE model can maintain consistent prediction accuracy without map information, while the baseline VAE model has a significant drop in performance. It implies that the baseline VAE model does not properly utilize historical social context, but cannot assert that social posterior collapse occurs due to the lack of direct evidence.
dataset. However, it does not mean that social posterior
Nevertheless, we do demonstrate that social posterior collapse is not unique to sparse-GAMi module (Appx. E.T. Even if we switch to other aggregation functions, we can still find evidence suggesting its occurrence. Our sparse graph attention function, G-entmax, is convenient and flexible toolkit that allows us to monitor and analyze social posterior collapse without compromising performance. In the future, will work along this direction to explore alternative diagnosis toolkits that can be applied to detect social posterior collapse for broader range of models.
Connections to Related Works. We want to wrap up our discussion with a glance at those prior works on VAE-based multi-agent behavior modeling. We are curious about if any elements in their models have implicitly tackled this issue. However, we would like to emphasize that it is still necessary to explicitly study social posterior collapse under their settings in the future, which could provide helpful guidance to avoid social posterior collapse in model design. Due to the space limit, we only discuss works using techniques potentially related to social posterior collapse, especially those provided evidence that interacting behaviors were appropriately modeled (e.g., attention maps or visualized prediction results in interactive scenarios).
Trajectron 6 and Trajectron++ 4 which are formulated as CVAES establish the previous state-of- the-art results on ETH/UCY. Different from ours, they adopted discrete latent space to account for the multi-moddlity in human behavior. They did not examine how their models utilize social context. However, we think that discrete latent space could potentially alleviate the social posterior collapse issue. Compared to a continuous random variable, discrete one can only encode a limited amount of information. It prevents the model from bypassing social context since a discrete latent variable is not sufficient to encode all the information of a long trajectory. For the same reason, NRI [29, which adopted a discrete latent space for interacting system modeling, is also potentially relevant. PECNet 77 is another CVAE-based trajectory prediction model. Instead of conditioning trajectory, they proposed the Endpoint VAE where the posterior latent variables only condition on the endpoints. It could also be potential solution it limits the amount of information the latent space can encode from the future trajectory.
the entire future
In M. Suo et al. proposed a multi-agent behavior model for traffic simulation under the CVAE frame- work. They demonstrated that their method was able to simulate complex and realistic interactive behaviors. In particular, they augmented ELBO with commonsense objective that regularizes the model from synthesizing undesired interactions (e.g., collisions). We think it plays a similar role as auxiliary prediction task. The last work we would like to discuss is the AgentFormer model [9] mentioned in Sec. [.2 Their results suggest that incorporating an autoregressive decoder and a future social context encoder could be more effective in interaction modeling, especially for systems without long-term dependency (e.g., pedestrians). However, as we have mentioned before, the autoregressive decoder introduces another issue if it is overly powerful.
7 Conclusion
In this work, we point out an under-explored issue, which we refer to as social posterior collapse, in the context of VAEs in multi-agent modeling. We argue that one of the commonly adopted formulations

---

Table 3: ETH/UCY Dataset Leave-one-out Testing Results
ETH
HOTEL
Model
AR(%))
minFDE
minADE
AR(%)
minFDE
minADE
VAE
74.0 + 7.33
0.98 + 0.07
0.63 : 0.03
29.0 = 28.1
0.28 = 0.01
0.19 0.01
CVAE
90.9 + 6.38
0.94 1 0.10
0.61 0.05
69.8 41.7
0.27
0.01
0.18
Ours
83.2 + 12.6
1.08 + 0.10
0.68 = 0.04
72.9 + 21.5
0.27 0.02
0.18
ZARAI
ZARA2
Model
AR(%)
minFDE
minADE
AR(%)
minFDE
minADE
VAE
40.3 + 22.8
0.38 + 0.01
0.22 0.01
30.1
0.32 0.01
0.18
0.01
CVAE
89.4 +: 7.65
0.39 + 0.01
0.22 + 0.01
64.2 + 25.6
0.35 + 0.01
0.19 + 0.01
Ours
61.9 + 5.71
0.38 + 0.01
0.01
58.6 15.7
0.37
0.02
0.20 0.01
UNIV
Average
Model
AR(%)
minFDE
minADE
AR(%)
minFDE
minADE
VAE
0.06 + 0.09
0.55 + 0.01
0.31 + 0.01
34.7 + 29.4
0.50 + 0.27
0.30 + 0.17
CVAE
41.2 + 25.1
0.63 + 0.05
0.35 0.02
71.1 29.5
0.51 0.25
0.31 0.17
Ours
36.3 +: 14.4
0.63 + 0.02
0.35 + 0.01
62.6 21.0
0.54 0.29
0.33 + 0.19
The results are presented in the format of mean + std computed over multiple trials.
of VAEs in multi-agent modeling is prone to ignoring historical social context when predicting the future trajectory of an agent. We analyze the reason behind and propose several measures to alleviate social posterior collapse. Afterward, we design GNN-based realization of the general framework incorporating proposed measures, which we refer as social-C -VAE. particular, social-CVAE incorporates novel GAMP layer which helps detect and analyze social posterior collapse. In our experiments, we show that social posterior collapse occurs in real-worke trajectory prediction problems and that the proposed measures effectivvlyy alleviate the issue. Also, the experimental results imply that social posterior collapse could cause poor generalization performance in novel scenarios the future movement of the agents indeed depends their historical social context. In the future, will utilize the toolkit developed in this work to explore social posterior collapse in a broader range of model architectures and interacting systems.
Acknowledgements
We would like to thank Liting Sun, Xiaosong Jia, and Jiachen Li for insightful discussion. We thank Vade Shah for helping us with the experiments. We thank Chenfeng Xu and Hengbo Ma for their valuable feedback. We would also like to thank the anonymous reviewers for their helpful review comments. This work supported by Denso International America, Inc.
References
[1] S. Suo, Regalado, S. Casas, and R. Urtasun, "TrafficSim: Learning to simulate realistic multi-agent behaviors." arXiv preprint MxXiv:201100657 2021.
[2] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi, "Social GAN: Socially acceptable trajectories with generative adversarial networks." in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[3] V. Kosaraju, A. Sadeghian, R. Martín-Martin, I. Reid, H. Rezatofighi, and S. Savarese, "Social- BiGAT: Multimodal trajectory forecasting using bicycle-GAr and graph attention networks," in Advances in Neural Information Processing Systems, 2019.
[4] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone, "Trajectron++:: Dynanically-feasine trajectory forecasting with heterogencous data." arXiv preprint arXiv:2001.03093. 2020.
[5] D. P. Kingma and M. Welling, "Auto-encoding variational Bayes," 1322v11312.6114, 2013.
arXiv preprint
10

---

[6] B. Ivanovic and M. Pavone, "The Trajectron: Probabilistic multi-agent trajectory modeling with dynamic temporal graphs." in IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2374-2384, 2019.
[7] K. Mangalam, H. Girase, S. Agarwal, K.-H. Lee. E. Adcli. J. Malik, and A. Gaidon, "It is not the journey but the destination: Endpoint conditioned trajectory prediction," in European Conference Computer Vision (ECCV), pp. 759-776, Springer, 2020.
[8] J. Li, H. Ma, Z. Zhang, and M. Tomizuka, prediction via wasserstein graph double-attention network," arXiv preprint arXiv:2002.06244 06244,, 2020.
Interaction- trajectory
[9] Y. Yuan, X. Weng, Y. Ou, and K. Kitani, "AgentFormer: Agent-aware transformers for socio- temporal multi-agent forecasting," arXiv preprint arXiv:2103. 14023, 2021.
[10] X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, and Abbeel, "Variational lossy autoencoder," arXiv preprint arXiv:
2016.
[11] S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio, "Generating sentences from continuous space," arXiv preprint arXiv:1511.06344 2015.
[12] M. Fraccaro, S. K. Sønderby, U. Paquet, and O. Winther, "Sequential neural models with stochastic layers," arXiv preprint 1605.07777 2016.
[13] I. Serban, A. Sordoni, R. Lowe, L. Charlin, J. Pincau, A. Courville, and Y. Bengio, "A hierarchi- cal latent variable encoder-decoder model for generating dialogues," in AAAI Conference on Artificial Intelligence, 2017.
[14] Y. Hu, W. Zhan, L. Sun, and M. Tomizuka, "Multi-modal probabilistic prediction of interactive behavior via an interpretable model," in IEEE Intelligent Vehicles Symposium (IV), pp. 557-563, IEEE, 2019.
[15] C. Tang, N. Srishankar, S. Martin, and M. Tomizuka, "Grounded relational inference: domain knowledge driven explainable autonomous driving," arXiv preprint arXiv:2102.1190. 2021. 2021.
[16] P. de Haan, D. Jayaraman, and S. Levine, "Causal confusion in imitation learning," arXiv preprint arXiv: /905. / 1979, 2019.
[17] A. Alahi, K. Goel, V. Ramanathan, A. Robiequet, L. Fei-Fei, and S. Savarese, "Social LSTM: Human trajectory prediction in crowded spaces," in /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[18] S. Khandelwal, W. Qi, J. Singh, A. Hartnett, and D. Ramanan, "What-if motion prediction for autonomous driving," arXiv preprint arXiv:2008. 10587, 2020.
[19] H. Fu, C. Li, X. Liu, Gao, A. Celikyilmaz, and L. Carin, "Cyclical annealing schedule: A simple approach to mitigating KL vanishing." " in Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
[20] K. Sohn, H. Lee, and X. Yan, "Learning structured output representation using deep conditional generative models," Advances in Neural Information Processing Systems, vol. 28, pp. 3483- 3491, 2015.
[21] B. Ivanovic, K. Leung, F. Schmerling, and M. Pavone, "Multimodal deep generative models for trajectory prediction: A conditional variational autoencoder approach,"" IEEE Robotics and Automation Letters (RA-L), vol. 6, no. 2. pp. 295-302, 2020.
[22] B. Peters, V. Niculae, and A. F. Martins, "Sparse sequence-to to-nue-ett--suunnee models," arXiv preprint 19050020, 2019.
[23] A. Martins and R. Astudillo, "From softmax to sparsemax: A sparse model of attention and multi-label classification," in International Conference on Machine Learning (ICML), pp. 1614 1623, PMLR, 2016.
[24] G. M. Correia, V. Niculae, W. Aziz, and A. F. Martins, "Efficient marginalization of discrete and structured latent variables via sparsity," arXiv preprint arXiv:2007.01919, 2020.
[25] M. Itkina, B. Ivanovic, R. Senanayake, M. J. Kochenderfer, and M. Pavone, "Evidential sparsi- fication of multimodal latent spaces in conditional variational autoencoders," arXiv preprint arXiv:2010.0916 2020.

---

[26] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., "Pytorch: An imperative style, performmee deep learning library," arXiv preprint arXiv: 19119911711111 703, 2019.
[27] A. Goyal, A. Lamb, J. Hoffmann, S. Sodhani, S. Levine, Y. Bengio, and B. Schölkopf, "Recur- rent independent mechanisms," arXiv preprint arXiv: 1909. 10893, 2019.
[28] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, "Neural message passing for quantum chemistry," in International Conference on Machine Learning (ICML), pp. 1263-1272,
PMLR, 2017.
[29] T. Kipf, E. Fetaya, K.-C. Wang, M. Welling, and R. Zemel, "Neural relational inference for interacting systems," International Conference on Machine Learning (ICML), pp. 2688-2697 PMLR, 2018.
[30] H. Zhao, J. Gao,, T. Lan, C. Sun, B. Sapp, B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, al., "TNT: Target-driven trajectory prediction," arXiv preprint arXiv:2008.088) 08294, 2020.
[31] M. Liang, B. Yang, R. Hu, Y. Chen, R. Liao, S. Feng, and R. Urtasun. "Learning lane graph representations for motion forecasting," Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12347 LNCS,
pp. 541-566, 2020.
[32] W. Zeng, M. Liang, R. Liao, and R. Urtasun, "LaneRCNN: Distributed representations for graph-centric motion forecasting," arXiv preprint arXiv:2101.06633 2021.
[33] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and
Y. Bengio. "Learning phrase representations using rnn encoder-decoder for statistical machine translation," arXiv preprint arXiv:14066 100.1008, 2014.
[34] J. Gao, C. Sun, H. Zhao, Y. Shen, D. Anguelov, C. Li, and C. Schmid, "VectorNet: Encoding hd maps and agent dynamics from vectorized representation," in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1552-11530, 2020.
[35] Bender, Ziegler, and C. Stiller, "Lanelets: Efficient map representation for autonomous driving," in IEEE Intelligent Vehicles Symposium (IV), pp. 425, 2014.
[36] S. Jain and B. C. Wallace, "Attention is not explanation," in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Shori Pupers), pp. 3543-3556, 2019.
[37] S. Wiegreffe and Y. Pinter, "Attention is not not explanation." in Conference on Empirical Methods in Natural Language Processing and International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 11-20, 2019.
[38] W. Zhan, L. Sun, D. Wang. H. Shi, A. Clausse, M. Naumann, J. Kummerle, H. Konigshof, C. Stiller, A. de La Fortelle, el al., "Interaction dataset: An international, adversarial and cooperative motion dataset interactive driving scenarios with semantic maps," arXiv preprint arXiv:1910.0300 2019.
[39] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, er al., "Argoverse: 3d tracking and forecasting with rich maps." in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8748-8757, 2019.
[40] S. Pellegrini, A. Ess, K. Schindler, and L. van Gool, "You'll 'll never walk alone: Modeling social behavior for (ICCV), pp. 261-268, 2009.
tracking," in IEEE/CVF International Conference on Computer Vision
[41] A. Lerner, Y. Chrysanthou, and D. Lischinski, "Crowds by example," in Computer Graphics Forum, vol. 26, pp. 655-664, Wiley Online Library, 2007.
[42] A. Sadeghian. V. Kosaraju, A. Sadeghian, N. Hirose, H. Rezatofighi, and S. Savarese, "So- phie: An attentive GAN for predicting paths compliant to social and physical constraints." in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1349-1358, 2019.
[43] J. Pearl, Probabilistic reasoning in intelligent systems: Networks of plausible inference. Elsevier, 2014.
[44] M. Fey and E. Lenssen, "Fast graph representation learning with PyTorch Geometric," in ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
12

---

[45] R. Liaw, E. Liang, R. Nishihara, P. Moritz, J. E. Gonzalez. and I. Stoica, "Tune: A research platform for distributed model selection and training," arXiv preprint arXiv:1807.05115 2018.
[46] J. L. Ba, J. R. Kiros, and G. E. Hinton, "Layer normalization," arXiv preprint arXiv:1607.06453 2016.
[47] D. P. Kingma and Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv: arXiv:1412.66 1412.6980, 2014.
[48] M. Ye, T. Cao, and Q. Chen, "TPCN: Temporal point cloud networks for motion forecasting," T arXiv preprini arXiv:2 003.03067, 2021.
[49] F. Giuliari, I. Hasan, M. Cristani, and F. Galasso, "Transformer networks for trajectory forecast-
ing," in International Conference on Pattern Recognition (ICPR), pp. 10335-10342, 2021.
[50] C. Yu, X. Ma, Ren, H. Zhao, and Yi, "Spatio-temporal graph transformer networks for pedestrian trajectory prediction," in European Conference on Computer Vision (ECCV), pp. 507-523, Springer, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In Sec. we mention that our findings are limited to the particular model architecture and tasks studied in this work.
(c) Did you discuss any potential negative societal impacts of your work? [No] We believe our work should not have any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] See supplementary materials.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as a URL)? [No] Because the code is proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplementary materials.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We do not estimate the amount of computation but provide the specification of the computational resources.
4. If you using existing assets code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See supplementary materials.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent
obtained from people whose data you're using/curating? [Yes] Yes, we mention how we obtained the data in the supplementary materials. Most datasets and packages we used are open-source. For the remaining INTERACTION dataset which is not open-source, we mention that we have been granted the access for research usage.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] We checked the format of the information provided by the datasets. There are no elements from the dataset that could potentially contain any personal information or offensive content.
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "wYhaR8Cw2TO", "reviewer": "Reviewer_q4Va", "summary": "This paper focuses on analyzing VAE-based frameworks for multi-agent trajectory forecasting from a relationship-modeling perspective. Specifically, it argues that standard VAE frameworks can potentially lead to a situation where the interactions between entities are ignored when predicting future trajectories, which they call “social posterior collapse.” To address this, they augment this framework in two ways: first, they convert the model to a conditional VAE (CVAE) where the prior on latent variables is conditioned on input trajectories; second, they add an auxiliary training task wherein samples from the prior are used to train a second trajectory decoder. To analyze the “social posterior collapse” issue, a graph attention layer that uses sparse attention is developed (titled “Sparse-GAMP”), and a metric called Agent Ratio (AR) is defined which consists of the proportion of surrounding agents to which the target agent is attending. Experiments are run on traffic and pedestrian trajectory prediction datasets, where it is shown that, in situations where social posterior collapse is shown to be present, their model is able to alleviate this issue. Furthermore, the auxiliary prediction task can lead to better model performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality: \nthe study undertaken in the paper is the first to my knowledge which attempts to explore the ability of VAE models for multiagent trajectory prediction tasks to properly model interactions between agents. The VAE framework that forms its basis is commonly used in prior work and appropriate for this study (and proper citations are made throughout).\n\n\n### Quality:\nMuch care has been taken to develop the social-CVAE framework, from motivation through derivation. Additionally, the development of the AR metric to evaluate the ability of a model to model interactions between agents is useful for understanding the social posterior collapse issue. The experimentation is thorough, with multiple trials for every setting being run and important ablations being included in the appendix. Overall, the presented results are trustworthy, and the presented social-CVAE approach seems to mitigate social posterior collapse in situations where it appears in other approaches.\n\nThat being said, there is one set of experimental results presented in the supplementary material that ought to be mentioned in the main paper, as they alter the conclusions one should draw from this work. Specifically, the results presented in Table 6 indicate that the representation of context used by the model significantly impacts whether social posterior collapse occurs or not. When a lane graph-based representation is not included, no models suffer from social posterior collapse; furthermore, in this situation, the AR metric does not seem to correlate with the distance-based error metrics. These conclusions are additionally supported by the pedestrian dataset results presented in Table 3, where no map context is used. This dependence of social posterior collapse on context representation is important, but as far as I could tell was not mentioned in the main paper anywhere.\n\n\n\n### Clarity: \nThe paper is written clearly, and is organized well. It is clear which components are drawn from prior work and which are novel developments. Sufficient detail is presented for reproducibility. As mentioned in the previous section, though, I think the results from Table 6 should be alluded to in the main paper.\n\n### Significance: \nAs this paper cites, there are a number of papers which use VAE frameworks for multi-agent trajectory prediction. Hence, understanding the properties of these models, including how they model interactions between agents, is very important for the development of improved models. This paper takes a first step in this direction; though the results are not state-of-the-art, the analysis of social posterior collapse is interesting and may be useful for designing future models.\n\n\n### Overall:\nThis work is a promising first step in understanding how to ensure interactions are encoded properly when using VAE-based multi agent trajectory prediction approaches. I do think that the lack of discussion regarding the impact of the context encoding representation hurts this paper somewhat, as it is possible to draw some incorrect conclusions without it.\n\n#POST-REBUTTAL UPDATE\nThanks for your comments. Based on your responses, I am keeping my recommendation the same - I think this paper should be accepted. Please add the extra clarifications you mentioned in your response, as they will benefit the text.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors fully acknowledge that their results are not state-of-the-art, and they acknowledge that this is a limited study whose results might not apply to all models or domains - the results are not over-sold. There is no mention of potential negative societal impacts in this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "tTpLgOwf77J", "reviewer": "Reviewer_3CCN", "summary": "This work discuss the problem of posterior collapse in VAE/CVAE problems and extend this when it is used in the concept of multi-agent  interaction modelling and trajectory prediction problem. They argues that the model designed naively based on VAE for this task is prone to ignore agent interaction when predicting the future trajectory of an agent. The framework proposes a novel sparse graph attention message-passing (sparse-GAMP) layer to address this problem. ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem discussed in this paper is very interesting and the authors well justified this critical issue with VAEs theoretically. My main concern is that the experiments and evaluations are not sufficient to validate the problem and efficacy of the solution. I suggest that the paper should include one experiment without any joint (social) interaction (encoding and decoding for each agent independently) and proves that the results using VAE in social setting is the same (or even worse) than this baseline. Also to show the efficacy of the solution, the authors could try their solution in many frameworks using VAE (trajectron, trajectron++ etc) and show that their sparse-GAMP layer can improve the performance of the state-of-the-art methods for this problem. Without these experiments, it is hard to support the acceptance of the paper with the theoretical arguments and a weak experimental validation only.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, they haven't. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "l9_YtGl1l-E", "reviewer": "Reviewer_2YSC", "summary": "The proposed paper introduces the concept of social posterior collapse in variational autoencoder (VAE) architectures trained for trajectory prediction in interactive environments. Social posterior collapse refers to the VAE ignoring interaction information during training, thus making trajectory predictions solely from the predicted agent's history and map information. The authors present a means to quantify the extent of social posterior collapse through a sparse attention mechanism and propose architectural changes to alleviate the issue.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really like the new perspective taken by the authors in investigating existing difficulties of training VAEs in the specific context of trajectory prediction in interactive environments. The authors provide valuable insight with respect to the connections between general machine learning findings and trajectory prediction. The authors do an excellent job in contextualizing the work in terms of existing literature. I would recommend two additional papers in the space of sparse distributions and VAEs from NeurIPS last year to include in the literature review.  \n\n[1] Correia, Gonçalo, et al. \"Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity.\" Advances in Neural Information Processing Systems (2020).\n\n[2] Itkina, Masha, et al. \"Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders.\" Advances in Neural Information Processing Systems (2020).\n\nEmploying sparse attention to quantify the extent of the social context used by the VAE to predict future trajectories is a good idea. The claim of quantifying the social context used to make predictions by the CVAE is well supported by the INTERACTION/Argoverse experiments. The authors also provide extensive experimental analysis in the supplementary material.\n\nI appreciated the transparency and the insightful discussion dedicated to the limitations of the proposed method. Although portraying negative results is valuable, and a rare practice in machine learning papers, I am concerned that the pedestrian experiments actually detract from and confuse the message of the paper in the current framing. On page 2 of the paper, the authors frame the task of interest as predicting multiple trajectories simultaneously (lines 40-41). The problem with current trajectory prediction models is defined as their tendency to ignore interactions over the predicted horizon (lines 72-75). However, the successful experiments focus on a single vehicle trajectory prediction task, demonstrating that the interaction history is important for the prediction. The unsuccessful pedestrian dataset experiments, despite providing significant insight into both the dataset and the problem, do not validate the original hypothesis of the paper. I would reframe the beginning of the paper to focus on ensuring that the \\textit{historical social context}, rather than future interaction, is accounted for in the models for predicting single trajectories at a time. That way the successful experiments validate this idea. The pedestrian experiments can be kept as a case study extension in the main paper. Alternatively, successful experiments demonstrating the simultaneous prediction of all trajectories in the scene should be included in the paper, and the pedestrian experiments moved to supplementary.\n\nI have some concerns with respect to the clarity of some of the presented material. Could you clarify the connection between the agent nodes and the $T_{i}$ context encodings? Since $T_{i}$ incorporates information from all of the observed agents by agent $i$, while sparse attention determines which agent connections are being used by the model, I would like to better understand the relationship between the two concepts.\n\nSeveral variables were not defined (e.g., $\\mathbf{s}$ on line 158, $I$ on line 110). It is great to see that the authors did their due diligence by running the experiments on multiple random seems. However, I have some questions regarding the standard deviation reported in Tables 1-3, but particularly Table 1. First, are these standard deviations or standard error? The latter is usually reported as a metric of result significance. Second, if these are standard errors, they are rather large, potentially demonstrating the results to not be significant? Third, why are there standard errors reported for only one of the columns in Table 1? Also in Table 1, could you clarify what the difference was between the models 'Ours' and 'Ours*'? I am not sure what the significance of the discussion of the latter model is in lines 261-265?\n\nThe presentation of the paper needs improvement in that it is riddled with typos and grammatical errors. Some examples are as follows, but these are far from exhaustive:\n\n- 'the model is prone to ignore social context' (should be 'prone to ignoring').\n- 'a model that models each agent's behavior separately without social context. which might suffer from over-estimated variance and large prediction error.' (punctuation/half-sentence typo).\n- 'to interfere the decoding procedure' (missing 'with').\n\nThe references should also be proofread for consistency in conference naming, capitalization (e.g., 'gan' -> 'GAN', 'bayes' -> 'Bayes'), etc.\n\nOverall, the authors present an interesting approach to evaluating social posterior collapse in VAE models for trajectory prediction. I really liked the idea and the in-depth discussion provided in the paper. With improvements to the presentation of the paper and the framing of the experiments with respect to the problem statement, this work will be a good contribution!", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not indicate any negative societal impact to their work, but did contextualize the limitations of the work in great detail. I would encourage the authors to think about potential inadvertent consequences such as the amplification of implicit bias in the data in the process of sparsification or concerns regarding deployment of neural network architectures in safety critical systems.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "8CQD6-gowVT", "reviewer": "Reviewer_8WnM", "summary": "This paper analyzes the reason behind the social posterior collapse caused in typical VAEs for multi-agent modeling and proposes improvement schemes. The paper test the proposed model on real-world multi-agent trajectory prediction datasets. This paper also proposes a novel sparse graph attention message-passing.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "originality: good\nquality: need to improve\nclarity: some designs are not easy to understand\nsignificance: just ok\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1 What's the meaning of Position 2 in the sparse graph attention message-passing towards social posterior collapse detection? \n\n2 The inferred sparse attention is only used in e2v process. The v2e process still causes large storage and complexity.\n\n3 What is the effects of the sparse graph attention message-passing, compared to the ordinary neural-message-passing without sparse attention? From the aspects of error and efficiency?\n\n4 Many excellent previous works are not introduced for performance comparison, some of which are based on CVAE or neural-message-passing, including:\n   \n[1] Social lstm:  Human trajectory prediction in crowded spaces\n\n[2] Social attention: Modeling attention in human crowds\n   \n[3] Social gan:  Socially acceptable trajectories with generative adversarial networks.\n\n[4] Stgat: Modeling spatial-temporal interactions for human trajectory prediction\n   \n[5] Sophie: An attentive gan for predicting paths compliant to social and physical constraints\n   \n[6] Multi-agent tensor fusion for contextual trajectory prediction\n   \n[7] Encoding crowd interaction with deep neural network for pedestrian trajectory prediction\n   \n[8] Collaborative motion prediction via neural motion message passing\n   \n[9] Trajectron++:  Multi-agent generative trajectory forecasting with heterogeneous data for control\n   \n[10] Conditional flow variational autoencoders for structured sequence prediction.\n   \n[11] Neural relational inference for interacting systems.\n   \n[12] Social-stgcnn: A social414spatio-temporal graph convolutional neural network for human trajectory prediction.\n\nThe results in this paper still need to be improved", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 day"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper identifies a failure mode of a common approach to modeling interaction using Variational Autoencoders in multi-agent behavior and trajectory forecasting, which the authors term "Social Posterior Collapse". This identified tendency to underfit in settings where relevant social context could improve prediction and generalization performance. The authors propose an approach to solving this problem using a contextual VAE formulation and auxiliary prediction task, and demonstrate that these lead to the expected empirical improvements in multi-agent interaction settings.


Reviewers assessed the focus of the papers as an important and interesting problem that is well diagnosed theoretically. Given that VAEs are commonly used to model multi-agent interactions, the identified failure mode, proposed metric for diagnosing it, and the novel solution to this problem have good potential for impact. 

A number of concerns were raised in the initial reviews. In particular, reviewers suggested additional empirical validation to more clearly pin point the failure case, and to probe its generality as well as the generality of the new proposed solution. In addition, reviewers made valuable suggestions for improving clarity as well as further increasing the contribution of the paper by moving an experiment from the appendix to the main paper.

In the rebuttal and discussion, reviewers indicated that their concerns were largely addressed. The most negative reviewer unfortunately did not follow up on the discussion. The AC reviewed their concerns and the author response, and assessed the concerns as sufficiently addressed. While the paper can be improved further, remaining issues can be addressed in the camera ready version and do not prevent acceptance of the paper. The AC recommends that the paper is accepted, and strongly encourages the authors to take on board all reviewer suggestions to further improve the camera ready version.
Decision: Accept (Poster)