Stochastic Multi-Person 3D Motion Forecasting
Authors: Sirui Xu
Yu-Xiong Wang
Liangyan Gui
Venue: ICLR 2023 notable top 25% (2023)
Keywords: stochastic forecasting
multi-person 3D motion
dual-level generative modeling

Abstract: This paper aims to deal with the ignored real-world complexities in prior work on human motion forecasting, emphasizing the social properties of multi-person motion, the diversity of motion and social interactions, and the complexity of articulated motion. To this end, we introduce a novel task of stochastic multi-person 3D motion forecasting. We propose a dual-level generative modeling framework that separately models independent individual motion at the local level and social interactions at the global level. Notably, this dual-level modeling mechanism can be achieved within a shared generative model, through introducing learnable latent codes that represent intents of future motion and switching the codes' modes of operation at different levels. Our framework is general; we instantiate it with different generative models, including generative adversarial networks and diffusion models, and various multi-person forecasting models. Extensive experiments on CMU-Mocap, MuPoTS-3D, and SoMoF benchmarks show that our approach produces diverse and accurate multi-person predictions, significantly outperforming the state of the art. 

Published as a conference paper at ICLR 2023
STOCHASTIC MULTI-PERSON 3D MOTION FORECAST- ING
Sirui Xu
Yu-Xiong Wang" Liang-Yan Yan Gui*
University of Illinois at Urbana- Champaagn
(siruixus yxw, igui ) e 11111111 noiss edu https://siruis github. 1 DDMMF
ABSTRACT
This paper aims deal with the ignored real-world complexities in prior work on human motion forecasting, emphasizing the social properties of multi-person motion, the diversity of motion and social interactions, and the complexity of articulated motion. To this end, we introduce novel task stochastic multi-person 3D motion forecasting. We propose dual-level generative modeling framework that separately models independent individual motion at the local level and social interactions the global level. Notably, this dual-level modeling mechanism can be achieved within a shared generative model, through introducing learnable latent codes that represent intents of future motion and switching the codes' modes of operation at different levels. Our framework is general; we instantiate it with different generative models, including generative adversarial networks and diffusion models, and various multi-person forecasting models. Extensive experiments on CMU-Mocap, MuPoTS-3D and SoMoF benchmarks show that our approach produces diverse and accurate multi-person predictions, significantly outperforming the state of the art.
INTRODUCTION
One of the hallmarks of human intelligence is the ability to predict the evolution of the physical world over time given historical information. For example, humans naturally anticipate the flow of people in public areas, react, and plan their own behavior based social rules, such avoiding collisions. Effective forecasting of human motion has thus become a crucial task in computer vision and robotics, e.g., in autonomous driving (Paden al., 2016) and robot navigation (Rudenko et al., 2018). This task, however, is challenging. First, human motion is structured with respect to both body physics and social norms, and is highly dependent on the surrounding environment and its changes. Second, human motion is inherently uncertain and multi-modal, especially over long time horizons.
Previous work on human motion forecasting often focuses on simplified scenarios. Perhaps the most widely adopted setting is on stochastic local motion prediction of a single person (Mao et al., 2021; Yuan & Kitani, 2020), which ignores human interactions with the environment and other people in the environment. Another related task is deterministic multi-person motion forecasting (Wang al., 2021b; Adeli et al., 2020; 2021; Guo et al., 2022). However, does not take into account the diversity of individual movements and social interactions. In addition, stochastic forecasting of human trajectories in crowds (Alahi et al., 2014) has shown progress in modeling social interactions, e.g., with the use of attention models (Kosaraju et al., 2019; Vemula al., 2018; Zhang al., 2019) and spatial-temporal graph models (Huang et al., 2019; Ivanovic & Pavone, 2019; Salzmann et al., 2020; Yu et al., 2020). Nevertheless, this task only considers motion and interactions at the trajectory level. Modeling articulated 3D poses involves richer human-like social interactions than trajectory forecasting which only needs to account for trajectory collisions.
To overcome these limitations, we introduce a novel task of stochastic multi-person 3D motion forecasting, aiming jointly tackle the aforementioned aspects ignored in the previous work the social properties of multi-person motion, the multi- multi-modalit of motion and social interactions, and
Yuu-Xiong Wang and Liang-Yan Yan Gui contributed equally this work.

---

Published as a conference paper at ICLR 2023
Starting Poses
Ending Poses
(a) Single-Person Fidelity
(b) Multi-Person Fidelity
(c) Overall Diversity
Figure I: Illustration of the multifaceted challenges in the proposed task of stochastic multi pelliiperron 3D motion forecasting. (a) Single-person fidelity: for each person, the predicted pose and trajectory should be realistic and consistent with each other, e-g., to avoid foot floating and skating. (b) Multi- person fidelity: multi-person motion in a scene inherently involves social interactions, egg., to avoid motion collisions. (c) Overall diversity: long-term human motion is uncertain and stochastic; address this intrinsic multi-modality. while existing work (Wang et al., 2021b; Adeli et al., 2020; 2021: Guo al., 2022) simplifies deterministic prediction.
the complexity of articulated motion. As shown in Figure I, this task requires the predicted multiple motion sequences of multi-person to satisfy the following conditions (a) single-person fidelíty: for example, all single-person person motion should be continuous, and the articulated properties should be preserved under the physical rules; (b) multi-person fidelity: the predicted motion should be socially-aware, with the consideration of the interactions between predictions from different people; and (c) overall diversity: the movement of the human body should be as varied as possible, but within the constraints of conditions (a) and (b).
Due to the substantially increased complexity of our task, it becomes challenging to optimize all three objectives simultaneously. We observe simply extending existing work such deterministic motion forecasting cannot address the proposed task. This difficulty motivates us to adopt divide- and-conquer strategy, together with the observation that single-person fidelity and multi-person fidelity can be viewed as relatively independent goals, while there is an inherent trade-off between fidelity and diversity. Therefore, we propose Dual-level generative modeling framework for Multi-person Motion Forecasting (DuMMF). At the local level, we model motion for different people independently under relaxed conditions, thus satisfying single-person fidelity and diversity. Meanwhile, at the global level, we model social interactions by considering the correlation between all motion, thereby further improving nulti-person fidelity. Notably, this dual-level modeling mechanism can be achieved within shared generative model, through simply switching the modes of operation of the motion intent codes (i.e., latent codes of the generative model) at different levels. By optimizing these codes with level-specific objectives, produce diverse and realistic multi-person predictions.
Our contributions can summarized as follows. (a) To the best of our knowledge, we are the first (b) We propose simple yet effective dual-level learning framework to address this task. (c) We introduce discrete learnable social intents dual levels to improve the realism and diversity of predictions. (d) Our framework is general and can be operationalized with various generative models, including generative adversarial networks and diffusion models, and different types of multi-person motion forecasting models. Notably, it can be generalized to challenging more-person (e.g., 18-person) scenarios that are unseen during training.
to investigate the task of stochastic multi-perss multi-person 3D motion forecasting.
2 RELATED WORK
Stochastic Human Motion Forecasting. There have been many advances in stochastic human motion forecasting, many of which (Walker et al., 2017; Yan et al., 2018; Barsoum et al., 2018) are based on the adaptation and improvement of deep generative models such variational autoencoders
2

---

Published as a conference paper at ICLR 2023
(VAEs) (Kingma & Welling, 2013), generative adversarial networks (GANs) (Goodfellow et al., 2014), normalizing flows (NFs) (Rezende & Mohamed, 2015), and diffusion models (Sohl-Dickstein et al., 2015; Song et al., 2020; Ho et al., 2020). Some recent approaches (Bhattacharyya et al., 2018; Dilokthanakul et al., 2016; Gurumurthy et al., 2017; Yuan & Kitani, 2019; 2020; Zhang et al., 2021; Mao al., 2021; Xu al., 2022b; Petrovich al., 2022) emphasize on the promotion of diversity. Mao et al. (2021) sequentially generate the different parts of pose for better controllability of diversity. Xu et al. (2022b) introduce learnable anchors in the latent space to guide the samples with sufficient diversity. Although these methods can predict very diverse human motion sequences, most of them are limited to local motion and ignore the global trajectory. Some of their produced motion sequences are actually unrealistic; in particular, incorporating global trajectories may result in severe foot skating. Predicting human motion under the constraint of scene context (Cao et al., 2020; Hassan et al., 2021; Zhang & Tang, 2022) has recently been explored, where the effect of global trajectories and scenes on human motion is considered. Instead of predicting single-perse movement, our work focuses on diverse multi-person movements and social interactions.
Multi-Person Forecasting. So far, research on multi-person forecasting has mainly focused on global trajectory forecasting (Helbing & Molnar, 1995; Mehran et al., 2009; Pellegrini al., 2009; Yamaguchi et al., 2011; Zhou et al., 2012; Alahi et al., 2014; 2016; Lee et al., 2017; Gupta et al., 2018; Sadeghian al., 2019; Amirian et al., 2019; Kosaraju al., 2019; Sun et al., 2020; Mangalam et al., 2020; Sun et al., 2021; Kothari et al., 2021; Tsao et al., 2022). Gupta et al. (2018) utilize a tanner-takes-all loss (Rupprecht al., 2017) with a socially-aware GAN. We also adopt these two loss functions in our dual--evel modeling. Kothari et al. (2021) introduce discrete social anchors and revise the distribution of trajectories by the predefined anchors. Unlike their hand-crafted anchors, our discrete social intent codes are learnable components obtained through the dual-level optimization. Some recent attempts address deterministic multi person 3D motion forecasting. Adeli al. (2020; 2021) use additional contextual information to aid multi-person forecasting and propose Social Motion Forecasting (SoMoF) Benchmark. Guo et al. (2022) utilize cross-attention for two-person motion forecasting, while Wang et al. (2021b) introduce can be generalized to more persons. Please refer to Sec. of the Appendix for additional discussion.
multi-person
transformer architecture that
METHODOLOGY
In this section, explain the proposed dual-level generative modeling framework (DuMMF) for our task. As illustrated in Figure 2, our key insight is to decouple the modeling of independent individual movements at the local level and social interactions the global level (Sec. 3.1). Within shared forecasting model, we achieve this by (a) introducing learnable latent codes that represent intents of future movement (Sec. 3.2). (b) switching the codes' modes of operation different levels (Sec. 3.1 and Sec. 3.2), and (c) training with level-specific objectives (Sec. 3.3). Our framework is general and can be operationalized with different types of motion forecasting models, and we summarize the multi-persen motion predictors used in this paper (Sec. 3.4).
Problem Formulation. We denote the input motion sequence of length Th, for N persons in a scene as (X,, 1 where [4) is the pose of n-th person at time step f. We aim to predict M future motion sequences of length Tp. denoted as { {( J. 1m=-1 where Y [ + 1)..... Ý [TT +Tpll is the m-th predicted motion of n--hh person. We use the 3D coordinates to represent the absolute joint position of V joints, hence Vn, m. t, [ [f]. [t] E IR Vx3 We assume to be given the ground truth motion of N persons as (Y,) -- Our goal is to forecast multiple realistic yet diverse future motion sequences, such that (a) all the M predictions represent human-like motion, simultaneously satisfying single-person fidelity and multi-person fidelit;
(b) the predictions are diverse (overall diversity); and (c) one of the predicted sequences is as close to the ground truth possible.
3.1 DUAL-LEVEL STOCHASTIC MULTI-PERSON MOTION FORECASTING: OVERVIEW
Basic Generative Modeling Framework. Stochastic multi-person future motion can be modeled as a joint distribution using deep generative models (Goodfellow et al., 2014). Accordingly, we denote this joint distribution of the future motion of N persons as p( (YY, Yz...., YN X1, X Xv). where all future movements (Y,)) > are conditioned
the past sequences {X,, ) of all per- sons. Typically, we can use a latent code z ~ p(z) to reparameterize this joint distribution as p( ((YY))= (x,,) ) = p( (Y, N. x |z, (X, J.V (o=))p(z) (z) Here, the latent code can be inter-
3

---

Published as a conference paper at ICLR 2023
Decoder
Continous Intents
Discrete Intents
Sampler
Multi-penson Embeddings
Encoder
Input
Ourput
(c) Sampler with discrete/leamaalee and continuous intents
(a) Local-Level Modeling
Decoder
Embooking X,
Sampler
Intents
Multi-person
Local Encoder
Global Encoder
Emheddings
Encoder
Inpul
Outpul
(b) Global-Level Modeling
(d) Encoder
Figure 2: Overview of our proposed dual-level generative modeling with motion inputs of three persons as an illustration. (a) We combine encoded multi-pere pelli-perronn embeddings with independent intent codes at the local level of modeling individual motion. (b) Differently, the global level of modeling social interactions requires all latent codes to be same. (c) The latent codes comprise both discrete intent codes, which are learned from the data and represented (d) We abstract the encoder of a multi-person predictor as the combination of a local branch that encodes single-person motion and global branch that encodes multi-person motions (see Table A of the Appendix for instantiations).
set, and continuous intent codes.
preted social intent that guides the future movements of multiple persons. With this social intent z sampled from the given distribution p(z). a deterministic neural network 9, with parameters can map the historical conditions { to prediction (Y)) under the latent distribution po( ( IN
( i2, [X.) J. which is formulated as
~~ p(z), (Y)) Go(z, (X,, I
(1)
Given the complexity of our task, it becomes challenging to simultaneously ensure all objectives (i.e., the fidelity of a single person, the fidelity of multiple persons, and the overall diversity). To overcome this difficulty, introduce dual-level modeling mechanism that explicitly decomposes the task objectives into local modeling of independent individual movements and global modeling of social interactions. Notably, we achieve this by simply switching the modes of operation for the latent codes z w.r.t. different levels of modeling, without any change to the model architecture G.
Local-Level Modeling: Individual Motion. At this level, the generative model Go models all human bodies as independent of each other, and we aim to improve the overall diversity and the single-person fidelity, alleviating problems such as predicting unrealistic poses. Here, the joint distribution of future human motions can be rewritten in the form of all single-person marginal distributions, i.e., P([Y,, n n== n M==11X,, J = II p(Y,, X,). To this end, as shown in Figure 2(a), we consider Zn independently drawn from p(z) to generate
leveraging N different individual intents 21, independent future movements, denoted
Zi. Zn ~ p(z), (Y IN
Go((2n) N (X,) I
(2)
Social Interactions. Going beyond the local individual level, the generative model Go at the global level takes into account the social behavior of multiple people to model their joint distribution. The goal is to further improve the multi-person fidelity, e.g., promoting the overall accuracy. As illustrated in Figure 2(b), to maintain the network architecture G unchanged, we still use N individual intents as input. However, different from the local level, we constrain these N individual intents to be the same, representing social intents that stand for correlations between the intents of multiple persons. Formally, we have
Global-Level Modeling:
p(z): (Y (o((2) (X,)
4

---

Published as a conference paper at ICLR 2023
Note that, without additional constraints, this dual-Ievel modeling scheme by itself is not guaranteed to enforce the latent codes to behave in the designed manner. To this end, we introduce learnable latent intent codes (Sec. 3.2), jointly optimize the codes and the forecasting model G guided by the level-specific training objectives (Sec. 3.3).
3.2 DISCRETE LEARNABLE HUMAN INTENT CODES
Intuitively, arbitrary, albeit identical, individual intent in Eq. 3 may not adequately lead to valid social intent. We thus hypothesize that social intent is formed when all individual intents are the same and belong within some range of " "options." This can typically be achieved through discrete choice models [Aguirregabiria & Mira, 2010; Ryan & Gerard, 2003; Bhat et al., 2008; Leonardi, 1984)- - an effective tool that predicts choices from a set of available options created by hand-crafted rules. Here, we formulate the correlation of multiple persons at the global level by using the same discrete code. However, the intent options for social interactions are more subtle and difficult to define manually than those in other applications such as for trajectories (Kothari et al., 2021). Therefore, we use a set of learnable codes z" E {Znh L.M Im=1 to represent social intents, inspired by Xu et al. (2022b). However, introduce different training strategies that are tailored to this new task (Sec. 3.3). Our motivations are: (a) Subject to physical constraints and social laws, the intents of future movements should share some deterministic patterns. For example, all intents should avoid imminent collisions anticipated from the history, even these intents refer to different motions. We assume that such deterministic properties shared by social intents can be represented by set of shareable codes learned directly from the data. (b) It will be easier for the predictor to identify and implement different levels of functionality by jointly optimizing discrete intents and the predictor. To further enhance the expressiveness of the codes, we retain the original continuous Gaussian noise z" ~ p(z) of the generative model Go and bundle the discrete intent with the noise to represent the final intent, as shown in Figure 2(c). Now the global-level modeling of social interactions in Eq. 3 is reformulated as
o zn ~ p(z), zd E (Z 1 IM=1,
(Yr = Go((2, + z d (
(4)
And correspondingly, the local-level modeling of individual motion in Eq. 2 becomes
ii ....... 22 p(z), 2 ......, 2 E ( Jm=1: (Y Go (1z + + (X,,
1M
(5)
3.3 TRAINING UNDER THE GUIDANCE OF LEVEL-SPECCIIC OBJECTIVES
Using only the same discrete intent does not naturally and necessarily inherit the multi-person correlation. Thus, we optimize both the parameters of the predictor and the discrete codes with a level-specific training strategy. We jointly train both levels of individual movement modeling and social interaction modeling, but with each level guided by its own objective. each forward pass, we explicitly produce different output predictions from different intents of the two levels. And then, in the backward pass, the discrete intent codes z are optimized separately at different levels, while the parameters 0 of the forecasting model G are updated based on the fused losses from the two levels.
Lecel-Level Training. At the local level, we train the model without social interactions. Given independent multi- celli--person motion data ((X,, J.W (Y,, } we first randomly sample the discrete intent codes and merge them with independently sampled continuous intent codes into M x N different latent codes {(z," J 1m We then use each intent and the past motion of each person 1== J for each person (X,) Local-level objectives consider single-person fidelity and overall diversity respectively.
to predict M future motion sequences ({Y.)) N
Global-Level Training. Meanwhile, training is also conducted at the global level to enable the modeling of social interactions. The difference from the local setting is that we incorporate the discrete and continuous codes into only A distinct latent codes {{zT) [< > latent codes zzZ' of all N individuals are the same for certain m-th prediction. Here, we introduce learning objectives to facilitate multi-person fidelity and accuracy.
1M JM hence, the discrete
Please refer Sec. of the Appendix for more detail on the aforementioned level-specific learning objectives. Also, Sec. G of the Appendix further demonstrates that these learning objectives are important, and in some cases critical, to the fidelity and diversity of
motion.
Inference. During inference, we only use the global-level strategy by sampling the same intent for all individuals present in the scene. Note that the uncertainty of human motion substantially
5

---

Published as a conference paper at ICLR 2023
Starting Poses
Ending Poses
Figure 3: Qualitative results of DuMMF with a DDPM. We demonstrate the generalizability of our method to handle significantly more complex scenario with 18 persons. Note that our model is trained only on 3-person data. We visualize the predicted final poses at 2 seconds.
Table 1: Quantitative results of DuMMF with DDPM. Both the baseline and our models are trained using SMPL-X representations AMASS, and we convert them skeletons for evaluation. Using the same backbone and generative model, our DuMMF framework significantly provides more accurate predictions with more intents.
Method, of Insents
@25 Frimes
655 Frumes
#77 frames
ADE
FDE
FDE
DDPM (Ho et al., 2120), NiA
2,021
4.008
3,786
7.585
5.434
11.111
DuMME (Ouns),
2.405
4.814
3.992
7.338
5.197
8.010
(Ours),
3.316
2.700
4.823
increases with the time horizons of the forecasting. So instead of predicting fixed number of diverse outputs (Mao al., 2021; Yuan Kitani, 2020), in our evaluation, the number grows with the length of the prediction. To this end, we employ autoregressive inference (Wang et al., 2021b) with progressive diversity. Given the past motion sequence (X, 1N and M sampled intent codes, the model outputs M predictions for each person { {(Ym) 1MM
history, the previous predictions {(X,, Ym) M/ In=1 m==1- And then, given a combination of outputs M predictions {{ {(Ym) L }M2
Jmm and M new intent codes as input, the model
Jn=1 m= etc.
3.4 NETWORK ARCHITECTURE
Our dual-level modeling framework in conjunction with the discrete learnable intent codes is general and, in principle, does not not rely specific network architectures generative models. To demonstrate this, we combine our framework with various types of deterministic multi-person motion predictors and different generative models, yielding consistent and significant improvements across all baselines (see Sec. 4). As shown in Figure 2, we abstract the encoder of the multi-person motion predictor into two parts: the local part is responsible for encoding single-person motion, while the global part is responsible for encoding multi-person motion and its interactions. A summary of the multi-person predictors used in the paper is given in Table A of the Appendix.
EXPERIMENTS
Datasets. In the main paper, we show the evaluation on two motion capture datasets, CMU- Mocap (CMU) and MuPoTS-3D (Mehta et al., 2018). CMU-Mocap consists of movement sequences with up to two subjects for each scene. It contains 2,235 recordings performed by 144 different subjects, eight of which include double-person motion. We directly adopt these two-person motions for comparisons in two-person scenarios. For skeletal representation, we follow Wang et al. (202lb) for the train/test split and the preprocess mix single-person and double-person motion together to synthesize a 3-person motion each scene. Moreover, we construct an even more challenging scenario where mix up more people per scene. Please refer Sec. H of the Appendix for the detail of this generalized setting. For meshes generated from SMPL-X representations (Pavlakos al., 2019), we extract the multi-person data in CMU-Mocap from AMASS (Mahmood al., 2019)

---

Published as a conference paper at ICLR 2023
Start
GT
DuMMF (Ours)
MRT
Figure 4: Qualitative results of DuMMF with CGAN model. The leftmost column is the ground truth starting poses, and the second column is the ground truth poses three seconds later. We show our five sampled 3-second predictions in the middle. Our model produces diverse predictions, with one closer to the ground truth (highlighted by the red box) compared with MRT (rightmost column).
Table 2: Quantitative comparison between our DuMMF and deterministic forecasting baselines and their CGAN stochastic variants, using skeletal representations on CMU-Mocap. The number of intents is set to 5 for stochastic forecasting in 3-person (top) and 2-person (bottom) scenarios. DuMMF significantly improves multi-person accuracy and diversity across various architectures and deterministic predictors. Additionally, our discrete and continuous intent codes are complementary to each other in most cases.
Archrecture
Predictor
Divzrstber
Vartanss
FFPIT
RT 25
XTIF
Detmmimaii:
NiA
1:25T
2:315
1.51%
1:779
3.282
RNN
MPF (Adeli 2020)
CGAN+ (Oura) (Oura)
we Continouss Full
wi Discrnee
1.051
2.05
0.778
0.735
0.811
1.368
1.213
1.225
2,000
1.730
1,589
1.370
2.230
1.325
1,287
3.320
1.513
1.791
2.500
2.377)
1.011
0.702
1.135
2,035
1.081
1.092
3/967
1.388
2. 3,082
Detminaii
CGAN
N/A
2.654
104
0.681
07300
1.111
11137
1.036
NFA
1,000
1-131
1.640
1-306
1.740
1473
1.845
1.5055
17507
2.004
(Adeli 2020)
Wil Sequraam
wlo
3021
07718
1.110
1.302
2.304
1.3015
1.351
1.206
1.563
Tranfonuz
Deleminatic: CCAN
NAA
Fill
0.88
0.242
NAA
0.671
0,881
0.007
1.055
1.125
1.886
0.82
1.797
N6A
1,082
1/223
1.206
1.367
0.716
1:144
N5A
1.427
1.287
2.456
1,900
1.911
MRT (Wang et 2021N)
CGAN- (Ours)
WTO Sepeanion
wh Diserziic
Continuous
9,003
1.6
2231
0,272
1.110
1.085
1.09%
10076
1,000
1.224
1.584
1.671
1.:14
3.301
1.325
1.555
27021
Delemansti
CCAN
NIA
0.055
NDA
0.688
1.135
1.419
1.591
10A
1.302
21224
1.250
N6A
1.024
1:485
3.201
MRT (Wang al,
Sepuraton
07155
1009
1.250
1.224
1.200
CGAN
2.277
1,010
1.700
0.971
Tranfoome:
Detemmnslic CCAN
N/A
Full
2.18
1211
N/A
07270
0.660
0.656
11100
1.052
1,68
2.595
1.206
NDA
0.992
1.044
1.22
1.470
10700
1.651
3,718
1.660
N7/5
1.000
1.510
1.235
3.727
1.798
2.111
XIA (Gin 2022)
CGAN-DOMMEE (Ours)
BID Meporaionn
wio
1121
0.415
0700
1005
1.64.
1015
1.530
1.663
17.00
1.565
1:11
3701
Full
1.145
1.010
1.429
1.197
1.764
and follow the same strategy to mix single- and double-persed motion together. MuPoTS-3D consists of more than 8,000 frames with up to three subjects. We convert the data to the same 15-joint human skeleton and length units as CMU-Mocap, and evaluate the generalization MuPoTS-3D of model trained only on CMU-Mocap. We also report our performance on the SoMoF benchmark (Adeli al., 2020; 2021) in Sec. G of the Appendix.
Metries. For evaluating multi-person motion accuracy and diversity, we adopt the common metrics used in stochastic forecasting (Mao et al., 2021; Yuan & Kitani, 2020: 2019; Salzmann al., 2020; Kothari et al., 2021) as follows. For accuracy measurement, we follow the Best-of-N (BoN) evaluation and use (a) Average Displacement Error (ADE): the average & distance over time between the ground truth and the prediction closest to the ground truth; (b) Final Displacement Error (FDE): the l2 distance between the final pose of the ground truth and the last predicted pose closest to the ground truth. For diversity measurement, we employ (c) Final Pairwise Distance (FPD): the average & distance between all predicted final pose pairs. We disentangle the local pose and the global trajectory of the motion and measure their accuracy and diversity separately by defining the following metrics: rootADE, rootFDE, AD, PDseFDE, rootFPD, and PoseFPD. To measure three different aspects including single-person person fidelity, multi person fidelity, and overall diversity comprehensively. we provide a summary and analysis of all metries for this novel task in Table F and Sec. F of the Appendix. We include results on all above metrics in Sec. G of the Appendix.
multi-person
7

---

Published as a conference paper at ICLR 2023
Figure 5: Qualitative results of DuMMF with a DDPM model. We visualize the predicted frames for two different 3-person input motions, which listed on the left and right respectively. For each input, we generate four sampled motions, arranged in a single column and listed sequentially in time at 0, 1, 2, and seconds. Our method effectively produces diverse human-like social interactions.
Baselines. Given that we propose new task, there is no readily available baseline from existing work. For comparison purposes, we customize several baselines from deterministic multi-person motion forecasting which we summarize in Table A, and extend them to stochastic forecasting. (a) RNN+SC-MPF (Adeli et al., 2020): We use Social Pooling (Alahi et al., 2016) to integrate the multi-person information. We follow the implementation in Adeli et al. (2020) and reproduce the results. For the Social Pooling module, we select the maximum pooling for its best performance as ablated in Adeli al. (2020). (b) Transformer+SC-MPE. For a fair comparison, we introduce transformer- based encoder architecture to combine with the Social Pooling module. (c) XIA (Guo al., 2022): We follow their implementation and use Cross Interaction Attention (XIA) to encode 2-person information.
(d) MRT (Wang et al., 2021b): We use multi-range transformer (MRT) as our main architecture for its state-of-the-an performance. We further extend MRT 2-person scenarios to compare with XIA. For stochastic forecasting, we formulate a conditional generative adversarial network (CGAN) (Mirza Osindero, 2014) and denoising diffusion probabilistic model (DDPM) (Ho et al., 2020), with or without dual-level modeling and discrete intents.
Implementation Details. For the skeletal representation, following Wang et al. (2021b), we train the model to predict a 15-frame sequence of 3 people given the ground truth 15, 30, and 45 past frames at 15Hz, as the encoder (RNN/Transformer) accepts different input lengths. We use a 6-layer transformer (or RNNs), where we set the feature dimension to 128. For evaluation, we recursively predict the next 15 frames times given all past frames generated, as illustrated in Sec. 3.3. Thus, given the number of intents to be M, the model outputs M, 2 and M different predictions in sequence. For the SMPL representation, we train the model to predict a 25-frame sequence people given the 10 past frames at 30Hz. We use an 8-layer transformer, where we set the feature dimension to 512. For evaluation, similarly, we use 10 frames as the past motion and recursively predict the next 25 frames. Additional implementation details are provided in Sec. E of the Appendix.
We compare our method with a pure DDPM in Table Our improvement is significant. Notably, even in the case of single intent, where we only evaluate one prediction, our method outperforms DDPM in long-term generation. As the number of intents increases, our method provides more accurate results, especially in long-term prediction. In Table 2, we demonstrate that our DuMMF framework benefits all predictor variants. We observe that the simple combination of deterministic predictors and CGAN results in very low diversity and accuracy. By contrast, our full approach significantly outperforms CGAN baselines on both diversity and accuracy across all backbones. In Table of the Appendix, we further show that DuMMF achieves the best generalization results across all predictors MuPoTS-3D. highlighting its generality and superiority.
Quantitative Results.
A Ablation: Effectiveness of Dual-Level Modeling, Tables 2 and show the effectiveness of our dual-level framework. First, we investigate the settings with only single-person motion modeling or only social interaction modeling. In Table 3, compared with our full method, modeling independent multi-person motion ('w/o Social') provides higher diversity but leads to inaccurate poses, since the social restrictions are not considered. With only social interaction modeling, the model ('w/o Individual") cannot output sufficiently diverse predictions, which also makes predictions inaccurate,
8

---

Published as a conference paper at ICLR 2023
Table 3: Ablation study of our DuMMF with CGAN and MRT (Wang al., 2021b) using skeletal representations on CMU-Mocap and MuPoTS-3D. We report both accuracy and diversity for root and pose separately. The results show the effectiveness of our dual-level modeling along with discrete motion intents, and complementariness of local-Ievel and global-level modeling.
CMU-Mocap (CMU)
Generalized to MuPoTS-3D (Mohia et al., 2018)
Method @t- 3x
ROOT
POSE
ROOT
POSE
FPD1 1
ADR
FDR
ADE
PDE
FPD 1
ADE
PDE
FPD 1
ADE
PDE
MRT (Wang et 2021b)
N/A
NIA
N/A
0.603
0.220
0.461
Ours, Deteministte
N/A
0.738
0.916
N/A
0.288
0.526
N/A
0.587
0.809
N/A
0.203
0.411
w/o Individual
0.178
0.733
0.896
0.339
0.264
0.453
0.120
0.584
0.797
0.253
0.180
0.331
weo Social
1661
0.751
0.909
1,077
0.264
0.413
2,100
0.605
0.818
1.283
0.183
0.318
w/o Separation
0.167
0.293
0.271
0.448
0.105
0.596
0.215
0.202
0.986
Full
0.249
0.734
0.898
0.562
0.243
0.390
0.187
0.588
0.796
0.444
0.170
0.305
w/o Discrete
0:114
0.761
0.999
0.196
0.279
0,490
0.083
0.620
0.857
0.158
0.212
0411
wio Continious
0.2118
0.746
0.905
0.381
0.265
0.439
0.159
0.994
0.801
0.328
0.196
0.378
Full
0.249
0.734
0.898
0.562
0.243
0,390
0.187
0.588
0.796
0,444
0.170
0.305
as more varied outputs have a better chance of covering the ground truth. Note that the results of "CGAN' and 'w/o Separation' in Table 2 and 'w/o Separation' in Table 3 are worse since they simply use all the learning objectives together without disentangling the two levels of modeling, while 'w/o Separation' is slightly better due to the use of discrete intents. Under our dual-level framework with level-specific motion intents and learning objectives, the model more effectively incorporate the benefits of both levels, thus leading to improved accuracy and diversity. In Sec. G of the Appendix, we further demonstrate that our dual-Ievel benefits different predictor variants.
Ablation: Effectiveness of Discrete Human Intents. In Tables and 3, we also demonstrate that discrete human intents are effective and crucial. We observe the best results when using both discrete and continuous intents, indicating that they are complementary. In the absence of discrete intents ('w/o Discrete'), the performance is only comparable with the baseline ('CGAN'). Importantly, with the help of discrete intents, the improvement of dual--leel modeling ('Full') over 'w/o Separation' more pronounced, compared with the improvement of 'w/o Discrete' over 'CGAN Therefore, discrete learnable intents are essential for effectively integrating the advantages of both levels during training. The performance without continuous intents ('w/o Continuous') is slightly worse than the full method. Our hypothesis is that relying solely on discrete intents is limiting, because they only support a finite number of outputs. In Sec. G of the Appendix, we further investigate how the number of discrete intents impacts stochastic forecasting.
Qualitative Results. Consistent with the quantitative evaluation above, we observe that our method provides diverse multi-person person motion, and produces predictions closer to the ground truth compared with the deterministic method MRT in Figure 4. In Figure 5, we qualitatively show that our gener- ated results in meshes reflect real-world diversity of social interactions. Furthermore, we provide qualitative results for more-person scenarios in Figure 3, and others in Figure A and Figure B of the Appendix. Please refer to Sec. of the Appendix for more detail on the more-person setting.
Limitation. Although our dual-level framework has proven effective in producing high-quality and diverse predictions, we have observed artifacts such as foot skating in some predicted motion sequences. This is because our model relies solely on loss functions to constrain motion, rather than explicitly modeling articulated motion. As this is a common issue in learning- based methods, we plan to exploit physical simulator to further improve the plausibility of our predicted motion.
5 CONCLUSION
We formulate a novel task called stochastic the real-world human motion complexities. To simultaneously achieve single-person fidelity, social realism, and overall diversity, we propose dual-level generative modeling framework (DuMMF) with learnable latent intent codes. Compared with prior work on deterministic or single-person prediction, our model learns to generate diverse and realistic human motion and interactions. Notably, our framework is model-agnostic and generalizes to unseen more-persoe scenarios.
3D motion forecasting, which better reflects

---

Published as a conference paper at ICLR 2023
Ethics Statement. Our proposed technique is useful in many applications, such selffdriving to avoid crowds. The potential negative societal impacts include: (a) our approach can be used to synthesize highly realistic human motion, which might lead to the spread of false information; (b) our approach requires real behavioral information as input, which may raise privacy concerns and result in the disclosure of sensitive identity information. Nevertheless, processed human skeleton representation that contains minimal identifying information, unlike raw data. On the positive side, this can be seen as a privacy-enhancing feature.
model operates the
Acknowledgement. This work was supported in part by NSF Grant 2106825, NIFA Award 2020- 67021-32799, the Jump ARCHES endowment through the Health Care Engineering Systems Center, the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana- Champaign through the NCSA Fellows program, the IBM-IIlinoss Discovery Accelerator Institute, the Illinois-Insper Partnership, and the Amazon Research Award. This work used NVIDIA GPUs at NCSA Delta through allocation CIS220014 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF Grants #2138259, #2138286, #2138307, #2137603, and #2138296.
REFERENCES
CMU graphics lab motion capture database. http: http::///cocpp../// cs. cmu. edu/. 6, 17, 19,
Vida Adeli, Ehsan Adeli, Ian Reid, Juan Carlos Niebles, and Hamid Rezatofighi. Socially and contextually aware human motion and pose forecasting. RAL, 2020. 1, 3, 7, 8, 14, 15, 17, 18, 19
Vida Adeli, Mahsa Ehsanpour, Ian Reid, Juan Carlos Niebles, Silvio Savarese, Ehsan Adeli, and Hamid Rezatofighi. TRiPOD: Human trajectory and pose dynamics forecasting in the wild. arXiv preprint SttTv:2104.00022 2021. 1, 3, 7, 14, 18, 19
Victor Aguirregabiria and Pedro Mira. Dynamic discrete choice structural models: A survey, Journal of Econometrics, 2010.
Alexandre Alahi, Vignesh Ramanathan, and Li Fei-Fei. Socially-aware large-scale crowd forecasting. In CVPR, 2014. 3, 14
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social LSTM: Human trajectory prediction in crowded spaces. In CVPR, 2016. 3, 8
Javad Amirian, Bernndd Hayet, and Julien Pettré. Social Ways: Learning multi-modal distribu- tions of pedestrian trajectories with gans. In CVPRW, 2019. 3
German Barquero, Johnny Núnez, Sergio Escalera, Zhen Xu, Wei-Wei Tu, Isabelle Guyon, and
Cristina Palmero. Didn't 't see that coming: a survey on non-verbal social human behavior forecasting. In Understanding Social Behavior in Dyadic and Small Group Interactions. PMLR, 2022. 14
Emad Barsoum, John R. Kender, and Zicheng Liu. HP-GAN: Probabilistic 3d human motion prediction via gan. In CVPRW, 2018. 2
Chandra Bhat, Naveen Eluru, and Rachel Copperman. Flexible model structures for discrete choice analysis. Handbook of Transport Modelling, 2008. 5
Apratim Bhattacharyya, Mario Fritz, and Bernt Schiele. Accurate and diverse sampling of sequences based on "best of many" sample objective. In CVPR, 2018. 3
Zhe Cao, Hang Gao, Karttikeya Mangalam. Qi-Zhi Cai, Minh Vo, and Jitendra Malik. Long-tern human motion prediction with scene context. In ECCV, 2020. 3
Nat Dilokthanakul, Pedro A. M. Mediano, Marta Garnelo, M. J. Lee, Hugh Salimbeni, Kai Arulku- maran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02645. 2016. 3
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NewrIPS, 2014. 3
10

---

Published as a conference paper at ICLR 2023
Tianpei Gu, Guangyi Chen, Junlong Li, Chunze Lin, Yongming Rao, Jie Zhou, and Jiwen Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. In CVPR, 2022. 15
Wen Guo, Xiaoyu Bie, Xavier Alameda-Pinees Pineda, and Francesc Moreno-Noguer. Multi-person extreme motion prediction. In CVPR, 2022. 3, 7, 8, 14, 15,
Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi. Social GAN: Socially acceptable trajectories with generative adversarial networks. In CVPR, 2018. 3
Swaminathan Gurumurthy, Ravi Kiran Sarvadevabhatla, and R. Venkatesh Babu. DeLiGAN
Generative adversarial networks for diverse and limited data. In CVPR, 2017. 3
Abner Guzmán-rivera, Dhruv Batra, and Pushmeet Kohli. Multiple choice learning: Learning to produce multiple structured outputs. In NeurlPS, 2012. 15
Mohamed Hassan, Duygu Ceylan, Ruben Villegas, Jun Saito, Jimei Yang, Yi Zhou, and Michael Black. Stochastic scene-aware motion prediction. In ICCV, 2021.
Dirk Helbing and Peter Molnar. Social force model for pedestrian dynamics. Physical review E, 1995.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020. 3, 6, 8, 16
Yingfan Huang, Huikun Bi, Zhaoxin Li, Tianlu Mao, and Zhaoqi Wang. STGAT: Modeling spatial- temporal interactions for human trajectory prediction. In ICCV, 2019.
Boris Ivanovic and Marco Pavone. The Trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs. In ICCV, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014. 17
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In arXiv preprint arXiv:1312.6114, 2013. 3
Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black. VIBE: Video inference for human body pose and shape estimation. In CVPR, 2020. 16
Vineet Kosaraju, Amir Sadeghian, Roberto Martín Martin,, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-BiGAT: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. arXiv preprini arXiv: 1907.03355 2019. 1,
Parth Kothari, Sven Kreiss, and Alexandre Alahi. Human trajectory forecasting in crowds: A deep
learning perspective. In arXiv preprint arXiv:2007.03633 2020. 14, 18
Parth Kothari, Brian Sifringer, and Alexandre Alahi. Interpretable social anchors for human trajectory
forecasting in crowds. In CVPR, 2021. 3, 5, 7, 14
Namhoon Lee, Wongun Choi, Paul Vernaza, Christopher B Choy, Philip HS Torr, and Manmohan Chandraker. Desire: Distant future prediction in dynamic scenes with interacting agents. In CVPR, 2017. 3
Giorgio Leonardi. The stucture of random utility models in the light of the asymptotic theory of extremes. Theory and decision, 1984. 5
Jiachen Li, Fan Yang, Masayoshi Tomizuka, and Chiho Choi. EvolveGraph: Multi-agent trajectory prediction with dynamic relational reasoning. In NewrIPS, 2020. 14
Naureen Mahmood, Nima Ghorbani. Nikolaus F. Troje, Gerard Pons-Moll, and Michael Black. AMASS: Archive of motion capture as surface shapes. In ICCV, 2019. 6
Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. is not the journey but the destination: Endpoint conditioned trajectory prediction. In ECCV, 2020. 3
11

---

Published as a conference paper at ICLR 2023
Wei Mao, Miaomiao Liu, and Mathieu Salzmann. Generating smooth pose sequences for diverse human motion prediction. CVPR, 2021. 1,3, 6, 15, 18
Ramin Mehran, Alexis Oyama, and Mubarak Shah. Abnormal crowd behavior detection using social force model. In CVPR, 2009. 3
Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, and Christian Theobalt. Single-shot multi-person 3D pose estimation from monocular RGB. 3DV, 2018. 6, 9, 17, 19
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprini arXiv:1411.1884 2014. 8, 16
Brian Paden, Michal Cáp, Sze Zheng Yong, Dmitry S. Yershov, and Emilio Frazzoli. A survey of motion planning and control techniques for self-driving urban vehicles. IEEE Transactions on Intelligent Vehicles, 2016. 1
Behnam Parsaeifard, Saeed Saadatnejad, Yuejiang Liu, Taylor Mordan, and Alexandre Alahi. Learn- ing decoupled representations for human pose forecasting. In ICCV, 2021.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, 2019. 6
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll never walk alone: Modeling social behavior for multi-target tracking. In ICCV, 2009. 3
Mathis Petrovich, Michael J. Black, and Gül Varol. TEMOS: Generating diverse human motions from textual descriptions. In ECCV, 2022. 3
Danilo Jimenez Rezende and Shakir Mohamed, Variational inference with normalizing flows. In ICLR, 2015. 3
Andrey Rudenko, Luigi Palmieri, and Kai Oliver Arras. Joint long-term prediction of human motion using a planning-based social force approach. In ICRA, 2018.
Christian Rupprecht, Iro Laina, Robert DiPietro, Maximilian Baust, Federico Tombari, Nassir Navab, and Gregory D Hager. Learning in uncertain world: Representing ambiguity through multiple hypotheses. In ICCV, 2017.
Mandy Ryan and Karen Gerard. Using discrete choice experiments to value health care programmes: current practice and future research reflections. Applied health economics and health policy, 2003.
Armin Saadat, Nima Fathi, and Saeed Saadatnejad. Towards human pose prediction using the encoder-decoder LSTM. In ICCVW, 2021. 14
Amir Sadeghian, Vineet Kosaraju, Ali Sadeghian, Noriaki Hirose, Hamid Rezatofighi, and Silvio Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints.
In CVPR, 2019. 3
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++: Dynamically- feasible trajectory forecasting with heterogencous data. In ECCV, 2020. 1, 14
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015. 3
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020. 3
Jianhua Sun, Qinhong Jiang, and Cewu Lu. Recursive social behavior graph for trajectory prediction. In CVPR, 2020. 3
Jianhua Sun, Yuxuan Li, Hao-Shu Fang, and Cewu Lu. Three steps to multimodal trajectory prediction: Modality clustering, classification and synthesis. arXiv preprint arXiv:2103.07885 2021. 3
12

---

Published as a conference paper at ICLR 2023
Li-Wu Tsao, Kai Wang. Hao-Siang Lin, Shuai Hong- Han, Lai-Kuan Wong, and Wen-Huang Cheng. Social-SSL: Self-supervised crosensequence representation learning based on transformers for multi-agent trajectory prediction. In ECCV, 2022.
Anirudh Vemula, Katharina Muelling, and Jean Oh. Social Attention: Modeling attention in human crowds. In ICRA, 2018.
Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll, Recovering accurate 3d human pose in the wild using imus and moving camera. In ECCV, 2018.
19
Jacob Walker, Kenneth Marino, Abhinav Gupta, and Martial Hebert. The pose knows: Video forecasting by generating pose futures. In ICCV, 2017.
Chenxi Wang. Yunfeng Wang, Zixuan Huang, and Zhiwen Chen. Simple baseline for single human motion forecasting. In ICCV, 202 14
Jiashun Wang, Huazhe Xu, Medhini Narasimhan, and Xiaolong Wang. Multi-person motion prediction with multi-range transformers. In NeurlPS, 2021b. 1, 2, 3, 6, 7, 8, 9, 14, 15, 17, 18, 19, 20
Chenxin Xu, Maosen Li, Zhenyang Ni, Ya Zhang, and Siheng Chen. GroupNet: Multiscale hy- pergraph neural networks for trajectory prediction with relational reasoning. In CVPR, 2022a.
15
Sirui Xu, Yu-Xiong Wang, and Liang- Yan Gui. Diverse human motion prediction guided by multi- level apatial-temporee anchors. ECCV, 2022b. 3.5
Yi Xu, Dongchun Ren, Mingxia Li, Yuchai Chen, Mingyu Fan, and Huaxia Xia. Tra2Tra: Trajectory- to-trajectory prediction with global social spatial-temporal attentive neural network. IEEE Robotics and Automation Letters, 2021.
Yi Xu, Lichen Wang, Yizhou Wang, and Yun Fu. Adaptive trajectory prediction via transferable gnn. In CVPR, 2022c. 15
Kota Yamaguchi, Alexander C Berg, Luis E Ortiz, and Tamara L Berg. Who are you with and where are you going? In CVPR, 2011.3
Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap, Ersin Yumer, and Honglak Lee. MT-VAE: Learning motion transformations to generate multimodal human dynamics. In ECCV, 2018. 2
Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In ECCV, 2020.
Ye Yuan and Kris Kitani. Diverse trajectory forecasting with determinantal point processes. arXiv preprint arXiv:1907.cm 190700677 2019. 3, 7,
Ye Yuan and Kris Kitani. DLow: Diversifying latent flows for diverse human motion prediction. In ECCV, 2020. 1, 3, 6, 7, 15, 18
Pu Zhang, Wanli Ouyang, Pengfei Zhang, Jianru Xue, and Nanning Zheng. SR-LSTM: State refinement for Istm towards pedestrian trajectory prediction. In CVPR, 2019.
Yan Zhang and Siyu Tang. The wanderings of odysseus in 3d scenes. In CVPR, 2022. 3
Yan Zhang, Michael J Black, and Siyu Tang. We are more than our joints: Predicting how 3d bodies move. In CVPR, 2021. 3, 18
Bolei Zhou, Xiaogang Wang, and Xiaoou Tang. Understanding collective crowd behaviors: Learning mixture model of dynamic pedestrian-agents. In CVPR, 2012. 3
13

---

Published as a conference paper at ICLR 2023
In this Appendix, we include additional method details and experimental results that are not included in the main paper due to limited space as follows. 1) We provide a visualization video as additional qualitative results, and the details explained in Sec. A. 2) We include further discussion related work in Sec. B. 3) We explain different generative models incorporated in our proposed dual-level modeling framework in Sec. D; 4) We provide additional details of the experimental implementation in Sec. E and the summary of evaluation metrics in Sec. F. 5) To elaborate the effectiveness of our method, we provide additional ablation experiments with qualitative and quantitative analysis in Sec. G, and we evaluate our approach in more challenging scenarios with significantly increased number of people in Sec. H.
A VISUALIZATION VIDEO
In addition Figure 4 and Figure 5 in the main paper and more qualitative results in this Appendix (Figures A and B), we provide video to demonstrate more comprehensive visualizations of multi- person 3D motion forecasting at https: / /s rui xui gi thub. i /DDDMMMF / mages / demo. mp4. In this video, we illustrate that our method DuMMF generates diverse multi-person motion and social interactions, as well as taking into account both single-person and mullti-person fidelity. We also show that model is scalable and provide effective predictions in more challenging scenarios with a significantly increased number of people and associated more complex interactions. We also highlight the impact and effectiveness of our dual-level modeling framework.
B ADDITIONAL DISCUSSION ON RELATED WORK
As we demonstrate in the main paper, our proposed stochastic multi-person motion forecasting needs to simultaneously take into account single-person pose fidelity. consistency of pose and trajectory, social interactions between poses, and overall diversity of motion, while prior work including stochastic multi-person trajectory forecasting (Alahi et al., 2014; Kothari et al., 2020; 2021) and deterministic multi-person motion forecasting (Adeli et al., 2020; 2021; Guo al., 2022: Wang et al., 2021b) focuses on simplified scenarios. A survey (Barquero et al., 2022) summarizes the progress in this area. Our problem reveals real-world complexity with substantially increased, multi-faced challenges which have not been, and cannot be. jointly tackled by any of the prior literature. In particular, in our stochastic scenario, the fidelity and interactions need to be both satisfied and diversified for all the predictions, which is very challenging and cannot be addressed by a simple extension of existing work on stochastic multi-person trajectory forecasting and deterministic multi multi-person motion forecasting as shown in Sec. G.
For deterministic multi-person motion forecasting, in Sec. of the main paper, we have highlighted our key difference with Adeli et al. (2020; 2021); Guo et al. (2022), and here we discuss in more detail. Adeli et al. (2020) introduce a social pooling module to integrate social information. Adeli et al. (2021) propose a graph attentional network jointly model numan-human and human-object interactions. Both methods utilize additional contextual information to aid deterministic prediction with their proposed Social Motion Forecasting (SoMoF) benchmark (Adeli et al., 2020; 2021). Guo et al. (2022) propose cross- inssrinteraction attention mechanism to predict cross dependencies between two pose sequences, making it applicable only to 2-person scenarios.
There are many methods achieving strong performance with simple and non-transformer-base frameworks on the SoMoF leaderboard. Specifically, futuremotion_ ICCV21 1 (Wang al., 2021a) develops simple but effective framework based on combination of graph convolutional networks and multiple motion optimization techniques. DViTA (Parsaeifard et al., 2021) decouples the human pose into trajectory and local pose and employs a VAE to learn representation of the local pose dynamics. LSTMV_LAST (Saadat et al., 2021) proposes a sequence-to-sseuueee L.STM via keypoint- representation.
Complementing Sec. of the main paper, we here discuss in more depth the trajectory-level multi-person forecasting. Trajectron++ (Salzmann et al., 2020) constructs a spatio-temporal scene graph where nodes represent individuals and edges represent their interactions, and global and local modeling is embedded in this graph structure. EvolveGraph (Li et al., 2020) introduces an effective dynamic mechanism to evolve interaction graphs, which can flexibly model dynamically changing interactions. Tra2tra (Xu et al., 2021) introduces global social spatio-temporal attention neural
14

---

Published as a conference paper at ICLR 2023
Table A: Summary of methods for encoding and integrating multi-person motions and interactions. We follow to abstract the encoder into local and global part as Figure 2(d). As we use the same decoder for baselines (Sec. of the main paper), we only discuss encoders of three predictors in this paper. Note that XIA (Guo et al., 2022) can only be applied to two-person scenarios.
Method
Local Encoder
Global Encoder
SC-MPF (Adeli et al., 2020)
GRU(X))
MaxPooll (GRU(X.)))))) H==))
MRT (Wang al,, 2021b)
MuhiHead X, X, Xi)
MultiHead (x,,) W (X.) m111 (xn)==))
XIA (Guo al., 2022)
N/A
[MultiHead X2, MultHeaad X2, X1)
network encode both spatial interactions and temporal features. GroupNet (Xu et al., 2022a) employs multi-scale hypergraph neural network that models group-based interactions and facilitates more comprehensive relational reasoning. T-GNN (Xu al., 2022c) introduces transferable graph neural network that allows not only trajectory prediction but also domain alignment of potential distribution differences. MID (Gu et al., 2022) employs a diffusion model to model the variation of indeterminacy for trajectory prediction.
C ADDITIONAL DETAILS OF LEVEL-SPPECIFC OBJECTIVES
We use A to represent the residual of the motion sequence. For example, AY; = [Y + [T,), ( + YT + I]...., Y. + Tyl T + +T 1]]
Local-Level Objectives. We adopt the multiple output loss (Guzmán-rivera et al., 2012) and extend it to the local reconstruction loss of multiple people LIR, which is used to optimize the most accurate prediction of each person while maintaining diversity. We highlight the structure of the human skeleton by introducing the limb loss (Mao et al., 2021) Lt. Specifically,
N
LIR
min
JA
AY, 12
(6)
N
M
n=1
L1
L
L 12
N
m=1
where the vector L,, represents the ground truth distance between all pairs of joints that are physically connected in the n-th human body and L includes the limb length for all the Tn poses in Y
We further develop multimodal reconstruction loss LmmR to provide additional supervision for all outputs {[A Vm [ We first construct pseudo future motion [YT)}= for each historical sequence X, Different from (Yuan & Kitani, 2020; Mao et al., 2021), we additionally consider
translation T g and rotation R E 33x of the pose. Specifically, given a threshold e, we cluster future motion with similar start pose and train the model with their residuals as
(P (YP
min
[TX] T) [TTllll & e).
(8)
N
P
LumR N+P n=1p=1
EX
JAY AYP m=L...,.M
min
(9)
To explicitly encourage diversity, we adopt diversity- promoting loss (Yuan & Kitani, 2020), which directly promotes the pairwise distance between the predictions of a single person. We decompose this loss into two parts, promoting the diversity of local pose and the global root separately. Supposing that YTT (l) and (g) are the local pose and the global root joint extracted from the global pose ym respectively, and a and B are two hyperparameters, this diversity prommoiing loss is denoted as
Ym (g)
( + exp(
Y (1) 4+ }]
CD N M(M
ΣΣ
Σ
(exp(
a
k=m+1
(10)
15

---

Published as a conference paper at ICLR 2023
Table B: Summary of the complementary evaluation metrics in the multi-person 3D motion forecasting task, with each focusing on evaluating different aspects of predicted motion. Here for simplicity. we show the metrics without alignment. We also provide ADE, FDE, and FPD with alignment to evaluate the pose and trajectory separately, explained in Sec. 4 of the main paper.
Type
Metrics
Dehnition
Local Average Disphacement Frro (IADE)
N Ensi minm Y Yalla
Single-Person Fidelity
Local Final Displacement Froor (IFDF)
E minm || [Tp]- Ys (Tilla
Foot Skating Ratio (FSR)
average ralio of frames where both fool joints close the ground (S 5cm) and fast (2 75mm/s)
(Global) Average Displacement Ertor (ADE)
miDm N75
>
!! Y
Multi-Person Fidelity
(Global) Final Displacement Eeror (FDE)
minm x >
[Tp Y, T 2
Trajectory Collision Ratio (TCR)
average ratio of frames where there is collision between any trajectories
Average Human Displacement (AHD)
NM Σ> EM [T)]
Overall Diversily
Final Pairwise Distance (FPD)
MM(M-IJ
Di-1
DM
Dt-m. (
Y 22lll
For CGAN, a GAN loss (Kocabas et al., 2020) is leveraged to train the model and the local discrimi- nator Dj for individual body realism. Suppose {Y:) is the set of real motion clips sampled from the data, we have the following.
M ΣΣ
LIGAN
(Y m=
ID
1 N
Σ (D()) - 11..
(11)
N M
Global-Level Objectives. Since in this setting we treat N individuals as a whole, the reconstruction loss is reformulated as
CgR
min
Σ IAY A N
(12)
For CGAN, global GAN loss is further leveraged to promote the realism of social motion, where the global discriminator D, takes the motion of all N people as input. Suppose (Y Jn=1 is the multi-person motion clip sampled from the data, and we have
M
LEGAN = M
Σ m==1
1D
In ME 11 1
(13)
Table C: Quantitative results (w/ error bar) of our DuMMF on single-person accuracy.
2s
1 3s
Method, # of Intents
IADE
IFDE L
IADE
IFDE
IADE
IFDE
DuMMF (Ouns),
0.663 + 0.007
1.088 + 0.019
1.029 +: 0.019
1.612 + 0.043
1.308 + 0.031
2.014 + 0.086
DuMMF (Ours),
0.656
0.007
1.056 + 0.007
0.996
0.009
1.503
0.032
1.244 + 0.021
1.883 + 0.076
DuMMF (Ours),
0.648
0.011
1.033 4 0.022
0.963 0. 0.020
1.400
0.035
1.176 - 0.025
1.672 - 0.052
D ADDITIONAL. DETAILS OF GENERATIVE MODELS
In this paper, we mainly evaluate the generalizability of our method by incorporating it with a
CGAN (Mirza & Osindero, 2014) DDPM (Ho al., 2020) model. For CGAN, we introduce local discriminator and global discriminator for objectives at local and global levels, respectively. For the model incorporated with DDPM, disregard these two GAN losses.
16

---

Published as a conference paper at ICLR 2023
Local Discriminator. To ensure the fidelity of the motion, especially to address the problem of foot skating, where the feet appear to slide in the ground plane, we concatenate the predicted motion of each person Y? with the feet velocities A as input to a local discriminator Dj. The local discriminator uses local-range transformer encoder adopted from Wang al. (2021b).
Global Discriminator. To ensure the realism of social interactions and avoid motion collisions, we propose a global discriminator D that encodes all motion of N people n== the same time and outputs fidelity score. The global discriminator uses a global-range transformer encoder adopted from Wang et al. (2021b).
E ADDITIONAL IMPLEMENTATION DETAILS
Here, we provide more details on the implementation of our method DuMMF. The two hyperpa- rameters (a, 33 in diversity promoting loss (Sec. C) are set to (50, 100). The model is trained using a batch size of 32 for 50 epoch, with 6000 training examples per epoch. We use ADAM (Kingma & Ba, 2014) to train the model. The code is based on PyTorch. On one NVIDIA GeForce GTX TITAN X GPU, training an epoch takes approximately 5 minutes. For license, CMU-Mocap (CMU) free for all users; MuPoTS-3I (Mehta et al., 2018) is for noncommercial purposes. Part of our code is based on AMCParser (MIT license), iiteniin-is-all-yo---eee -ytorhhh (MIT license), and MRT (Wang al., 2021b) (not specified), and XIA (Guo et al., 2022) (GPL, license).
Table D: Quantitative comparison on MuPoTS-3D between our DuMMF and deterministic forecasting baselines and their CGAN variants. All models trained only using skeletal representations on CMU-Mocap and we compare their generalizations on MuPoTS-3D here. The number of
intents set to 5 for stochastic forecasting 3-person (top) and 2-person (bottom) scenarios.
DuMMF significantly improves mulli-person accuracy and diversity across various architectures and deterministic predictors.
Architettue
Prodictor
Divenifer
Varianta
FDET
FDE
OGAN
NIA
0.756
01846
0.753
1:50
1.207
0.167
1,514
RNN
SC MPF (Adeli 2020)
CGAN-DOMME (Ours)
wil Unntinauss
wio Discrste
3,077
8.36
0.355
0.002
0,478
10221
0.798
19092
1.534
1-112
0,770
1,084
1.202
1.258
1,800
2,600
9.307
2,1071
1.005
1:476
9.568
17200
1.815
Full
0.747
0.456
0.744
1.441
0.707
1.109
2,200
0.893
3.964
NiA
3.004
N7A
07100
0.090
11:31
0.529
03733
1,077
17A
039075
0.557
1-761
MPP (Adeli 2020)
CGAN-IIMIMP (Oura)
we Continaru
we Discrete
0.917
0,507
0.410
0.106
0.707
0.090
2.124
0.585
0.557
1.163
1,070
3.835
0.024
0.557
1,382
1.573
Tranfoomer
Deleminssic CCAN
NIA
Fill
0.865
176
N/A
0.402
0.425
0.491
0.991
0.711
0.247
0.459
1,816
N/A
0.636
0.755
0.747
1.071
1.354
1.188
0.501
0.275
N(A
0.852
1.071
1.00
2.043
1,5.0
1,3552
MRT (Wang al,
2114
1L:13
0.741
1.207
1:1077
1:704
CGAN (Ours)
0.230
1.342
Full
0.510
0.711
1.185
Detemtnatic
NAA
NDA
01222
0.503
1.200
1:315
MIT 2021b)
CGAN.DAMMMF (Ours)
We Continauus Full
wel Disertte
3009
3.000
3,01
0.413
0.175
0.510
R1712
0.86
0.231
1.452
122
0.674
0.543
0.713
1.100
1:101
1.395
2.338
0.410
03522
3.885
1.192
1.433
2.307
Tranfome::
Denmanani CCAN
NIA
0.680
1155
NAA
0.459
0.173
0.73
0.78
11500
2,285
NA
0.706
0.S.5
0.591
1.155
1.106
1.145
3,875
N7A
0.015
1.225
0.591
2.305
1,444
1.884
XIA (Gias 2002)
CGAN-DOMME (Durs)
we
Sepurazion
0,215
0.754
0.242
0.705
1.355
1.192
1.002
0.917
1.586
Full
0.885
0.151
0.751
190
0.690
1.082
3.057
0.579
1,335
F SUMMARY OF EVALUATION METRICS
In the main paper, we mainly focused on evaluating the fidelity and diversity of the predicted multi- person motion based on Average Displacement Error (ADE), Final Displacement Error (FDE), and Final Pairwise Distance (FPD), and briefly discussed other metrics. Here, we explain these and additional metrics in detail and also provide systematic summary in Table B for better understanding. Additional comparisons based on these metrics are shown in the following sections.
As summarized in Table B, we group the metrics into thee types, with each type evaluating different aspects of predicted motion (discussed in Sec. of the main paper) single-person fidelity, multi- person fidelity, and overall diversity.
17

---

Published as a conference paper at ICLR 2023
Table E: Quantitative comparison SoMoF Benchmark. Here, we only show the deterministic forecasting results, Our method with MRT predictor significantly outperform two deterministic baselines. We use VIM (Adeli et al., 2020) as the metric. means we directly report the results from the benchmark leaderboard.
Prediction Time
Method, # Intents
100 ms
240 ms
500 ms
640 ms
900 ms
SC MPF (Adelli al., 2020), N/A
46.28
73.88
130.23
160.83
208. 44
TRIPOD" (Adeli al,, 2021), 1), N/A
30.26
85.08
104.78
146.33
MRT (Wang et al., 2021b), NIA
22.99 22.93
42.35
79.41
98.02
137.99
Table F: Quantitative comparisons between our DuMMF and deterministic forecasting baselines and their CGAN variants on SoMoF benchmark. The number of intents is set to 5 for stochastic forecasting, Our DuMMF significantly improves multi-person accuracy and diversity.
Architecture
Predictor
Diversifier
Variants
0F2 1s
A
FDEL
Deterministic CGAN
N/A
N/A
0.375
0.961
0.501
0.582
Transformer
MRT (Wang al., 2021b)
W/o Separation
0.864
0.3101
0.969
CGAN+DuMMF (Ours)
Discrete
1.075
0.567
w/o Continnous
1.063
0.587
1.018
Full
0.969
0.564
0.935
Specifically, Local Average Displacement Error (IADE) and Local Final Displacement Error (IFDE) are proposed further evaluate the single-person fidelity. They compute the average distance between the individual ground truth and the individual prediction closest to the individual ground truth. Note that ADE (or IADE) and FDE (or IFDE) only measure the best predictions all outputs. While we can average ADE (or IADE) and FDE (or IFDE) by computing the distance between multiple predictions and a single ground truth, this way of assessing the overall prediction quality cannot reflect motion realism. For example, a very realistic but diverse set of outputs may have poor average ADE and FDE. Therefore, we do not use such metrics for our evaluation.
For diversity evaluation, common metric is Average Pairwise Distance (APD) (Mao et al., 2021; Yuan & Kitani, 2020; 2019) that is the average &2 distance between all predicted motion pairs. In this paper, we formulate diverse forecasting This progressive generation is actually closer to reality because the multi-modality of motion should be more pronounced after further time. However, in this case, APD cannot reflect the diversity well, since many predictions will share the same previous segment. Therefore, we only examine the diversity of the last pose (FPD), as the last pose should not the same.
producing more forecasts over time (see See. 3.3).
Moreover, we introduce three tailored metrics to evaluate specific aspects of predicted motion, which correspond to the unique challenges in multi-person motion forecasting as discussed in the main paper. (a) Foot Skating Ratio (Zhang et al., 2021): the average ratio of frames with foot skating. (b) Trajectory Collision Ratio: the average ratio of predictions that is considered have collision (Kothari et al., 2020) between any two trajectories in the scene. (c) Average Human Displacement: Average displacement of the predicted human body between the last frame and the first frame, reflecting the properties of the predicted motion distribution.
G ADDITIONAL EXPERIMENTAL RESULTS
Additional Quantitative Results. We compare our method with MRT (Wang et al., 2021b) in Table G. We show that the improvement is significant by performing each experiment five times with different random seeds and reporting error bars. When the number of intents is one, equivalent to a deterministic setting, method marginally outperforms MRT for all metrics on CMU-Mocap and also generalizes better on MuPoTS- 3D.. This suggests the generality of our approach, which is also advantageous for deterministic processes.
In Table C, we use IADE and IFDE to compare single- personneonnnnno fidelity, and we observe that our model also significantly outperforms the baseline.
18

---

Published as a conference paper at ICLR 2023
Table G: Quantitative results (w/ error bar) of DuMMF with CGAN and the baseline MRT (Wang et al., 2021b). The baseline and our models are trained only on CMU-Mocap, and are tested on CMU-Mocap (top) and MuPoTS-3D (bottom). With the same backbone, our DuMMF framework significantly outperforms MRT on the deterministic prediction, and provides more accurate and diverse predictions with more intents and predictions.
P10
PIT
AZ30
1110
ADD:
FIXC
MRT (Niazs ctil. 302 NIA
N0A
B.822 0.56
1,127 (L100)
N/A
LIK2 21100
1,258
2: 20121
NIA
1.456
0654
2.603 BL167
DDMMM (Surs)
30A
0,670 2 0.003
L.11720.014
NAA
1,061 0.000
1,700
0.028
NIA
1,380
= 0.017
2,2919 =: 0.003
DuMME
2006
2.00
1.66
17.7400.00
DUMME (Ous).3
0.395-2 00000
0.664 0.067
L007 0.000
1.002
LOL9 0.00
1,556 6- 2/25
27172
1.556
D6000
1999
0.073
DOMMF (Ous).
0631: 0.022
0.301
1.00
0.,00
1.86 0.311
0,092 2.0000
2.1:2-2-0.24
1.500
0/056
1,00
0.00
(RorS 2l. 201th5
NIA
0.739 1000
3.377 10133
NIA
DUMMI (Shm).2
0:34 2.110
1.11E: 0.005
1L7221 11001
0.713 2..83
1.233 0UAR
JJ00 20337
IL504) 16636
1 1.783::1 11.10
(
2 IL003
3.075
1/13
(4041
Start
End Poses, DuMMF (Ours)
Pra
f i8
0.336
0.678
18
1.58a
1.87a
2.335
2.675
30
3.38s
3.67s
Figure A: Qualitative results on CMU-Mocap. We evaluate the generalizability and scalability of our model to predict 3-second motion on the constructed 6- 6-person motion test data. The top row the historical pose, and the five end poses predicted by our model; and we show the sequence of one of the predictions the bottom two rows (highlighted by the blue dashed box).
Additional Comparisons SoMoF Benchmark. We investigate the performance of our framework on the Social Movement Prediction Challenge (SoMoF) (Adeli et al., 2020; 2021). We specifically use labeled trajectories and poses (13-joint
the 3DPW dataset (von Marcard et al., 2018), where human skeleton), but we do not use videos of given scenes as input. We discard the multi-modal reconstruction loss since the 3DPW provided by SoMoF benchmark is relatively small. In Table E, we provide results on our implemented deterministic prediction and compare with baselines directly reported from the leaderboard on the SoMoF benchmark. In Table stochastic multi multi-person forecasting. Note that ADE and FDE require access to ground truth data, which is not publicly available from SoMoF benchmark. Thus, we report the results on validation set. We observe similar performance as on CMU-Mocap (CMU) and MuPoTS-3D (Mehta et al., 201 8), specifically, using DuMMF, the model has significantly better accuracy and diversity in stochastic multi-person forecasting.
use SoMoF benchmark for
Additional Comparison with Single-Person Forecasting Methods. In Table H, we provide a comparison by applying stochastic single-person forecasting baseline. For fair comparison, we formulate this single-perse forecasting architecture (named SRT) with a transformer encoder- decoder adapted from MRT (Wang et al., 2021b), where modified the encoder and decoder to be individually independent. Table H illustrates that (1) the modeling of social interactions is crucial, since the single-person forecasting baseline has much worse performance; (2) the improvement from
19

---

Published as a conference paper at ICLR 2023
Start
End Poses, DuMMF (Ours)
0.336
0.67s
16
1,58s
1.87s
2,33:
2.67s
30
3.33s
3.67s
Figure B: Qualitative results CMU-Mocap. We evaluate the generalizability and scalability of our model to predict 3-second motion on the constructed 9 9-person motion test data. The top row the historical pose, and the five end poses predicted by our model; and we show the sequence of one of the predictions in the bottom two rows (highlighted by the blue dashed box).
Table H: Quantitative comparison on CMU-Mocap between our DuMMF based on MRT (Wang et al., 202 Ib) and a single-person person forecasting baseline adapted from MRT. The number of intents is set to for stochastic forecasting 3-person. For DuMMF with SRT, we use the variant of "w/o Separation" to skip the social interaction modeling in DuMMF, making it full single-person prediction baseline.
Anchisostnne
Prodictor
Docenffar
Varimts
FPD
ADE.
FDE
FPD
07
ADE
24
FDE
FPD
GF
ADE
FDE
Trandoemer
(Wang al,,
(Wing 20011))
SKCE
CGANADEMME (Ours)
CGAN
Negarattio
NIA
0.989
NDA
61011
01077
1220
1966
03066
0.607
N7A
1306
1,03
1.705
1110
1-100
N0A
1.292
1337
1.420
1.987
2.45
1.919
our DuMMF for the multi mulli-pperoon forecasting model is much higher than that for the single-person forecasting model.
Additional Ablation on Impact of Learning Objectives. In Table we evaluate the impact each loss term within the dual-level framework. In general, using all loss functions yields the best results, since its results are either the best or the second best. We observe that local and global discriminators not only make predictions more accurate, but also more diverse. Note that the reconstruction losses Lin and LgR optimize only the most accurate prediction of the outputs. supervision for other predictions that not the best. We observe that limb loss LL is crucial, it is the only loss function that provides supervision for all outputs. The multi-modal reconstruction loss also has large performance impact, since it provides supervision for more than output.
important to provide
Additional Analysis of the Number of Discrete Latent Codes. Note that the number of discrete latent codes not restricted to the same number the number of predictions per second. We chose
Table I: Ablation study of our DuMMF model on CMU-Mocap using skeletal representations. We report the accuracy and diversity with error bar for root and pose respectively. The results show the impact of different learning objectives. Best results are bolded, and next best results are underlined.
Ca
Cr
DRAAN
Co
CyNAN
ronFPD
poseFPD
matADE
romFFF
paseADE L
poseFDE
0:251
0.044
0.745
0.0000
0.015
0.069
0:437 1 0.1.6
0.781
0.983 0.075
0.274 0.015
0.499
0.030
0.241
0.026
0.500
4 0.055
0.735
0.005
0.897 :: 0.000
0.250 0.004
0.407
0.014
/
0.218
+ 0.074
0.468
+-0.866
0.748 +00011
0.920 + 0-007
0.254 0.012
0.488
0.039
/
0.240
0.002
0.537 0.077
0.738 0.005
0.900 + 0.010
0.241 + 0.000
0.353 0.021
0.232
- 0.068
0.506 0.1.11
0.700 0.000
0.966 1: 0.012
0.217 2: 0.008
0.402 0.025
0.077
0.738
0.05
20

---

Published as a conference paper at ICLR 2023
Table J: Ablation study of the number of discrete latent codes CMU-Mocap using skeletal representations. For producing predictions per second, we observe that both the accuracy and diversity of predictions decrease significantly as the number of discrete intents increases.
# of Intents
@(=1a 1s
=
35
FPD
ADE
FDE
FPD
ADE
FDE
IPD
ADE
FDE
0.716
0.658
1.435
6
0. 0.198
0.607
1.087
1.086
1.033
1.578
1.814
1.300
1.971
7
0.465
0.671
1.003
0.960
1.032
1.582
1.502
1.298
1.986
9
0. 0.263
0.689
1.087
0.565
1.027
1,578
0.941
1.295
2.028
15
0.231
0.678
1.127
0.582
1,70
1.704
0.968
1.384
2.209
them to be the same for simplicity and better trade-off between prediction performance and training efficiency. We find that this setup also achieved the best performance. To ensure that all predictions come from different discrete codes, the number of discrete latent codes should not be less than the number of predictions. In Table J, we provide an ablation study when the number of predictions per second is 5. If the number of discrete intents is greater than 5, we randomly select discrete code without replacement (excluding any of the previously selected intents) to generate prediction. We observe that both the accuracy and diversity of the predictions decrease as the number of intents increases. An explanation of such behavior could be: When the number of discrete latent codes is the same as the number of predictions, each code explicitly and fully optimized for a particular prediction, leading to the best prediction accuracy and diversity; whereas, when the number of intents increases, the random selection strategy may hurt performance, as the probability of selecting the best five discrete codes decreases, We hypothesize that a better training and selection strategy might improve performance with more discrete codes.
H GENERALIZABILITY AND SCALABILITY: EVALUATION ON MORE-PERSON SCENARIOS
Datasets with More People per Scene. In the main paper, constructed multi-person motion data with people per scene. Here, we construct more challenging datasets with significantly increased number of people; this also increases the complexity of social interactions. Specifically, first sample 2-person and 1-person motion data from CMU-Mocap (CMU), and then compose them together. We use handcraft rules filter out scenes with trajectory collisions. Instead of retraining the model in the novel setting with more people per scene, we directly evaluate the model trained on 3-person motion data, and test if our model is able scale to scenarios with more people.
Qualitative Results. Similar to Figure of the main paper, we provide visualizations for 6-person and 9- 9-person scenarios in Figure and Figure of the end poses of predictions of second. We randomly select a prediction and visualize the corresponding full motion sequence as well. Note that our model trained on only 3-person data without any fine-tuning on more-person data, but it still performs well on more-person data, suggesting the generalizability and scalability of our approach.
21


Reviews:
[{"review_id": "yQV8boGbrp", "reviewer": "Reviewer_Lood", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose a dual-level generative model, considering the individual movements and social interactions. The proposed method achieves good performance on CMU-Mocap, MuPoTS-3D, and SoMoF benchmarks.", "summary_of_the_review": "I list my concerns in [Weaknesses], I am happy to discuss and increase the rating if my concerns are addressed.", "strength_and_weaknesses": "**Strengths**:\nOverall, it is a good paper with strong motivation. The method is clearly introduced and the experiments are sufficient and convincing.\n\n\n**Weaknesses**:  \n1. In the proposed method, why are the latent codes of the social interactions defined as the same? What's the insight here? In my opinion, the interactions are also dynamically changing during the observations, which is also pointed in [1,2]. Some explanations should be included here. In addition, why consider social interactions on a global level? How about the local social interactions?\n\n2. In Sec. 3.1, Local-Level Modeling, $N$ different intents $z_{i}$ are extracted to generate independent future movements. Does $N$ represent the total number of people in one sequence or one batch or the whole dataset? I presume it is the number of people in one sequence or batch, then $N$ must vary in different sequences or batches. How to deal with this situation? More details should be included here.\n\n\n3. Missing related works, some latest multi-person forecasting trajectory/prediction [1, 2, 3]. Since this work focuses on multi-person motion prediction, I think part of the discussion in Sec. B of the Appendix should be summarized in Related Work Section to highlight the differences between the proposed method and existing ones.\n\n[1] Evolvegraph: Multi-Agent Trajectory Prediction with Dynamic Relational Reasoning. NeurIPS2020  \n[2] Tra2tra: Trajectory-to-Trajectory Prediction with A Global Social Spatial-Temporal Attentive Neural Network. ICRA 2021  \n[3] GroupNet: Multiscale Hypergraph Neural Networks for Trajectory Prediction With Relational Reasoning. CVPR2022   \n[4] Adaptive Trajectory Prediction via Transferable GNN. CVPR2022  \n[5] Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion. CVPR2022 \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qz3L6qIjgdg", "reviewer": "Reviewer_UYRH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem of multi-person 3D motion forecasting. The proposed framework uses a dual-level transformer-based generative model, encoding the local (per person) and global (multi-person) motion histories to decode each individual future body motion. The framework has been comprehensively evaluated on different datasets such as CMU-Mocap, 3DPW and MuPoTS-3D and also a benchmark, SoMoF, and demonstrates its superior performance against SOTA frameworks.\n", "summary_of_the_review": "Although the technical contribution is not substantial, I think the paper has a good quality in the terms of presentation, experiments and results. Considering these aspects, I support the paper to be accepted in this venue. However I have some minor comments\n\n1-  on the presentation of qualitative results: it can barely understand anything from Figs 3-5. The Errors and the good results could be highlighted. The figures are also too small to distinguish anything meaningful. \n\n2- I am not sure if I found out how did the predicted samples from generative models are evaluated during inference? I think eventually one sample from a distribution should represent the final output and this should be decided independently from evaluating them against ground truth. Do you calculate the mode of the generated samples? If so how?\n\n3-  while the model is compared with most recent conference papers published on SoMoF benchmark, it would be great if the authors can comment on other published frameworks (eg workshop ones) which use very simple and non-transformer based frameworks with superior results. For comparison purpose, It would best if the authors make their results on SoMoF leaderboard published . ", "strength_and_weaknesses": "Strengths:\n\n1- The idea and motivation are clear and the submission is well presented, 2- a comprehensive literature review with relevant literature and research topics is covered 3- extensive experiments on different dataset and one related benchmark is provided, 4- the method achieved SOTA results on these datasets \n\nWeakness:\n\n1-considering most generative models are well studied in multi agent trajectory forecasting, the paper’s technical contribution is yet relatively limited compared to the literature (though the problem here is not identical and the overall framework is yet different)\n\n 2- a better qualitative visualisation of results would be helpful for better understanding of the framework behaviour ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ahqLOe4MVOv", "reviewer": "Reviewer_Eu4r", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The work presents a deep GAN model to predict the 3d motion of humans in a social interaction. It introduces a dynamic intent model that models the interaction between the representation of the individual motions.", "summary_of_the_review": "The work is good. It need to re-visit several claims in the introduction part. \nThe method needs a better illustration. \nThe evaluation misses a core metrics and clarification of how the comparison versus deterministic models was constructed. ", "strength_and_weaknesses": "Introduction level discussion: \n\n“stochastic forecasting of human trajectories in crowds …”\nAdvanced a lot with modeling the social interactions and multiple works showed this. \nFor example, Trajectron++… etc. Authors should re-visit this statement in the introduction.\n\nThe “multi-person 3D motion forecasting” task already exists. See “Multi-Person 3D Motion Prediction with Multi-Range Transformers” . Claiming it is “novel” because the word “stochastic” is not accurate. Motion forecasting/ prediction is a generic task that can be solved deterministically or stochastically. \n\nThe concept of local and global dealing with motion forecasting already existed in prior works such as Trajectron++ and follow works… etc. \n\n\nMethod discussion: \nThe discrete intent is positive of this work. Though, is not sampling the same intent for all individuals without conditioning on the individual trajectory might lead to mode collapse? \n\nResults discussion: \n- Are the ADE/FDE metrics best of N (B-o-N)? \n- How many samples used in calculating the performance? Does the results vary with respect to the number of samples?\n- When evaluating 3D person motion prediction we are expected  to use the 3D path error and 3D pose error to understand the accuracy of the joints. Please see “Long-term human motion prediction with scene context “ and later works for these metrics. \n- What is the divergence of the prediction between 1,2,3 seconds? \n- Ablation study is positive. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TBNQA3FgmS", "reviewer": "Reviewer_r2ic", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel architecture to performe multi-person 3D pose forecasting. The importance of pose forecasting is well known, useful in quite complex situations like autonomous driving.\nThe model is quite simple but effective, taking into account an important variable of the human behaviour which is the context and the others in the environment.\nThe paper is integrated with an extensive experimental section.\n\n", "summary_of_the_review": "The model is quite simple but effective, taking into account an important variable of the human behaviour which is the context and the others in the environment. It could have been ok to show some experiments on single pose forecasting too\n", "strength_and_weaknesses": "strenghts:\n- The model is quite simple, it is based on a generative model which based on two steps: the first one aims to predict the single-person pose in the future, the second one aims to analize the social interactions among people in order to refine the single-person pose.\n- the model is able to predict interactions between people and can learn for example how to react to collisions.\n- the paper opens up to new tasks and problems, it can introduce a new branch of pose forecasting leading to new models capable of better understandings of human behaviours.\n- details on implementation of the code are well described in supplementary material\n\nweaknesses: \n- Metrics used in my opinion aren't easy to understand from a practical point of view. It is difficult to have an idea on how the model can perform in terms of mm of error.\nI suggest to use metrics like MPJPE or MAE to better understand the quantitative performances of the model.\n- improving models with social aspect of the human movement can be really effective for the model performances. I would like to see an ablative study on how this model can improve performances compared to standard  single person forecasting models.\nMy doubt about this system is that performances compared to sota single person forecasting models would be lower, on standard datasets that offer even multi person poses (CMU panoptic, 3DPW).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%