REMIPS: Physically Consistent 3D Reconstruction of Multiple Interacting People under Weak Supervision
Authors: Mihai Fieraru
Mihai Zanfir
Teodor Alexandru Szente
Eduard Gabriel Bazavan
Vlad Olaru
Cristian Sminchisescu
Venue: NeurIPS 2021 Poster (2021)
Keywords: human
pose
shape
reconstruction
3d
people
transformer
mesh

Abstract: The three-dimensional reconstruction of multiple interacting humans given a monocular image is crucial for the general task of scene understanding, as capturing the subtleties of interaction is often the very reason for taking a picture. Current 3D human reconstruction methods either treat each person independently, ignoring most of the context, or reconstruct people jointly, but cannot recover interactions correctly when people are in close proximity. In this work, we introduce \textbf{REMIPS}, a model for 3D \underline{Re}construction of \underline{M}ultiple \underline{I}nteracting \underline{P}eople under Weak \underline{S}upervision. \textbf{REMIPS} can reconstruct a variable number of people directly from monocular images. At the core of our methodology stands a novel transformer network that combines unordered person tokens (one for each detected human) with positional-encoded tokens from image features patches. We introduce a novel unified model for self- and interpenetration-collisions based on a mesh approximation computed by applying decimation operators. We rely on self-supervised losses for flexibility and generalisation in-the-wild and incorporate self-contact and interaction-contact losses directly into the learning process. With \textbf{REMIPS}, we report state-of-the-art quantitative results on common benchmarks even in cases where no 3D supervision is used. Additionally, qualitative visual results show that our reconstructions are plausible in terms of pose and shape and coherent for challenging images, collected in-the-wild, where people are often interacting.

REMIPS: Physically Consistent 3D Reconstruction of Multiple Interacting People under Weak Supervision
Mihai Fieraru mihai. fierarulimar... ro
Mihai Zanfir m mihaiz@google.oom com
Teodor Alexandru Szente teosz&google.oom com
Eduard Gabriel Bazavan egbazavan@google. com
Vlad Olaru vlad. olaru@imar ro
Cristian Sminchisescu sminchiseescu@googee.. com
GGooll Research Lund University 'Institute of Mathematics of the Romanian Academy
Abstract
The three-dimensional reconstruction of multiple interacting humans given a monocular image is crucial for the general task of scene understanding, cap- turing the subtleties of interaction is often the very reason for taking a picture. Current 3D human reconstruction methods either treat each person independently, ignoring most of the context, or reconstruct people jointly, but cannot recover inter- actions correctly when people are in close proximity. In this work, we introduce REMIPS, a model for 3D Reconstruction of Multiple Interacting People under Weak Supervision. REMIPS can reconstruct variable number of people directly from monocular images. At the core of our methodology stands a novel transformer network that combines unordered person tokens (one for each detected human) with positional-encolee unified model for self- and pyppeenttrationncolliioooo based on a mesh approxi- mation computed by applying decimation operators. We rely self-supervised losses for flexibility and generalisation in-the-wild and incorporate self-contact and interaction-contaee losses directly into the learning process. With REMIPS, we report state-of-the- quantitative results on common benchmarks even in cases where 3D supervision is used. Additionally, qualitative visual results show that our reconstructions are plausible in terms of pose and shape and coherent for challenging images, collected in-the-wild wild, where people are often interacting.
tokens from image features patches. We introduce novel
Introduction
Reconstructing tree-dimensional models multiple interacting people from images is an important computer vision task with applications behavior analysis, automatic video analysis of sport events, or collaborative augmented reality applications. As the demands on the depth of analysis increase, beyond qualitative pose, often seeks know whether people in contact not, what is the nature of that contact, and how long it lasts. As such questions become important, it is clear that very coarse estimates of pose even shape no longer sufficient. One needs accurate models of shape and contact with predictable response over time. This is difficult due to the various degree of occlusion that occurs, the depth ambiguities particularly given only monocular images (but more generally given that occlusion and self-occlusion for people are frequent), the high degree of variability of even valid human poses, their different scales and underlying spatial image support, ranging from the large body parts like torso or thighs informed by larger image regions to hands, with parts typically accounted for by smaller spatial support, comparatively.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Most of the volumetric human pose and shape recovery methods focus on a single person and require detecting the humans in the image and then running the inference model on each positive response. This approach has achieved good results and scales linearly in the number of people in the scene. There are multi people reconstruction methods which either predict 3D pose only and defer the shape reconstruction step to a later optimization stepl] or rely on an orthographic camera model 21. However, it is nontrivial to obtain a realistic 3D scene placement for all the reconstructed human models such that their 3D spatial relations are plausible and the alignment with the image evidence is good. All aforementioned methods train using full 3D supervision coming from mocap datasets. However, this often lacks scenarios of interactions or fine grained self-contact Recent work such as 3 designed datasets for interaction analysis, with additional interaction signature annotations, and showed that models able to represent those produce more realistic reconstructions. However, they only work in an optimization framework and reconstruct only pairs of people in close interaction. Ideally would like to be able to precisely model contacts, rely attention models to identify the important element of interaction, have a framework that can accommodate more than two people and learn such models from data, using only weak supervision.
Our main contributions can be summarized as proposing the following: (1) fast, accurate and unified 3d self collision and interpenetration models for multiple people; (2) a novel vision transformer architecture to predict 3D pose and shape for multiple people; (3) weakly supervised models which do not require 3D annotations during training: (4) state of the art results challenging datasets, with favorable performance compared to competing predictive or optimization based methods.
In this paper, we propose REMIPS, hybrid mnnvolutional-ranfformmee where the output of a human detector combined with high-level convolutional image features. They are processed by a series of transformer encoder layers that iteratively refine the 3D pose and shape reconstruction estimates for all detected humans in the scene, irrespective of their number. Decoupling the detection component has the advantage of benefiting from innovations in that space without necessarily paying performance cost (detection models are already quite fast, close to real time, and providing accurate results). Similar to 5] we use a full perspective camera model, allowing us to infer translations for all humans in the same camera coordinate system. We propose novel interpenetration and self-collision losses that are amenable to deep-learning-bas based training process and work in superv-su regime.
2 Related Work
The field of 3D human pose and shape reconstruction from images has received increasing attention in recent years. This has been driven by progress in 3D statistical and articulated human body models 7 and advances in 2D 8 D T0 and 3D [I pose estimation. Most reconstruction methods focus on estimating the 3D pose and shape of a single person [S] 12 recently, several papers focused on reconstructing multiple people coherently 22
13 14 15 although, more
I7].
Recent work started focusing on modeling inter-human contact 3 or self-contact 4 18] developing
methodology for more principled, contact aware reconstructions and releasing datasets with ground truth interaction (3] and self- self-contact [ vertex-level annotations. We use these datasets in our training and evaluation pipeline, together with other various supervision signals.
Physically plausible human mesh reconstructions require collision avoidance for single and multiple interacting people. Prior work related to self-collisions [19] 20 21 22 rely on bounding volume hierarchies (23) to detect the set of colliding triangles. We propose novel, efficient and faster human mesh collision losses which are more appropriate for deep learning training pipelines. See $3.4 for more details.
have increased the capabilities of neural network architectures for handling image features, and their ability to produce accurate reconstructions for images in-the-wild [26 27. We take inspiration from this as well.
Recent innovations in the space of visual transformers 24 25
3 Methodology
3.1 Statistical 3D Human Body Models
We use a recently introduced statistical articulated 3D human body model called GHUM 6 to represent the pose and the shape of the human body. The model has been trained end-to-end, in

---

deep learning framework, using a large corpus of human shapes and motions. The model has generative body shape and facial expressions , (B3, (3y) represented using deep variational auto-encoders and generative pose 0 = (0., in, 0rh) ) for the body, left and right hands, respectively,
represented normalizing flows 12 The pelvis translation and rotation are controlled separately,
and represented by a 6D rotation
28 E R6x1 and translation vector E R w.r. the origin
(0, O, 0). The GHUM mesh, M facets F E 22035233 expression parameters B , and the left and right hand pose parameters On. OrA. as we focus only on reconstructing the body pose and shape. We also drop the b subscript for convenience.
V, F). consists of set of vertices V E R 10168x3 and set of
experiments we infer the GHUM parameters (0., r, t) omitting facial
3.2 Camera Model
We use perspective projection operator II characterized by camera intrinsics C= [f= fy. Cz, Cyl which assumes that the camera coordinate system aligned with the world reference frame using an identity extrinsic rotation matrix and zero translation. This corresponds to a left handed coordinate system with positive z values forward. All predictions are made in the camera coordinate system. When processing the camera intrinsics for a crop we use the same approach as in [5
3.3 Architecture
In fig. we show an overview of REMIPS, our proposed hybrid learning architecture for monocular multi-person 3D body pose and shape estimation. We draw inspiration from vision transformers 24), as we also use a hybrid convolutional trrnsfoomrrr architecture, and from [3], as we explore the idea of iteratively refining estimates by relying on cascaded, input-sensitive processing blocks, with homogeneous parameters, as an end-to-end learnable surrogate for non- con-linner optimization.
Our network receives as input an image I E W containing multiple people, together with the pseudo ground-truth camera intrinsics C E L of the image (see 3.2). We apply convolutional neural network (CNN) on the input image and extract a downsampled feature map representation W. H/ x #
F
E
R
We flatten the feature map along the spatial dimensions to get sequence of 32 tokens. This sequence is linearly embedded by means of matrix E E R DxD' where
N W
D' is the embedding dimensionality. Next, learnable positional embeddings Epos E R N are added to the sequence. In addition to image feature tokens, we also use sequence of Np = 16 maximum tokens for the people the image. We first run an off-the-shelf human detector on the input image and collect the bounding box coordinates for each person index p. b" E 1x4 For each person p. we compute its initial GHUM parameters sh = (On, Bo; ro, th) where we compute the translation such that the default GHUM mesh projects in the center of the corresponding bounding box ba We denote by So (s). ... =="'] "r the initial state of GHUM parameters for all the people in the image. For each person, we concatenate the bounding person coordinates, the initial GHUM state and the camera intrinsics and construct the sequence of tokens corresponding to the persons in the image P E PDxDp, We linearly project the people token sequence P by means of matrix Ep E D The two sequences of tokens are concatenated into a single input sequence Zu and iteratively transformed through single shared transformer encoder layer (25). TL. for number of L. steps. We collect at each step E (1... L}} a refinement update AS, from each transformed representation Z, using a shared MI.P applied on the corresponding transformed representation of the people token sequence.
Zg = F,E + E Epos: PEp]
Z, (Z
(2)
AS; MLP(Z,NNN N.N+o.
The refinement updates AS, are added to the initial state GHUM parameters, So, as
S = So + AD
(4)
where S, = (0,, By, R,, T,} are the final GHUM parameter estimates for all the people detected in the image and is a weighting term for the residual estimates.
Note that the maximum number of person tokens Np = 16 chosen specifically for training purposes, to batch training examples together. Yet, at inference time, we can actually use as many
3

---

Person Detector
CNN
Spatial image features
Bounding box coordiineles and initial GHUM state Sur
Linear project
and and add positional embeddings
Append camera intrinsics C and linear project
Transformer Encoder Layer ())
Lx
MLP
AS
S. Σ AS
+ S
Figure 1: Overview of REMIPS, our proposed architecture to reconstruct the 3D pose and shape of multiple interacting people. Starting from a single input image, we use an off- off-the-shelf detector to extract the human bounding boxes. We create sequence of person tokens from these detections which attach initial GHUM estimate So. On separate hranch, starting from the image, we backhone convolutional neural network architecture and create an additional sequence spatial image feature tokens, F.. We concatenate two sequence representations and iteratively refine this joint representation through single transformer encoder layer for a number of J. stages. At the end of each stage I, we collect the transformed representation for the token sequence associated with the people and apply MLP regress the residual GHUM state estimates AS,. Our final estimation given by weighted sum of all the residual state updates and initial state. The network is trained weakly-supervised on various datasets with 2D annotations. We use contact and collision losses defined over the recovered geometries to ensure physical plausibility.
person tokens as the person detector outputs (even more than 16), as long as the quadratic memory requirements for the attention mechanism are satisfied (i.e. in the order of thousands persons). We observe that, in practice, the differences in inference speed for variable number of persons (between 1 and 32) be almost negligible, i.e. at most 10% increase running time.
We train the network by optimizing the following loss function:
L Lse + Lic + Lics + Lues + La Lka Lo + Lgg
(5)
where the first 5 loss terms correspond the physical plausibility of the reconstructed meshes as follows: Exe and Lie are the self-collision loss and the interpenetration loss, respectively; Liss and Lucs correspond the supervised self-contact 4 and interaction-contar [3] losses; La is the depth-aware ordering loss from 2 These losses are discussed and explained in more detail in subsequent sections.
We use a standard keypoint alignment loss, Lka, that measures the error between the 3D joints from the GHUM reconstructions, projected onto the image plane,
and the available 2D

---

keypoints annotations in the image lie fjihi=
K - - (J,, - 1
Cka
(6)
K
Note that for some datasets we do not have 2D pose annotations available. In that case supervise with the predictions of an off-the- the-shelf 2D pose estimator.
For body shape and pose, regularization losses:
Le = 0 LB = 1,112
3.4 Physical Collision for GHUM
In methodology, use approximation for the GHUM mesh topology, which denote by M FV. This mesh topology is obtained by applying decimation operators on the default GHUM model resulting in mesh with N₂ = 512 vertices and N; = 1020 triangles. We refer to this low resolution approximation of the GHUM mesh as PHUM. As V is a subset of the GHUM vertices, this easily allows the PHUM mesh be reposed and reshaped with respect the underlying GHUM mesh it approximates. The ICP loss of our approximation (i.e. for each vertex on the GHUM mesh not included in PHUM, we compute the distance the closest point on the PHUM surface) is of 2.8mm. By using PHUM, we drastically reduce the computation cost of our proposed interpenetration losses as they are quadratic in the number of vertices. We use generalized winding numbers 29 to rapidly test whether 3D points are inside/outside relative to a closed mesh topology.
and
Given a GHUM mesh, M, we compute its PHUM approximation M. We apply the generalized winding number test L = Φi (V) on its set of vertices V, with respect to its own mesh topology M and gather the binary inside/outside labeling
Self-collision loss
{0, 1] The vertices marked as inside. V V44. with L+ E L 1}. are pushed out of the mesh in the direction of their nearest neighbor vertices, NN(V L. M):
Lec Σ M)]2
(
IEL+
The nearest neighbors are extruded outside the mesh by small amount (i.e. 5mm) in the direction of their normal before computing the loss, such that inside vertices get pushed outside the mesh. For the GHUM mesh, we have body part labeling information available which is transferred onto PHUM. That is, for each vertex in the mesh we have its associated body part label class from a total of 14 body parts. When computing nearest neighbor vertex either consider vertices that exterior or on another body part than the query vertex, and we choose the one with the smallest distance. The body part labeling allows us avoid situations where interior vertex can choose exit the mesh geometry through a vertex from the same body part as itself.
Interpenetration loss pair of meshes M, and M2. we compute the generalized winding number test for both L ΦM, (Vi) and L &MM, (V2). The loss is then given by:
Interpenetration is straightforward to handle by our representation. Given
Lic Σ II - -NNNNN( 1.1, M2)
Σ - 10,--NN NN(V M,)
IEL.+
1EL
This time, when we compute the nearest neighbor for inside vertices of a mesh, we query from all vertices in the colliding mesh.
Relation Prior Work Whereas in our formulation self-collision and interpenetration collision are handled based on common unified representation, in the literature different approaches are used for each type of collision. For self-collision, most related work [9 20 21 22 uses bounding volume hierarchies (BVH) 23 to detect list of colliding triangles. Using local conic 3D distance fields, penetrations are penalized by the depth of the intrusion. The choice of penalizing collisions only for colliding triangles (at the border) is done solely for performance reasons (as pointed out in Section
5

---

3.2.4 from (20)). Ideally, one would use the full set of points inside the mesh and their distance to the surface, which our case. We compare the running time of the forward pass of the self- collision loss proposed in [2I] with that of our self-collision loss, based on PHUM and generalized winding numbers. For a batch size of 1, on one NVIDIA Tesla V100 32GB, our self-collision loss computation takes 33.5ms, while the one proposed in (2 takes 50.8ms. Differently from the BVH implementation, we did not write any custom CUDA kernel, but instead rely on standard TensorFlow 30 operations with batch support.
For interpenetration collision, the most popular approach 22 to first voxelize the two meshes and then compute a signed distance field (SDF) for each of them. The SDF of each mesh is queried by the vertices of the other mesh to compute the loss. The voxelization step takes around 45ms for mesh and the SDF computation is implemented using custom CUDA kernel. The running time is more than an order of magnitude higher than our generalized winding number computation and has considerably more complex implementation.
3.5 Depth-Ordering Loss
We follow the same procedure in 2 train using a depth-ordering loss. We use ground-truth instance segmentation masks from COCO [3] during training. For our differentiable mesh rasterizer we use DIRT 332 render the depth maps, DR. for each person. We use the depth ordinal loss:
Ld =
Σ PES
log(1 + exp( v(p) (p) - DD(p)(p))
(10)
where S = [p E v()) 0, (p)) > 0, v(p) * {p)),. I is the RGB image, v(p) is the index of the person occupying pixel p and j(p) is the ground-truth index of the person occupying pixel
3.6 Contact Losses
Similarly to 3. we employ contact losses when ground truth annotations are available. For a pair of people, Pı, P2. in contact in an image L, we define the contact signature (1, at the facet level (3] as a matrix where cf (I, Pı, P₂) 1 if facet S1 E M (P1) is in contact with facet ies
fe M (P2) and 0 otherwise, where M (P) the mesh corresponding to person P. The distance between two facets fi and fe is defined as the Euclidean distance d (fi. fz) between the centers of the facets 33. We define the following interaction-contac signature loss
PIP (.EM(P))/EEM(P3)
Σ
(11)
In similar fashion, the self contact signature Csep (I,P) 4 of a person P in an image is defined as Coop (I, P) Cics (T, P. P), where Cfi.fa (T, P) facet f2 M (P). Subsequently, the self-contact signature loss is defined as
facet fi E M (P) in contact with
Lecs P)
Σ
Coop (1, P)d(fi f2)
(12)
(EM{()).AEM(P)
In practice, contact signature matrices are sparse (have less than 100 non zero values) so the computa- tion of Lics and Leop while training models is efficient.
4 Experiments
In this section, we offer implementation details regarding the network architectures and training hyperparameters, we describe the datasets used for training and evaluating models, and we present sate-of-theeant results on the challenging Panoptic (33 and CHI3D 3 datasets, as well as several studies illustrating the importance of the proposed components and design choices.
4.1 Implementation details
For all our models we use a ResNet50 34 backbone pretrained on ImageNet 355 The network has total number of 26M parameters, out of which 23M parameters for the backbone and 3M parameters

---

for the transformer layers. We use L = 4 stages, an embedding size of 512 and 8 heads for the MultHeeaAAttentioon layer. The network is trained for 150 epochs with batch size of 32, starting learning rate of le 4 and exponential decay of 0.98. Our code is implemented in TensorFlow and we employ random scaling, rotation, horizontal flipping and cropping augmentations during training. All images are resized to 480 x 480 while preserving the original aspect ratio using padding when necessary, All networks are trained on a single V100 GPU with 32GB of memory.
4.2 Datasets
In our experiments, we train weakly supervised on various datasets with 2D supervision in the form of annotated body joints and instance segmentation masks. When available, we use vertex-level self-contact and person-to-persee contact supervision. Each dataset usually contains only subset of annotation types, so we employ the corresponding losses accordingly.
MPII (3) is an indoor/outdoor image dataset of 25K images containing multiple people with around 40K 2D pose annotations. We only use a subset of 10k images of this dataset for training.
Pose Track 37 is an in-the-wild dataset containing 1.3K video sequences with 2D poses for multiple people annotated across multiple frames. We use a subset of 25K images and 160K 2D pose annotations from the trainval subset of this dataset, for training only.
LSP 38 and LSP Extended (39 are in-the-wild datasets with single person 2D pose annotations. We use the training subsets consisting of approximately IK, respectively 10K images with ground truth 2D pose annotations.
COCO 31 is a dataset containing 40K images with multiple people. In total, 100K annotated poses are used for training our models. Similarly to [2, the instance person segmentation masks are used for computing the depth-ordering loss, in training only.
FlickrCI3D 3 and FlickrSC3D 4 contain in-the-wild images with multiple people engaged in various activities involving human interactions and self-contacts. Ground truth interactions and self-contact annotations are available for this dataset at facet granularity. These datasets have annotations for 2D keypoints, so we generate them by running a 2D human pose predictor on each of the images obtain pseudo ground-truth 2D pose annotations. We use total of 15K images with more than 25K interactions or self-contact annotations for training.
The Panoptic studio [33 is an indoor multiple people dataset captured in multi-view laboratory system. We use the same data selection as [16] 2) resulting in 9.6K images containing ground truth 3D pose for multiple people in each image. We use this dataset for evaluation purposes only.
CHI3D 3 is an indoor dataset with 3D ground truth joints from mocap containing images from 631 sequences and 4 cameras. Each sequence contains 2 people in various interactions and contact ground-truth annotations exist for each image. We use this dataset for evaluation purposes only.
Human3.6M 40 is a single-person 3d pose dataset containing sequences of people performing various activities, containing more than 3 million 3d skeletons. We use it only in one experiment (see table for evaluation under protocol P1 and use no 3d ground truth during training.
4.3 Results
Although our method is designed to reconstruct any number of people, we first evaluate REMIPS on the popular Human3.6M dataset (40] to ensure it is competitive in the simpler single-person scenario. For this experiment only, REMIPS is fine-tuned on the training images from the P1 protocol of Human3. 6m. We do this only self-supervised regime, i.e. fine-tuning only with supervision from predicted 2d keypoints, no ground truth 3d information being used. We compare favorably with I and [5] (table M]. which also report results of their methods trained without full 3d supervision.
We now start the multi perisopesonnn evaluation of our model on the challenging Panoptic (3) dataset. We follow the evaluation protocol from [16] I 2 (without training on any data from the Panoptic dataset). We present ofate-of -ftte-----h--att the-art art results in table showing that REMIPS performs better than other optimization- [16] inference-based multiple people reconstruction methods 2
We continue our evaluation on the CHI3D 3 dataset, which is not used during training. CHI3D consists of sequences of two interacting people, together with ground-truth motion capture markers,
7

---

Method
MPIPP-PA
MPIPE
HMR [3]
67.4
106.8
HUND
66.0
91.8
REMIPS (ours)
64.3
96.1
Table 1: Evaluation on Human3.6M, dataset [40]
protocol PI. We report mean per joint positional error (MPJPE) and its version including Procrustes Analysis (MPJPE-PA). All errors are reported in mm.
Method
Haggling
Mafia
Ultimate
Pizza
Mean
Zanfir al. [16]
140.0
165.9
150.7
156.0
153.4
Zanfir et al. [
141.4
152.3
145.0
162.5
150.3
Jiang et al. [2]
129.6
133.5
153.0
156.7
143.2
REMIPS (ours)
121.6
137.1
146.4
148.0
138.3
Table 2: Performance on the Panoptic 33
dataset for pose and shape reconstruction methods focusing on multiple people. All errors represent mean per per joint position errors and reported in mm, relative the root Our model achieves lower error compared to
joint. The evaluation protocol the same the one [6 [2] optimization- or learning- -aanin approaches.
3d joints and GHUM parameters available for one of the two. For each sequence, one contact frame is annotated with an interaction contact signature. We use the same evaluation protocol as in Fieraru al. K and, for the contact frames, we compute the 3D joint error for the prediction which has the highest 2D bounding box overlap with the ground truth. In table we present ofate-of-the-- the-art results when we compare again against optimization and inference only [2] methods. To run the model from on the CHI3D B dataset, we use their publicly available code repository.
Method
MPJPE
MPVPE
Translation Error
#2D
#3D
Fieraru
3
125.4
368.0
N/A
N/A
Jiang et al. (2
136.0
N/A
N/A
100K
300K
REMIPS (ours)
120.8
134.7
284.1
115K
0
Table 3: Performance on the CHI3D 33 dataset for multiple person pose and shape reconstruction methods. (MPJPE), the mean per vertex position (MPVPE) and the translation error. All errors are reported in mm and are relative to the root joint. Our method has lower errors compared to the other optimization and inference based methods. We also compare the number of #2D and #3D annotations used as supervision during the training of the different models. Our models use no #3D and achieve better performance the challenging dataset CHT3D 3
In columns 2, 3, 4 we show the
per joint position
In table we present series of ablations on the CHI3D 3 dataset for the loss components in eq, that are used during training. We also show the results for version of REMIPS that does not take pseudo ground-truth truth camera intrinsics parameters input. The lowest error is achieved using all of the introduced losses and inputting camera intrinsics, which supports their individual importance.
Method
MPJPE
MPVPE
REMIPS (w/o Cic)
122.7
134.9
REMIPS (w/o L.c)
123.5
137.1
REMIPS (w/o Cics)
127.1
142.1
REMIPS (w/o Lecs)
125.3
140.0
REMIPS (w/o camera instrinsics)
139.0
158.0
REMIPS (ours)
120.8
134.7
Table 4: Ablation study on the CHI3D dataset for multiple person pose and shape reconstruction methods. We report mean per joint positional error (MPJPE) and mean per vertex positional error (MPVPE). All errors are reported in mm and are relative to the root joint. The model trained with all proposed losses Lie, Lac, Lies Lues and taking camera intrinsics as input achieves the lowest error.
To study how the number of people in the scene and the spatial context affect the performance of REMIPS, we run our method in three different ways and introduce a new baseline based on the single person body and pose estimator SPIN [4]. We evaluate each scenario on the Panoptic dataset [3 where we report the MPJPE under the same protocol as in [66 2, averaged across all frames and actions. We propose the following experiments: (1) multi-person setting where the input image is crop around all detected persons and bounding boxes are provided to the transformer together

---

for all persons in the crop (this is the original version of REMIPS); (2) single-perse setting with context where the input image is a crop around all detected persons and each bounding box is provided separately to REMIPS (i.e. one bounding box token at a time); (3) single-person setting without context- where each input image consists of a crop of only one detected person and REMIPS receives only its bounding box (i.e. like in a standard single-person 3d pose reconstruction setting, such as in SPIN [14], HMR [33, HUND [5)): (4) single-person setting without context (baseline with SPIN (4) where each input image consists of a crop of only one detected person that, this Comparing scenarios (1) and (2),
time, is processed by SPIN 14. Results are illustrated in table 5
Method
MPJPEE
(1) pertii (REMIPS)
138.3
(2) single-person w. context (REMIPS)
138.2
(3) single-person w/o context (REMIPS)
141.7
(4) single-person w/o context (SPIN)
164.9
Table 5: Performance comparison on the Panoptic 333 dataset for different scenarios of REMIPS and a SPIN baseline. Scenarios (1) and (2) suggest that pose accuracy is not affected by the number of people detections. Comparing (2) and (3), we observe the benefits of providing the large context of the image as input, while comparing (3) with (4) highlights the accuracy of REMIPS over recent methods, even in the tight crop single-person scenario.
results are almost identical in terms of pose accuracy, leading to suspect that, at inference time, attention to the image feature tokens is the most important part of the transformer architecture, given larger context. This means that spatial image features already encode information regarding the other persons in the scene, due to the way the network is trained (i.e. reasoning about all the persons once). Pose accuracy is not affected by the number of people in the input as long as the context is large/informative enough. Comparing scenarios (2) and (3), we see the benefit of providing the large context of the scene as input (3.5mm accuracy improvement). Comparing scenarios (3) and (4), we observe that even in a single-person scenario with crops tight around each person, REMIPS performs much better than SPIN significant difference of 23. 2mm). Note that neither of the two methods was trained on Panoptic, and our method has not used any 3d ground-truth mesh supervision. We observe large errors for SPIN in cases of occlusions and close people interactions.
To study the effect that the distance between a person and the camera has on REMIPS, we perform another study the Panoptic dataset. We downscale our input image by factors of 2, 4 and 8, and then upscale it back to our working resolution of 480 x 480, to simulate increasing the distance from the camera. The results, reported in MPJPE over all frames of the Panoptic dataset, are as follows: 138.3mm (factor of 1- no downscaling), 140. 2mm (factor of 2), 15m.7mm (factor of 4) and 210. 110.5mm (factor of 8). As expected, higher resolution of the input image correlates with better pose estimates. Critical degradation is observed only for very small people occupying a few pixels (factor of 8).
show visual reconstructions of REMIPS on in-the-will wild photos from the COCO validation set. Note the reconstruction quality even in high occlusion cases (cither partial views or occlusion by the environment) and the robustness to the size of the humans in the image, ranging from very far away to very close to the camera. Additionally, in figure 3 we show reconstruction results samples from the test split of CHI3D, with and without using the collision and contact losses. It is to be noted the better physical plausibility when using all losses.
In figure 2,
Conclusion
To better reconstruct and analyse scenes with multiple interacting people, we have emphasized the importance of attention modeling and the role of weak supervision for scalability and good generalisation. We have introduced REMIPS, new transformer- for multiple person 3D pose and shape estimation from RGB images, which relies on GHUM [6]. a generative and articulated full body statistical 3D model. Our network produces physically plausible human mesh reconstructions with realistic scene placement under a fully perspective camera projection model. We also introduce new collision losses uniformly applicable to both self-contacts and inter-person contact, and more efficient than previously proposed models by an order of magnitude. We demonstrate state-of-the-art 3d human pose and shape reconstruction results on challenging datasets like CHI3D 3 or Panoptic 33 which contain multiple people involved in complex interactions.
neural network architecture
9

---

Figure 2: 3D human pose and shape predictions on the COCO validation set (rows 1-5) for in-the- wild images. We show the initial image together with an overlaid reconstruction of the meshes as well as rendering from a different viewpoint which better illustrates the physical consistency of the REMIPS reconstructions. Best seen color.
Figure 3: Visual 3D human pose and shape predictions on the CHI3D datasets. On columns 2-3 we show the results of a model which is not using any the 4 losses Lies Lses Lies, Lses, while on columns 4-5 we illustrate the predictions of our full REMIPS model, trained using the previously mentioned losses. We observe more physically plausible results when training with all losses. Best seen in color.
10

---

Acknowledgments and Disclosure of Funding
This work was supported in part by the ERC Consolidator grant SEED, CNCS-UEFISCDI (PN-IIII P4-DDPCCCF-201--1119 and SSF.
References
[1] Andrei Zanfir, Elisabeta Marinoiu, Mihai Zanfir, Alin-lonut Popa, and Cristian Sminchisescu. Deep network for the integrated 3d sensing of multiple people in natural images. In NIPS, 2018.
[] Wen Jiang, Nikos Kolotouros, Georgios Pavlakos. Xiaowei Zhou, Kostas Daniilidis. Coherent reconstruction of multiple humans from single image. In CVPR, pages 5579-5588, 2020.
Mihai Fieraru, Mihai Zanfir, Elisabeta Oneata, Alin-Ionut Popa, Vlad Olaru, and Cristian Sminchisescu. Three-dimensional reconstruction of human interactions. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[4] Mihai Fieraru, Mihai Zanfir, Elisabcta Oneata, Alin-lonut Popa, Vlad Olaru, and Cristian Sminchisescu. Learning complex 3d human self-contact. In Thirty- Fifth AAAI Conf. on Artificial Intelligence (AAAI'21), 2021.
[5] Andrei Zanfir, Eduard Gabriel Bazavan, Mihai Zanfir, William Freeman, Rahul Sukthankar, Cristian Sminchisescu. Neural descent for visual 3d human pose and shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages (4484-1449) June 2021.
[6] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, Bill Freeman, Rahul Sukthankar, and Cristian Sminchisescu. GHUM & GHUML: Generative 3D human shape and articulated pose models. CVPR, 2020.
[7] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: skinned multi-person linear model. (SIGGRAPH), 34(6))248:1116, 2015.
[8] Shih-En Wei. Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In CVPR, June 2016.
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: realtime multi-person 2d pose estimation using part affinity fields. arXiv preprint arXiv: 18/2.0808,,, 2018.
[10] Natalia Neverova, Riza Alp Guler, and lasonas Kokkinos. Dense pose transfer. In (ECCV), September 2018.
[11] A.-L. Popa, M. Zanfir, and C. Sminchisescu. Deep multitask architecture for integrated 2d and 3d human sensing. CVPR, 2017.
[12] Andrei Zanfir, Eduard Gabriel Bazavan, Hongyi Xu, Bill Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Weakly supervised 3d human posc and shape reconstruction with normalizing flows. ECCV, 2020.
[13] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018
[14] Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, and Kostas Danilidis. Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In ICCV, 2019.
[15] Vasilcios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Monocular expressive body regression through body-driven attention. In European Conference on Computer Vision (ECCV), 2020.
[16] Andrei Zanfir, Elisabeta Marinoiu, and Cristian Sminchisescu. Monocular 3d pose and shape estimation of multiple people in natural scenes the importance of multiple scene constraints. In CVPR, 2018.
[17] Jianfeng Zhang, Dongdong Yu, Jun Hao Liew, Xuecheng Nie, Jiashi Feng. Body meshes as points. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 546-556, 2021.
[18] Lea Müller, Ahmed A. A. Osman, Siyu Tang, Chun-Han P. Huang, and Michael Black. On self-contact and human pose. In Proceedings IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), June 2021.

---

[19] Luca Ballan, Aparna Taneja, Jürgen Gall, Lue Van Gool, and Mare Pollefeys. Motion capture of hands in action using discriminative salient points. In ECCV, 2012.
[20] Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Mare Pollefeys, and Juergen Gall. Capturing hands in action using discriminative salient points and physics simulation. I/CV, 2016.
[21] Georgios Pavlakos, Vasileios Choutas, Nima Chorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black, Expressive body capture: 3d hands, face, and body from single image. In CVPR, 2019.
[22] Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J. Black. Resolving 3D human pose ambiguities with 3D scene constraints. In ICCV, 2019.
[23] Matthias Teschner, Stefan Kimmerle, Bruno Heidelberger, Gabriel Zachmann, Laks Raghupathi, Arnulph Thalmann, Wolfgang Strasser, et al. Collision
Fuhrmann, M-P Cani, François Faure, Nadia Magnenat-Thalmann, detection for deformable objects. In Computer graphics forum, volume 24, pages 61-81, 81. Wiley Online Library, 2005.
[24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010.111
2020.
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
[26] Mihai Zanfir, Andrei Zanfir, Eduard Gabriel Bazavan, William T. Freeman, Rahul Sukthankar, and Cristian Sminchisescu. Thundr: Transformer-based 3d human reconstruction with markers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 12971-12980, October 2021.
[27] Kevin Lin, Lijuan Wang, and 7icheng Liu. End-to-end human pose and mesh reconstruction with transformers. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 1954-1963, 2021.
[28] Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang. and Hao Li. On continuity rotation representa- tions in neural networks. arXiv preprint arXiv:1812.070 27035, 2018.
[29] Alec Jacobson, Ladislav Kavan, and Olga Corkine-Hornung Robust inside-outside segmentation using generalized winding numbers. ACM Transactions on Graphics (TOG), 32(4):1-12, 2013.
[30] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jezefowicz. Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015, Software available from tensorflow.org.
[31] Tsung- Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zimicck. Microsoft coco: Common objects context. In ECCV, 2014.
[32] Paul Henderson and Vittorio Ferrari. Learning single-image 3D reconstruction by generative modelling of shape, pose and shading. International Journal of Computer Vision, 2019.
[33] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In ICCV, 2015.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[35] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR, 2009.
[36] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014.
[37] M. Andriluka, U. Igbal, E. Ensafutdinov, L. Pishchulin, A. Milan, J. Gall, and Schiele B. PoseTrack: A benchmark for human pose estimation and tracking. In CVPR, 2018.
12

---

[38] Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC, 2010.
[39] Sam Johnson and Mark Everingham. Learning effective human pose estimation from inaccurate annotation. In CVPR 2011, pages
IEEE, 2011.
[40] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3.66 3.6M: Large scale datasets and predictive methods for 3d human sensing in natural environments. PAMI, 2014.
13


Reviews:
[{"review_id": "qBGJ5D7lswj", "reviewer": "Reviewer_mtiJ", "summary": "This paper proposes a method for estimating multiple human pose & shape from monocular RGB input, which leverages a novel convolutional-transformer structure to optimize the detector-based human pose, shape and RT information. Also, it introduces a mesh-approx. based method for jointly formulating self-collision and interpenetration, which tackles the problem of treating two kinds of collisions in the same framework. Combined those methods, this paper tackles the problem of multiple human pose & shape estimation under challenging scenes.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows very well innovation, reasonable technical ideas with robust results and sufficient analysis. The reference is comprehensive.\nAccording to the demonstrated results, this paper shows robust results dealing with different challenging scenes, including multiple human strong occlusion, incomplete human body capturing, and strong interactions between humans. The reconstructed human poses and 3D position relations between different humans are visually accurate according to the input images. The comparisons show its robustness compared with previous works.\nThe proposed convolutional-transformer structure combines the architecture of CNN and transformers, and takes the CNN features and human-detection features to initialize the transformer inputs, which may inspire future works on studying better detection-reconstruction combination methods. The proposed mesh-approx. based collision resolving method moves a step forward for formulating 3D collision in deep learning networks.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The first concern is that, although the pose and positions are estimated accurately, the estimated human shapes are slightly inaccurate, which may be further improved. \nAnother concern is that, although it is not fair to compare this work with monocular single-person pose & shape estimation methods, this paper will show more profundity by comparing with those methods and analyzing the results.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "jgLGo0aScAL", "reviewer": "Reviewer_rGCr", "summary": "This paper presents a network and learning scheme for reconstructing multiple people from monocular images. The proposed approach includes a transformer to iteratively refine the reconstruction results, self-collision and interpenetration-collision losses based on the generalized winding numbers, and self-contact and interaction-contact losses. Qualitative results are impressive and strong.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This submission basically includes incremental upgrades of existing solutions for single/multiple person reconstruction and mesh collision detection, though these solutions bring performance gains over previous approaches for multiple person reconstruction.\n\nMethodologically, though using the transformer to leverage spatial features and update the parameters of the reconstruction is somewhat interesting, the transformer for mesh reconstruction has been investigated in [Lin et al.], which is not discussed in the paper. \n\n[Lin et al.] End-to-End Human Pose and Mesh Reconstruction with Transformers \n\nFor collision detection, exploiting generalized winding numbers is appreciated, which provides a more efficient solution for self-collision and interpenetration penalization in multiple person reconstruction. However, it only affects the runtime during training. There is no ablation study or discussion on the performance of different solutions for collision detection. \n\nOther minor issues:\n\nSection 3.3 introduces the network architecture of using a transformer to predict the reconstruction results. Adding an illustration figure for the transformation of different features should make this part more readable.\n\nThere are some typos in the manuscript:\nL35: deffer -> defer\nL38: \"so their 3D spatial relations are plausible, and the alignment with the image evidence is good.”?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In Line 114, does the $N_P = 16$ mean that the maximum number of people in an image is 16? Do the number of people in the input image affect the performance of the transformer? Moreover, how do the solution and scale of the person in the image affect the reconstruction?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "i0XinN8Wbw", "reviewer": "Reviewer_y5WN", "summary": "This paper presents a method for 3D shape and pose estimation of multiple persons from RGB images. The network utilizes numerous off-the-shelf techniques available from the literature and employs a transformer network to combine this information aided by a self-collision and an interpenetration loss.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The paper is fairly well written.\n2. The paper addresses an important and a valid problem.\n3. The model seems to achieve good results in challenging conditions.\n\nWeaknesses:\n\n1. The paper uses a collection of state-of-the-art models, but the actual novelty of the method itself is limited.\n2. The performance of the approach is directly limited by the performance of used off-the-shelf methods.\n3. The method seems to consist of a lot of steps without proper explanations or motivations (at least the authors have failed to describe them). For example, Sec. 3.3. presents a lot of design choices but lacks an explanation on \"why?\". For each design choice given in Sec. 3.3, there are many alternatives. The authors should justify these choices either theoretically or using a thorough ablation study. Table 3 presents an ablation study, but it only shows the impact of loss components, which is not enough.\n4. There are many different moving parts, hyper-parameters, and design choices associated with the presented method. This raises serious doubts about the robustness of the method against the slightest changes of hyperparameters and conditions (or different datasets). At least the authors should have discussed some guidelines or motivations (or at least the effects of) toward choosing hyper-parameters. In general, the performance of any model is sensitive to hyper-parameters and design choices. But in a model that is a collection of a lot of different components, this is a significant problem that should be clearly discussed. \n5. The experiments section is not thorough. More experimental validation is necessary to conclude the usefulness of the model. If the paper presented a theoretically rigorous model, it is acceptable to weaken the experiments section. However, since the proposed model is mainly based on heuristics, the experiment section needs to highlight the strengths of the approach more. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper has not discussed the limitations of the method adequately. The authors should discuss limitations associated with the points mentioned under \"weaknesses\" more comprehensively.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "bR7ZTDAIzKv", "reviewer": "Reviewer_ueL1", "summary": "This paper presents a method to reconstruct multi-people in 3D from a single image. Different from popular approaches where a neural network pose estimator focuses on a single person given a bounding box, the proposed method estimates multiple people given multiple boxes at once, in particular considering their spatial layout and physical consistency. For that objective, the paper presents several novel aspects including a transformer framework, collision and interpenetration loss functions. Comparisons over previous approaches in the similar domain (multi-person) are performed in the public benchmarks (Panoptic[32] and CHI3d[3]) and the proposed framework outperforms the previous approaches. \n\nOverall, the paper is in a good shape, containing several interesting components. Additional experiments on the other more popular dataset (3DPW) would be still needed. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The use of transformer architecture for multi-people 3D pose estimation purposes is interesting. I think that this is the first paper in that direction. The use of global image features with box features are convincing for me, to better incorporate the contextual information of the scene. \n\n- The experiments on public benchmarks (Table 1 and 2) support the strength of the method over approaches in the same domain. The ablation study performed in Table 3 is also helpful to better understand the proposed models. \n\n- This paper also puts effort into other details such as introducing self-collision loss and interpenetration loss. This is encouraging, even though additional experiments over alternative options would be necessary to show clearer advantages. \n\n- I also like the fact that the paper is based on GHUM over the more popular human model, SMPL, particularly in terms of diversifying the framework in the 3D human pose estimation field. \n\nWeaknesses:\n\n- Missing references: There are recent work where transformer architectures are used in different ways. It would be helpful to include them with discussions. \nPose and Mesh Reconstruction with Transformers, Lin et al., CVPR 2021\nLifting Transformer for 3D Human Pose Estimation in Video, Li et al., 2021\n\n- The paper ignores other popular 3D pose estimation benchmarks such as H36M and 3DPW. Even though the paper focuses on multi people scenarios, it would be still necessary to evaluate its performance in these datasets. In particular, note that the similar goal can be achieved via regression + optimization, for example, by (1) first estimating individual people per box (2) adjusting their spatial layout via an optimization. See the following paper:\n\nPerceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild, Zhang et al., 2020\n\nI understand the proposed method is trained without any 3D dataset (e.g., no H36m)? In this case,  I am concerned about its accuracy over the recent SOTA paper such as SPIN or VIBE. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. In section 6. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received 4 diverging final ratings: 4, 6, 7, 6.
On the positive side, the reviewers mentioned importance of the problem, an overall interesting approach to the multi-person setting, strong performance and clear presentation. At the same time, some of the reviewers were skeptical about overall novelty (collection of known components), insufficient motivation of certain design choices and high overall complexity, lack of important ablations and evaluation on common datasets. Some of these concerns were addressed in the rebuttal, as acknowledged by the reviewers. The most skeptical reviewer did not engage in further discussion with the authors.
Overall, the AC agrees with the positively-inclined reviewers that the strengths of this work outweigh its weaknesses, and the paper presents an interesting approach that can potentially have an impact. The authors are highly encouraged to address the remaining concerns in the camera ready version. The final recommendation is to accept as a poster.
Decision: Accept (Poster)