REMIPS: Physically Consistent 3D Reconstruction of Multiple Interacting People under Weak Supervision

1 Introduction . Reconstructing three-dimensional models of multiple interacting people from images is an important computer vision task with applications in behavior analysis , automatic video analysis of sport events , or collaborative augmented reality applications . As the demands on the depth of analysis increase , beyond qualitative pose , one often seeks to know whether people are in contact or not , what is the nature of that contact , and how long it lasts . As such questions become important , it is clear that very coarse estimates of pose or even shape are no longer sufficient . One needs accurate models of shape and contact with predictable response over time . This is difficult due to the various degree of occlusion that occurs , the depth ambiguities particularly given only monocular images ( but more generally given that occlusion and self-occlusion for people are frequent ) , the high degree of variability of even valid human poses , their different scales and underlying spatial image support , ranging from the large body parts like torso or thighs – informed by larger image regions – to hands , with parts typically accounted for by smaller spatial support , comparatively . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Most of the volumetric human pose and shape recovery methods focus on a single person and require detecting the humans in the image and then running the inference model on each positive response . This approach has achieved good results and scales linearly in the number of people in the scene . There are multi people reconstruction methods which either predict 3D pose only and defer the shape reconstruction step to a later optimization step [ 1 ] or rely on an orthographic camera model [ 2 ] . However , it is nontrivial to obtain a realistic 3D scene placement for all the reconstructed human models such that their 3D spatial relations are plausible and the alignment with the image evidence is good . All aforementioned methods train using full 3D supervision coming from mocap datasets . However , this often lacks scenarios of interactions or fine grained self-contact . Recent work such as [ 3 , 4 ] designed datasets for interaction analysis , with additional interaction signature annotations , and showed that models able to represent those produce more realistic reconstructions . However , they only work in an optimization framework and reconstruct only pairs of people in close interaction . Ideally one would like to be able to precisely model contacts , rely on attention models to identify the important element of interaction , have a framework that can accommodate more than two people and learn such models from data , using only weak supervision . Our main contributions can be summarized as proposing the following : ( 1 ) fast , accurate and unified 3d self collision and interpenetration models for multiple people ; ( 2 ) a novel vision transformer architecture to predict 3D pose and shape for multiple people ; ( 3 ) weakly supervised models which do not require 3D annotations during training ; ( 4 ) state of the art results on challenging datasets , with favorable performance compared to competing predictive or optimization-based methods . In this paper , we propose REMIPS , a hybrid convolutional-transformer where the output of a human detector is combined with high-level convolutional image features . They are processed by a series of transformer encoder layers that iteratively refine the 3D pose and shape reconstruction estimates for all detected humans in the scene , irrespective of their number . Decoupling the detection component has the advantage of benefiting from innovations in that space without necessarily paying a performance cost ( detection models are already quite fast , close to real time , and providing accurate results ) . Similar to [ 5 ] , we use a full perspective camera model , allowing us to infer translations for all humans in the same camera coordinate system . We propose novel interpenetration and self-collision losses that are amenable to a deep-learning-based training process and work in a weakly-supervised regime . 2 Related Work . The field of 3D human pose and shape reconstruction from images has received increasing attention in recent years . This has been driven by progress in 3D statistical and articulated human body models [ 6 , 7 ] and advances in 2D [ 8 , 9 , 10 ] and 3D [ 1 , 11 ] pose estimation . Most reconstruction methods focus on estimating the 3D pose and shape of a single person [ 5 , 12 , 13 , 14 , 15 ] although , more recently , several papers focused on reconstructing multiple people coherently [ 2 , 16 , 1 , 17 ] . Recent work started focusing on modeling inter-human contact [ 3 ] or self-contact [ 4 , 18 ] , developing methodology for more principled , contact aware reconstructions and releasing datasets with ground truth interaction [ 3 ] and self-contact [ 4 ] vertex-level annotations . We use these datasets in our training and evaluation pipeline , together with other various supervision signals . Physically plausible human mesh reconstructions require collision avoidance for single and multiple interacting people . Prior work related to self-collisions [ 19 , 20 , 21 , 22 ] rely on bounding volume hierarchies [ 23 ] to detect the set of colliding triangles . We propose novel , efficient and faster human mesh collision losses which are more appropriate for deep learning training pipelines . See §3.4 for more details . Recent innovations in the space of visual transformers [ 24 , 25 ] have increased the capabilities of neural network architectures for handling image features , and their ability to produce accurate reconstructions for images in-the-wild [ 26 , 27 ] . We take inspiration from this as well . 3 Methodology . 3.1 Statistical 3D Human Body Models . We use a recently introduced statistical articulated 3D human body model called GHUM [ 6 ] , to represent the pose and the shape of the human body . The model has been trained end-to-end , in a deep learning framework , using a large corpus of human shapes and motions . The model has generative body shape and facial expressions β = ( βb , βf ) represented using deep variational auto-encoders and generative pose θ = ( θb , θlh , θrh ) for the body , left and right hands , respectively , represented as normalizing flows [ 12 ] . The pelvis translation and rotation are controlled separately , and represented by a 6D rotation [ 28 ] r ∈ R6×1 and a translation vector t ∈ R3×1 w.r.t the origin ( 0 , 0 , 0 ) . The GHUM mesh , M = ( V , F ) , consists of a set of vertices V ∈ R10168×3 and a set of facets F ∈ R20332×3 . In experiments we infer the GHUM parameters ( θb , βb , r , t ) omitting facial expression parameters βf and the left and right hand pose parameters θlh , θrh , as we focus only on reconstructing the body pose and shape . We also drop the b subscript for convenience . 3.2 Camera Model . We use a perspective projection operator Π characterized by camera intrinsics C = [ fx , fy , cx , cy ] which assumes that the camera coordinate system is aligned with the world reference frame using an identity extrinsic rotation matrix and zero translation . This corresponds to a left handed coordinate system with positive z values forward . All predictions are made in the camera coordinate system . When processing the camera intrinsics for a crop we use the same approach as in [ 5 ] . 3.3 Architecture . In fig . 1 we show an overview of REMIPS , our proposed hybrid learning architecture for monocular multi-person 3D body pose and shape estimation . We draw inspiration from vision transformers [ 24 ] , as we also use a hybrid convolutional-transformer architecture , and from [ 5 ] , as we explore the idea of iteratively refining estimates by relying on cascaded , input-sensitive processing blocks , with homogeneous parameters , as an end-to-end learnable surrogate for non-linear optimization . Our network receives as input an image I ∈ RW×H×3 containing multiple people , together with the pseudo ground-truth camera intrinsics C ∈ R1×4 of the image ( see § 3.2 ) . We apply a convolutional neural network ( CNN ) on the input image and extract a downsampled feature map representation F ∈ RW32×H32×D . We flatten the feature map along the spatial dimensions to get a sequence of N = W32 × H 32 tokens . This sequence is linearly embedded by means of matrix E ∈ R D×D′ , where D′ is the embedding dimensionality . Next , learnable positional embeddings Epos ∈ RN×D ′ are added to the sequence . In addition to image feature tokens , we also use a sequence of NP = 16 maximum tokens for the people in the image . We first run an off-the-shelf human detector on the input image and collect the bounding box coordinates for each person index p , bp ∈ R1×4 . For each person p , we compute its initial GHUM parameters sp0 = { θ0 , β0 , r0 , t p 0 } where we compute the translation such that the default GHUM mesh projects in the center of the corresponding bounding box bp . We denote by S0 = [ s10 , . . . s NP 0 ] the initial state of GHUM parameters for all the people in the image . For each person , we concatenate the bounding person coordinates , the initial GHUM state and the camera intrinsics and construct the sequence of tokens corresponding to the persons in the image P ∈ RNP×DP . We linearly project the people token sequence P by means of matrix EP ∈ RDP×D ′ . The two sequences of tokens are concatenated into a single input sequence Z0 and iteratively transformed through a single shared transformer encoder layer [ 25 ] , TL , for a number of L steps . We collect at each step l ∈ { 1 . . . L } a refinement update ∆Sl from each transformed representation Zl , using a shared MLP applied on the corresponding transformed representation of the people token sequence . Z0 = [ FsE + Epos , PEP ] ( 1 ) Zl = TL ( Zl−1 ) ( 2 ) ∆Sl = MLP ( ZN : N+NPl ) . ( 3 ) The refinement updates ∆Sl are added to the initial state GHUM parameters , S0 , as SL = S0 + λΣ L l=1∆Sl ( 4 ) where SL = { ΘL , BL , RL , TL } are the final GHUM parameter estimates for all the people detected in the image and λ is a weighting term for the residual estimates . Note that the maximum number of person tokens NP = 16 is chosen specifically for training purposes , to batch training examples together . Yet , at inference time , we can actually use as many person tokens as the person detector outputs ( even more than 16 ) , as long as the quadratic memory requirements for the attention mechanism are satisfied ( i.e . in the order of thousands of persons ) . We observe that , in practice , the differences in inference speed for a variable number of persons ( between 1 and 32 ) to be almost negligible , i.e . at most 10 % increase in running time . We train the network by optimizing the following loss function : L = Lsc + Lic + Lics + Lscs + Ld + Lka + Lθ + Lβ ( 5 ) where the first 5 loss terms correspond to the physical plausibility of the reconstructed meshes as follows : Lsc and Lic are the self-collision loss and the interpenetration loss , respectively ; Lics and Lscs correspond to the supervised self-contact [ 4 ] and interaction-contact [ 3 ] losses ; Ld is the depth-aware ordering loss from [ 2 ] . These losses are discussed and explained in more detail in subsequent sections . We use a standard keypoint alignment loss , Lka , that measures the error between the 3D joints from the GHUM reconstructions , projected onto the image plane , J = { Ji } i=1 , ... , K , and the available 2D keypoints annotations in the image { ji } i=1 , ... , K Lka = 1 K K∑ i=1 ‖ji −Π ( Ji , C ) ‖22 . ( 6 ) Note that for some datasets we do not have 2D pose annotations available . In that case we supervise with the predictions of an off-the-shelf 2D pose estimator . For body shape and pose , we use regularization losses : Lθ = ‖θ‖22 , Lβ = ‖β‖22 . ( 7 )

Summary: This paper presents a method to reconstruct multi-people in 3D from a single image. Different from popular approaches where a neural network pose estimator focuses on a single person given a bounding box, the proposed method estimates multiple people given multiple boxes at once, in particular considering their spatial layout and physical consistency. For that objective, the paper presents several novel aspects including a transformer framework, collision and interpenetration loss functions. Comparisons over previous approaches in the similar domain (multi-person) are performed in the public benchmarks (Panoptic[32] and CHI3d[3]) and the proposed framework outperforms the previous approaches.   Overall, the paper is in a good shape, containing several interesting components. Additional experiments on the other more popular dataset (3DPW) would be still needed.