REMIPS: Physically Consistent 3D Reconstruction of Multiple Interacting People under Weak Supervision

1 Introduction . Reconstructing three-dimensional models of multiple interacting people from images is an important computer vision task with applications in behavior analysis , automatic video analysis of sport events , or collaborative augmented reality applications . As the demands on the depth of analysis increase , beyond qualitative pose , one often seeks to know whether people are in contact or not , what is the nature of that contact , and how long it lasts . As such questions become important , it is clear that very coarse estimates of pose or even shape are no longer sufficient . One needs accurate models of shape and contact with predictable response over time . This is difficult due to the various degree of occlusion that occurs , the depth ambiguities particularly given only monocular images ( but more generally given that occlusion and self-occlusion for people are frequent ) , the high degree of variability of even valid human poses , their different scales and underlying spatial image support , ranging from the large body parts like torso or thighs – informed by larger image regions – to hands , with parts typically accounted for by smaller spatial support , comparatively . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Most of the volumetric human pose and shape recovery methods focus on a single person and require detecting the humans in the image and then running the inference model on each positive response . This approach has achieved good results and scales linearly in the number of people in the scene . There are multi people reconstruction methods which either predict 3D pose only and defer the shape reconstruction step to a later optimization step [ 1 ] or rely on an orthographic camera model [ 2 ] . However , it is nontrivial to obtain a realistic 3D scene placement for all the reconstructed human models such that their 3D spatial relations are plausible and the alignment with the image evidence is good . All aforementioned methods train using full 3D supervision coming from mocap datasets . However , this often lacks scenarios of interactions or fine grained self-contact . Recent work such as [ 3 , 4 ] designed datasets for interaction analysis , with additional interaction signature annotations , and showed that models able to represent those produce more realistic reconstructions . However , they only work in an optimization framework and reconstruct only pairs of people in close interaction . Ideally one would like to be able to precisely model contacts , rely on attention models to identify the important element of interaction , have a framework that can accommodate more than two people and learn such models from data , using only weak supervision . Our main contributions can be summarized as proposing the following : ( 1 ) fast , accurate and unified 3d self collision and interpenetration models for multiple people ; ( 2 ) a novel vision transformer architecture to predict 3D pose and shape for multiple people ; ( 3 ) weakly supervised models which do not require 3D annotations during training ; ( 4 ) state of the art results on challenging datasets , with favorable performance compared to competing predictive or optimization-based methods . In this paper , we propose REMIPS , a hybrid convolutional-transformer where the output of a human detector is combined with high-level convolutional image features . They are processed by a series of transformer encoder layers that iteratively refine the 3D pose and shape reconstruction estimates for all detected humans in the scene , irrespective of their number . Decoupling the detection component has the advantage of benefiting from innovations in that space without necessarily paying a performance cost ( detection models are already quite fast , close to real time , and providing accurate results ) . Similar to [ 5 ] , we use a full perspective camera model , allowing us to infer translations for all humans in the same camera coordinate system . We propose novel interpenetration and self-collision losses that are amenable to a deep-learning-based training process and work in a weakly-supervised regime . 2 Related Work . The field of 3D human pose and shape reconstruction from images has received increasing attention in recent years . This has been driven by progress in 3D statistical and articulated human body models [ 6 , 7 ] and advances in 2D [ 8 , 9 , 10 ] and 3D [ 1 , 11 ] pose estimation . Most reconstruction methods focus on estimating the 3D pose and shape of a single person [ 5 , 12 , 13 , 14 , 15 ] although , more recently , several papers focused on reconstructing multiple people coherently [ 2 , 16 , 1 , 17 ] . Recent work started focusing on modeling inter-human contact [ 3 ] or self-contact [ 4 , 18 ] , developing methodology for more principled , contact aware reconstructions and releasing datasets with ground truth interaction [ 3 ] and self-contact [ 4 ] vertex-level annotations . We use these datasets in our training and evaluation pipeline , together with other various supervision signals . Physically plausible human mesh reconstructions require collision avoidance for single and multiple interacting people . Prior work related to self-collisions [ 19 , 20 , 21 , 22 ] rely on bounding volume hierarchies [ 23 ] to detect the set of colliding triangles . We propose novel , efficient and faster human mesh collision losses which are more appropriate for deep learning training pipelines . See §3.4 for more details . Recent innovations in the space of visual transformers [ 24 , 25 ] have increased the capabilities of neural network architectures for handling image features , and their ability to produce accurate reconstructions for images in-the-wild [ 26 , 27 ] . We take inspiration from this as well . 3 Methodology . 3.1 Statistical 3D Human Body Models . We use a recently introduced statistical articulated 3D human body model called GHUM [ 6 ] , to represent the pose and the shape of the human body . The model has been trained end-to-end , in a deep learning framework , using a large corpus of human shapes and motions . The model has generative body shape and facial expressions β = ( βb , βf ) represented using deep variational auto-encoders and generative pose θ = ( θb , θlh , θrh ) for the body , left and right hands , respectively , represented as normalizing flows [ 12 ] . The pelvis translation and rotation are controlled separately , and represented by a 6D rotation [ 28 ] r ∈ R6×1 and a translation vector t ∈ R3×1 w.r.t the origin ( 0 , 0 , 0 ) . The GHUM mesh , M = ( V , F ) , consists of a set of vertices V ∈ R10168×3 and a set of facets F ∈ R20332×3 . In experiments we infer the GHUM parameters ( θb , βb , r , t ) omitting facial expression parameters βf and the left and right hand pose parameters θlh , θrh , as we focus only on reconstructing the body pose and shape . We also drop the b subscript for convenience . 3.2 Camera Model . We use a perspective projection operator Π characterized by camera intrinsics C = [ fx , fy , cx , cy ] which assumes that the camera coordinate system is aligned with the world reference frame using an identity extrinsic rotation matrix and zero translation . This corresponds to a left handed coordinate system with positive z values forward . All predictions are made in the camera coordinate system . When processing the camera intrinsics for a crop we use the same approach as in [ 5 ] . 3.3 Architecture . In fig . 1 we show an overview of REMIPS , our proposed hybrid learning architecture for monocular multi-person 3D body pose and shape estimation . We draw inspiration from vision transformers [ 24 ] , as we also use a hybrid convolutional-transformer architecture , and from [ 5 ] , as we explore the idea of iteratively refining estimates by relying on cascaded , input-sensitive processing blocks , with homogeneous parameters , as an end-to-end learnable surrogate for non-linear optimization . Our network receives as input an image I ∈ RW×H×3 containing multiple people , together with the pseudo ground-truth camera intrinsics C ∈ R1×4 of the image ( see § 3.2 ) . We apply a convolutional neural network ( CNN ) on the input image and extract a downsampled feature map representation F ∈ RW32×H32×D . We flatten the feature map along the spatial dimensions to get a sequence of N = W32 × H 32 tokens . This sequence is linearly embedded by means of matrix E ∈ R D×D′ , where D′ is the embedding dimensionality . Next , learnable positional embeddings Epos ∈ RN×D ′ are added to the sequence . In addition to image feature tokens , we also use a sequence of NP = 16 maximum tokens for the people in the image . We first run an off-the-shelf human detector on the input image and collect the bounding box coordinates for each person index p , bp ∈ R1×4 . For each person p , we compute its initial GHUM parameters sp0 = { θ0 , β0 , r0 , t p 0 } where we compute the translation such that the default GHUM mesh projects in the center of the corresponding bounding box bp . We denote by S0 = [ s10 , . . . s NP 0 ] the initial state of GHUM parameters for all the people in the image . For each person , we concatenate the bounding person coordinates , the initial GHUM state and the camera intrinsics and construct the sequence of tokens corresponding to the persons in the image P ∈ RNP×DP . We linearly project the people token sequence P by means of matrix EP ∈ RDP×D ′ . The two sequences of tokens are concatenated into a single input sequence Z0 and iteratively transformed through a single shared transformer encoder layer [ 25 ] , TL , for a number of L steps . We collect at each step l ∈ { 1 . . . L } a refinement update ∆Sl from each transformed representation Zl , using a shared MLP applied on the corresponding transformed representation of the people token sequence . Z0 = [ FsE + Epos , PEP ] ( 1 ) Zl = TL ( Zl−1 ) ( 2 ) ∆Sl = MLP ( ZN : N+NPl ) . ( 3 ) The refinement updates ∆Sl are added to the initial state GHUM parameters , S0 , as SL = S0 + λΣ L l=1∆Sl ( 4 ) where SL = { ΘL , BL , RL , TL } are the final GHUM parameter estimates for all the people detected in the image and λ is a weighting term for the residual estimates . Note that the maximum number of person tokens NP = 16 is chosen specifically for training purposes , to batch training examples together . Yet , at inference time , we can actually use as many person tokens as the person detector outputs ( even more than 16 ) , as long as the quadratic memory requirements for the attention mechanism are satisfied ( i.e . in the order of thousands of persons ) . We observe that , in practice , the differences in inference speed for a variable number of persons ( between 1 and 32 ) to be almost negligible , i.e . at most 10 % increase in running time . We train the network by optimizing the following loss function : L = Lsc + Lic + Lics + Lscs + Ld + Lka + Lθ + Lβ ( 5 ) where the first 5 loss terms correspond to the physical plausibility of the reconstructed meshes as follows : Lsc and Lic are the self-collision loss and the interpenetration loss , respectively ; Lics and Lscs correspond to the supervised self-contact [ 4 ] and interaction-contact [ 3 ] losses ; Ld is the depth-aware ordering loss from [ 2 ] . These losses are discussed and explained in more detail in subsequent sections . We use a standard keypoint alignment loss , Lka , that measures the error between the 3D joints from the GHUM reconstructions , projected onto the image plane , J = { Ji } i=1 , ... , K , and the available 2D keypoints annotations in the image { ji } i=1 , ... , K Lka = 1 K K∑ i=1 ‖ji −Π ( Ji , C ) ‖22 . ( 6 ) Note that for some datasets we do not have 2D pose annotations available . In that case we supervise with the predictions of an off-the-shelf 2D pose estimator . For body shape and pose , we use regularization losses : Lθ = ‖θ‖22 , Lβ = ‖β‖22 . ( 7 )

Summary: This paper proposes a method for estimating multiple human pose & shape from monocular RGB input, which leverages a novel convolutional-transformer structure to optimize the detector-based human pose, shape and RT information. Also, it introduces a mesh-approx. based method for jointly formulating self-collision and interpenetration, which tackles the problem of treating two kinds of collisions in the same framework. Combined those methods, this paper tackles the problem of multiple human pose & shape estimation under challenging scenes.