Aligning Silhouette Topology for Self-Adaptive 3D Human Pose Recovery

1 Introduction . Human pose recovery has garnered a lot of research interest in the vision community . It finds extensive use in a wide range of applications such as , augmented reality , virtual shopping , humanrobot interaction , etc . Recent advances in human pose recovery is largely attributed to the availability of 3D pose supervision from large-scale datasets [ 19 , 42 , 57 ] . Although , the models achieve superior performance on in-studio benchmarks , they usually suffer from poor cross-dataset performance . Training on synthetic and in-studio datasets , which lack diversity in subject appearance , lighting , background variation , among more , can inherently induce a domain-bias [ 36 , 27 ] thereby restricting generalizability . This calls for the question : how to bridge performance gaps across diverse data domains ? Some works use weaker forms of supervision such as 2D pose keypoints , either directly [ 23 , 43 ] from the manually annotated large-scale datasets [ 1 , 21 ] or indirectly [ 5 , 26 ] via an off-the-shelf 2D pose detector network . Further , multi-view [ 51 , 20 ] cues have also been used to learn a reliable 2D-to-3D mapping . However , acquiring such additional supervision often comes at a cost , such as laborious manual skeletal-joint annotation or cumbersome calibrated multi-camera setups . This severely limits the deployability of a model in a novel , in-the-wild target environment . Images in a target deployment scenario can vary from those used in training setup in several ways , from simple changes in lighting or weather , to large domain shifts such as thermal or near-infrared ( NIR ) imagery . It is highly impractical to gather pose annotations for every novel deployment scenario . ∗Equal contribution . | Webpage : https : //sites.google.com/view/align-topo-human 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Motivated by these observations , we aim to realize a self-adaptive monocular 3D human pose recovery framework that does not rely on any strong ground-truth ( 2D/3D pose ) or auxiliary target supervision ( multi-view , or depth ) . The proposed adaptation framework ( Fig . 1 ) aims to transfer the task knowledge from a labeled source domain to an unlabeled target ( Fig . 1B ) , while relying only on foreground ( FG ) silhouettes . We observe that the traditional FG segmentation techniques are mostly unaffected by input domain shifts ( Fig . 1C ) and thus the FG masks are usually robust and easy to obtain in diverse target scenarios . Consider a deployment scenario where a static camera is capturing a moving object . Here , one can rely on classical vision based background subtraction techniques to obtain a FG silhouette while being considerably robust to a wide range of domain shifts ( including gray-scale , thermal , NIR imaging , etc ) . Certain deep learning based class-agnostic FG segmentation methods [ 61 , 39 ] ( motion and salient-object segmentation methods ) also exhibit reasonable robustness to domain-shifts . Hence , an adaptation framework that can effectively use silhouette supervision would open up the scope for adaptation to diverse deployment scenarios . Prior human mesh recovery works [ 26 , 20 , 23 , 24 ] have demonstrated considerable performance gains by training with articulation-centric annotations such as 3D and 2D pose supervision . At the same time , silhouettes have found a relegated use , mostly towards enforcing auxiliary shape-centric objectives [ 9 , 47 ] . Although these weaker objectives employ silhouettes , they do not offer strong supervisory signal in isolation , hence are not self-sufficient . In this paper , we aim to realize a stronger , self-sufficient and articulation-centric supervision from raw silhouettes . The key challenge is that the articulation information is usually lost in direct pixel-level losses , as raw silhouettes mostly provide 2D shape information . Our key insight is to extract a skeleton-like 2D topology information from silhouettes ( see Fig . 1D ) , which can provide a stronger articulation-centric supervision . We further propose a topological alignment objective that is inspired from 3D Chamfer distance [ 11 ] and can directly work on binary images . We leverage this objective to drive our target adaptation training using just silhouettes , and demonstrate the applicability of our framework across various domains . We make the following main contributions : • We develop a series of convolution-friendly spatial transformations in order to disentangle a topological-skeleton representation from the raw silhouette . • We devise a Chamfer-inspired spatial topology alignment loss via distance field computation , while effectively avoiding any gradient hindering spatial-to-pointset mapping . • Our self-supervised adaptation framework achieves state-of-the-art performance on multiple challenging domains , such as low-resolution domain ( LR ) , universal adversary perturbed domain ( UAP ) , in-studio Human3.6M [ 19 ] , and in-the-wild 3DPW [ 58 ] . 2 Related work . Human mesh recovery . Supervised human mesh recovery on monocular RGB images has been well explored in recent years , works such as [ 26 , 23 , 24 , 47 ] achieve impressive performance on standard benchmarks . These works heavily rely on large-scale annotated in-studio datasets [ 19 , 42 ] for the bulk of their training , however they report and agree on the lack of generalized performance on unseen in-the-wild data . Most works [ 23 , 26 , 49 , 47 , 24 , 33 , 29 , 5 ] try to address the issue by utilizing manually annotated 2D supervision from in-the-wild datasets [ 22 , 1 , 35 ] , and some additionally finetune their models by enforcing temporal [ 24 , 17 ] or multiview [ 34 , 49 , 17 ] constraints . Kolotouros et al . [ 26 ] has also explored integrating pre-trained off-the-shelf 2D pose detectors into the training loop , aimed towards improving in-the-wild performance . Despite the diversity in approaches , prior works mainly focus on applicability to a single target domain i.e . the ideal in-the-wild RGB data . Due to such a viewpoint , even the weakly-supervised approaches [ 10 , 55 , 47 ] render themselves irrelevant towards extending to new unlabeled target domains , as their methods implicitly assume access to significant amount of annotated samples from the final target domain . Xu et al . [ 60 ] proposed to overcome the performance-gap due to low-resolution ( LR ) imagery , but required labeled targets with simultaneous access to high-resolution source data . Later , Zhang et al . [ 62 ] proposed to address the domain induced performance degradation via self-supervised inference stage optimization . But their requirement of 2D pose keypoints as input , all the while only addressing 3D pose distribution shifts , severely restricts their applicability . Recognising this deficiency in prior works , our aim is to provide a self-supervised framework which can facilitate the overcoming of performance gaps across a wide array of domains . Use of silhouettes . Silhouettes have found extensive use in works which aim to obtain 3D human body shape from images . Sigal et al . [ 53 ] is one of the earliest works attempting to estimate the high quality 3D human shape , via fitting a parametric human-body model ( i.e. , SCAPE [ 2 ] ) to ground truth silhouettes . Subsequently , Guan et al . [ 15 ] used silhouettes , shading and edges as cues towards the body-shape fitting process , but required a user-given 2D pose initialization . Lassner et al . [ 33 ] proposed a silhouette based fitting approach for SMPL [ 38 ] human-body model , by using an off-the-shelf segmentation network . These works primarily aim for shape fitting based optimization and attempt to fit a 3D parametric mesh to a set of 2D evidence . However , solving these iterative optimization formulations for each and every instance can be prohibitively slow at test-time , and are also prone to getting stuck in local minimas . In contrast to these shape-centric works , we primarily aim to recover accurate posture of the person and consider shape estimation as a useful by-product of our formulation . A few works [ 55 , 47 , 30 ] formulate encoder-decoder based architectures with silhouette and 2D keypoint reconstruction as primary training objectives for their final regressor network . Silhouettes have also been used [ 56 , 29 , 10 , 54 , 3 ] as weak forms of auxiliary supervision usually in tandem with optical-flow , motion or other self-supervised objectives , some of these even relying on synthetic data for pre-training their networks . Although these works remain relevant in terms of performance on ideal in-the-wild RGB data , they fail to offer a fast , self-supervised end-to-end framework capable of adapting to diverse unseen target domains ( see Table 1 ) . 3 Approach . We aim to accurately recover the 3D human pose for unlabeled target domain samples , while relying only on silhouette supervision to adapt a source-trained model-based regressor . 3.1 Background and motivation . Traditionally , weakly-supervised human mesh estimation has been framed as a bundle-adjustment problem [ 5 ] . Given the SMPL body model [ 38 ] , an iterative fitting algorithm aims to adjust the pose , shape and camera parameters to minimize the reprojection loss against the given 2D observations . 3.1.1 Model-based pose regression network . Given an input image I , a CNN regressor outputs the SMPL [ 38 ] parameters ( pose θ , shape β ) alongside the camera parameters , Rc = { R , s , t } ( as shown in Fig . 2A ) . Here , Rc includes global orientation R ∈ R3×3 , scale s ∈ R , and translation t ∈ R2 . Following this , the SMPL module generates the ordered mesh vertices in the canonical space , i.e . V̂ ∈ R6890×3 =M ( θ , β ) . The mesh vertices V̂ are mapped to 3D pose Ŷ ( i.e . vertex-to-3DPose ) with k joints using a pre-defined linear regression : Ŷ ∈ Rk×3 = WpV̂ . We can then project 3D pose Ŷ to 2D pose Ẑ ∈ Rk×2 using weak-perspective camera projection : Ẑ = π ( RŶ , s , t ) . 3.1.2 2D pose supervision . In the presence of 2D pose ground-truth ( GT ) , prior approaches [ 23 , 47 , 5 , 33 ] aim to obtain the same representation for the prediction branch in order to employ a direct loss . Here , the vertex-to-3DPose mapping is already well specified in the SMPL module via fixed regression parameters Wp . Such approaches heavily benefit from this pre-specified body-joint structure . As a result , the mesh fitting problem boils down to minimizing the distance between two ordered point-sets ( i.e . a one-to-one mapping between the 2D joint locations ) which is easily achieved via a direct L1/L2 loss . Note that in such cases , post SMPL-regression , all the intermediate mappings operate on pre-specified ordered point-set representations , bypassing the need for a self-sufficient articulation-centric gradient via a rendering pipeline ( i.e . a mapping from point-set to spatial-maps ) . 3.1.3 Silhouette supervision . Unlike 2D pose supervision , learning from raw silhouette GT comes with a multitude of challenges . Though one can easily obtain the spatial silhouette from the predicted SMPL mesh via differentiable rendering [ 50 ] , devising an effective self-sufficient loss stands as the major bottleneck . A trivial loss to align the two binary silhouette masks would be a pixel-level L1 or cross-entropy loss , as employed in general segmentation tasks . However , such a loss fails to capture any gradient information along the spatial direction ( see Fig . 3A ) . Certain prior-arts [ 17 , 33 , 12 ] employ a chamfer based distance between two point-sets , termed as fitting-loss . Here , the two pointsets are basically the FG pixel-locations obtained from the GT and predicted silhouettes . However , this is meant for a better shape fitting and is almost always used in tandem with direct 2D or 3D pose supervision . When employed in isolation , it often lead to sub-optimal degenerate solutions , specifically in the absence of any pose-related articulation component . Based on the above observations , we propose a new representation space , termed as topologicalskeleton , which can facilitate seamless morphological silhouette alignment via effective spatial distance computations . To this end , we introduce the following : a ) Distance-map , D. For a given silhouette-mask S , its distance-map is defined as a spatial map D ( u ) , whose intensity at each pixel-location u∈U represents its distance from the closest maskoutline pixel of S ( see Fig . 2B ) . We utilize two different distance-maps : a ) inwards distance map , Din , and b ) outwards distance map , Dout . The inwards distance-map holds zero intensity for BG pixels ( i.e . pixels outside the active region of S ) , and vice-versa for the outwards distance-map . Mathematically , Din ( u ) = { min { u′ : S ( u′ ) =0 } |u− u′| , if S ( u ) =1 , 0 , otherwise . Dout ( u ) = { min { u′ : S ( u′ ) =1 } |u− u′| , if S ( u ) =0 , 0 , otherwise . ( 1 ) b ) Topological-skeleton , T . Topological-skeleton is a thin-lined pattern that represents the geometric and structural core of a silhouette mask . This can also be interpreted as the ridges-lines [ 4 , 13 , 8 ] of Din extracted for a given silhouette ( see Fig . 2B ) . For example , people with arms wide open would have the same structural topology , irrespective of their body-build type . In the absence of direct pose-centric supervision ( i.e . 2D or 3D pose annotations ) , we propose to treat topological-skeleton T as a form of pose-centric influence that can be extracted from the raw silhouette . Further , we aim to realize T in the form of a binary spatial map , unlike the point-set form of 2D pose keypoints . Here , T ( u ) =1 indicates that pixel u∈U is present in the topological ridge . We rely on distance-map to ; a ) construct the topological-skeleton via inwards distance-map of the silhouette ( see Fig . 2B ) as discussed in Sec . 3.2.1 , and b ) devise a Chamfer-inspired training objective via outwards distance-map of the topological-skeleton ( see Fig . 2C ) as discussed in Sec . 3.3 . Note that , the outwards distance-map of the topological-skeleton is defined by replacing S with T in Eq.1 .

Summary: Human mesh recovery methods commonly resort to 2D keypoint supervision to be able to handle in-the-wild images, as these often lack 3D ground truth. This submission argues that binary silhouettes are a better source of weak supervision and that classical segmentation techniques are less susceptible to domain shifts that 2D keypoint detectors. As such, silhouettes can be used to train shape and pose regressors on more heterogenous data without the need for annotating and retraining keypoint detectors, which this submission does: A SPIN-like method (Kolotouros et al., 2019) is proposed that replaces the keypoint-based loss and refinement, with a silhouette-based loss that allows for comparisons between the model and observed silhouettes. Key here is a differentiable skeletonisation operation that pares the silhouette down before applying the loss. The proposed method compares favourably against the state-of-the-art on a variety of datasets. The submission additionally demonstrates that the resulting scheme is more robust to adversarial perturbations than competing approaches.