Aligning Silhouette Topology for Self-Adaptive 3D Human Pose Recovery
Authors: Mugalodi Rakesh
Jogendra Nath Kundu
Varun Jampani
Venkatesh Babu Radhakrishnan
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D Human pose estimation
Human mesh recovery
3D from Single Images
Domain adaptation
Silhouette topology extraction

Abstract: Articulation-centric 2D/3D pose supervision forms the core training objective in most existing 3D human pose estimation techniques. Except for synthetic source environments, acquiring such rich supervision for each real target domain at deployment is highly inconvenient. However, we realize that standard foreground silhouette estimation techniques (on static camera feeds) remain unaffected by domain-shifts. Motivated by this, we propose a novel target adaptation framework that relies only on silhouette supervision to adapt a source-trained model-based regressor. However, in the absence of any auxiliary cue (multi-view, depth, or 2D pose), an isolated silhouette loss fails to provide a reliable pose-specific gradient and requires to be employed in tandem with a topology-centric loss. To this end, we develop a series of convolution-friendly spatial transformations in order to disentangle a topological-skeleton representation from the raw silhouette. Such a design paves the way to devise a Chamfer-inspired spatial topological-alignment loss via distance field computation, while effectively avoiding any gradient hindering spatial-to-pointset mapping. Experimental results demonstrate our superiority against prior-arts in self-adapting a source trained model to diverse unlabeled target domains, such as a) in-the-wild datasets, b) low-resolution image domains, and c) adversarially perturbed image domains (via UAP). 

Aligning Silhouette Topology for Self-Adaptive 3D Human Pose Recovery
Mugalodi Rakesh
Jogendra Nath Kundu
Varun Jampani
Venkatesh Babu
Indian Institute of Science, Bangalore
2 Google Research
Abstract
Articulation-centtia 2D/3D pose supervision forms the core training objective most existing 3D human posc estimation techniques. Except for synthetic source environments, acquiring such rich supervision for each real target domain at de- ployment is highly inconvenient. However, we realize that standard foreground silhouette estimation techniques (on static camera feeds) remain unaffected by domain-shifts. Motivated by this, propose novel target adaptation framework that relies only on silhouette supervision to adapt a source- trained model-based regressor. However, in the absence of any auxiliary cue (multi view, depth, 2D pose), an isolated silhouette loss fails to provide a reliable pose-specific gradient and requires to be employed in tandem with topology-centee ccnttiic loss. To this end, we develop a series of convolution-friendy spatial transformations in order to disentangle topological-skeltorr representation from the raw silhouette. Such a design paves the way to devise Chamfer-inspiree spatial aeooooical-aalgnmeen loss via distance field computation, while effectively avoiding any gradient hinder- ing spatial-to- t0-poinnset mapping. Experimental results demonstrate our superiority against prior-arts in self-adapting source trained model to diverse unlabeled target domains, such as a) in-the-wild datasets, b) now-resolution image domains. and c) adversarially perturbed image domains (via UAP).
Introduction
Human pose recovery has garnered a lot of research interest in the vision community. It finds extensive use in a wide range of applications such as, augmented reality, virtual shopping, human- robot interaction, etc. Recent advances in human pose recovery is largely attributed to the availability of 3D pose supervision from large-scale datasets [19, 42, 57]. Although, the models achieve superior performance in-studio benchmarks, they usually suffer from poor cross-dataset performance. Training on synthetic and in-studio datasets, which lack diversity in subject appearance, lighting, background variation, among more, can inherently induce domain-bias [36, 27] thereby restricting generalizability. This calls for the question: how to bridge performance gaps across diverse data domains?
Some works use weaker forms of supervision such as 2D pose keypoints, either directly [23, 43] from the manually annotated large-scale datasets [1, 21] indirectly [5, 26] via off-the-shelf 2D pose detector network. Further, multi-view [51, 20] cues have also been used to learn a reliable 2D-to-3D mapping. However, acquiring such additional supervision often comes cost, such as laborious manual skeletal-joint annotation or cumbersome calibrated multi-camera setups. This severely limits the deployability of model in novel, in-the-wild target environment. Images in target deployment scenario can vary from those used in training setup in several ways, from simple changes in lighting weather, large domain shifts such thermal near-infrared (NIR) imagery. It is highly impractical to gather pose annotations for every novel deployment scenario.
Equaa contribution. Webpage: https://sites www.le..ooooe...oooooooooo.oe.. www.gle.com/vie////e/eoooo www.w//ww/ww/..ooo.oooooo...eeee algganniitttt wow.gnnttooiiiiee human
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A
B
Domain-1 (in (i-tho-wid)
Source superviaion
Target adaptation
Domain 2 (Lowires)
extractt
Silhoustte Extraction
source data
Labeled
CNN
Domein-3 (Thermal)
domain-shiff
unaffected by
Tepole
Unlabeled target R
domains
The proposed self-supervised adaptation relies only on silhouette supervision (unaffected by domain- shift) to adapt the source trained pose regressor novel target deployment scenarios. Topology extraction from raw foreground silhouettes is one of the key design aspects of framework.
Figure 1:
Motivated by these observations, we aim realize self-adaptive monocular 3D human pose recovery framework that does not rely on any strong ground-truth (2D/3D pose) or auxiliary target supervision (multi-view, the task knowledge from a labeled source domain to an unlabeled target (Fig. 1B), while relying only on foreground (FG) silhouettes. We observe that the traditional FG segmentation techniques mostly unaffected by input domain shifts (Fig. IC) and thus the FG masks are usually robust and easy obtain in diverse target scenarios. Consider a deployment scenario where static camera is capturing moving object. Here, one can rely on classical vision based background subtraction techniques to obtain a FG silhouette while being considerably robust wide range of domain shifts (including gray-scale. thermal, NIR imaging, etc). Certain deep learning based class-agnostic FG segmentation methods [61, 39] (motion and salient-object segmentation methods) also exhibit reasonable robustness to comain-shifts. Hence, an adaptation framework that can effectively use silhouette supervision would open up the scope for adaptation to diverse deployment scenarios.
depth). The proposed adaptation framework (Fig. 1) aims to transfer
Prior human mesh recovery works [26, 20, 23, 24] have demonstrated considerable performance gains by training with articulation-centric annotations such 3D and 2D pose supervision. At the same time, silhouettes have found a relegated use, mostly towards enforcing auxiliary shape-centric objectives [9, 47]. Although these weaker objectives employ silhouettes, they do not offer strong supervisory signal in isolation, hence are not self-sufficient. In this paper, we aim to realize a stronger, self-sufficient and articulation-centric supervision from raw silhouettes. The key challenge is that the articulation information is usually lost in direct pixel-level losses, as raw silhouettes mostly provide 2D shape information. Our key insight is extract a skeleton-like 2D topology information from silhouettes (see Fig. ID), which can provide stronger articulation-centir supervision. We further propose topological alignment objective that is inspired from 3D Chamfer distance [11] and can directly work on binary images. We leverage this objective to drive our target adaptation training using just silhouettes, and demonstrate the applicability of our framework across various domains. We make the following main contributions:
We develop a series of convolution-frient friendlly spatial transformations in order to disentangle topological-skelttnn representation from the raw silhouette.
We devise a Chamfer-inspired spatial topology alignment loss via distance field computation, while effectively avoiding any gradient hindering
mapping.
Our self-supervised adaptation framework achieves state-of-the-ar art performance on multiple chal- lenging domains, such as low-resolution domain (LR), universal adversary perturbed domain (UAP), in-studio Human3.6M [19], and in-the-wild 3DPW [58].
Related work
Human mesh recovery. Supervised human mesh recovery on monocular RGB images has been well explored in recent years, works such as [26, 23, 24, 47] achieve impressive performance on standard benchmarks. These works heavily rely on scage annotated in-studio datasets [19, 42] for the bulk of their training, however they report and agree the lack of generalized performance unseen in wild data. Most works [23, 26, 49, 47, 24, 33, 29, 5] try to address the issue by utilizing manually annotated 2D supervision from in-the-wild datasets [22, 1, 35], and some additionally fine- tune their models by enforcing temporal [24, 17] or multiview [34, 49, 17] constraints. Kolotouros er al. [26] has also explored integrating pre-trained off-the-sheel 2D pose detectors into the training loop, aimed towards improving in-the-wild performance.
Despite the diversity in approaches, prior works mainly focus on applicability to single target domain i.e. the ideal in-the-wild RGB data. Due to such a viewpoint, even the weakly-supervised approaches [10, 55, 47] render themselves irrelevant towards extending to new unlabeled target

---

domains, as their methods implicitly assume access to significant amount of annotated samples from the final target domain. Xu al. [60] proposed to overcome the performance- due to low-resolution (IRR) imagery, but required labeled targets with simultaneous access to high-resolution source data. Later, Zhang et al. [62] proposed to address the domain induced performance degradation via self-supervised inference stage optimization. But their requirement of 2D pose keypoints as input, all the while only addressing 3D pose distribution shifts, severely restricts their applicability. Recognising this deficiency in prior works, our aim is to provide self-supervised framework which can facilitate the overcoming of performance gaps across a wide array of domains.
Use of silhouettes. Silhouettes have found ex- tensive use in works which aim to obtain 3D hu- man body shape from images. Sigal et al. [53] one of the earliest works attempting to es- timate the high quality 3D human shape, via
Table I: Characteristic comparison of our method against prior arts, separated by access to direct (paired) supervision for adaptation to new domains. MV: multi-view. Sil.: silhouette Paired sup.
Model-hased Methods
Self-sup. Adaptation
fitting& a parametric human-body model (i.e., SCAPE (2]) to ground truth silhouettes. Subse- quently, Guan et al. [15] used silhouettes, shad- ing and edges cues towards the body-shape fit- ting process, but required a user-given 2D pose initialization. Lassner al. [33] proposed a sil-
2D/3D
MV
Sil.
pose
cams
15, 23, 26]
x
X
X
[34, 49, 17]
V
/
x
[33, 15, 47, 55]
x
Ours
x
houctte based fitting approach for SMPL [38]
human-body model, by using off- the-shelf segmentation network. These works primarily aim for shape fitting based optimization and attempt to a 3D parametric mesh to a set of 2D evidence. However, solving these iterative optimization formulations for each and every instance can be pro- hibitively slow at test-time, and are also prone to getting stuck in local minimas. In contrast to these shape-centric works, we primarily aim to recover accurate posture of the person and consider shape estimation as a useful by-product of our formulation.
A few works [55, 47, 30] formulate encoder-decoder based architectures with silhouette and 2D keypoint reconstruction as primary training objectives for their final regressor network. Silhouettes have also been used [56, 29, 10, 54, 3] as weak forms of auxiliary supervision usually in tandem with optical-flow motion or other self-supervised objectives, some of these even relying on synthetic data for pre-training their networks. Although these works remain relevant in terms of performance on ideal in-the-wild RGB data, they fail to offer a fast, self-supervised end-to-end framework capable of adapting diverse unseen target domains (see Table 1).
3 Approach
We aim to accurately recover the 3D human pose for unlabeled target domain samples, while relying only on silhouette supervision to adapt a source-trained model-based regressor.
3.1 Background and motivation
Traditionally, weakly-supervisedd human mesh estimation has been framed as a bundle- adjustment problem [5]. Given the SMPL body model [38], an iterative fitting algorithm aims to adjust the pose, shape and camera parameters to minimize the reprojection loss against the given 2D observations.
3.1.1 Model- pose regression network. Given an input image I, CNN regressor outputs the SMPL [38] parameters (pose 0, shape 3) alongside the camera parameters, R. = R, s, () (as shown in Fig. 2A). Here, R. includes global orientation R E R3x3, scale E IR, and translation t E RR² Following this, the SMPL module generates the ordered mesh vertices in the canonical space, Le. E R = _M0, 3). The mesh vertices V are mapped to 3D pose Ý (i.e. vertex-to-3DPose with l joints using pre-defined linear regression: Y e R = W.V. We can then project 3D pose Y to 2D pose 2 E *x* using weak-perspectivy camera projection: 2 = = TRRÝ,
3.1.2 2D pose supervision. In the presence of 2D pose ground-trute (GT), prior approaches [23, 47, 5, 33] aim to obtain the same representation for the prediction branch in order to employ a direct loss. Here, the vertex-to-DDPose mapping is already well specified the SMPL module via fixed regression parameters W,. Such approaches heavily benefit from this pre-specified body-joint structure. As a result, the mesh fitting problem boils down to minimizing the distance between two ordered point-sets (i.e. one-to-one mapping between the 2D joint locations) which is casily achieved
3

---

cipp
M25
S2D
D21
D21
S2D
Topology extraction
Topology extraction
SMPL model
Prediction branch
S2D
S2D
GT
Camera
Shape
Pose
Traversing the latent pose manifold
Alignmett-support
Figure 2: Framework overview highlighting the modules and the series of transformationn to realize the topology aligmment objective. A. Image to SMPL regression (Sec. 3.1.1). B. Extracting topological saallggicll-ssletooo skeleton via inwards distance-ma map of silhouette mask (Sec. 3.2). C. and D. Obtaining outwards distance- map of topological skeleton support spatial- chamfer (Sec. 3.3). E. Chamfer insprredddddddd opplogy-alignment alignment objective (Sec. 3.3).
via direct L1/L2 L2 loss. Note that in such cases, post SMPL-regression, all the intermediate mappings operate on pre-specified ordered point-set set representations, bypassing the need for a self-sufficient articulation-centric gradient via rendering pipeline (i.e. mapping from point-set to spatial-maps).
3.1.3 Silhouette supervision. Unlike 2D pose supervision, learning from silhouette GT comes with multitude of challenges. Though one can easily obtain the spatial silhouette from the predicted SMPL mesh via differentiable rendering [50], devising effective self-sufficient loss stands as the major bottleneck. A trivial loss to align the two binary silhouette masks would be pixel-level L1 or cross-entropy loss, employed in general segmentation tasks. However, such a loss fails to capture any gradient information along the spatial direction (see Fig. 3A). Certain prior-arts [17, 33, 12] employ chamfer based distance between two point-sets, termed as fitting-loos loss. Here, the two point- sets are basically the FG pixel-locee pixel-locations obtained from the GT and predicted silhouettes. However, this is meant for better shape fitting and is almost always used in tandem with direct 2D or pose supervision. When employed in isolation, it often lead to sub-optimal degenerate solutions, specifically in the absence of any pose-related articulation component.
Based on the above observations, we propose new representation space, termed as topological- skeleton, which can facilitate seamless morphological silhouette alignment via effective spatial distance computations. To this end, we introduce the following:
a) Distance-map, D. For given silhouette-main S, its distance-may mapp is defined spatial map Du)), whose intensity at each pixel-location u E represents its distance from the closest mask- outline pixel of S (see Fig. 2B). We utilize two different distance-mant maps: inwards distance map, Di.- and b) outwards distance map, Dour- The inwards distance-map holds zero intensity for BG pixels (i.e. pixels outside the active region of 5), and vice-versa for the outwards distance-map. Mathematically,
min Iu - w'1, if S(u)=1, (wiSS("))0) 0.
min Iu u'), if S(u)=0, Dome(u) (u':S(m))=) (0
(mluu)
otherwise.
otherwise.
(1)
b) Topological-skeleton, T. Topological-skeleton is a thin-lined pattern that represents the geometric and structural core of silhouette mask. This also be interpreted the ridges-lines [4, 13, of Di,, extracted for a given silhouette (see Fig. 2B). For example, people with arms wide open would have the same structural topology, irrespective of their body-build type. In the absence of direct pose-centric supervision (i.e. 2D or 3D pose annotations), we propose to treat topological-skelene T form of pose-centric influence that can be extracted from the raw silhouette. Further, we aim to realize T in the form of binary spatial map, unlike the point-set form of 2D pose keypoints. Here, T(u) 1 indicates that pixel UU is present in the topological ridge.
We rely on distance-man map to; a) construct the pooologiallsskeettee via inwards distance-mar of the silhouette (see Fig. 2B) discussed in Sec. 3.2.1, and b) devise Chamfer-inspired training objective via outwards distance-map of the topological-skeles
(see Fig. 2C) as discussed Sec. 3.3. Note that, the outwards distance-map of the topological-skellton is defined by replacing with T in Eq.

---

52D
M25
Distance Transform
lterative Thinning
Siguum
(i)
Flat
D2T
valley
Thinning operation
10.11
(iiy Sharp
valley
Figure 3: A. representative spatial-gradient landscape for different kind of silhouette losses (left: Z-axis indicates gradient magniuue,, right: intensity indicates gradient magnitude) such as, i) Pixel-level L11 or cross- entropy loss, ii) Chamfer inspired silhouette loss, iii) Chamfer-inspine inspired
loss. B. Details of the M2S module (see Sec. 3.2). C. Details of the S2D module (see Sec. 3.2.1) where D. shows details of single convolutional thinning. E. Details the D2T module (see Sec. 3.2.2). Pixel intensity range shown on top-right.
3.2 Silhouette topology extraction
Here, the primary objective to formalize a series of transformations order to extract the respective lopological-seeetone for the predicted and GT silhouette masks, as shown in Fig. 2.
In the prediction pathway, the silhouette is obtained via a fast differentiable renderer [50] which operates on the mesh vertices V. However, non-binary silhouette is not suitable for distance-man map and topology definitions as discussed in Sec. 3.1. Thus, we employ a binary activation function [18]
as shown in Fig. 3B. The final outcome is represented as S = M2S(VV) While formulating the later transformation modules, one must pay special attention to design constraints discussed below.
Design constraints. We intend to update the CNN by back back-propagating a loss which would be computed at the topological representation, / (see Fig. 2). An ineffective regressor output Óre to f mapping results in vanishing gradient issues. Additionally, parallelizable operations remain critical and desirable for faster training on modern GPUs. Hence, we aim to formulate each transformation via wwtvollution-rrinnyyy spatial operations, thereby avoiding any spatial-map to point-set mapping while implementing the topology-extraction and alignment objectives.
3.2.1 Silhouette distance-map, S2D. Restricting to spatial convolution-frient
computation, we devise a recursive erosion (or thinning) operation in order to gradually shave off the boundary regions of the silhouette mask. To this end, we compute the number of active neighbours of pixel by convolving the binary silhouette input with an n X neighbourhood kernel N, as shown in Fig. 3D. The boundary pixels are expected have less than maximum neighbours (i.e. < neighbours for n = 3), these non non-maximal pixels are then purged via the ReLU activation to output a thinned binary silhouette map, i.e. S ReLL @ S, m). Here, @ denotes a convolution operator and n2 22 Following this, all the series of thinned binary maps are added to realize distance map equivalent,
D S2D(S) = S2D Et where: So = and S ReLU S; m)
(2)
Interestingly, the outwards distance-map of S can also be computed as D =
D2T. We aim to realize the thin-lined topological- skeleton as the ridge-line of the distance-may Din. In other words, the ridge-line pixels are the local maxima of Din. In view of convolution-frieet friendly backpropable implementation, a pixel would be selected as ridge-pixel if its distance-map intensity matches with the local MaxPooled map of the same (i.e. where D MaxPool(D,, O). Following this, the topological-skeltton is realized as:
3.2.2 Distance-map to topological-skeleet
T = D2T(D in: 5) ReLU(D,, in Maazpol((Dn) + 1 S
(3)
Where @ represents element dot product. Note that, intensities D are discrete values in the range [0, I] i.e., 0, 1, 2.,, Thus, the +1 term followed by ReLU selects the ridge-pixels along with the false- aalse-positives from the BG region, which are later pruned by masking using S (see Fig. 3E). The resultant T is also a binary map by formulation, i.e T(u) E {0.1}
In summary, the prediction pathway involves series of transformations, M2S H2S-S22DDD21 to realize V - S- D.n - (see Fig. 2). Similarly, the GT pathways involves S2D-D2T realize S D.,. -T.
5

---

3.3 Topological loss
Here, we discuss the formulation of our topological loss, which aims to quantify the misalignment between T and T. We draw inspiration from Chamfer distance [11] which quantifies the misalignment between the predicted and GT point-sets. Let T and T2 be the point-set representation of T and T respectively, implying Tn = {u: T(u)= 13. Note that, unlike 2D pose keypoints, (T + [Tpl and there exist no point-to-point correspondence. distance of each point in Tp against all points in Tp Thus, the two-way Chamfer is expressed as,
measures the sum of the shortest
(1)) (l) CT - T. + - TT-T, WEIp
E min |u w v'122 w'ET,
min ||u Σ MET,
n'112
a'ET,
However, in the view of convolution-friendt implementation we aim to avoid any spatial-map to point-set mapping. Implying, can not obtain Tp Tp as required to compute the above loss term. To this end, we propose to make use of the distance-map representation to formalize a similar loss term that would directly operate spatial-maps. We recognize that the outwards distance-man map of T, iee D (u), stores the shortest distance of each BG pixel u against the active pixels in T. Thus, an equivalent form of ((P) of the vectorized matrix. Thus, the final coumfer-inspired
TT-TT be computed as; [I @ 7) where 1|-I11,1 computes the L1-norm
loss is realized as,
c( 1. O 11, ofil,
where: DTH S2D and D S2D (5)
Note that, S2D(1 -) -) yields the outwards distance-map (see Fig. 2) which is inline with Eq. 1. The above loss formulation effectively addresses all the design and gradient related difficulties thereby facilitating& a seamless morphological silhouette alignment.
3.4 Adaptation training
We posit the training as self-supervised domain adaptation problem. To this end, we first train the CNN regressor by supervising on labeled source samples, Oree) E Din which is typically a graphics-based synthetic domain or an in-studio dataset. Upon deployment to an unknown target environment, we gather the image and silhouette pairs, S) Digr where S is obtained via classical
FG-mask estimation techniques. Alongside the topology alignment objective (L.e. minimizing c(SP)), we adopt the following to regularize the adaptation process.
a) Enforcing natural priors. Most works rely on adversarial prior-enforcing losses restrict the CNN predictions within the realm of natural pose and shape distributions. To simplify adaptation process, we train an adversarial auto-encoder [40, 31, 28, 48] to learn latent pose space de [-1, 1, 1] whose decoder, V generates plausible 3D pose vectors 0 E 3 (Ø)) Thus, just integrating this frozen decoder into our framework constrains the pose to vary within the plausibility limits. We
denote 2.yg = {3, 6, R.} and regularize 3 to remain close to the mean shape, inline with [26].
b) Optimization in the loop. Motivated by the ben- efits of combining regression and optimization based training routines [26], we propose target adaptation procedure as shown Algo. 1. We first initialize reg
Ocww: Source training initialized CNN weights (lam) SMPL parameters initialized from θ,g for iter < Maxiter do if iter (mod K) * 0 then
by inferring ,,eg from the current state of the source- trained CNN. These regression- -nnitaaaiiee @japt) param- eters undergo an iterative fitting procedure which aims to minimize c(Sp) and c.SP) without updating the CNN parameters. Here, -( represents the spatial chamfer- based loss directly applied on raw-silhouettes, obtained by replacing T and 7 with and S in Eq. 5. This loss further improves pose and shape fitting.
Update θcww by optimizing c(SP) c(SP)
clse
for iter Maxlter do
Fit 6/ to minimize cS'D), c(
Update Ocww by optimizing Le.
Algorithm 1: Proposed adaptation procedure.
Subsequently, the fitted mg is used to form a supervised loss, Le = | -O/p0/) to update the CNN weights, Ocw. Although, we primarily rely on the end-to-end training of the CNN regressor by directly minimizing the proposed alignment objectives, alternating between regression and fitting based routines allows the model to self-improve faster.
=

---

Internet (in-the-wlid)
B H36M (in-studio)
3DPW (in-the-wild)
Adaptation Progression
R-LRR
CAP
10k
100K
Figure 4: A. B. and C. Qualitative results of Ours(S on the respective datasets. Failure cases are highlighted in magenta. D. Progression of our iicculation-centre adaptation across various domains (across columns).
Table Evaluation on the 3DPW dataset, none of the works train on 3DPW. For fair comparison, we separate works based on access to target supervision.
Table 2: Comparison against works on Human3.6M (P-2). Table is divided based on access to supervision. We beat prior arts at comparable supervision levels.
Sup.
Method
MPJPE(J) PA-MPIPE(()
Sup.
Method
PA-MPIPECL)
HMR [23]
128.1
81.3
Full
Kanazawa et al. [24]
116.5
72.6
Pavlakos al. [47]
75.9
SPIN [26]
98.6
59.2
Full
HMR [23]
56.8
Martinez et al. [41]
157.0
SPIN [26]
41.1
SMPLify [5]
199.2
106.1
Weak
HMR (unpaired) [23]
66.5
Dnersch et (RGB+2D) [10]
82.4
Weak
SPIN (unpaired) [26]
62.0
Ourx(S >R, weak)
126.3
79.1
Ours(S +R, weak)
58.1
Doersch et (DANN) [10]
103.0
Unsup.
Ours(S
Kundu al. (unsup) [29]
+R)
90.5
81.3
Unsup. Duersch et al. (Flow) [10]
Kundu al. (unsup) [29]
187.1
100.1
102.7
Ours(S -R)
159.0
95.1
4 Experiments
We perform a thorough empirical analysis demonstrating our superiority for cross domain adaptation.
Implementation details. We use an ImageNet [52] pre-trained ResNer-50 [16] network as our CNN backbone for the regressor. The final layer features are average pooled and subsequently passed through a series of fully- conne-co layers to regress the latent pose encoding ø, shape and camera parameters. We use the Adam optimizer [25] with a learning rate 1e-6 and batch size of 16, while
setting Maxlter to 10 and K=4. We use separate optimizers for ((s)) and c(*P), A single iteration of the iterative fitting procedure takes nearly 12ms
Titan-RTX GPU.
Datasets. Moshed [37] CMU-MoCap [32] and H3. 6.66 training-ser set [ [19] form our unpaired 3D pose data, which is used to train the 3D pose-prior. Our adaptation settings use the following datasets.
a) Synthetic (S): We use SURREAL [57] to train our synthetic source model. is large-scale dataset with synthetically generated images of humans, rendered from 3D pose sequences of [32].
b) Real (R): We use mixture of Human3.66 [19] and MPII [1] as our Real-domain dataset. This domain is used as both source and target in different adaptation settings.
UAP-H3M (UAP): Universal Adversarial Perturbation (UAP) [44] is an instance- agnoosic per- turbation aimed at attacking a deployed model, thereby inflicting a drop in the task performance. Typically, the adversarial noise is added with the clean source domain images construct the adver- sarially perturbed target domain samples. We craft single UAP perturbation (L.o and c 8/255) using an equivalent 3D pose estimation network while following the procedure from [45]. We aim to evaluate the effectiveness of the proposed self supervupeee adaptation as a defence mechanism against such perturbed domains. We perturb the clean Human3.66 6.M [19] samples to construct such target domain, denoted as UAP-H3M. Further, we experiment on three perturbation strengths while varying as 4/255, 8/2 8/255, and 16/ 255 (see Table 5). d) LR-3DPW (LR)): We use low-resolution (LR) variants of 3DPW [58] dataset as another target domain, denoted LR-3DPW. LR holds practical significance as they are common in surveillance and real-time streaming applications. Several works [46, 60) acknowledge the performance drop in models trained on high-resolution (HR) data, while evaluating on LR targets. Addressing this, Xu et al. [60] construct an ---specific architecture and assume simultaneous access to pose-labeled samples from both L.R and HR domains, hence not comparable our setting. We aim to evaluate the proposed self-supervised adaptation technique remedy to such domain-shifis in the absence of target
7

---

I/P: Thermal
B Input Domain: LW
Input Domain: UAP-H3M
Input
Pre-fit
Post-fit
Input
SPIN
A2
Pre
Post Ours(R -LR)
Input
SPIN
A2
Post Ours(R-
Pre
R-UAP
Figure 5:
Inference-tine timee fitting on Thermal images. B. Results on LR-3DPW) C. Results on UAP-H3M).
Yellow ellipses highlight the region of articulation errors (refer Table 5 caption for comparison).
domain pose-labels. We construct three different LR domains by downsampling the full-resolution image w w where w is set as 96, 52 and 32 (see Table 5). Each LR domain image upsampled (bi-cubic) back to 224 x 224 before forwarding them through the CNN regressor.
Adaptation settings. We evaluate our approach on the following adaptation settings. We use full supervision for source pre- tre-training and use only silhouettes for target adaptation. In static-camera moving-object scenarios, silhouettes obtained via BG subtraction remain unaffected by domain shifts.
a) Ours(S--R) and Ours(S-aa RR weak). In this setting, we initialize our Synthetic source model by accessing full supervision from SURREAL [57] and undertake appoloiial-alignmeent based adaptation on the Real domain. R)) does not access any Real 2D/3D pose annotation. However, the weakly-superviee
variant, Ours(S weak), utilizes 2D pose supervision.
b) Ours(R-UAA)) and Purs(R-IAR) Here, we adapt the Real domain source model, Ours(R), to the UAP-H3M and LR-3DPW domains respectively, by only accessing silhouettes from the target domains.
4.1 Ablative study
We define three baseline models that use various silhouette-based losses for self-supervised adaptation. In Table 4, Ours(S) reports the pre-adaptation per- formance in order to gauge the corresponding post- adaption improvements. B1, a model adapted using only the pixel-level L2 loss (i.e. ||5-$1)) shows very poor performance. This shows that L2 based losses fall short of offering an independent articulation-centie ob- jective. On the other hand, replacing it with our spatial chamfer-based alignment loss c c(SP) on raw-silhouettes, gives reasonable adaptation performance, shown as B2.
Table 4: Ablation experiments 3DPW for S-R. BI, B2, and B3 are adaptation baselines that use different loss variants as listed below.
Method
Loss on S
Loss
PA-MPJPP
on T
(1)
Ours(S) BI(S-R)
134.7
L2
129.6
B2(S->R)
c
106.2
B3(S-R)
cl
as L.2
105.5
Ours(S
C c(sp)
95.1
We observe that naively supplementing B2 with an L2 loss topological tupological-skeleton (i.e. 11 yields very minimal benefit, despite containing useful articulation-cennie centric information. Finally, compare these baselines with our proposed model to clearly establish the effectiveness of our Chamfer inspired oopolgiiall-aignneee loss towards driving articulation-centiic learning.
skeletonn
4.2 Comparison against prior works
Adhering to standard metrics, we compute the mean per joint position error (MPJPE) [19] and Procrustes aligned [14] mean per joint position error (PA-MPJPE) to evaluate the pose specific adaptation performance. We evaluate on Human3,6N, [19] following Protocol-2 [23].
a) Adaptation from Synthetic Real. We evaluate the proposed approach against prior human mesh recovery works on both Human3.66 3M6 [19] (Protocol-2) and 3DPW [58] datasets. Our model achieves state-of-the-ari performance on self-supervised and weakly-supervised settings (see Table and Table 3). Though we access synthetic domain data similar to Doersch et al. [10], we deem ourselves comparable as other listed works [23, 26, 47, 24] benefit from access to additional Real domain datasets such as LSP, LSP- Extended [22] and COCO [35].
b) A Adaptation from Real to UAP- We obtain pre-adaptation performance of the source trained networks via direct inference on the shifted target (first 2 rows of Table 5). We report these in order
to gauge the severity of the domain-gap. AI, SPIN [26] is finetuned on UAP-H3M targets by using 2D pose predictions from tff-the-shelff OpenPose [7, 6, 59] network. Only the high-confidence 2D
8

---

Table 5: Evaluation on UAP-H3M (P2) and LR-3DPV. We compare against two strong adaptation baselines: using AI: 2D pose from off-the-shelf network [6], 2) A2: additional point-set based silhouette fitting-loss [17].
Comparing A1, A2, and Ours pre-to-post performance recovery, we observe that the effectiveness of
approach increases with increasing domain shift. + represents adaptation using the specified loss.
Adapiation from R to UAP-H3M
Aduptation from R to LR-3DPW
Method
MPJPE (4)
PA-MPJPE (L)
MPJPE (J)
PA-MPJPE (J)
4/255
8/255
16/255
4/255
8/255
16/255
52
32
96
52
32
SPIN [26] Ours(R)
65.8
98.2
160.1
44.6
60.8
90.7
104.3
120.3
176,4
63.7
71.1
87.9
Pre-Adapt.
67.7
103.9
161.8
46.9
63.6
91.2
110.8
127.5
178.1
68.6
76.3
88.2
SPIN+L22
64.5
94.0
151.2
43.4
59.5
89.8
100.2
117.0
153.6
61.7
85.4
Post-Adapt. A2: SPIN+C20+ $
64.1
89.1
136.5
43.4
58.9
85.1
100.1
115.2
147.5
61.5
69.8
82.3
Ours(R -UAP)
63.6
84.7
125.2
43.2
57.6
79.4
Ours(R -LR)
99.8
114.7
134.2
61.3 68.7 78.9
predictions are used, inline with [26]. To create strong baseline, in A2, we supplement the Al setting with additional silhouette supervision, via the fitting-loss [17]. As shown in last two rows of Table 5, the proposed adaptation method clearly imparts superior recovery of target performance, despite accessing only silhouettes. The baselines Al and A2, fail to recover from domain shift despite having access to stronger supervision. We observe that, unlike the silhouette extraction process, OpenPose itself suffers from domain shift issues, thus is ineffective towards improving target performance.
<) Adaptation from Real to LR-3DPW. Here, we undertake adaptation to the LR-3DPW domain and evaluate against the same baseline settings as in R U-P task. But unlike in UAP, the target silhouettes here of lower resolution. Despite this, in Table 5, we clearly superior adaptation performance against the baselines (A1 and A2) which access additional 2D-pose predictions obtained via OpenPose [7, 6, 59]. These observations shine light on the ineffectiveness of network- -ased pseudo supervision (via OpenPose) towards recovering from domain-shift issues, and voices the need and usefulness of our framework in practical deployment scenarios.
Qualitative results. Fig. 5 shows qualitative results on LR-3DPW and UAP-H3M. Interestingly, our formulation of fitting in-the-loop with silhouette based objectives allows us to fit meshes using only silhouettes. Hence enabling us to perform fast inference-time fitting on diverse unseen domains such thermal images, without requiring any 2D/3D pose annotations. We show few such fittings in Fig. 5A. We also observe that in contemporary ENN-based approaches, domain primarily manifests articulation degradation, while person- -oieonttiienttiion remains relatively robust. Additionally, in Fig. 4, we show a few pose recovery results of Ours(S RR) on Human3.6, 66, 3DPW and other in-the-wild images. In Fig. 4D, visualize the gradual improvement in the predicted mesh output across intermediate iterations of the training.
Limitations. Silhouettes offer significantly weaker supervision than 2D/3D pose and hence the model fails in presence of multi-level inter- part occlusions. Some complex poses also elicit limb-swapped articulations (shown in magenta in Fig. 4). Though, an only fitting based approach severely fails to solve such cases, optimization coupled with CNN training fairly recovers this deficiency as a result of the inductive bias instilled in the source trained model. Here, the inductive bias can be improved by training on multi-source data, such as combining S and R for adapting to extreme domain shifts like thermal or NIR. In future, it remains interesting to explore ways to address such deficiencies via temporal and multi-frame articulation consistencies.
Negative societal impacts. Although our approach in itself does not pose any direct negative societal impacts, we understand that the task at hand (i.e. 3D human pose recovery) may be directed towards unwarranted and intrusive human activity surveillance. We strongly urge the practitioner to confine the applicability of the proposed method to strictly ethical and legal use-cases.
5 Conclusion
In this paper, we introduce a seff-supervised domain adaptation framework that relies only on silhouette supervision. The proposed convolution friendly spatial distance field computation helps us disentangle the topological-skeltoon thereby facilitating seamless morphological silhouette alignment. The empirical evaluations clearly highlight the usefulness of the framework for practical deployment scenarios, even as a remedy to adversarial attack. Integrating temporal aspects, such as optical-flone flow, in our framework remains an interesting future direction.

---

Acknowledgments and Disclosure of Funding
This work was supported by Qualcomm Innovation Fellowship (Jogendra) and a project grant from MeitY (No.4(16/2019-TTEM) Govt. of India. We would also like to thank the anonymous reviewers for their valuable suggestions.
References
[1] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014. 1,2,7
[2] Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sehastian Thrun, Jim Rodgers, and James Davis. Scape: shape completion and animation of people. In ACM SIGGRAPH, 2005. 3
[3] Benjamin Biggs, Thomas Roddick, Andrew Fitzgibbon, and Roberto Cipolla. Creatures great and smal: Recovering the shape and motion of animals from video. In ACCV, 2018. 3
[4] Harry Blum. Transformation for Extracting New Descriptors of Shape. M.I.T. Press, 1967.
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael Black. Keep smpl: Automatic estimation of human pose and shape from a single image. In ECCV, 2016. 1, 2, 3,
[6] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In CVPR, 2017. 8,
[7] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih en Wei, and Yaser Sheikh. Openpose: Realtime multi- person 2d pose estimation using part affinity fields. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2019. 8,
[8] Sukmoon Chang. Extracting skeletons from distance maps. International Journal of Computer Science and Network Security, 2007. 4
[9] Endri Dibra, Himanshu Jain, Cengiz Öztireli, Remo Ziegler, and Markus Gross, Human shape from silhouettes using generative hks deseriptors and cross-modal neural networks. In CVPR, 2017. 2
[10] Carl Doersch and Andrew Zisserman. Sim2real transfer learning for 3d human pose estimation: motion to the rescue. In NeurlPS, 2019. 2, 7,8
[11] Haoqiang Fan, Hao Su, and Loonidas Guibas. A point set generation network for 3d object reconstruction from a single image. In CVPR, 2017. 2,
[12] Dariu M. Gavrila and Larry S. Davis. 3-d model-hased tracking of humans action: multi-view approach. In CVPR, 1996. 4
[13] Polina Golland, W. Eric, and L. Grimson. Fixed topology skeletons. In CVPR, 2000. 4
[14] John C Gower. Generalized procrustes analysis. Psychometrika, 40(1):33-51, 1975. 8
[15] Peng Guan, Alexander Weiss, Alexandru O Balan, and Michael Black. Estimating human shape pose from a single image. In ICCV, 2009. 3
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. In ECCV, 2016. 7
[17] Yinghao Huang, Federica Bogo, Christoph Lassner. Angjoo Kanazawa, Peter Gehler, Javier Romero, Ijaz Akhter, and Michael Black. Towards accurate marker-less human shape and pose estimation over time. In 3DV, 2017. 2, 3, 9,
[18] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio, Binarized neural networks. In NeurIPS, 2016. 5
[19] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 2013. 1, 2, 7, 8
[20] Umar Iqbal, Paylo Molchanov, and Jan Kautz. Veakly-superviae 3d human pose learning via multi-view images in the wild. In CVPR, June 2020. 1.2
10

---

[21] Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC, 2010.
[22] Sam Johnson and Mark Everingham. Learning effective human pose estimation from inaccurate annotation. In CVPR, 2011. 2,8
[23] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR. 2018. 3,
[24] Angjoo Kanazawa, Jason Y Zhang, Panna Folsen, and Jitendra Malik. Learning 3d human dynamics from video. In CVPR, 2019. 7,8
[25] Diederik Kingma and Jimmy Ba. Adam: method stochastic optimization. arXiv preprint arXiv:1412.660 2014.
[26] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In ICCV, 2019. 1, 6, 8,
[27] Jogendra Nath Kundu, Nishank Lakkakula, and R Venkatesh Babu. Um-adapt: Unsupervised multi-task adaptation using adversarial cross-task distillation. In ICCV, 2019. 1
[28] Jogendra Nath Kundu, Jay Patravali. Venkatesh Babu. Unsupervised cross-dataset adaptation via probabilistic amodal 3d human pose completion. In WACV, 2020. 6
[29] Jogendra Nath Kundu, Mugalodi Rakesh, Varun Jampani, Rahul M Venkatesh, and R. Venkatesh Babu. Appearance consensus driven self-supervised human mesh recovery. In ECCV, 2020. 2, 3.7
[30] Jogendra Nath Kundu, Siddharth Seth, Varun Jampani, Mugalodi Rakesh, R Venkatesh Babu, and Anirban Chakraborty. sele-suee 3d human pose estimation via part guided novel image synthesis. In CVPR. 2020. 3
[31] Jogendra Nath Kundu, Siddharth Seth, MV Rahul, Mugalodi Rakesh, R Venkatesh Babu, and Anirban Chakraborty. sineematiesttuucuur-rrrseet representation for unsupervised 3d human pose estimation. In AAAI, 2020. 6
[32] CMU Graphics Lab. Motion capture library, http:///moccap.csccmuuuu/
[33] Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael Black, Peter Gehler.
Unite the people: Closing the loop between 3d and 2d human representations. In CVPR, 2017. 2, 3,4
[34] Junbang Liang and Ming C Lin. Shape-aware human pose and shape reconstruction using multi-view images. In ICCV, 2019. 2,
[35] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. 2,8
[36] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICML, 2015.
[37] Matthew Loper, Naureen Mahmood, and Michael J Black. Mosh: Motion and shape capture from sparse markers. ACM Transactions Graphics, 2014.
[38] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: skinned multi-person linear model. ACM Transactions on Graphics, 2015. 3
[39] Xiankai Lu, Wenguan Wang, Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised video object segmentation with co-attention siamese networks. In CVPR, 2019. 2
[40] Alireza Makhzani. Jonathon Shlens. Navdeep Jaitly, lan Goodfellow, and Brendan Frey, Adversarial autoencoders. arXiv preprint 1r150:151..05644, 2015. 6
[41] Julicta Martinez, Rayat Hossain, Javier Romero, and James Little. simple yet effective baseline for 3d human pose estimation. /CCV, 2017. 7
[42] Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko. Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved can supervision. In 3DV, 2017. 1,2

---

[43] Dushyant Mehta, Srinath Sridhar, Oleksandr Sotaychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel. Weipeng Xu, Dan Casas, and Christian Theobalt. Vnect: Real-time 3d human pose estimation with single rgb camera. ACM Transuctions on Gruphics, 2017.
[44] Seyed Dezsavi- Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversar- ial perturbations. CVPR, 2017.
[45] Konda Reddy Mopuri, Aditya Ganeshan, and R Venkatesh Babu. Generalizable data-free objective crafting universal adversarial perturbations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018. 7
[46] Lukáš Neumann and Andrea Vedaldi. Tiny people pose. In ACCV, 2018.
[47] Georgios Pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Daniliddss Learning estimate 3d human pose and shape from a single color image. In CVPR, 2018. 2, 7.8
[48] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from single image. In CVPR, 2019. 6
[49] Georgios Pavlakos, Nikos Kolotouros, and Kostas Daniilidis. Texturepose: Supervising human mesh estimation with texture consistency. In ICCV, 2019. 2,
[50] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv:2007.0850) 2020. 4,
[51] Helge Rhodin, Jörg Spöri, Isinsu Katircioglu, Victor Constantin, Frédéric Meyer. Erich Müller, Mathieu Salzmann, and Pascal Fua. Learning monocular 3d humaa pose estimation from multi-view images. In CVPR, 2018.
[52] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Audrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 2015.
[53] Leonid Sigal, Alexandru Balan, and Michael Black. Combined discriminative and generative articulated pose and non-rigid shape estimation. In NewrlPS, 2008. 3
[54] Cristian Sminchisescu and Alexandru Cristian Telea Human pose estimation from silhouettes a consistent approach using distance level sets. In WSCG, 2002.
[55] Vince Tan, Ignas Budvytis, and Roberto Cipolla. Indirect deep structured learning for 3d human body shape and pose prediction. In BMVC, 2017. 2,
[56] Hsiao-Ya Yu Tung, Hsiao-We Tung, Ersin Yumer, and Katerina Fragkiadaki. Self-supervised learning of motion capture. In NeurlPS, 2017.
[57] Gul Schmid. Learning from synthetic humans. In CVPR, 2017. 7.
Romero, Xavier Martin, Naureen Mahmond, Michael Black, Ivan Laptev, and Cordelia
[58] Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll, Re- covering accurate 3d human pose in the wild using imus and moving camera. In ECCV, 2018.
[59] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In CVPR, 2016. 8,
[60] Xiangyu Xu, Hao Chen, Francese Moreno-Noguer, László A Jeni, and Fernando De la Torre. 3d human shape and pose from a single low-resolution image with sule-supe learning. In ECCV, 2020. 3, 7
[61] Zhao Yang, Qiang Wang. Luca Bertinetto, Weiming Hu, Song Bai, and Philip HS Tom. Anchor diffusion for unsupervised video object segmentation. /CCV, 2019.
[62] Jianfeng Zhang, Xuecheng Nie, and Jiashi Feng. Inference stage optimization for cross-scenario 3d human pose estimation. NewrIPS, 2020.
12


Reviews:
[{"review_id": "rMfCj-N5SwC", "reviewer": "Reviewer_oYg4", "summary": "The paper proposes an online adaptation method to adapt pretrained models to new datasets through the silhouette and their proposed topological skeleton via inward distance map and outward distance map. They also propose a Chamfer-inspired topology-alignment objective to align the silhouette better. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The results of the paper are compelling and the generalization is good. The optimization on Thermal and low-res images is also amazing. However, as the loss function is only limited to T, I am not sure why not added the previous D and S as loss functions?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Will the method work on occluded cases?\nlike images in https://arxiv.org/abs/1905.07718 or https://prox.is.tue.mpg.de ?\n\nWhy the loss function only involves T as Eq 4?\nCan D and S as loss function?\n\nThe writing and organization are poor and a little hard to follow. \nFor examples, citation [5] [6] is the same person, however, the names are cited differently.\nline 197: double the.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 "}, "extra_text": {}}, {"review_id": "_b41lYDoGoP", "reviewer": "Reviewer_xChc", "summary": "This work proposes a novel method to tackle the lack of annotated data for 3D human pose recovery. The core observation is that silhouette estimation is not prone to domain gaps. Hence, the authors develop the topological skeleton, obtained from silhouette, to guide the training. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well-written. It is a joy reading this paper.\n\n- The proposed method is sound and interesting. Specifically, 1) transferring chamfer distance, a normally 3D loss to images is innovative; 2)  computation of the ridge-lines via convolution-friendly operations is carefully designed and beautifully implemented. \n\n- Thorough ablation studies. \n\n- Insightful discussions. The observation on the robustness of silhouette estimation across domains is highly thought-provoking. Also, it is less relevant but important to highlight that articulation degradation is the key result of the domain gap (Line 345). \n\nWeaknesses:\n-  Please explain the setting used for HMR and SPIN in Table 2, under the weak sup. setting. What does \"unpaired\" mean?\n\n- Optimization in the loop is used. However, this involves using a well-established regressor. It raises a concern that the network regressor may play an over-important role in determining the performance of the method, because the regressor is trained in a supervised way, usually making it much better even if there exist domain gaps.\n\n- As stated in the Limitation section that \"inter-part occlusion\" remains a challenge, is this an inherent limitation of the proposed method? i.e. impossible to solve?\n\n- Unlike many fields in which unsupervised learning can produce close results to the supervised learning counterpart, the unsupervised learning in 3D human pose recovery is, however, far from being practical since there is a large performance gap. I would like to hear the authors' comments on this.\n\nMinor:\n- Line 332: it may be more convincing to provide some qualitative evidence to support the statement.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have stated clearly the limitations and societal impact, and provide sound solutions to address them.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "YZNdQwU0m8", "reviewer": "Reviewer_DMVp", "summary": "The authors present a method for human pose and shape estimation that uses silhouette information as an intermediate representation that is well suited for domain shifts. Albeit not new as a concept, the proposed implementation contains several novelties (computation of distance map, efficient approximation of Chamfer-like distance metric), is sound, and effective.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Missing related work (pose estimation from silhouette, as proposed here):\nCreatures great and SMAL: Recovering the shape and motion of animals from video\n\nThe approximation of distance maps via convolution and pooling operation is creative, effective, and novel. Are there any corner cases (corners?) where the approximation fails? Could you quantify the approximation to a true distance map and visualize some examples qualitatively too?\n\nHow much faster is this method to the non-convolutional computation of distance maps and chamfer distance (each measured individually). I don't doubt that it is faster, but like to know how much.\n\nThe ablation study is extensive in terms of accuracy.\n\nOverall the paper has sufficient novelty, is easy to follow, and the evaluation is sufficient; showing clear improvements of model components and fairs well in comparison to related work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n", "review_text": "", "extra_scores": {"ethical_concerns": "n", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1.5h"}, "extra_text": {}}, {"review_id": "KIF8kS7EK_1", "reviewer": "Reviewer_fpzf", "summary": "Human mesh recovery methods commonly resort to 2D keypoint supervision to be able to handle in-the-wild images, as these often lack 3D ground truth. This submission argues that binary silhouettes are a better source of weak supervision and that classical segmentation techniques are less susceptible to domain shifts that 2D keypoint detectors. As such, silhouettes can be used to train shape and pose regressors on more heterogenous data without the need for annotating and retraining keypoint detectors, which this submission does: A SPIN-like method (Kolotouros et al., 2019) is proposed that replaces the keypoint-based loss and refinement, with a silhouette-based loss that allows for comparisons between the model and observed silhouettes. Key here is a differentiable skeletonisation operation that pares the silhouette down before applying the loss. The proposed method compares favourably against the state-of-the-art on a variety of datasets. The submission additionally demonstrates that the resulting scheme is more robust to adversarial perturbations than competing approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I find this to be an interesting approach. I am somehow impressed that the submission makes the case for background subtraction as a replacement for keypoint-based fitting. In principle, I actually agree that keypoints by themselves are not reliable enough especially due to the domain adaptation problem as pointed out by the authors. The results also partly bear this out, especially when adversarial perturbations are applied. In general, incorporating low-level information can be very useful for pose estimation and indeed help address domain shifts. Silhouettes also provide information that keypoints don't include (e.g. shape, depth relations). \n\nNonetheless, I completely disagree with the following: [L41]: \"traditional FG [foreground] segmentation techniques are mostly unaffected by input domain shifts (Fig. 1C) and thus the FG masks are usually robust and easy to obtain in diverse target scenarios\". Even if we assume this to be the case, it's notoriously hard to get reliable foreground segmentations without learning. It's especially hard in crowded scenarios or in poor viewing conditions. There's also the problem of self-occlusion. This is briefly mentioned at the very of the paper in the \"Limitations\" paragraph, but it's a really major problem that should somehow be addressed much more explicitly when the method is being introduction. In most of the visual examples, the depicted people conveniently do not self-occlude much, but in Fig. 1 C/D, the 3rd example demonstrates the problem: The skeletonisation procedure misses the right arm entirely. \n\nSo while I agree that keypoint detection has its limitations that can partially be overcome by alternate sources of supervision, foreground segmentation (without learning) has different and partially more severe limitations. The submission also argues (or at least implies) that the lack of point-to-point correspondences in the proposed loss is an advantage over keypoint-based losses (e.g. L213), but this is not the case. If your 3D regressors provides you with a poor initialisation, the lack of correspondences is a problem. It is not a problem and can even be helpful when you are close to the solution and do not want to fit your model to observation noise but there is a trade-off.\n\nThe results are good enough for me, but I'm not sure about the fairness of the comparison against the state-of-the-art. Table 2 compares against other methods on H36M, and this approach outperforms others in the weakly- and unsupervised setting. However, it is very important especially in this area to have some comparisons that control for differences in the training data. Having the right pose distribution is at least as important as being able to handle shifts in appearance. As such, it would be good to have direct comparisons against HMR or SPIN using the same training setup. Given that SPIN is used for the ablation study later, that would probably be a good choice. The comparison against 3DPW is similarly problematic. Why aren't HMR and SPIN also compared in the weakly-supervised setting? It is not very surprising that SMPLify is significantly worse, given that it relies on optimising the method. Otherwise I think the rest of the experiments are nice, e.g. the ablation study comparing different losses as well as the ones with adversarial perturbations / challenging low-resolution images. What I'm missing however is a description of how the silhouettes are obtained on the UAP-H36M and LR-3DPW datasets. This is a critical detail that will help me judge these results better.\n\nSilhouettes have a very long history in this area, and it's great that the submission cites some older papers. However, there are a lot of other relevant references that need to be taken into consideration. These need not all be cited, but I mention them here because of certain inaccuracies in the text: One very early method that is relevant is the 2002 paper of Sminchisescu and Telea \"Human Pose Estimation from Silhouettes: A Consistent Approach Using Distance Level Sets\". If I'm not mistaken, they also use something very similar to the \"Distance-map\" described in L153-L159.  Figure 3 in that paper shows the \"outwards distance-map\" as it is referred to in this submission, and the full objective is described in section 4. As such, it is inaccurate to that this has been \"introduce[d]\" here (L152). I also think the description of \"chamfer[-]based distance[s]\" (L145-L149) is inaccurate, as many rely on distance transforms as well, e.g. Gavrila & Davis (1996) \"3-D Model-based tracking of humans in action: a multi-view approach\" (Fig. 6 and section 4.1), described in more detail I think in Gavrila (1998) \"Multi-feature Hierarchical Template Matching Using Distance Transforms\": \"The chamfer distance consists thus of a correlation between T and the distance image of I, followed by a division.\" A couple of newer methods that use silhouettes include Alldieck et al., (2017) \"Optical Flow-based 3D Human Motion Estimation from Monocular Video\" who use a distance-transform-based loss as well, and Omran et al. (2018) who use part segmentations as an intermediate step before lifting to 3D.\n\nJust to be clear, I understand that a key difference here is the skeletonisation step, rather than the direct comparison of distance transforms applied to silhouettes, and that this extra step is helpful here (Table 4). The text however still needs to be amended as suggested. It would also probably not hurt to refer to a few works on skeletonisation based on distance transforms, as this also has a long history, e.g. Wright et al. (1995) \"Skeletonization using an extended Euclidean distance transform\" or Latecki et al. (2007) \"Skeletonization using SSM of the Distance Transform\". Maybe there are more appropriate references. What appears to be novel here as well is the differentiable skeletonisation but I am not sufficiently familiar with this problem area not to mention the use of distance transforms and skeletonisation outside of pose estimation, so I'm not sure.\n\nI'm not sure why the binary activation function described in section 3.2 is necessary. The submission justifies this by stating that \"a non-binary silhouette is not suitable for distance-map and topology definitions\" (L176). Why would a \"silhouette obtained via a fast differentiable renderer\" (L174) be non-binary? If every mesh vertex has the same colour and no lighting is enabled, the resulting image will be binary and would not break differentiability.\n\nEditing suggestions:\n- the caption in Figure 2: The description of the topological-skeleton extraction in B is also in 3.1.3 (not just 3.2).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address the potential misuses of 3D mesh recovery for surveillance and caution against such applications.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission has received 4 positive final ratings: 6, 7, 6, 7.
The reviewers appreciated overall novelty, extensive experiments, strong empirical performance and clear presentation.
The remaining minor questions and concerns were addressed by the authors in the rebuttal, as acknowledged by the reviewers.
As a result, the final recommendation is to accept for poster presentation.
Decision: Accept (Poster)