Synthetic-to-Real Pose Estimation with Geometric Reconstruction
Authors: Qiuxia Lin
Kerui Gu
Linlin Yang
Angela Yao
Venue: NeurIPS 2023 poster (2023)
Keywords: pose estimation
domain adaptation

Abstract: Pose estimation is remarkably successful under supervised learning, but obtaining annotations, especially for new deployments, is costly and time-consuming. This work tackles adapting models trained on synthetic data to real-world target domains with only unlabelled data. A common approach is model fine-tuning with pseudo-labels from the target domain; yet many pseudo-labelling strategies cannot provide sufficient high-quality pose labels. This work proposes a reconstruction-based strategy as a complement to pseudo-labelling for synthetic-to-real domain adaptation. We generate the driving image by geometrically transforming a base image according to the predicted keypoints and enforce a reconstruction loss to refine the predictions. It provides a novel solution to effectively correct confident yet inaccurate keypoint locations through image reconstruction in domain adaptation. Our approach outperforms the previous state-of-the-arts by 8% for PCK on four large-scale hand and human real-world datasets. In particular, we excel on endpoints such as fingertips and head, with 7.2% and 29.9% improvements in PCK.

Synthetic-to-Real Pose Estimation with Geometric Reconstruction
Qiuxia Lin'
Kerui Gu Linlin Yang
Angela Yao'
Department of Computer Science, National University of Singapore 2 State Key Laboratory of Media Convergence and Communication, CUC School of Information and Communication Engineering, CUC fqiuxia, keruigu, ayomp_comp. mus edu.
Abstract
Pose estimation is remarkably successful under supervised learning, but obtaining annotations, especially for new deployments, is costly and time-consuming. This work tackles adapting models trained on synthetic data to real-world target do- mains with only unlabelled data. A common approach is model fine-tuning with pseudo-labels from the target domain; yet many pseudo-labelling strategies cannot provide sufficient high-quality pose labels. This work proposes reconstruction- based strategy as a complement to pscudo-labeling for synthetie-to-reet domain adaptation. We generate the driving image by geometrically transforming base image according to the predicted keypoints and enforce a reconstruction loss to refine the predictions. It provides novel solution effectively correct confident yet inaccurate keypoint locations through image reconstruction in domain adap- tation. Our approach outperforms the previous oftte-of-the-are by 8% for PCK on four large-scale hand and human real-world datasets. In particular, we excel endpoints such fingertips and head, with 7.2% and 29.9% improvements PCK.
1 Introduction
Training deep models for human pose estimation, be it of the full body [GLL
2g 2
hands [+ E.]
requires large-scale labelled datasets. Synthetic data [43, [T] has become more favoured due to the ease of generation and the challenges of accurately annotating real-world data. Despite advance- ments in realistic rendering [23, [5], models trained on synthetic data do not generalize well to real- world settings. The gap arises due to variations in pose, appearance, lighting, and other low-level differences between the two data domains. This paper aims to close this gap and targets learning from labelled synthetic and unlabelled real-world data for application real-world settings.
Domain adaptation studies the transfer of models from synthetic to real data. Recent works on domain adaptive pose estimation [177 5R, [S] learn from target pscudo-labels. However, the gains from pseedo-labae ascuuo-labelling can be limited. There is an inherent contradiction only labels which are suffi- ciently accurate benefit the learning, and it is important ensure label quality through selection [TS]] and or correction schemes [[66, [8]. Yet the ability to obtain high-quality pseudo-labels implies that the model is sufficiently capable of estimating the poses of said samples. The cross domain setting makes it difficult to obtain high-quality labels. The challenge is compounded by additional gaps like different subjects' anatomical sizings, different annotation standards across datasets, etc.
Instead of estimating labels for the given target data, what if we try to reconstruct the data itself? This latter strategy is often used in self-supervised pose estimation, where learning is guided by reconstruction losses 6 [2]], The challenge lies in linking the reconstruction scheme back to
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the underlying pose". While it is feasible to use a differentiable renderer, there are several drawbacks, such as the complexity of the renderer [| and the difficulty of model fitting [[66]
With this motivation, we design a simple yet effective reconstruction scheme that connects keypoint prediction via bone maps. Specifically, given base and driving image pair with different poses, we estimate the keypoints and region-wise geometric transformations between the two poses, where bone linking two joints. The transformation is then applied with the predicted mask to obtain fine-grained warping (as shown in Fig. I.. The complete reconstruction conducted in the generation module, which translates the base image into the driving image, specifically performing feature warping and inpainting in downsampling and upsampling structures. Intuitively, the more accurate the estimated keypoints and geometric transfommation in the image pair, the better the reconstruction. Differences between the reconstruction and the original driving image can serve as a supervisory signal to improve keypoint localization.
each region corresponds
A key benefit of the reconstruction scheme is that it can leverage almost all the samples in the unlabelled target dataset and give refinement according to the reconstruction loss. Pseudo-labeled samples are often selected based on some confidence measure like heatmap response, but given that most neural networks are poorly calibrated [22, 19], the confidence is often unsatisfactorily aligned with pose accuracy. Many samples can be highly confident yet inaccurate and the predictions remain inaccurate in this way. This is a key reason why pseudo-labelling methods easily saturate (see Table B (b)). In contrast, our reconstruction scheme requires only the two images containing the same object (hand//uuman) and rigid background. As such, we are able improve those confident but wrong keypoints based on 2D evidence in the target domain that arises from the reconstruction loss (see Table (c)). Also, for keypoints that are both confident and accurate, the reconstruction will not degrade the result since it has perfectly provided useful information for reconstruction.
The supervision through reconstruction is indirect and is weaker source of information than pose pseudo-labels. However, given that the two approaches are complementary, they can also be used in conjunction, i.e., limited but strong supervision from pseudo-labelling in combination with larger and diverse source of weak supervision from the reconstruction. To that end, we additionally propose new selection criteria for pseudo-label that integrates 2D evidence and 3D kinematic constraints. In combination, we are able to outperform state-of-the-ar domain adaptation methods by 8%, with significant improvements on difficult keypoints, like fingertips for the hand, and head for humans by 29.9%. Our contributions can be summarized as
A novel reconstruction-basel approach for domain adaptive pose estimation. We introduce a reconstruction method to make up for the limitations of the pseudo-labelling, and realize the connection between image reconstruction and keypoint prediction by building bone maps, which enables the correction of keypoints through optimizing reconstruction loss.
An improved pseudo-labelling selection criteria that integrates both 2D) evidence and 31) kinematic constraints. The new criteria improves the quality of the pseudo-labels by enlarg- ing the set of usable pseudo-labels by 13%.
Domain adaptation results from synthetic to real-world data for both hand and human body show that outperform tice-of-theerrr by large margin, with much 29.9% im- provement on difficult keypoints like fingertips and head and an overall 8% improvement.
2 Related Work
2.1 Cross-domain Pose Estimation
Cross-domain challenges in pose estimation, especially in the context of synthetic-to-real, are best known for animal pose estimation [I]. 20. [Z]], where it is challenging to collect and label sizable train- ing data. Similar challenges, however, also exist for the human body [+]]] and hand [GG] T. [X as deep learning algorithms become more data-hungry. The standard strategy is to apply consistency constraints [20, [36] or aseudo-labelling [[[7 [SS] to learn effective features for the target domain. Consistency constraints can be applied in equivariant augmentations [20]], and cross cross-modality [36]]
'We make distinction between data synthesis and reconstruction. The former generates image observations based on given poses; the latter recreates image observations without knowledge of the underlying pose.

---

To generate less-noisy aseudo-labeld and guide the network fine-tuning, [|]]] utilized the normal- ized output from a teacher network and [T] proposed a progressive self self-paced pseudo-label updating strategy to make the guidance more stable. Both works denoise pseudo-labels based the heatmap activations while [36, [8] reduced noise with pose correction procedure. Fither strategy alone can lead to the neglect of additional correlation of depth or the accumulation of prediction errors. We therefore leverage both strategies simultaneously and embrace their union of pseudo-labels indica- tors when their consensus is high to enrich guidance on the target domain.
2.2 Conditional Image Reconstruction
Reconstructing images conditioned on paired images is an effective technique to concentrate on spe- cific attributes (e.g. keypoints) while ignoring irrelevant ones in pose estimation, where the focus can be summarized conditions design [M I1 2 Jakab et al. [TTO] proposed to disentangle key- point heatmaps from appearance. The heatmaps, as conditions to generate images, can be optimized by the reconstruction loss. A follow-up work [T]]] utilized skeleton image to facilitate the gener- ation process. Furthermore, recent work [25] regressed affine parameters to deform a pre-defined shape template conditions. However, they all extract appearance and keypoints from different im- ages separately and overlook the geometric relationship across images, which can provide explicit guidance image reconstruction.
Using geometric relationships across images to perform reconstruction has been seen in image ani- mation [27] 26, 28]. They predict motion-specific keypoints [27, [6] or semantic regions [28], and obtain transformation parameters to deform one image to resemble another image. Those motion- specific keypoints or region-based methods are generalizable in predicting transformation parame- ters and modeling complex movements. But they target achieving the mean and deviation of each region, which is unrelated to our purpose of joint locations.
To emphasize the geometric relationship (i.e.. the transformations between the two images) per- form reconstruction for refining joint locations, we connect semantic regions and joint locations via bone maps, which enables the correction of poses through optimizing the reconstruction loss. Natu- rally, the reconstruction becomes complement to pseudo-labelling, and both of them contribute to domain adaptive pose estimation.
3 Method
3.1 Problem Formulation
Consider labelled data D, = {[Iss M from source domain and unlabelled data D; {(I+)) from target domain t, where denotes an RGB image, y the pose label, and N, and N₂ are the number of samples in the source and target datasets. The objective is to learn pose estimation system fap = (fenc o faec), trained on {D., Dr} and apply it to hold-out test data on the target domain for estimating 2D or 3D keypoints.
We follow other domain adaptive pose estimation methods, and pre-train on the labelled source domain (synthetic) and then fine-tune the unlabelled target domain (real-world). In pre-training, (Sec. B.2), we use the labelled synthetic data D. 2 to learn an initial pose estimation network. For fine-tuning on unlabelled real-world data DD (Sec. B.3), we introduce pseudo- Additionally, we propose a geometric The overall pipeline
strategy to
correct 2D pscudo-labels via 3D constraints in Sec. L reconstruction to correct keypoint locations from image signals in for fine-tuning is shown in Fig. MI
3.2 Pre-training on the Source Domain
Keypoint Predictor The keypoint predictor fkp = (fone o fass) is first pre-trained on the synthetic source images I, to estimate a heatmap ho and a relative depth d. As the synthetic images have ground truth labels, fep can be learned in supervised manner with ground-truths (1.) d:);
Cup N.
1 + Aa lad - -d21)2)) >
(1)
3

---

Bhaar, base/ /rrvving bone
db
Integrated pacudo-labels
Clmmh inpainting map
Derehh displacement
Adresh transformation parmmeeerrs
pire
fenr
fave
A
C.romb
normalized courdinates
W
I hrdrs; keypolnt locations
Rdr
Ddiv-b
O: multiplication
M: mask
C: deformation
jdrv
(a) Fine-tuning on Target Domain
[Bury SVD [B
Ec
Eq.
(Alreey Expand
[M,)//
(A drvmb
{A (Ad
PP
Ddrvmb
Rp
(b) Warping Estimation Module
Figure 1: Illustration of pipeline modules fine-tuning. (a) Fine-tuning on larget domain. Given base and driving images sharing the same background but different poses, we forward them into keypoint predictor Su = (Sexe [ax) produce keypoint heatmaps hn and relative depth predictions d. Afterward, our integrated pscudo-labellng labelling PP (See Sec. 6.3.3) is introduced to generate pscudo-labels, while the warping cstimation module W and the generation module G (See Sec. (x6))))) are used to reconstruct driving images from base images. (b) Warping estimation modulc. Bone maps derived from keypoints transformation parameters via SVD and Expand. With the transformation and base images, hourglass used generate inpainting map Carveb and region-wise displacement field D dave-b that will work on upsampled feature maps in the generator g.
used to calculate coarse
where da is hyper-parameter that trades off depth loss and heatmap loss. The ground-truth heatmap he is constructed as 2D Gaussian centered at the ground-truth coordinate (u vdt) and a standard deviation of 2. The ground-trute d the depth relative to the root keypoint. We follow the same practice in [Р] to regress 2.5D and lift it to 3D pose. To elaborate, we employ soft-argmax operation on hp and then perform Hadamard product with depth value maps to derive the relative depth d. The expected outcome of this operation yields the 2D coordinates, denoted as (ū, t).
Invariant Feature Learning Models trained only on synthetic data D, are unlikely to perform well on target data D, (see Source only Table I... One reason the domain gap between source and target data; typically, synthetic data used for pose estimation is not fully photorealistic, so the model does not generalize well real-world data [9], 3II prevent Sup from overfitting to source- specific representations, which is common [38, 21J.. we augment the source data to resemble target data and align the original and augmented source features using contrastive learning [3] Specifically, we train
StyleNet from [ which transfers the style of the target domain to the source, while preserving
the source content, i.e., the pose. The keypoint predictor fkp is then trained with the supervised keypoint loss and an alignment on features with the following loss:
B
Calign
Σ 1=1
log
(2)
El= k (e((= + (((x.=t)\))))
with batch size B, temperature hyperparameter 7 0.5, and (-) denotes the cosine similarity. In the above equation, the features (z, x are derived from the source and augmented source image.
[T55] also applies a StyleNet to augment source samples but only supervises the network output. Our work extends the supervision to the latent features, as it can further benefit task-specific feature learning by building a consistent embedding space. is feasible to also add target data into invariant feature learning with the style of source data, but we empirically find no obvious improvement. Therefore, we only perform alignment between source and augmented source features considering the limited memory.

---

3.3 Fine-tuning on the Target Domain
The fine-tuning on the target domain consists integrated pseudo-labelinn and geometrie recon- struction. The integrated pseudo-labelling extracts confident pesudo-labels with guidance from both 2D heatmaps and 3D constraints. Since the label can still be noisy, we also have geometric recon- struction strategy that generates driving image based on the base image and pose transformation, which provides supervision simply from 2D image signals to improve those confident yet inaccurate keypoints.
3.3.1 Integrated Pseudo-labelling
commonly used strategy to adapt source models to a target domain is pseudo-labelling. ensur- ing that pscudo-labels are valid, existing works have leveraged the heatmap activations in hip [LS] or kinematic feasibility of the pose [[66]. Both strategies have drawbacks; the former ignores cor- relations that may exist in d, while the latter uses greedy strategy that may accumulate errors down kinematic chains. With these considerations mind, we propose an integrated pseudo-lahelling strategy that combines the use of the 2D heatmap and the kinematic feasibility of 3D predictions. Specifically, given a predicted heatmap hip we generate a 2D confidence mask MaD RBxK as indicators to select keypoints with heatmap activations larger than the threshold "au in each batch during training. The 2D pseudo-label hip is the gaussian heatmap hp centered on the predicted 2D coordinate, as defined in [[S]]] For the 3D pseudo-labels, the prediction ý (û, 0, d) lifted into 3D coordinates and rectified as pose y with reasonable feasibility based on the pose correction [36]]
The 3D confidence mask MaD E TR. 1gxx indicates the qualified keypoints based on the comparison of 1.2 norm between the 2D projected coordinates before and after correction on ÿ to the threshold 73D-
For the first T epochs of fine-tuning, the pseudo-labels hip and y generated by 2D evidence and 3D correction. respectively, are both not confident, so we require them to train independently with their corresponding mask. Then we integrate the confidence mask MID and MaD complement each other and enrich the training set as M = MMD U MaD, which considers more pseudo-labels for adaptation to both 2D and 3D. Overall, we optimize the following loss,
MM EMD, 11 Tilll M Em T + N,y Ú Øl12)
Maol EMm, 11 Ull epoch T
Ay
Cpesado
(3)
epoch > T
where IM indicates the number of elements in M and Ay 0.1 coefficient that trades off pseulo-la between 2D heatmap loss and 3D loss. We find out that as fine- fine-tuning proceeds, the overlap of confidence masks MID and MaD will increase, at which point a single strategy can also provide valuable indicators for the other strategy.
3.3.2 Geometric Reconstruction
Although the pseudo-labelling strategy is improved by utilizing both 2D and 3D information, the amount of pseudo-labels is limited and some confident labels are still incorrect. This is because these labels are generated from the model calibrated on the source domain, we therefore propose a geometric reconstruction that solely exploits specific features in the target domain. The perfor- mance of reconstructions is weaker compared to high-level pseudo-labeling. However, can serve as complementary supervision by including all target data and correcting confident yet inaccurate keypoints by levelevel evidence.
The geometric reconstruction consists of Warping Estimation Module W and Generation Module G, shown in Fig. IIKa). Given a pair of images from target domain containing human hand (I0, ddrv sampled in video, The aim is to reconstruct driving image jdr xx from base image 16 E 3 xW where the image size is H x W. To do so, the generation module G warps base image I6 based on specifications Ddrv+ E R x W x2 and Cdiveb E RIx 4 x 4h estimated by the warping estimation module W:
dr = D drve- Cdlve--b,,
where [ d by (ihhh,nrr
(4)
5

---

In Eq. B. Ddvve and Cdive-b denote region-wise displacement field and an inpainting map respec- tively. The displacement field D is dense vector field that specifies each pixel location in Id with its corresponding location in Is And Cdrveb is an attention map ranging from 0 to 1, where the should do more inpainting on pixels with lower values. We apply a Perceptual Loss to penalize the semantic differences between the reconstructed image īdrv and its original image 1dr to refine the keypoints. The details of each component (in bold) as follows.
Warping Estimation Module W Given the heatmap hp estimated from fkg. we decode it into coordinates by using soft-argmax followed by an expectation: this combination is the standard approach followed in coordinate or integral regression methods [8, 29, 71 15). Specifically, we obtain the k-th 2D keypoint locations μt in all K keypoints while keeping the pretrained heatmaps:
eh
F Σp.hp. where hp PEN
Ep'cn e'
is the area of the heatmaps, which are generated by fup.
Following the coarse-to-fine pipeline, to obtain coarse motion of each body part, we need to obtain the pose transformation parameters. Previous works [25% [27] use the network to do direct
regression but proved to have a poor generalization [28]. We thus follow [28] to use singular value
decomposition (SVD) to compute transformation parameters of the bone map with respect to a reference heatmap. which is whitened and has zero mean and identity covariance. However, different from [28], where the bone maps are predicted by the network, we generate the bone map according to keypoint locations μ4 and the kinematic tree. Formally, we assume there are J chains in the kinematic tree and we denote the parent and child node of the kinematic chain Cy as μtp and ji respectively. The value of bone map B, at pixel p of the j-th kinematic chain can be formulated
by(p) e dd((p//r²
where a is hyperparameter that controls the thickness of the edge and dj denotes the distance between the pixel p and the edge determined by Hip and pje- Accordingly, the transformation parameters A of can be obtained by
A U,S psl, where U,S,V,
bb,(D)e - pt)(p ,,,)),,;), PEN
SVD
where pe is the mean of Uip and Uje- By Eq. M A deve-ref Finally, the geometric transformation of bone j from driving to base image is calculated by A dr A rel (A (bb-ree))
get the transformations A be-ref and
To generate a dense shift to deform Is to approach 1dv, we next convert J/ transformation parameters fine-graing -reinga displacement field along with background transformation parameters. Specifically, with the input of I", Bh and Buh an hourglass network outputs J J+1 1 masks M, indicating one background and J bones. Mo corresponds to the background. (p) denotes the possibility of p belonging to the j-th region when j >> O. To ensure that the possibilities of every channel sum up to one, softmax is applied across the channels. Then, the displacement field Ddivtb((p) (p) E TR.2, is
generated by
D Ddlveb(p) (p) (p)=
[D)=EM,nn ( A
P P
(8)
1=0
Expand
Through region-wise displacement field Ddrv+b, the network gains insight into the specific locations on the base image that require deformation to match the driving image. Nonetheless, certain regions may be absent in the base image while present the driving image. To identify these areas in need of inpainting by the network, we add an additional layer after the same hourglass to generate the inpainting map d
We only need to rotate, translate, and scale to realize the transformation between the designed bone maps, which is different from the previous works based on affine transformation.

---

Generation Module G We build G with downsampling and upsampling, interconnected through skip connections. Similar to [28]] the feature map from the skip connection is warped by displace- ment field Ddrvb and then activated with inpainting map Ca We add the processed feature map to the Cdiv+b activated feature map of the previous layer, and the sum is passed to the next layer. In this way, 9 can generate driving images through feature warping and limited inpainting from a similar image without the need for extensive rendering.
Perceptual Loss Following [T4] B5J, we apply multi-scale perceptual & loss with pre-trained VGG-19 19 network. Mathematically, the reconstruction loss Cree is formulated as
Lree Σ (d,((dm)) o5 (I*M)),
where (I) is the intermediate output of i-th layer of VGG-19.
Training Procedure The overall training of our framework is done in two stages. First is the pre- training on the source domain with supervised learning in Eq. W and invariant feature learning in Eq. with weighting hyperparameter Aa:
Spre Ssup + Xa Lalign-
(10)
The second stage finetunes in the target domain, with pseudo-labelling from Eq. K and geometric reconstruction from Eq. B together with supervision from the source domain to avoid over-fitting:
Line tune Loup + X Lree + Xp
(11)
where X, and Ap are balancing hyper-parameters. We have a warmstart in the fine-tuning where Loup and Lpseudo first used and then add free together update the overall pipeline.
4 Experiments
4.1 Datasets & Evaluation Metric
Hand Pose Datasets. For the source domain, we consider the synthetic dataset RHD [X]] with 44k rendered images for training. For the unlabeled target domain, we consider real-world datasets HBD [[]] and MVHand [3%]; these have I 1lk/2 and 42k/42k training/testing splits respectively. For H3D, we consider only the subset with one-handed gestures. H3D and MVHand are both multi-view datasets; we treat each viewpoint as a "video" source.
Human Pose Datasets. SURREAL [GT]] is synthetic human dataset with 6 million annotations
including keypoint locations. Human3. 3.6M [X] annotates 3.6 million real-world indoor human poses. We follow protocol 1, using S1 and S5-S8 for training and S9 and S11 for testing. 3DPW [32] is a challenging outdoor dataset with 24 videos for training and 24 videos for testing. keypoints by projecting the corresponding 3D ground truth.
We obtain 2D
Evaluation Metric. The convention of domain adaptive pose estimation works [2 is eval- uate 2D keypoint detection with Percentage of Correct Keypoint (PCK). We report PCK @0.05, which tabulates the percentage of correct predictions within the range of 5% of the input image size: a higher PCK indicates better performance. For 3D keypoint estimation, we evaluate with mean end end-point-erts (EPE); a lower EPE indicates better performance.
4.2 Implementation Details
Our framework is shown in Fig. II We use ResNet-101 as the feature extractor fene and deconvolu- tional layers as the decoder faee to produce a heatmap of size 2K 64 64 for 2D and depth. Each input image is cropped around the hand or the human and resized to 256x The initial learning rate is le-4, and we used Adam optimizer and degraded the learning rate with 0.1 steps 60 and 75. Pre- Pre-training is done with batch size of 64 for 40 epochs; fine-tuning is done with a batch size of 32 for 80 epochs. The hyperparameters Xa Xr, X are set as 0. We compare with state-of-the-art domain adaptation methods: RegDA [02], CC-SSL [20]] UniFrame [[S], SemiHand [[6]], and Du- alNet [18]. Where possible, results are reported directly from the corresponding papers; otherwise, results are based on officially released code. In pre-training, RHD [ and SURREAL [G]]
7

---

Method
RHD-+130
BEID--MVHand
MCP
PIP
DIP
Fin
All
MCP
PIP
DIP
Fin
All
Source only (Coup
67.4
64.2
63.3
54.8
61.8
32.1
70.5
64.6
39.8
52.7
SemiHand [E6]
67.2
56.3
DualNet
74.9
68.9
CC-SSL
81.5
79.9
74.4
64.0
75.1
60.2
RegDA [|]]
79.6
74.4
71.2
62.9
72.5
60.1
UniFrame
86.7
84.6
78.9
68.1
79.6
45.0
77.7
71.0
57.6
62.9
Ours
89.4
88.3
81.9
73.9
84.1
76.4
76.9
68.7
61.0
70.8
Table 1: Hand 2D keypoint detection. Our method has the best average PCK 0000 both datasets and the most significant improvements on MCP and
Mothod
SURREAL- SERREAL-HHRIB3 6M
SURREAL
Head
Sid
Elb
Wrist
Hip
Knee
Ankle
All
Head
Sid
Fih
Wrist
Knoe
Ankle
All
Source Only
51,3
69.4
75.4
66.4
37.9
77.3
77.7
67.3
40,9
59.2
65.6
64.2
64.4
56.4
39.8
57.9
CC-SSL 100
68.5
62.3
57.8
51.7
41.0
58.5
67.4
47.8
62.5
RegDA [T]]
73.3
86.4
72.8
54.8
82.0
84.4
75.6
41.8
65.8
73.3
68.1
72.8
65.0
49.3
64.8
UniPrame [S]
63.5
78.1
89.5
81.1
52.6
85.3
87.1
79.0
54.6
68.9
77.5
74,8
75.5
67.8
60.3
70.5
87.5
89.8
90.1
78.7
69.5
85.4
84.8
83.7
66.6
86.6
77.8
75.8
83.6
71,9
65.3
76.7
Table 2:
The comparison PCK @0.05 for 2D human keypoint detection. We achieve the best average performance both datasets with the most significant improvements on the head.
used as labeled source dataset DD in hand and human pose estimation, respectively. In fine-tuning, the training of H3D [II]], MVHand [39], Human3.6M [8] and 3DPW [32]] are individually used as unlabeled target dataset DD for each pose adaptation experiment. And we evaluate on the corresponding test set.
4.3 Comparison with the State-of-the-Art
Hand Pose. Table 0 shows that for 2D keypoint detection on the hand, our method outperforms the previous best method by 5.7% and 12.6% the H3D and MVHand datasets respectively. For more in-depth analysis, we also report the individual joint results averaged across the fingers i.e. MCP, PIP, DIP, Fin Adapting RHD H3D gives consistent improvement, with increases of 3.1% to 8.5%. On RHD + MVHand, ours still achieves a significant improvement PCK on MCP, increasing from 45.0 to 76.4.
Table E (b) shows that our method is state-of-the-art on 3D hand pose estimation for both RHD H3D and RHD > MVHand. Especially notable is that we surpass DualNet [18], as use only RGB images, while they incorporate additional modalities like background masks and depth images.
Human Body Pose. Table Q shows, for SURREAL + Human3.66 6M and SURREAL + 3DPW that our method outperforms kate-of-the-ar UniFrame [|S]] by 5.9% for Human3. 66M and 8.8% for 3DPW. The biggest improvements come in the head, shoulders, and hips. In particular, we observe the hip problem arising from the gap annotation convention between SURREAL and Human3. M.M so all methods (including ours) have lower accuracy on the hips. However, we outperform others by 32. 1 on Human3.6., indicating our geometric reconstruction can help the model less affected by the annotation gap.
4.4 Analysis
Effectiveness of Components. We made two core contributions: geometric reconstruction and integrated pseudo-labelling. We here provide the gain of each component in Table E (a). LUiiPrame means we use the pseudo-labelling strategy in [[S]]. The comparison with LniiFrame reveals the effectiveness of integrated eseudo-labelling with 1.2 PCK improvement. Together with geometric reconstruction, we can further boost the increase to 2.7. Cacamap means that we remove the warping estimation module and purely use heatmaps both images to reconstruct idr The results show that using explicit guidance in the generation module could help effectively link reconstruction back to the underlying poses.
'Hand joints moving away from the palm: MetaCarpoPhalangeal (MCP), PIP (Proximal InterPhalangeal), DIP (Distal InterPhalnngeal) and Fin (Fingertips).
8

---

Human3.60 .6M
H3D
before
after
GT
before
after
GT
Figure 2: Comparisons before and after applying geometric reconstruction. Our method can correct the
predictions that are on the background (left) and alleviate the annotation gap (right).
Method
RIDD 113D
SURREAL-> [lumnnd 6M
67.2
69.0
80.2
79.8
$1.6
$1.2
Accameo
Epach
Typa Respiinss
Table 3: (a) Ablation study the proposed components and the reconstruction madule contributes the final performance. (b) Percentage of correct pseudo-labels in training MVHand dataset. Both strategies saturate but the proposed integrated pseudo-labelling induces more correct labels. (c) Accuracy comparison before and after applying reconstruction loss Liee for specific set of confident keypoints on Hu- man3.66 6.M training After applying Lie., able improve the aseudo-labe of those keypoints previ- ously proposed using only Eperdd and Lup.
Keypoint Correction by Geometric Reconstruction. We analyze the impact of geometric recon- struction on confident yet inaccurate keypoints. We first train the model with Csup and Lpscudo until convergence, and then save the indices of confident keypoints. Next, the reconstruction loss Cree is added to fine-tune the model. We show the accuracy of the pseudo-labels for the keypoints associ- ated with the saved indices, both before and after adding the reconstruction loss Table G (c). The reconstruction loss obviously improves the confident yet inaccurate keypoints (i.e., head, shoulder and hip), while having negligible influence on other keypoints. This is because confident yet in- accurate keypoints will cause large reconstruction errors based image evidence while others will not. Also, we observe that those confident yet inaccurate keypoints tend to be specific keypoints, and speculate it is due to the annotation gap.
The Amount of Correct Pseudo-labels. We select pseudo-labels with high confidence. However, we cannot assure that their accuracy is within PCK @0.05 for effective adaptation. We therefore tabulate the percentage of correct pseudo-labels, whose accuracy within PCK @C.00.05. in unlabeled training data aggregated by the last ten epochs. As shown in Table B (b), the percentage saturates 59% for the speudo-labelling strategy and to 72% for the proposed integrated pseudo-labelling. Since the pseudo-labels are explored on training data with augmentations, their accuracy might not align consistently with the accuracy observed on the evaluation set. evident that both our ap- proach and Uniframe eventually reach a saturation point for accurate pseudo-labels, but our method involves more usable pseudo-labels and leads to a better result.
Fairness & Stride Strategy. In our method, we need pair of images with similar background and the same object (human/hand). In contrast, existing state-of-the-art methods only require single image during training on the target domain. For fair comparison, we first consider the most common module to utilize adjacent paired input, i.e., temporal consistency by applying & loss on the predicted heatmaps || h++ ll of two consecutive frames I, I,+1. Results in Table E (a) show can only gain marginal improvement. Another consideration is whether the difference between the two poses affects the effectiveness of the reconstruction. We attempted to amplify the pose difference two images from small to large by adjusting the stride from [0.5] [30, 50). As shown in Table E our method performs best [5.15]. This makes sense that degenerately close

---

Stride
[0.
[0. 15]
[6. 15]
[15, 30)
[30,50
SURREAL-~ Human3.6M
81.8
83.5
83.7
83.1
82.5
Table 4: Ablation study on frame strides for the task of SURREAL - Humann366 6M evaluated by PCK We can see that all strides lead to improvement, and our method is robust to the selection of strides within certain range.
Method
RHD- H3D
RHD- MVHand
Method
RHD-MVHHad
SURREAL--Humnn.dk
Source only
27.77
21.21
RegDA
60.701 0.6)
76.4(t 0.8)
SemiHand]USSI
19.19
19.75
UniPrame[
63.4Kt 0.5)
79.3(f 0.3)
Dual
17.08
16.45
Ours
70.8
83.7
Ours
16.92
16.37
a
(b)
Table 5: (a) We add temporal consistency ensure equivalence in the input information, which only brings minor improvement. (b) The comparison EPE (mm) for 3D hand pose. By only improving 2D and applying a simple 3D pseudo-labelling strategy, we beat the state-of-the-art methods.
frames will make the network find the shortcut to self-reconstrruction and far frames will make the generation too difficult to have meaningful perceptual loss on keypoint locations.
4.5 Qualitative Results
Hand Pose. Fig. D (left) shows a common case in only using pseudo-labelling, where the fingertip is predicted on the background, although it is near the ground truth location. Applying geometric reconstruction results more accurate fingertip predictions. Besides, the wrong prediction of the pinky fingertip falling on the palm also be rectified via the perceptual loss in our reconstruction module.
Human Body Pose. Fig. A (right) shows the comparison of our method on Human3.6M sample before and after applying geometric reconstruction, where the shoulders are much lower in "be- fore" compared to "GT". One possible cause is the difference in the annotation conventions across datasets, c.g. The annotation of the shoulders in SURREAL is closer to the chest, while Human3. 6M localizes them at reasonable shoulder position (Sec Supp.). Similar differences due to the annota- tion gap are reflected in the accuracy of head and hips. as shown in Table Our method, however, refine the pose prediction by leveraging the correlation between bone map and reconstruction, which is independent of the pose prior learned in the source domain. More analysis can be seen in Supp.
5 Conclusion
Pseudo-labelling is a common strategy for domain adaptive pose estimation which selects confident predictions to supervise the network. However, the number of accurate labels under this selection will saturate with the progress of training and some of the wrong predictions will remain unchanged. We hereby introduce a novel geometric reconstruction module to relieve this problem that aims to reconstruct the driving image according to keypoint locations and geometric transformations from the base image to the driving image. The incredible improvement especially comes from keypoints that generalize extremely poorly to target domain by only pscudo-labelling, We hope this will inspire future work to explore reconstruction-bet based methods to bridge the gap between source and target domains.
This research project is supported by the Ministry of Education, Singapore, under its MOE Academic Research Fund Tier 2 (STEM RIE2025 MOE- M0E-22P22020--00155
Acknowledgments
References
[1] Jinkun Cao, Hongyang Tang, Hao-Shu Fang. Xiaoyong Shen, Cewu Lu, and Yu- Wing Tai. Cross-domain adaptation for animal pose estimation. In /CCV, 2019.
10

---

[2] Ching-Hang Chen and Deva Ramanan. 3d human pose estimation= 2d pose estimation+ matching. In CVPR, 2017.
[3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. ICMI, 2020.
[4] Wuyang Chen, Zhiding Yu, Zhangyang Wang, and Animashrce Anandkumar. Automated synthetic-to- real generalization. In ICML, 2020.
[5] Salehe Erfanian Ebadi, Saurav Dhakad, Sanjay Vishwakarma, Chunpu Wang. You-Cyuan Jhang, Maciek Chociej, Adam Crespi, Alex Thaman, and Sujoy Ganguly. Psp-hdri synthetic dataset generator for pre-training human-centric computer vision models. ICML,, 2022.
[6] Kerui Gu. Linlin Yang, Michael Bi Mi, and Angela Yao. compecomeedd integral regression for human pose estimation. TPAMI, 2023.
[7] Kerui Gu, Linlin Yang. and Angela Yao. Removing the bias of integral pose regression. ICCV, pages 11067-11076, 2021.
[8] Catalin lonescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI, 36(7):1325- 1339, 2013.
[9] Umar Iqbal, Pavlo Molchanov, Thomas Breuel Juergen Gall, and Jan Kautz. Hand pose estimation via latent 2.5d heatmap regression. In ECCV, 2018.
[10] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of object land- marks through conditional image generation. In NeurlPS, volume 31, 2018.
[11] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Self-supervised learning of interpretable keypoints from unlabelled videos. In CVPR, pages 8787-8797, 2020.
[12] Junguang Jiang. Yifei Ji, Ximei Wang. Yufeng Liu, Jianmin Wang, and Mingsheng Long, Regressive domain adaptation for unsupervised keypoint detection. In CVPR, 2021.
[13] Rui Jin, Jing Zhang, Jianyu Yang. and Dacheng Tao. Multibranch adversarial regression for domain adaptative hand pose estimation. TCSVT, 32(9):6125 6136, 2022.
[14] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In ECCV. Springer, 2016.
[15] Donghyun Kim, Kaihong Wang, Kate Saenko, Margrit Betke, and Stan Sclaroff. A unified framework for domain adaptive pose estimation. In ECCV Springer, 2022.
[16] Nikos Kolotouros, Georgios Pavlakos, Michael Black, and Kostas Daniilidis. Learning reconstruct 3d human pose and shape via model-fitting the loop. In ICCV, pages 2252 2261, 2019.
[17] Chen Li and Gim Hee Lee. From synthetic real: Unsupervised domain adaptation for animal pose estimation. In CVPR, 2021.
[18] Qiuxia Lin, Linlin Yang, and Angela Yao. Cross-domain 3d hand pose estimation with dual modalities. In CVPR, 2023.
[19] Charlotte Loh, Rumen Dangovski, Shivchander Sudalairaj, Seungwook Han, Ligong Han, Leonid Karlin- sky, Marin Soljacie, and Akash Srivastava. On importance of calibration in semi-supervised learning. In arXiv preprint arXiv:2210.0478.. 2022.
[20] Jiteng Mu, Weichan Qiu, Gregory Hager, and Alan I, Yuille. Learning from synthetic animals. In CVPR, 2020.
[21] Hyconscob Nam, HyunJac Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR, pages 8690-8699, 2021.
[22] Pawel A Pierzchlewicz. James Cotton, Mohammad Bashiri, and Fabian Sinz. Multi-hypothesis 3d human pose estimation metrics favor miscalibrated distributions. In arXiv preprint 1rXiv:22/0.11179, 2022.
[23] Neng Qian, Jiayi Wang, Franziska Mueller, Florian Bernard Vladislav Golyanik, and Christian Theobalt. Html: A parametric hand texture model for 3d hand reconstruction and personalization. In ECCV, 2020.
11

---

[24] Dan Ruta, Saeid Motiaan, Baldo Faieta, Zhe Lin, Hailin Jin, Alex Filipkowski, Andrew Gilbert, and John Collomosse. Aladin: all layer adaptive instance normalization for fine-grained style similarity. In ICCV, 2021.
[25] Luca Schmidtke, Athanasios Vlontzos, Simon Ellershaw, Anna Lukens, Tomoki Arichi, and Bernhard Kainz. Unsupervised human pose estimation through transforming shape templates. CVPR, pages 2484-2494, 2021.
[26] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. CVPR, 2019.
[27] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In NeurIPS, 2019.
[28] Aliaksandr Siarohin. Oliver J Woodford. Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion represen- tations for articulated animation. In CVPR, pages 16663,1662, 2021.
[29] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV, 2018.
[30] Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In CVPR, 2014.
[31] Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael Black, Ivan Laptev, and Cordelia Schmid, Learning from synthetic humans. In CVPR, 2017.
[32] Timo Von Mareard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. Re- covering accurate 3d human pose in the wild using imus and a moving camera. In ECCV, 2018.
[33] Chengde Wan, Thomas Probst, Lue Van Gool, and Angela Yao. Self-supervised 3d hand pose estimation through training by fitting. In CVPR, 2019.
[34] Chengde Wan, Thomas Probst, Lue Van Gool, Angela Yao. Dual grid net: Hand mesh vertex regres- sion from single depth maps. In ECCV, pages 442-459. Springer, 2020.
[35] Wei Wang, Xavier Alameda-Pinedee Dan Xu, Pascal Fua, Elisa Ricci, and Nicu Sebe. Every smile unique: Landmark-guides diverse smile generation. In CVPR, pages 7083-7092, 2018.
[36] Linlin Yang, Shicheng Chen, and Angela Yao. Semihand: Semi-supervised hand pose estimation with consistency. In ICCV, 2021.
[37] Linlin Yang and Angela Yao. Disentangling latent hands for image synthesis and pose estimation. In CVPR, 2019.
[38] Yalan Ye, Ziwei Huang, Tongjie Pan, Jingjing Li, and Heng Tao Shen. Reducing bias to source samples for unsupervised domain adaptation. NN, 141:61-71, 2021.
[39] Ziwei Yu, Linlin Yang, Shicheng Chen, and Angela Yao. Local and global point cloud reconstruction for 3d hand pose estimation. In BMVC, 2021.
[40] Xiheng Zhang, Yongkang Wong, Mohan Kankanhalli, and Weidong Geng. Unsupervised domain adap- tation for 3d human pose estimation. In ACMMM, pages 926-934, 2019.
[41] Zhengyi Zhao, Tianyao Wang, Siyu Xia, and Yangang Wang. Hand-3d 3d-studio: A new multi-view system for 3d hand reconstruction. In ICASSP, 2020.
[42] Christian Zimmermann, Max Argus, and Thomas Brox. Contrastive representation learning for hand shape estimation. In GCPR, 2021.
[43] Christian Zimmermann and Thomas Brox. Learning to estimate 3d hand pose from single rgb images. In ICCV, 2017.
12


Reviews:
[{"review_id": "J2X4XVJUb0", "reviewer": "Reviewer_QXSS", "summary": "This paper addresses the task of learning human body and hand pose estimation from labeled synthetic data and unlabeled data of the target domain. The main idea is to improve the key point estimation pre-trained on synthetic data by reconstructing a video frame using another frame with key point estimation, affine transformation, optical frame estimation, and image synthesis. This paper also proposes a novel pseudo-labeling method using 2D head-map-based method [16] and 3D pose correction [41] jointly. The experiments demonstrate that the proposed method outperforms existing syn-to-real methods on several hand and human body datasets, and the effectiveness of the proposed two components (reconstruction and pseudo-labeling).", "questions": "- Please see also the weakness section.\n- Where did the \"12%\" in the abstract/introduction come from? This number does not appear in the experiment section.\n- Is invariant feature learning (p. 3.2) an original contribution? Or are there similar methods?\n- The curve in Table 3 (b) looks inconsistent with the description in the main body. Maybe the legends are reversed?\n", "limitations": "- Please see the weakness section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper proposes a method to exploit unlabeled data by image reconstruction with predicted keypoints. Since the most existing method in this task is to use pseudo-labeling, this method is in a new direction and the novelty would be high.\n- A novel pseudo-labeling strategy using 2D confidence [16] and 3D confidence with pose correction [41]. Although the proposed method is rather simple, it outperforms the SOTA pseudo-labeling method [16].\n- State-of-the-art accuracy on multiple benchmarks when the proposed reconstruction and pseudo-labeling are used together. It is also good that the two proposed methods (reconstruction and pseudo-labeling) are not only complementary, but rather reinforce each other. It has been shown that reconstruction improves the ratio of valid pseudo-labels.\n- The proposed method significantly improves the accuracy of hard semantic categories where annotation gap exists that existing methods do not work very well.\n", "weaknesses": "- There are several reconstruction-based methods [38,11,29,17] as described, but no experimental comparison has been provided, and the proposed method may be inferior. Is there any reason why these methods cannot be compared with the proposed method?\n- It is clearly demonstrated that the accuracy of the new technique is superior. However, the underlying mechanism seems less clear from the manuscript. In particular, why is the annotation gap between synthetic and real datasets overcome? What (and how) information or inductive bias is obtained in the reconstruction strategy?\n- Unlike existing methods, the proposed method requires a video dataset of the target domain. (However, I think this is a minor issue because this kind of data is easy to obtain).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Experimental comparisons to reconstruction-based methods [38,11,29,17].** \n\nA1: Thanks for your suggestions. We added experiments on [29] and opted to exclude others due to the inapplicable rendering [38] or additional requirements like 2D labels [11, 17].\n\nSpecifically, the rendering in [38] is designed for depth maps and is not applicable to our RGB task. Rendering RGB images is still challenging. For [11] and [17],  they both require additional information, such as unpaired or paired ground truth 2D, which gives them an unfair advantage. Instead, we only model the geometric transformations without other explicit supervision on the target domain.\n\nWe compared against [29] and will add the result to the revision. We find ours significantly outperforms [29] because [29] relies on a pre-defined template (front pose) to predict pose transformation, leading to failure on behind poses, or folded body poses, as shown in the bottom half of rebuttal pdf.\n\n\n**Q2: Underlying mechanism of improved accuracy. Why is the annotation gap between synthetic and real datasets overcome? What (and how) information or inductive bias is obtained in the reconstruction strategy?**\n\nA2: One gap comes from different annotation conventions -- for example, the hip keypoints in SURREAL are closer to the pelvis than Human3.6M, even though the two datasets use the same skeleton topology (refer to Fig. B in Supplementary). Methods that use keypoints directly for supervision, e.g., pseudo-labels, will therefore be more sensitive to this difference.  In our case, our supervisory signal comes from both the reconstructed image and the pseudo-labelling. Specifically, the proposed reconstruction refines the keypoints according to the shared skeleton topology and tends to satisfy both datasets. Therefore, it’s less sensitive to annotation differences and will help improve the keypoint predictions like hips (refer to the results in Table 2).\n\n**Q3: Unlike existing methods, the proposed method requires a video dataset of the target domain. (However, I think this is a minor issue because this kind of data is easy to obtain).**\n\nA3: In principle, our method requires only two images with the same target instance (human or hand).  However, diverse backgrounds and very different poses may present additional challenges so we advocate the use of video.  \n\n**Q4: Where did the \"12%\" in the abstract/introduction come from? This number does not appear in the experiment section.**\n\nA4: Thanks for pointing out this oversight. We tabulate 12% by averaging the most significant improvements (fingertips on hands, shoulders on humans):\n\n* 8.5% (73.9 vs. 68.1) on the RHD->H3D\n* 5.9% (61.0 vs. 57.6) on the RHD->MVHand\n* 15.0% (89.8 vs. 78.1) on the SURREAL->Human3.6M\n* 18.7% (75.7 vs. 63.8) on the SURREAL->3DPW.\n\nWe will add this tabulation to Section 4.4.\n\n**Q5: Is invariant feature learning (p. 3.2) an original contribution? Or are there similar methods?**\n\nA5: No, it is not an original contribution and we do not claim it as such. Invariant feature learning is standard practice for domain adaptation of classification [1], semantic segmentation [2]  and depth estimation [3]. Current works [4,5,6] in domain adaptation of pose estimation only utilize style transfer as data augmentations, we are the first to apply it to perform invariant feature learning.\n\n[1] Feature Stylization and Domain-aware Contrastive Learning for Domain Generalization, ACM MM’21\n\n[2] Learning Texture Invariant Representation for Domain Adaptation of Semantic Segmentation, CVPR’20\n\n[3] Real-Time Monocular Depth Estimation using Synthetic Data with Domain Adaptation via Image Style Transfer, CVPR’18\n\n[4] Enhancing Human Pose Estimation in Ancient Vase Paintings via Perceptually-grounded Style Transfer Learning, ACM Journal on Computing and Cultural Heritage’22\n\n[5] Sim2Real Instance-Level Style Transfer for 6D Pose Estimation, IROS’22\n\n[6] A Unified Framework for Domain Adaptive Pose Estimation, ECCV’22\n\n**Q6: The curve in Table 3 (b) looks inconsistent with the description in the main body. Maybe the legends are reversed?**\n\nA6: Thank you for pointing this out. Yes, they are reversed. We will correct them in the revision.\n\n\n\n\n\n "}, {"review_id": "Lli7va5q42", "reviewer": "Reviewer_kmmR", "summary": "This paper introduces a self-supervised synthetic-to-real domain adaptive pose estimation method. The approach utilizes a reconstruction-based strategy. It involves geometric transformations with an image generation network to reconstruct a driving image from a base image and uses pixel-wise reconstruction losses to ensure accurate keypoint localization. The method also incorporates an integrated pseudo-labelling technique that combines confidence masks from 2D heatmap and 3D predictions, resulting in more usable pseudo-labels. The performance of the approach is assessed using evaluation metrics including Percentage of Correct Keypoint (PCK) and end-point error (EPE). The study includes ablation studies as well as experiments in 2D hand/human keypoint detection and 3D hand pose. The experimental results demonstrate improvements over the selected state-of-the-art methods in terms of the main evaluation metrics.", "questions": "How many frames does the 3DPW dataset consist of?", "limitations": "The potential societal impact is not discussed.\nIt would be beneficial to include more visualized successful cases in the comparisons with state-of-the-art methods. For instance, in Figure A for Human3.6M, it is observed that the proposed method, contrary to the expectation based on Table 2, suffers from the same hip problem as other state-of-the-art methods.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper ist well-structured.\n- The incorporation of meaningful ablation studies strengthens the scientific validity.", "weaknesses": "- StyleNet was not adequately explained.\n\nTable 1:\n- It was not clarified that MCP, PIP, and DIP refer to the joints of the hand.\n- Although PIP and DIP in the MVHand dataset for the proposed method do not perform better than UniFrame, Fin in the proposed method outperforms the Fin in UniFrame. The reason for this improvement is unclear.\n\nTable 2: \n-The discussion regarding the results on the 3DPW dataset in comparison with the Human3.6M dataset was not sufficient.\n\n- It would be highly beneficial to provide more visualizations of the final results, at least in the supplementary material, particularly for the 3DPW dataset.\n\nThe explanation regarding the comparison of EPE in Table 5 (b) was insufficient.\n\nFigure A in the supplementary:\nFor Human3.6M, the head in UniFrame was detected much more accurately than in the proposed method (ours), contrary to the expectation based on Table 2, where the proposed method was expected to have better head detection.\n\nIt would enhance clarity if the paper provided a brief explanation of the state-of-the-art methods, particularly UniFrame. This would allow for a better understanding of the unique contributions of the paper in comparison to its competitors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate you highlighting the parts of our paper which could benefit from more explanation and discussion.  We will definitely include the suggested points in the revision and hope it will help you view our work more positively.   \n\n**Q1: Explanation of StyleNet.**\n\nA1: Thanks for pointing out this oversight. For style transfer, we follow UniFrame [2] and use Aladin [1]. The style transfer network is an encoder-decoder that adjusts the first and second-order moments in the adaptive instance normalization layers to transfer the target style to a source image while maintaining source semantics. We will add details to Section 3.2.\n\n[1] Aladin: all layer adaptive instance normalization for fine-grained style similarity, ICCV’21\n[2] A Unified Framework for Domain Adaptive Pose Estimation, ECCV’22\n\n**Q2: MCP, PIP, and DIP are joints of the hand.**\n\nA2: Thanks for pointing out this oversight. MCP, PIP, DIP and Fin refer to the MetaCarpoPhalangeal joints (MCP), Proximal InterPhalangeal joints (PIP), Distal InterPhalangeal joints (DIP), and hand fingertips (Fin). We will clarify it in the final version.\n\n**Q3: Why proposed method outperforms UniFrame for fingertips but not PIP and DIP joints.**\n\nA3: Thanks for pointing out this curious phenomenon. Our approach relies on the reconstructed image to serve as a supervisory cue. The reconstruction is less sensitive to intermediate joints like DIP and PIP as these can often compensate for each other. The reconstruction is more sensitive to endpoint locations, i.e. the fingertips. If the predicted fingertip is located on the background, then upstream segments will feature background pixels in the reconstruction. If the predicted fingertip is located on the hand but is incorrectly placed, certain foreground pixels will be unaccounted for. Both cases will be penalized in the reconstruction loss.\n\nWe will add this explanation to the revision.\n\n**Q4: More discussion on 3DPW results vs. Human3.6M. It would be highly beneficial to provide more visualizations of the final results, at least in the supplementary material, particularly for the 3DPW dataset.**\n\nA4: Thank you for pointing this out. We added visualizations from 3DPW to the rebuttal pdf and will include more samples in the revised supplementary. Since 3DPW is collected in the wild, it is more challenging and features more complexity in pose, appearance and background compared to Human3.6M. Our paper is the first to verify that adaptation is feasible on 3DPW (previous approaches only target Human3.6M) and highlights the strength of our approach.\n\n**Q5: The explanation regarding the comparison of EPE in Table 5 (b) was insufficient.**\n\nA5: Thanks for your suggestion. We will add more explanations in the revision.\n\nTable 5(b) shows that all methods can reduce the domain gap to some extent. SemiHand, using pseudo-labelling and consistency learning, has improvements of 8.58 mm and 1.46 mm on the H3D and MVHand, respectively. DualNet builds on the framework of Semihand and adds depth inputs for additional improvements of 2.11 mm and 3.3 mm respectively. \n\nOur work, using only RGB inputs, emphasizes the benefits of reconstruction as a supervisory signal. Even without consistency or depth inputs, we can outperform SemiHand and DualNet with improvements of 10.85 mm and 4.84mm. This result highlights that 3D pose estimation can benefit from 2D reconstruction. Results of Fig. 3(a) show qualitatively that reconstruction complements pseudo-labelling, as it refines end-point errors such as the fingertips.\n\n**Q6: Figure A in the supplementary: For Human3.6M, the head in UniFrame was detected much more accurately than in the proposed method (ours), contrary to the expectation based on Table 2, where the proposed method was expected to have better head detection.**\n\nA6: Thanks for pointing out this inconsistency. Compared with Uniframe, our performance on the head is indeed much more competitive (63.5 vs. 86.5). We presented more representative samples in the rebuttal pdf, and we will provide more visualizations in the Supplementary.\n\n**Q7: It would enhance clarity if the paper provided a brief explanation of the state-of-the-art methods, particularly UniFrame. This would allow for a better understanding of the unique contributions of the paper in comparison to its competitors.**\n\nA7: Thanks for your suggestion. The state-of-the-art methods, including UniFrame, CCSSL, RegDA, focus on the pose space to directly refine the predicted keypoints. Specifically, for UniFrame, it uses style transfer to augment source data and align the predictions from the student model to the pseudo-labels of the teacher model. \n\nIn contrast, we focus on improving the keypoints indirectly through image reconstruction. Such an approach is less sensitive to annotation gaps and is especially effective for accurate endpoints.\n\n**Q8: How many frames does the 3DPW dataset consist of?**\n\nA8: 3DPW contains 23543 frames for training and 35515 frames for testing.\n\n**Q9: The potential societal impact is not discussed.**\n\nA9: We do not foresee direct negative societal impacts from our work, though accurate human/hand keypoint estimation may facilitate downstream tasks such as creating deep fakes.  We will urge readers to make ethical use of our work in the revision.\n\n**Q10: Include more visualized successful cases in the comparisons with state-of-the-art methods. For instance, in Figure A for Human3.6M, it is observed that the proposed method, contrary to the expectation based on Table 2, suffers from the same hip problem as other state-of-the-art methods.**\n\nA10: Thanks for your suggestion. We have added more successful cases in the rebuttal pdf and will include more in the revised supplementary material. The hip problem arises from differences in annotation convention so all methods (including ours) have lower accuracy on the hips. However, we outperform others by 15.4% on Human3.6M. \n\n\n\n"}, {"review_id": "ogmjOdxzSa", "reviewer": "Reviewer_RVKB", "summary": "This paper proposed a domain adaptation method from synthetic to real for 3D hand/human pose estimation. Specifically, two self-supervised losses are designed, one is a geometric reconstruction loss which measures a multi-scale perceptual loss between an image in target domain and a generated one with pairwise geometric constraint, another is an integrated pseudo-labeling loss which measures integrated deviations on both 2D heatmaps and 3D poses. Experiment has been done on a set of benchmark dataset and better performance has been achieved compared with existing methods.", "questions": "Please ref to the weakness part", "limitations": "I donot see any potential negative societal impact", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed affine transformation module for optical flow generation and image reconstruction is interesting. The paper is easy to follow and the experiment result is promising. The ablation study clearly demonstrates the effectiveness of the paper's technical contributions.\n", "weaknesses": "1. One question is on the occlusion map described in Sec 3.3.2, it is not clear whether this occlusion map represent the occlusion between hand/human and background? or the self-occlusions between fingers/palm? does the occlusion map predictor uses the same hourglass network backbone as the 'Pixel-wise Flow Predictor'? whether the proposed geometric reconstruction module can handle self-occlusion?\n2. Some details of comparison experiment maybe needed, i.e. whether the methods listed in table 1&2 use the same backbone network? is the computational cost and memory cost comparable, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Does the occlusion map in Sec 3.3.2, represent the occlusion between hand/human and background? or the self-occlusions between fingers/palm?**\n\nA1: The occlusion does not represent either. Instead, it marks out regions that should be inpainted by the base image, and is applied as a mask to diminish the impact of the existing features in these regions. As as you can see the occlusion map in Fig. 2, the base image needs to inpaint the fingers to background pixels to match the driving image, so they are black. However, the palm does not need to inpaint, so they remain almost white.\n\n**Q2: Does the occlusion map predictor uses the same hourglass network backbone as the 'Pixel-wise Flow Predictor'?**\n\nA2: Yes, it uses the same backbone, though we add one more convolution head. Please refer to the architecture details of hourglass in R4 Q1.\n\n**Q3: Does the proposed geometric reconstruction module handle self-occlusion?**\n\nA3: Unfortunately, our reconstruction module cannot fully address self-occlusion.  Our method is based only on a 2D reconstruction, and it is difficult to reason out occlusions without multi-view or 3D information. However, we are able to outperform pseudo-labelling strategies in many challenging occlusion cases. One example is given in Supplementary Fig. A, where pseudo-labelling mixes up the left and right-ankles, while we can prevent such inversions since we explicitly link child to parent nodes in the reconstruction.\n\nThanks for all these interesting and insightful questions. We have added clarifications from our answers to the revision.\n\n**Q4: Some details of comparison experiment maybe needed, i.e. whether the methods listed in table 1&2 use the same backbone network? is the computational cost and memory cost comparable, etc.**\n\nA4: Thank you for pointing this out. Yes, the methods listed in table 1&2 all use the same backbone network, i.e. ResNet101. The computation cost and memory cost during inference will be the same. We will clarify them in the revision.\n\n"}, {"review_id": "AZ9TIHLsE5", "reviewer": "Reviewer_mnmU", "summary": "The authors proposed a geometric reconstruction-based domain adaptation method for synthetic to real transfer of hand/human pose estimation models. Based on the geometric transfer and image generation, the reconstruction loss can help the model generalize to the target domain. The proposed method can benefit from a wider range of unlabeled data compared to pseudo-label methods. The proposed method achieved improved performance on standard benchmarks.", "questions": "1. What is the architecture of the generator $\\mathcal{G}$? Is the generator pretrained?\n2. Is the legend text in Table 3 (b) reversed?\n3. I thought in Table 3 (b) the performance should start from the pre-trained only performance in the training set (about 67.2 from Table 3 (a)) and improve. Why is the percentage of correct labels lower than 30% at the beginning of the adaptation?\n4. What’s the limitation of the proposed reconstruction-based method? Does the adaptation collapse?\n5. It seems that the authors used a different style transfer network than in UniFrame. Does the reconstruction-based adaptation work without the style transfer network or a worse initialization with a low percentage of the good pose estimations at the beginning?", "limitations": "1. See weakness 1: the idea of reconstruction is not entirely new.\n2. See question 1,3,4: the clarity of the writing can be improved.\n3. See weakness 2 and question 5: would be good to analyze the limitations of the model in other settings.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Only part of the unlabeled data can be used for pseudo labeling while most images are suitable for the proposed reconstruction scheme.\n2. The geometry-based reconstruction seems novel for hand/pose estimation, making the image generation easier and simplifying the training.\n3. Quantitative results showed that the proposed method is quite effective for both hand and human pose estimation.", "weaknesses": "1. Reconstruction from pseudo-labels for domain adaptation is not entirely new. However, the model in this work used optical flow and occlusion maps for inpainting-based reconstruction, which seems to be quite effective without a complex reconstruction network. However, some implementation of the proposed model is not explained very clearly.\n2. It’s unclear how much the perceptual reconstruction loss could help when predictions from the pretrained model are not very good. All results on reconstructions seem to be combined with pseudo labeling. For a sample with very inaccurate pose prediction, would the gradient from the perceptual reconstruction loss help? Or is a good pretrained model or a medium pretrained model + pseudo labeling necessary for the reconstruction module to work? Also see question 5.\n3. Is the perceptual reconstruction only used in domain adaptation or also used to refine the predicted pose? For the results visualized in Figure 3, are they “before” or “after” the reconstruction fine-tuning of the model or something else?\n4. Label cycle-consistency and refining predictions with the reconstruction were explored in related works and could also be of interest here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Reconstruction from pseudo-labels for domain adaptation is not entirely new. However, the model in this work used optical flow and occlusion maps for inpainting-based reconstruction, which seems to be quite effective without a complex reconstruction network. However, some implementation of the proposed model is not explained very clearly.**\n\nA1: Thanks for pointing this out. The pixel-wise flow predictor $\\mathcal{P}$ is based on an hourglass with five conv3×3 - bn - relu - pool2×2 in the encoders and five upsample2×2 - conv3×3 - bn - relu blocks in the decoders. After the hourglass, there are two convs to respectively generate masks and occlusion map. The masks are used to generate dense optical flow $\\rm{O}^{\\text{drv}\\leftarrow\\text{b}}$ (See Eq.8). The design of generation module can be seen in A5. The pose predictor $f_{\\text{kp}}$ consists of a feature extractor $ f_{\\text{enc}}$ using ResNet-101 and a decoder $ f_{\\text{dec}}$ with three convtranspose2d - bn - relu blocks.\n\nWe will provide more implementation details in the Supplementary, and release source code.\n\n**Q2: It’s unclear how much the perceptual reconstruction loss could help when predictions from the pretrained model are not very good. All results on reconstructions seem to be combined with pseudo labeling. For a sample with very inaccurate pose prediction, would the gradient from the perceptual reconstruction loss help? Or is a good pretrained model or a medium pretrained model + pseudo labeling necessary for the reconstruction module to work? Also see Q11.**\n\nA2: Thanks for this interesting question. The reconstruction module can still bring medium-trained models to comparable results against the state-of-the-art methods without pseudo-labelling.\n\nExperimentally, we adapt SURREAL to Human 3.6M and apply only the perceptual reconstruction loss to a checkpoint model of 52.1, which has a really bad performance. After running 50 epochs, PCK reaches 74.8, which is a competitive result against state-of-the-art methods. \n\n**Q3: Is the perceptual reconstruction only used in domain adaptation or also used to refine the predicted pose? For the results visualized in Figure 3, are they “before” or “after” the reconstruction fine-tuning of the model or something else?**\n\nA3: The perceptual reconstruction is only used in domain adaptation. Yes, Figure 3 shows the prediction “before” or “after” the geometric reconstruction fine-tuning.\n\n**Q4: Label cycle-consistency and refining predictions with the reconstruction were explored in related works and could also be of interest here.**\n\nA4: Thank you for pointing this out. We will add more literature about label cycle-consistency [1,2] and refining predictions [3,4] in the related work.\n\n[1] DualPoseNet: Category-level 6D Object Pose and Size Estimation Using Dual Pose Network with Refined Learning of Pose Consistency, ICCV’21\n\n[2] Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild, ICLR’23\n\n[3] Learning to Refine Human Pose Estimation, CVPR’18\n\n[4] Pose-Oriented Transformer with Uncertainty-Guided Refniement for 2D-to-3D Human Pose Estimation, AAAI’23\n\n\n**Q5: What is the architecture of the generator $\\mathcal{G}$? Is the generator pretrained?**\n\nA5: The generator is based on U-Net with five conv3×3 - bn - relu - avg − pool2×2 blocks in the encoders and five upsample2×2 - conv3×3 - bn - relu blocks in the decoders. In the generator network, we use the Johnson architecture [1] with two down-sampling blocks, six residual-blocks and two up-sampling blocks. The design follows [32]. The inputs are the base image, dense optical flow, and occlusion map. It downsampled 4x and upsampled 4x to get the output, i.e. the reconstructed image. \n\nYes, the generator is pretrained with predicted keypoints before applying the geometric reconstruction module. \n\nThank you for pointing out and we will include the above information in the final version. \n\n\n**Q6: Is the legend text in Table 3 (b) reversed?**\n\nA6: Thank you for pointing this out. Yes, they are reversed. We will correct them in the revision.\n\n**Q7: I thought in Table 3 (b) the performance should start from the pre-trained only performance in the training set (about 67.2 from Table 3 (a)) and improve. Why is the percentage of correct labels lower than 30% at the beginning of the adaptation?**\n\nA7: Sorry for the confusion. The plot given in Table 3(b) is conducted on MVHand while Table 3(a) shows results on H3D and Human3.6M. We will label this clearly in the revision.\n\nNote, for the plot in 3(b), the percentage of correct pseudo-labels at the beginning only represents the performance of partly training samples as the data loader cannot iterate the whole training samples in one epoch, which is keeping in line with the training scheme of Uniframe. \n\n\n**Q8: What’s the limitation of the proposed reconstruction-based method?**\n\nA8: One limitation is that our method cannot handle reconstruction ambiguity in intermediate keypoints, e.g. consider the case where the entire arm is straight, the location of the elbow is challenging for the reconstruction model. \n\n**Q9: Does the adaptation collapse?**\n\nA9: No, the adaptation will not collapse.\n\n**Q10: It seems that the authors used a different style transfer network than in UniFrame.**\n\nA10: No, we use Aladin [1] as the style transfer network, which is the same as UniFrame.\n\n[1] Aladin: all layer adaptive instance normalization for fine-grained style similarity, ICCV’21 \n\n**Q11: Does the reconstruction-based adaptation work without the style transfer network or a worse initialization with a low percentage of the good pose estimations at the beginning?**\n\nA11: Please see A2.\n\n\n\n\n\n\n\n\n\n\n\n\n"}]

Author rebuttal:
Dear reviewers and ACs,

Thank you for your careful and thoughtful reviews. R1 and R4 commented that the proposed geometric reconstruction for domain adaptive pose estimation is new and novel. All reviewers appreciated the empirical verifications to validate the effectiveness of the proposed methods.

---

We highlight the following to help understand our paper:

- As recognized by R1, our key novelty is the incorporation of image reconstruction for domain adaptive pose estimation -- existing works rely only on pseudo-labeling.

- The reconstruction complements pseudo-labelling by (1) emphasizing accurate endpoint placements (see R2 Q3 for details) (2) being less sensitive to different annotation conventions between source and target domains, e.g. shoulders for human (see R1 Q2 for details).

---

Since the concerns voiced by the reviewers are mainly non-overlapping, we offer detailed responses individually to each review. Also, we attach a pdf that contains the visualizations for the reviewers to refer to.

Meta-review: The paper seek to address the challenge of sim-to-real transfer in human body pose estimation. The initial reviews were mixed. The reviewers raised concerns on the clarity of technical details, as well as the positioning w.r.t. to prior methods. The authors provided a detailed response. The reviewers were satisfied with the response, resulting in overall positive final ratings. 

After going through the paper, the review, and the response, the AC values the key idea of the paper (considering geometric reconstruction for domain adaptation in pose estimation), and recommends the acceptance of the paper. The AC also recognizes that the technical components in the current manuscript were not particularly well described, and thus urges the authors to further improve the clarity of the presentation.
Decision: Accept (poster)