SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qiuxia Lin $^{1}$  Kerui Gu $^{1}$  Linlin Yang $^{2,3}$  Angela Yao $^{1}$ \
 $^{1}$ Department of Computer Science, National University of Singapore\
 $^{2}$ State Key Laboratory of Media Convergence and Communication, CUC\
 $^{3}$ School of Information and Communication Engineering, CUC\
{qiuxia, keruigu, ayao}@comp.nus.edu.sg
Pose estimation is remarkably successful under supervised learning, but obtaining annotations, especially for new deployments, is costly and time-consuming. This work tackles adapting models trained on synthetic data to real-world target domains with only unlabelled data. A common approach is model fine-tuning with pseudo-labels from the target domain; yet many pseudo-labelling strategies cannot provide sufficient high-quality pose labels. This work proposes a reconstruction-based strategy as a complement to pseudo-labelling for synthetic-to-real domain adaptation. We generate the driving image by geometrically transforming a base image according to the predicted keypoints and enforce a reconstruction loss to refine the predictions. It provides a novel solution to effectively correct confident yet inaccurate keypoint locations through image reconstruction in domain adaptation. Our approach outperforms the previous state-of-the-arts by  $8\%$  for PCK on four large-scale hand and human real-world datasets. In particular, we excel on endpoints such as fingertips and head, with  $7.2\%$  and  $29.9\%$  improvements in PCK.
Training deep models for human pose estimation, be it of the full body [30, 29, 2] or hands [37, 42], requires large-scale labelled datasets. Synthetic data [43, 31] has become more favoured due to the ease of generation and the challenges of accurately annotating real-world data. Despite advancements in realistic rendering [23, 5], models trained on synthetic data do not generalize well to real-world settings. The gap arises due to variations in pose, appearance, lighting, and other low-level differences between the two data domains. This paper aims to close this gap and targets learning from labelled synthetic and unlabelled real-world data for application to real-world settings.
Domain adaptation studies the transfer of models from synthetic to real data. Recent works on domain adaptive pose estimation [17, 36, 15] learn from target pseudo-labels. However, the gains from pseudo-labelling can be limited. There is an inherent contradiction – only labels which are sufficiently accurate benefit the learning, and it is important to ensure label quality through selection [15] and or correction schemes [36, 18]. Yet the ability to obtain high-quality pseudo-labels implies that the model is sufficiently capable of estimating the poses of said samples. The cross-domain setting makes it difficult to obtain high-quality labels. The challenge is compounded by additional gaps like different subjects' anatomical sizations, different annotation standards across datasets, etc.
Instead of estimating labels for the given target data, what if we try to reconstruct the data itself? This latter strategy is often used in self-supervised pose estimation, where learning is guided by reconstruction losses [33, 11, 34]. The challenge lies in linking the reconstruction scheme back to
the underlying pose<sup>1</sup>. While it is feasible to use a differentiable renderer, there are several drawbacks, such as the complexity of the renderer [23] and the difficulty of model fitting [16].
With this motivation, we design a simple yet effective reconstruction scheme that connects keypoint prediction via bone maps. Specifically, given a base and driving image pair with different poses, we estimate the keypoints and region-wise geometric transformations between the two poses, where each region corresponds to a bone linking two joints. The transformation is then applied with the predicted mask to obtain fine-grained warping (as shown in Fig. 1). The complete reconstruction is conducted in the generation module, which translates the base image into the driving image, specifically performing feature warping and inpainting in downsampling and upsampling structures. Intuitively, the more accurate the estimated keypoints and geometric transformation in the image pair, the better the reconstruction. Differences between the reconstruction and the original driving image can serve as a supervisory signal to improve keypoint localization.
A key benefit of the reconstruction scheme is that it can leverage almost all the samples in the unlabelled target dataset and give refinement according to the reconstruction loss. Pseudo-labeled samples are often selected based on some confidence measure like heatmap response, but given that most neural networks are poorly calibrated [22, 19], the confidence is often unsatisfactorily aligned with pose accuracy. Many samples can be highly confident yet inaccurate and the predictions remain inaccurate in this way. This is a key reason why pseudo-labelling methods easily saturate (see Table 3 (b)). In contrast, our reconstruction scheme requires only the two images containing the same object (hand/human) and a rigid background. As such, we are able to improve those confident but wrong keypoints based on 2D evidence in the target domain that arises from the reconstruction loss (see Table 3 (c)). Also, for keypoints that are both confident and accurate, the reconstruction will not degrade the result since it has perfectly provided useful information for reconstruction.
The supervision through reconstruction is indirect and is a weaker source of information than pose pseudo-labels. However, given that the two approaches are complementary, they can also be used in conjunction, i.e., a limited but strong supervision from pseudo-labelling in combination with a larger and diverse source of weak supervision from the reconstruction. To that end, we additionally propose a new selection criteria for pseudo-labels that integrates 2D evidence and 3D kinematic constraints. In combination, we are able to outperform state-of-the-art domain adaptation methods by  $8\%$ , with significant improvements on difficult keypoints, like fingertips for the hand, and head for humans by  $29.9\%$ . Our contributions can be summarized as
- A novel reconstruction-based approach for domain adaptive pose estimation. We introduce a reconstruction method to make up for the limitations of the pseudo-labelling, and realize the connection between image reconstruction and keypoint prediction by building bone maps, which enables the correction of keypoints through optimizing reconstruction loss.
- An improved pseudo-labelling selection criteria that integrates both 2D evidence and 3D kinematic constraints. The new criteria improves the quality of the pseudo-labels by enlarging the set of usable pseudo-labels by  $13\%$ .
- Domain adaptation results from synthetic to real-world data for both hand and human body show that we outperform state-of-the-art by a large margin, with as much as  $29.9\%$  improvement on difficult keypoints like fingertips and head and an overall  $8\%$  improvement.
2.1 Cross-domain Pose Estimation
Cross-domain challenges in pose estimation, especially in the context of synthetic-to-real, are best known for animal pose estimation [1, 20, 17], where it is challenging to collect and label sizable training data. Similar challenges, however, also exist for the human body [40] and hand [36, 18, 13], as deep learning algorithms become more data-hungry. The standard strategy is to apply consistency constraints [20, 36] or pseudo-labelling [17, 15] to learn effective features for the target domain. Consistency constraints can be applied in equivariant augmentations [20], and cross-modality [36].
<sup>1</sup>We make a distinction between data synthesis and reconstruction. The former generates image observations based on given poses; the latter recreates image observations without knowledge of the underlying pose.
To generate less-noisy pseudo-labels and guide the network fine-tuning, [15] utilized the normalized output from a teacher network and [1] proposed a progressive self-paced pseudo-label updating strategy to make the guidance more stable. Both works denoise pseudo-labels based on the heatmap activations while [36, 18] reduced noise with a pose correction procedure. Either strategy alone can lead to the neglect of additional correlation of depth or the accumulation of prediction errors. We therefore leverage both strategies simultaneously and embrace their union of pseudo-labels indicators when their consensus is high to enrich guidance on the target domain.
2.2 Conditional Image Reconstruction
Reconstructing images conditioned on paired images is an effective technique to concentrate on specific attributes (e.g., keypoints) while ignoring irrelevant ones in pose estimation, where the focus can be summarized on conditions design [10, 11, 25]. Jakab et al. [10] proposed to disentangle keypoint heatmaps from appearance. The heatmaps, as conditions to generate images, can be optimized by the reconstruction loss. A follow-up work [11] utilized a skeleton image to facilitate the generation process. Furthermore, recent work [25] regressed affine parameters to deform a pre-defined shape template as conditions. However, they all extract appearance and keypoints from different images separately and overlook the geometric relationship across images, which can provide explicit guidance on image reconstruction.
Using geometric relationships across images to perform reconstruction has been seen in image animation [27, 26, 28]. They predict motion-specific keypoints [27, 26] or semantic regions [28], and obtain transformation parameters to deform one image to resemble another image. Those motion-specific keypoints or region-based methods are generalizable in predicting transformation parameters and modeling complex movements. But they target achieving the mean and deviation of each region, which is unrelated to our purpose of joint locations.
To emphasize the geometric relationship (i.e., the transformations between the two images) to perform reconstruction for refining joint locations, we connect semantic regions and joint locations via bone maps, which enables the correction of poses through optimizing the reconstruction loss. Naturally, the reconstruction becomes a complement to pseudo-labelling, and both of them contribute to domain adaptive pose estimation.
Consider labelled data  $\mathcal{D}_s = \{(\mathbf{I}_{si},\pmb {y}_i)\}_{i = 1}^{N_s}$  from a source domain  $s$  and unlabelled data  $\mathcal{D}_t = \{(\mathbf{I}_{ti})\}_{i = 1}^{N_t}$  from target domain  $t$ , where  $\mathbf{I}$  denotes an RGB image,  $\pmb{y}$  the pose label, and  $N_{s}$  and  $N_{t}$  are the number of samples in the source and target datasets. The objective is to learn a pose estimation system  $f_{\mathrm{kp}} = (f_{\mathrm{enc}}\circ f_{\mathrm{dec}})$ , trained on  $\{\mathcal{D}_s,\mathcal{D}_t\}$  and apply it to hold-out test data on the target domain  $t$  for estimating 2D or 3D keypoints.
We follow other domain adaptive pose estimation methods, and pre-train on the labelled source domain (synthetic) and then fine-tune on the unlabelled target domain (real-world). In pre-training, (Sec. 3.2), we use the labelled synthetic data  $\mathcal{D}_s$  to learn an initial pose estimation network. For fine-tuning on unlabelled real-world data  $\mathcal{D}_t$  (Sec. 3.3), we introduce a pseudo-labelling strategy to correct 2D pseudo-labels via 3D constraints in Sec. 3.3.1. Additionally, we propose a geometric reconstruction to correct keypoint locations from image signals in Sec.3.3.2. The overall pipeline for fine-tuning is shown in Fig. 1.
3.2 Pre-training on the Source Domain
Keypoint Predictor The keypoint predictor  $f_{\mathrm{kp}} = (f_{\mathrm{enc}} \circ f_{\mathrm{dec}})$  is first pre-trained on the synthetic source images  $\mathbf{I}_s$  to estimate a heatmap  $\hat{h}_p$  and a relative depth  $\hat{d}$ . As the synthetic images have ground truth labels,  $f_{\mathrm{kp}}$  can be learned in a supervised manner with ground-truths  $(h_p^{\mathrm{gt}}, d^{\mathrm{gt}})$ :
$$
\mathcal {L} _ {\sup } = \frac {1}{N _ {s}} \sum_ {\mathcal {D} _ {s}} \left(\| \hat {h} _ {\boldsymbol {p}} - h _ {\boldsymbol {p}} ^ {\mathrm {g t}} \| _ {2} + \lambda_ {d} \| \hat {d} - d ^ {\mathrm {g t}} \| _ {2}\right), \tag {1}
$$
(a) Fine-tuning on Target Domain
(b) Warping Estimation Module
Figure 1: Illustration of pipeline and modules of fine-tuning. (a) Fine-tuning on target domain. Given the base and driving images sharing the same background but different poses, we forward them into keypoint predictor  $f_{\mathrm{kp}} = (f_{\mathrm{enc}} \circ f_{\mathrm{dec}})$  to produce keypoint heatmaps  $\hat{h}_p$  and relative depth predictions  $\hat{d}$ . Afterward, our integrated pseudo-labelling  $\mathcal{P}$  (See Sec. 3.3.1) is introduced to generate pseudo-labels, while the warping estimation module  $\mathcal{W}$  and the generation module  $\mathcal{G}$  (See Sec. 3.3.2) are used to reconstruct driving images from base images. (b) Warping estimation module. Bone maps derived from keypoints are used to calculate coarse transformation parameters via SVD and Expand. With the transformation and base images, a hourglass is used to generate inpainting map  $\mathbf{C}^{\mathrm{drv} \leftarrow \mathrm{b}}$  and region-wise displacement field  $\mathbf{D}^{\mathrm{drv} \leftarrow \mathrm{b}}$  that will work on upsampled feature maps in the generator  $\mathcal{G}$ .
where  $\lambda_{d}$  is a hyper-parameter that trades off depth loss and heatmap loss. The ground-truth heatmap  $h_p^{\mathrm{gt}}$  is constructed as a 2D Gaussian centered at the ground-truth coordinate  $(u^{\mathrm{gt}}, v^{\mathrm{gt}})$  and a standard deviation of 2. The ground-truth  $d^{\mathrm{gt}}$  is the depth relative to the root keypoint. We follow the same practice in [9] to regress 2.5D and lift it to 3D pose. To elaborate, we employ soft-argmax operation on  $\hat{h}_p$  and then perform Hadamard product with depth value maps to derive the relative depth  $\hat{d}$ . The expected outcome of this operation yields the 2D coordinates, denoted as  $(\hat{u}, \hat{v})$ .
Invariant Feature Learning Models trained only on synthetic data  $\mathcal{D}_s$  are unlikely to perform well on target data  $\mathcal{D}_t$  (see Source only in Table 1). One reason is the domain gap between source and target data; typically, synthetic data used for pose estimation is not fully photorealistic, so the model does not generalize well to real-world data [4, 31]. To prevent  $f_{\mathrm{kp}}$  from overfitting to source-specific representations, which is common [38, 21], we augment the source data to resemble target data and align the original and augmented source features using contrastive learning [3]. Specifically, we train a StyleNet from [24], which transfers the style of the target domain to the source, while preserving the source content, i.e., the pose. The keypoint predictor  $f_{\mathrm{kp}}$  is then trained with the supervised keypoint loss and an alignment on features with the following loss:
$$
\mathcal {L} _ {\text {a l i g n}} = - \sum_ {i = 1} ^ {B} \log \frac {e ^ {(\langle \boldsymbol {z} _ {i} , \boldsymbol {z} _ {i} ^ {+} \rangle / \tau)}}{\sum_ {k = 1} ^ {B} \mathbb {1} _ {[ k \neq i ]} \left(e ^ {(\langle \boldsymbol {z} _ {i} , \boldsymbol {z} _ {k} \rangle / \tau)} + e ^ {(\langle \boldsymbol {z} _ {i} , \boldsymbol {z} _ {k} ^ {+} \rangle / \tau)}\right)}, \tag {2}
$$
with batch size  $B$ , temperature hyperparameter  $\tau = 0.5$ , and  $\langle \cdot \rangle$  denotes the cosine similarity. In the above equation, the features  $(z,z^{+})$  are derived from the source and augmented source image.
[15] also applies a StyleNet to augment source samples but only supervises the network output. Our work extends the supervision to the latent features, as it can further benefit task-specific feature learning by building a consistent embedding space. It is feasible to also add target data into invariant feature learning with the style of source data, but we empirically find no obvious improvement. Therefore, we only perform alignment between source and augmented source features considering the limited memory.
3.3 Fine-tuning on the Target Domain
The fine-tuning on the target domain consists of integrated pseudo-labelling and geometric reconstruction. The integrated pseudo-labelling extracts confident pesoo-labels with guidance from both 2D heatmaps and 3D constraints. Since the label can still be noisy, we also have a geometric reconstruction strategy that generates a driving image based on the base image and pose transformation, which provides supervision simply from 2D image signals to improve those confident yet inaccurate keypoints.
3.3.1 Integrated Pseudo-labelling
A commonly used strategy to adapt source models to a target domain is pseudo-labelling. In ensuring that pseudo-labels are valid, existing works have leveraged the heatmap activations in  $\hat{h}_p$  [15], or kinematic feasibility of the pose  $\hat{\pmb{y}}$  [36]. Both strategies have drawbacks; the former ignores correlations that may exist in  $\hat{d}$ , while the latter uses a greedy strategy that may accumulate errors down kinematic chains. With these considerations in mind, we propose an integrated pseudo-labelling strategy that combines the use of the 2D heatmap and the kinematic feasibility of 3D predictions. Specifically, given a predicted heatmap  $\hat{h}_p$ , we generate a 2D confidence mask  $\mathcal{M}_{\mathrm{2D}} \in \mathbb{R}^{B \times K}$  as indicators to select keypoints with heatmap activations larger than the threshold  $\gamma_{\mathrm{2D}}$  in each batch during training. The 2D pseudo-label  $\widetilde{h}_p$  is the gaussian heatmap  $\hat{h}_p$  centered on the predicted 2D coordinate, as defined in [15]. For the 3D pseudo-labels, the prediction  $\hat{\pmb{y}} = (\hat{u}, \hat{v}, \hat{d})$  is lifted into 3D coordinates and rectified as pose  $\widetilde{\pmb{y}}$  with reasonable feasibility based on the pose correction [36]. The 3D confidence mask  $\mathcal{M}_{\mathrm{3D}} \in \mathbb{R}^{B \times K}$  indicates the qualified keypoints based on the comparison of L2 norm between the 2D projected coordinates before and after correction on  $\hat{\pmb{y}}$  to the threshold  $\gamma_{\mathrm{3D}}$ .
For the first  $T$  epochs of fine-tuning, the pseudo-labels  $\widetilde{h}_p$  and  $\widetilde{\boldsymbol{y}}$  generated by 2D evidence and 3D correction, respectively, are both not confident, so we require them to train independently with their corresponding mask. Then we integrate the confidence mask  $\mathcal{M}_{\mathrm{2D}}$  and  $\mathcal{M}_{\mathrm{3D}}$  to complement each other and enrich the training set as  $\mathcal{M} = \mathcal{M}_{\mathrm{2D}} \cup \mathcal{M}_{\mathrm{3D}}$ , which considers more pseudo-labels for adaptation to both 2D and 3D. Overall, we optimize the following loss,
$$
\mathcal {L} _ {\text {p e s u d o}} = \left\{ \begin{array}{l l} \frac {1}{| \mathcal {M} _ {2 D} |} \sum_ {\mathcal {M} _ {2 D}} \| \hat {h} _ {\boldsymbol {p}} - \widetilde {h} _ {\boldsymbol {p}} \| _ {2} + \frac {\lambda_ {y}}{| \mathcal {M} _ {3 D} |} \sum_ {\mathcal {M} _ {3 D}} \| \hat {\boldsymbol {y}} - \widetilde {\boldsymbol {y}} \| _ {2} & \text {e p o c h} <   T \\ \frac {1}{| \mathcal {M} |} \sum_ {\mathcal {M}} (\| \hat {h} _ {\boldsymbol {p}} - \widetilde {h} _ {\boldsymbol {p}} \| _ {2} + \lambda_ {y} \| \hat {\boldsymbol {y}} - \widetilde {\boldsymbol {y}} \| _ {2}) & \text {e p o c h} \geq T \end{array} , \right. \tag {3}
$$
where  $|\mathcal{M}|$  indicates the number of elements in  $\mathcal{M}$  and  $\lambda_y = 0.1$  is a coefficient that trades off pseudo-labelling between 2D heatmap loss and 3D loss. We find out that as fine-tuning proceeds, the overlap of confidence masks  $\mathcal{M}_{\mathrm{2D}}$  and  $\mathcal{M}_{\mathrm{3D}}$  will increase, at which point a single strategy can also provide valuable indicators for the other strategy.
3.3.2 Geometric Reconstruction
Although the pseudo-labelling strategy is improved by utilizing both 2D and 3D information, the amount of pseudo-labels is limited and some confident labels are still incorrect. This is because these labels are generated from the model calibrated on the source domain, we therefore propose a geometric reconstruction that solely exploits specific features in the target domain. The performance of reconstructions is weaker compared to high-level pseudo-labeling. However, it can serve as complementary supervision by including all target data and correcting confident yet inaccurate keypoints by low-level evidence.
The geometric reconstruction consists of Warping Estimation Module  $\mathcal{W}$  and Generation Module  $\mathcal{G}$ , as shown in Fig. 1(a). Given a pair of images from target domain  $t$  containing human or hand  $\{\mathbf{I}^{\mathrm{b}},\mathbf{I}^{\mathrm{drv}}\}$  sampled in a video, The aim is to reconstruct driving image  $\hat{\mathbf{I}}^{\mathrm{drv}}\in \mathbb{R}^{3\times H\times W}$  from base image  $\mathbf{I}^{\mathrm{b}}\in \mathbb{R}^{3\times H\times W}$ , where the image size is  $H\times W$ . To do so, the generation module  $\mathcal{G}$  warps base image  $\mathbf{I}^{\mathrm{b}}$  based on specifications  $\mathbf{D}^{\mathrm{drv}\leftarrow \mathrm{b}}\in \mathbb{R}^{\frac{H}{4}\times \frac{W}{4}\times 2}$  and  $\mathbf{C}^{\mathrm{drv}\leftarrow \mathrm{b}}\in \mathbb{R}^{1\times \frac{H}{4}\times \frac{W}{4}}$  estimated by the warping estimation module  $\mathcal{W}$ :
$$
\hat {\mathbf {I}} ^ {\mathrm {d r v}} = \mathcal {G} \left(\mathbf {I} ^ {\mathrm {b}}; \mathbf {D} ^ {\mathrm {d r v} \leftarrow \mathrm {b}}, \mathbf {C} ^ {\mathrm {d r v} \leftarrow \mathrm {b}}\right), \quad \text {w h e r e} \left[ \mathbf {D} ^ {\mathrm {d r v} \leftarrow \mathrm {b}}, \mathbf {C} ^ {\mathrm {d r v} \leftarrow \mathrm {b}} \right] = \mathcal {W} \left(\hat {h} _ {\boldsymbol {p}} ^ {\mathrm {b}}, \hat {h} _ {\boldsymbol {p}} ^ {\mathrm {d r v}}\right). \tag {4}
$$
In Eq. 4,  $\mathbf{D}^{\mathrm{drv}\leftarrow \mathrm{b}}$  and  $\mathbf{C}^{\mathrm{drv}\leftarrow \mathrm{b}}$  denote a region-wise displacement field and an inpainting map respectively. The displacement field  $\mathbf{D}^{\mathrm{drv}\leftarrow \mathrm{b}}$  is a dense vector field that specifies each pixel location in  $\mathbf{I}^{\mathrm{drv}}$  with its corresponding location in  $\mathbf{I}^{\mathrm{b}}$ . And  $\mathbf{C}^{\mathrm{drv}\leftarrow \mathrm{b}}$  is an attention map ranging from 0 to 1, where the  $\mathcal{G}$  should do more inpainting on pixels with lower values. We apply a Perceptual Loss to penalize the semantic differences between the reconstructed image  $\hat{\mathbf{I}}^{\mathrm{drv}}$  and its original image  $\mathbf{I}^{\mathrm{drv}}$  to refine the keypoints. The details of each component (in bold) as follows.
Warping Estimation Module  $\mathcal{W}$  Given the heatmap  $h_p$  estimated from  $f_{\mathrm{kp}}$ , we decode it into coordinates by using a soft-argmax followed by an expectation; this combination is the standard approach followed in coordinate or integral regression methods [9, 29, 7, 6]. Specifically, we obtain the  $k$ -th 2D keypoint locations  $\pmb{\mu}_k$  in all  $K$  keypoints while keeping the pretrained heatmaps:
$$
\boldsymbol {\mu} _ {k} = \sum_ {\boldsymbol {p} \in \Omega} \boldsymbol {p} \cdot \tilde {h} _ {\boldsymbol {p}}, \text {w h e r e} \tilde {h} _ {\boldsymbol {p}} = \frac {e ^ {h _ {\boldsymbol {p}}}}{\sum_ {\boldsymbol {p} ^ {\prime} \in \Omega} e ^ {h _ {\boldsymbol {p} ^ {\prime}}}}. \tag {5}
$$
$\Omega$  is the area of the heatmaps, which are generated by  $f_{\mathrm{kp}}$ .