Rethinking Self-Supervised Visual Representation Learning in Pre-training for 3D Human Pose and Shape Estimation
Authors: Hongsuk Choi
Hyeongjin Nam
Taeryung Lee
Gyeongsik Moon
Kyoung Mu Lee
Venue: ICLR 2023 poster (2023)
Keywords: pre-training
3D human pose and shape estimation
self-supervised representation learning

Abstract: Recently, a few self-supervised representation learning (SSL) methods have outperformed the ImageNet classification pre-training for vision tasks such as object detection. However, its effects on 3D human body pose and shape estimation (3DHPSE) are open to question, whose target is fixed to a unique class, the human, and has an inherent task gap with SSL. We empirically study and analyze the effects of SSL and further compare it with other pre-training alternatives for 3DHPSE. The alternatives are 2D annotation-based pre-training and synthetic data pre-training, which share the motivation of SSL that aims to reduce the labeling cost. They have been widely utilized as a source of weak-supervision or fine-tuning, but have not been remarked as a pre-training source. SSL methods underperform the conventional ImageNet classification pre-training on multiple 3DHPSE benchmarks by 7.7% on average. In contrast, despite a much less amount of pre-training data, the 2D annotation-based pre-training improves accuracy on all benchmarks and shows faster convergence during fine-tuning. Our observations challenge the naive application of the current SSL pre-training to 3DHPSE and relight the value of other data types in the pre-training aspect.

Published as a conference paper at ICLR 2023
RETHINKING SELF-SUPERVISED VISUAL REPRESEN- TATION LEARNING IN PRE-TRAINING FOR 3D HUMAN POSE AND SHAPE ESTIMATION
Hongsuk Choi 11.2, Hyeongjin Nam Taeryung Lee", Gyeongsik Moon', Kyoung Mu Lee2.4
1 Samsung Research America, New York
Dept. of ECE& & ASRI, Seoul National University, Korea
3 Meta Reality Labs Research
IPAI, Scoul National University, Korea
ABSTRACT
Recently, a few self-supervised representation learning (SSL) methods have out- performed the ImageNet classification pre-training for vision tasks such as object detection. However, its effects on 3D human body pose and shape estimation (3DHPSE) are open question, whose target is fixed to a unique class, the hu- man, and has an inherent task gap with SSL. We empirically study and analyze the effects of SSL and further compare it with other pre- pre-training alternatives for 3DH- PSE. The alternatives are 2D conotation-based pre-training and synthetic data pre- training, which share the motivation of SSL that aims reduce the labeling cost. They have been widely utilized as a source of weak-supervision or fine-tuning, but have not been remarked as a pre-training source. SSL methods underperform the conventional ImageNet classification pre-training on multiple 3DHPSE bench- marks by 7.7% on average. In contrast, despite a much less amount of pre-training pre- pre-training improves accuracy on all benchmarks
data, the 2D annotation-base and shows faster convergence during fine-tuning. Our observations challenge the naive application of the current SSL pre-training to 3DHPSE and relight the value other data types in the pre- pre-training aspect.
INTRODUCTION
Transferring the knowledge contained in one task and dataset to solve other downstream tasks (i.e., transfer learning) has proven very successful in range of computer vision tasks (Girshick et al.) 2014 Carreira Zisserman, 2017, He al. 2017). In practice, transfer learning done by pre- training a backbone (He et al. 2016) on source data to learn better visual representations for the target task. The ImageNet classification has been the de facto pre-training paradigm in computer vision, and the 3D human body pose and shape estimation (3DHPSE) literature has followed this.
Recently, self-supervised representation learning (SSL) has gained popularity the interest of re- ducing labeling costs (Chen et al.]] 2020aa Grill et al. 2020; He et al. [2200] Caron et al. (2020) Hénaff al. 202T). SSL pre-trains backbone using unlabeled arbitrary object images and fine-tunes the backbone target tasks. MoCo (He et al. 2020) and DetCon (Hénaff et al. 2021 surpassed the ImageNet classification pre-training for downstream tasks like object detection and instance segmen- tation arbitrary class objects. Motivated by them, PeCLR (Spurr et al. 2021) and HanCo (Zim-l mermann et al.] (02T) targeted a human hand and pre- pre-trained backbone on hand data without 3D labels. They showed the accuracy improvement for 3D hand pose and shape estimation from the controlled setting (Zimmermann et al., 2019), compared with random initialization (no pre-training) and the ImageNet classification pre-training While the results of PeCLR and HanCo are promis- ing for 3DHPSE, they have limited practical lessons. For example, the amounts of labeled hand data, which is fine-tuning data, are significantly smaller (~~64K) than that of the commonly used labeled body data (~480K). Also, the total training
fiiniinn&fin--tuiinng time of the different
"qqual contribution.
Most work was done when Hongsuk Choi Scoul National University.

---

Published as a conference paper at ICLR 2023
Pre-training data types
3DHPSE errors
ImageNat
magelea Ion steled magelle: steedd
juntabeled arbitrrry objectu)
MSCRCO post 20)
ImageNot antinaa otjects)
SURREAL (yythatic hurnon dars))
MSCOCO
(neil human data]
Desoi mmnuun (1)
o1g (ffea-turtngi specite
Figure 1: (Left) We pre-train a backbone (ResNet-50 (He et al. 2016)) with different data types: unlabeled arbitrary objects (Russakovsky et al. 2015), labeled arbitrary objects (Russakovsky al. 2015), synthetic 3D human data (Varol et al., 2017). and real 2D human data (Lin et al.) (014). (Right) 3DHPSE errors when initializing its backhone with differently pre-trained weights. We fine- tune PARE (Kocabas et al. (2021) on Human3. 6M (Ionescu et al. 2014) and MSCOCO (Lin et al., 2014 and evaluate on 3DPW (von Marcard et al. 2018).
approaches is not matched, which is critical to the final accuracy (He et al. 2019). Last, they require labeled data with bounding boxes of a hand.
This paper questions the effectiveness of SSL pre-training for 3DHPSE by thoroughly comparing with alternatives in multiple aspects (i.e. final accuracy, convergence speed, and cost-effectiveness)) We perform experiments by fixing the fine-tuning task 3DHPSE and changing the pre-training approach. The experiments are organized in three steps. First, we compare state-of-the-art SSL methods, pre-trained on ImageNet, with the ImageNet classification pre-training. Different object detection and instance segmentation, the SSL methods are outperformed by the classification pre- training in three 3DHPSE benchmarks with 7.7% margin on average. Interestingly. the accuracy of SSL is comparable to or even worse than the random initialization baseline. The results imply general visual representations learned by SSI. could be detrimental to 3DHPSE.
Second, we explore the reasons behind the current SSL methods' disappointing performance in depth by contriving a new pre-training approach. Modern SSL pre-training methods (Chen et al.) 2020a He et al. 2020) have two unfavorable factors on 3DHPSE; 1) they learn inconsistent repre- sentations for the same class instances as argued by (Khosla et al. 2020 which hinders learning high-level priors about specific class, and 2) SSL pre-training has instance-level learning char- acteristic (Le., a single attribute per image), which has an inherent task gap with 3DHPSE that requires understanding of the fine-level semantic information (i.e., multiple attributes per image), the human joints. We combine an SSL approach with 2D joint labels, which we call JointCon, to experimentally validate the two factors' effects. JointCon contrasts local image features of human joints instead of global image features of images.
Third, we compare SSL methods with 2D annotation-based pre-training and synthetic data pre- training on human data following PeCLR (Spurr et al. 2021] and HanCo (Zimmermann et al. 2021), and discuss cost-effectiveness data pre- pre-training are worth investigating in that they share the motivation of SSI., which is to benefit from data") with less collection cost (Rong et al., 2019 Patel et al. 2021 J. In our experiments on human data, 2D annotation-baser pre-training shows the highest accuracy and the fastest conver- gence speed among different pre-training approaches. Compared with the classification baseline, its final accuracy increased by 3.1% on 3DPW (von Marcard al. 2018) and 1.9% Hu- man3.6M (Ionescu et al.) 2014).. 3DPW, the convergence speed is approximately 2 faster. In the semi- semi-supervised setting, the accuracy improvement increases to 9.9% on 3DPW and 7.1% on Human3.6M, We assume rich pose and appearance information learned from the 2D pose data is the key to these improvements as expected. Synthetic data pre-training produces higher emror than the classification bascline. We conjecture that a domain gap between real and synthetic data inter-
Section 5 and B 2D annotation- based pre-training and synthetic
Unless otherwise noted, 'data' indicates laheled images
2

---

Published as a conference paper at ICLR 2023
rupts efficient transfer learning. Finally, SSL on human data also underperforms the classification pre-training. The overall results of SSL suggest that the current stage of SSL may not be enough to benefit 3DHPSE, which essentially requires high-level understanding of human kinematic structure.
Our main empirical results are summarized in Figure T The current SSL that pre-trains on unlabeled arbitrary object images is not effective for 3DHPSE. Despite the least amount of pre-training data, 2D annotation- based pre-training provides the best result. This paper has two significant empirical contributions: 1) we provide novel experimental evidence and discussion points for people rethink the naive application of SSL pre-training paradigm in 3DHPSE, and 2) we relight the value of other data types that have received relatively less attention in the pre-training aspect.
2 RELATED WORK
3D human pose and shape estimation. We focus on reviewing the body, not hand literature, where extensive works have been addressed for 3D pose and shape estimation from in-the-wild im- ages. HMR (Kanazawa et al. 2018) proposed an end-to-end trainable human mesh recovery system that introduced adversarial loss leverage MoCap data without images. GraphCMR (Kolotouros iet al., (019b) designed a graph convolutional network that takes the rest pose human mesh and image features as input and predicts mesh vertex coordinates. SPIN (Kolotouros et al.) (019a) com- bined neural network regressor (Kanazawa et al. (2018) and an iterative fitting framework (Bogo cet al., 2016). 12L-MeshNet (Moon & Lee| 2020) introduced a lixel-based ID heatmap to estimate mesh vertex coordinates. Poseetteess.PeeMMMhhhhh (Choi et al. 2020) proposed a graph convolutional network that recovers 3D human pose and mesh from a 2D human pose. VIBE (Kocabas et al. 2020) and
TCMR (Choi al. 2021) extended HMR to video input. METRO (Lin et al. [2021] improved GraphCMR (Kolotouros et al. (019b) with a transformer architecture. PARE (Kocabas et al., (02T) introduced part-guided attention mechanism for mesh parameter regression. PyMAF (Zhang al., 2021a) used mesh-aligned image features to iteratively refine prediction. 3DCrowdNer (Choi et al,, 2022 resolved the inter-person occlusion issue in crowded scenes with 2D pose guidance for image features and joint-based regressor (Moon et al. 2022a).
Human dataset. Human datasets for 3DHPSE can be broadly categorized into three types; Motion capture (MoCap) dataset, in-the-wild 2D dataset, and synthetic dataset. MoCap datasets (Ionescu) jtt al.] [014] MMehta et al.] (017) provide accurate 3D joint labels with images captured from a con- trolled multi-view studio. In-the-wild 2D datasets (Andriluka et al. 2014] Johnson & Everingham 2010 contain manually annotated 2D labels, which are usually 2D joints. MSCOCO (Lin et al.
2014 is the most widely used dataset that has rich 2D annotations, including part segmentation and DensePose (Güler et al. 2018). Synthetic datasets (Varol et al. 2017 Patel et al.)
202I) render 3D human avatars synthetic background images. Human poses from MoCap data and appearances from real scan data are exploited. Recent works (Patel et al.) 2021 Baradel et al., 2021 Cai et al., 202I) have shown that fine-tuning synthetic human data can improve accuracy real-world benchmarks. 3DPW (von Marcard et al., (2018) is an in-the-wild human benchmark with 3D body pose and mesh annotations. Since few in-the-wild 3D human datasets exist, evaluation on 3DPW is the current best way to evaluate 3DHPSE methods on in-the-wild images.
The concurrent work of (Pang et al. 2022) provides experimental results on pre-training on three datasets (classification on ImageNet, 2D pose estimation on MPII (Andriluka et al. 2014] and MSCOCO). However, the effects of SSL and synthetic data are still unexplored. Considering that SSL has shown a powerful impact on other vision tasks, the extensive experiments and analysis on the current SSL methods distinguish our work from (Pang et al. 2022).
Self-supervised representation learning. Recently, contrastive (earning-based methods are show- ing ofate-of-the-art performance among self-supervised approaches. The contrastive learning's 's fun- damental idea is to pull together an anchor and a "positive" sample in embedding space, and to push apart the anchor from many "negative" samples (Khosla et al. 2020). Since this strategy can be applied to unlabeled training data, assuming positive pair from data augmentations of the same sample, the research community has endeavored to use the learned representation for downstream transfer tasks. In practice, backbone is pre-traines on large-scale classification dataset (Rus- (sakossky et al. 2013) Mahajan et al.) 2018) without using labels, and fine- tuned for classification, object detection, or instance segmentation.
3

---

Published as a conference paper at ICLR 2023
Pre-training stage
'dog'
Backbonn
Head
Jummnn sisa
pre-training targets
huran data
Wwight Intialization
Fine-tuning stage (3DHPSE)
3DHP4E Regrossor
cemman-certered
human
Figure 2: Overview of the training procedure for 3DHPSE. We pre-train a backbone with each different data type (e.g.. labeled arbitrary object, synthetic 3D human data, and real 2D human data). From the pre-trained backhone, we fine-tune both the backbone and human mesh regressor an end-to-end manner.
MoCo (He et al. 2020 Chen et al. 2020ch interpreted the contrastive learning as building a dy- namic and large dictionary of embeddings with queue and moving-averaged key encoder. Sim- CLR (Chen et al., 22200a by introduced a nonlinear projection layer and proved that augmentation during pre-training should be stronger than supervised learning. SwAV (Caron al. 2020), Siam- Siam (Chen & Hc 2020). and BYOL (Grill et al.) (020) eliminated the requirements for negative samples, while maintaining the Siamese architecture. DetCon (Hénaff et al. 2021 proposed new contrastive objective that is based on unsupervised mask generation.
PeCLR (Spurr et al., 202T) and HanCo (Zimmermann al. 2021) are the recent works that adapted SSL to 3D hand pose and shape estimation. PeCLR proposed a contrastive objective equivari- ant to geometric transformations (e.g., rotation and translation), which models the transforma- tions in latent vector level similar (Rhodin et al. 2018) and NSD (Rhodin et al. 2019). It showed accuracy improvements over the random initialization baseline, but trained on small-scale dataset (~32K) (Zimmermann et al. 2019) during fine-tuning. although more hand data, for ample, MSCOCO (150K) and YT3D (47K), exists. HanCo applied MoCo (He et al. 2020) on background-augmenne unlabeled hand dataset, but the improvement marginal and like PeCLR, experiments were done in the controlled setting, not the in-the-wild environment.
CONVENTION OF 3D HUMAN POSE AND SHAPE ESTIMATION
Architecture. A 3DHPSE network consists of backbone and human mesh regressor, depicted in Figure 2 A backbone (He et al. [0166] Sun et al.) (2019) extracts image features from given human-centered image. The features are fed to human mesh regressor that estimates mesh defined by human models, such as SMPL (Loper et al. 2015).
Pre-training. Almost all 3DHPSE methods pre-train their backbones by ImageNet (Russakovsky et al. 2015) classification. They take pre-trained weights from the open source (e.g., torchvi- sion Paszke et al.) 2017)) and initialize their backbones with them in practice. Initializing back- bone's weights with the ImageNet classification -rreiattooo----rriinedddd weights is known to expedite training convergence and to bring better accuracy than random initialization.
During fine-tuning, a pre-trained backbone and human mesh regressor are trained an end-to-end manner by mix-using MoCap and in-the-wild 2D datasets. 2D annotations in- the-wild 2D datasets weakly supervise the 3D mesh prediction (Kanazawa et al., 2018). and the 3D pseudo-mesh labels generated from 2D joints (Pavlakos et al. 2019; Joo et al. 2021: Moon ett al. (022b) directly supervise the 3D output. The mixed use of the MoCap and the in the wild 2D datasets has enabled reasonable performance on in-the-wild 3DHPSE, despite the scarce in in-the-will 3D training data.
Fine-tuning.

---

Published as a conference paper at ICLR 2023
4 EXPERIMENT
4.1 SETTING
We adopt ResNet-50 (He et al. 2016) as backbone and SMPL (Loper et al. 2013 mesh as an estimation target. During fine-tuning on 3DHPSE, we mainly use PARE (Kocabas et al. 2 1)
as
a human mesh regressor and Human3.6M (lonescu et al., 2014) and MSCOCO (Lin et al. 2014)
as training datasets. For the semii-supervised setting, we exploit 10% of Human3.cm 6M and 10% of MSCOCO data. 3DPW (von Marcard al., (2018). Human36M (lonescu et al.) 2014). and MuPoTS-3D (Mehta et al., 2018) are used as evaluation benchmarks. We report PA-MPIPE for 3DPW, Human36M, and 3DPCK for MuPoTTS-3D following the convention. PA-MPJPE stands for the Procrustes-aligaas mean per-joint position error in millimeters. 3DPCK indicates the percentage of correct 3D keypoints.
Pre-training. For the SSL pre-training on ImageNet (Chen et al. 2020b Chen & He 2020 Caron et al.] 2020 Chen al.) 2020e Hénaff al. 2021), use the publicly released pre-trained ResNet- 50 weights. SSL pre-training on ImageNet typically requires much larger computational costs to be effective than the classification baseline. For pre-training on human datasets, we fix the pre-training length (total number of training epochs) at 140 epochs regardless of methods, which is longer than those of 3DHPSE adapted SSL papers (100 epochs) (Spurr et al. [202]] Zimmermann et al. 2021) and the same as that of (Xiao et al.] (018). To pre-train the proposed 2D pose-driven alternatives, we decay the learning rate by 10x at 90 and 120 epochs, starting from the initial value of 10-3 following (iiao et al. 2018).
Fine-tuning. For fine-tuning, we train a 3DHPSE network for 120 epochs and decay the learning rate by 10x at 90 epochs, which is originally 10 4 This schedule empirically showed full conver- gence for various networks initialized with different pre-trained weights. For the case of training from scratch (i.e., random initialization), following the spirit of (He et al., 2019), we extend the total training length to 240 epochs and decay the learning rate by 10x at 180 epochs from 10-4, roughly matches the total training length of the pre-training counterparts (140 epochs + 120 epochs) and empirically showed full convergence.
4.2 PRE-TRAINING ON IMAGENET
We first examine the effects of the current state-of-the-ar SSL methods (Chen al. 2020b Chen & He 2020 Caron et al., 2020) Chen et al.) 2020c Hénaff et al. 2021) that pre-train a backbone on ImageNet. As summarized
Table 1 and Figure 3] when full fine-tuning data is used, they show approximately 2 slower convergence and 7.7% higher errors (PA-MPJPE) than the ImageNet classification pre-training (He et al., 2016). Interestingly, the final accuracy even worse than the random initialization baseline, though the convergence is faster. In the semi-supervised setting, SSL methods outperform the random initialization baseline by 2,5% but still the classification pre- training provides the best overall accuracy.
We analyze the results that seemingly oppose the fact that SSL surpasses the random initialization and classification baselines in object detection and instance segmentation (Grill et al. 2020; Hénaff (i) Why SSL is worse than the random initialization baseline, when full fine-tuning data is used? We conjecture that data domain gap is of the reasons for the different results. While both object detection and instance segmentation target lo- calization of arbitrary class objects, 3DHPSE only targets a single class, the human. For inference on arbitrary class objects, learning a wide range of general features unlimited to labels of a dataset could be advantageous in the generalization aspect (Tendle Hasan 2021). However, for 3DHPSE, a backbone network is preferred to learn more about human features rather than features of arbitrary objects, given the limited learning capacity. Transferring knowledge about arbitrary objects could distract network from learning necessary human features for 3DHPSE. (ii) Why the classification pre-training outperforms the random initialization baseline, while SSL does not, when full fine- tuning data is used? The classification pre-training makes a backbone to learn high-level semantic representations, such as the global structure of objects, that could be beneficially transferred to infer- ence on humans. For example, (Yosinski et al. 2015) showed that AlexNet (Krizhevsky et al. 2017) trained on ImageNet can recognize important features of human faces, although ImageNet has no labels of human faces. On the contrary, visual representations learned by SSL are likely to lack high-
et al. 2021) by answering three questions.
5

---

Published as a conference paper at ICLR 2023
Table 1: Effects of self-supervised pre-training on ImageNet. The red and blue colors indicate the first and second best scores, respectively.
fine-tuning
pre-training data
3DPW PA-MPJPEL
H36M PA-MPJPEJ
MuPoTS 3DPCK1
pre- tre-training method
data
random init.
56.37
52.72
67.12
ImageNet
classification
55.65
48.36
67.76
(labeled)
H36M+ MSCOCO
SimCLR
Chen et al.
2020a
59.56
56.93
65.67
SimSiam
Chen & Hc. 2020
56.42
51.82
66.34
(100%)
ImageNet (unlabeled)
SwAV
Caron al.
2020
56.85
52.18
66.19
MoCo v2
(Ceen et al.
2020c
58.34
55.43
66.68
DetCon Hénaff ct al.
2021
64.54
58.83
63.83
random init.
73.37
67.59
57.32
ImageNet
classification
63.29
58.79
62.75
(labeled)
H36M+ MSCOCO (10%)
SimC1.R
(Chen
2020a
73.97
72.02
59.80
SimSiam
(Chen & He.
2020
66.94
62.45
63.34
ImageNet (unlabeled)
SwAV
Caron al.
2020
68.96
63.26
60.79
MoCo v2
(Chen et al
2020c
64.76
60.70
63.47
DetCon Hénaff et al.
2021
81.63
82.60
55.03
desssiicatoo McDD
Swudel
2.00
40
130
175
(fne-tuning)
(ine-tuning)
Figure 3: Learning curves of PA-MPIPE on 3DPW in the fine-tuning stage when using full fine- tuning data (Left) and 10% of fine-tuning data (Right). The backbone is initialized with different weights pre-traines pre-trained on ImageNet by different pre- pre-training methods.
level information that can be found objects with the same class. Instead. SSL focuses on learning representations that are invariant to data augmentations, which are performed on single object sample. These representations can be inconsistent over instances of the same class (Khosla al.) 2020). Considering that 3DHPSE essentially requires high-level priors that are shared across differ- ent human samples, inconsistent representations learned by SSL could detrimental to 3DHPSE. (iii) Why SSL is effective in the semi-supervised setting, compared with the random initialization? Inconsistent representations Icarned by SSL could a problem for 3DHPSE, but does not indicate obsolete. A convolutional neural network requires sufficient training data to extract meaningful low-level features (e.g., textures) from images. this perspective, insufficient training images deprive network of learning high-level representations, which in turn harms 3DH- PSE. In such circumstance, SSL can step in to provide sufficient amount of training images to network and make it learn necessary features.
their representations
4.3 ANALYSIS OF SSL FOR 3DHPSE.
We further investigate the reasons for the SSL's disappointing performance discussed in Section 4.2 with experimental evidence. We devise a new pre-training approach, JointCon, that combines the SSL approach with the 2D annotation- based approach. JointCon extracts joint-level features (Moon jet al. 2022a) by sampling image features based on GT 2D joint locations. It applies contrastive learning the joint-level features pull together anchor with "positive samples" and push

---

Published as a conference paper at ICLR 2023
Table 2: Effects of different representation-levee in pre-training via ablation study on JointCon. We use MSCOCO for pre-training. The red and blue colors indicate the first and second best scores, respectively.
fine-tuning data
3DPW
H36M PA-MPIPEL
MuPoTS 3DPCK1
variations of JointCon
PA-MPJPEL
H36M+ MSCOCO (100%)
JointCon(T): instance-level
58.29
53.17
67.06
JointCon(J): jonttoooo:joint-eeeel joint-level
54.25
45.52
68.87
JoiniCon(I+J):: instance-level joint-level
56.71
47.99
67.85
H36M+
JoiniCom(I): instance-level
67.71
62,04
62,64
MSCOCO
JointCon(J): joint-level
57.19
54.97
67.98
(10%)
JointCon(I+J): instance-level + joint-level
59.59
56.55
67.38
apart the anchor from "negative samples". We design three variations of JointCon as below: 1) JointCon(I) defines "positive samples" joint-level features extracted from the same image and "negative samples" as those from different images. 2) JointCon(J) defines "positive samples" as joint-level features extracted from the same joint label and "negative samples" as those from dif- ferent labels. Note that it treats joint-level features of the same joint class from different images as "positive samples". 3) JointCon([+J) defines "positive samples" as joint-level features extracted from the same image and the same joint label, and "negative samples" otherwise.
Table 2 shows that JointCon(J) outperforms the other variations in all three benchmarks. The ac- curacy gap between JointCon(J) and JointCon(I+J) proves that it is important to learn consistent representations across instances of the same class. Contrasting representations of the instances in the same class (i.e., objects or joints in the same class) hinders backbone from learning useful high-level priors and degrades the performance of 3DHPSE. As a result, the ImageNet classifi- cation pre-training rather learns consistent representations of the instances in the same class and achieves better accuracy than SSL, as shown in Table [ The other important observation is that both JointCon(J) and JointCon(I+J) outperform JointCon(I). The accuracy boost by simply adding joint information to the SSL system demonstrates that the current SSL's low performance is due to the instance-level learning characteristic rather than network architecture, data augmentation, losses.
4.4 PRE-TRAINING ON A HUMAN DATASET
We investigate three pre-training approaches, SSL, 2D annotation-based pre-training, and synthetic data pre-training. First of all, given the results of Section 4.2 we apply SSL human dataset following (Spurr et al. 202T Zimmermann et al. 202T) PeCLR (Spurr et al., 202T) that adapted SimCLR (Chen et al. 2020a
3D hand pose and shape estimation, MoCo v2 (Chen et al. 2020c), the method used by HanCo (Zimmermann et al.] (021). and Swav (Caron et al.) 2020). which is not based contrastive learning, are experimented. MSCOCO (Lin et al.) (014] used as a pre-training dataset. For synthetic data pre-training, we pre-train PARE (Kocabas et al. 2021) on AGORA (Patel al. (02I] and SURREAL (Varol al. 2017). For 2D annotation- based pre- training, we experiment with part segmentation estimation, DensePose (Güler et al. 2018) estima- tion, and 2D pose estimation on MSCOCO.
As shown in Table g and Figure 4 the SSL methods bring faster convergence, but the final accu- racy becomes on par with the random initialization baseline. also lower than the traditional classification pre-training SSL appears to be effective only when significantly less fine-tuning data (10% Human3.6N, + 10% MSCOCO, < 50K) used. The results coincide with the experimental results of PeCLR and HanCo: PeCLR was fine-tuned on ~322 labeled data (Zimmermann et al.) 2019), and HanCo was fine-tuned on ~64K labeled data (Zimmermann et al. 2021). Synthetic data pre-training shows similar tendency with SSL, though it outperforms SSL. On the contrary, 2D inootation-bassed pre-training improves accuracy against the random initialization and classification baselines all benchmarks. In the semi-supervised setting, the improvement increases up to 12.4% compared with the random initialization. The convergence speed is much faster than both random initialization and classification, especially in the semi-supervised setting.
7

---

Published as a conference paper at ICLR 2023
Table 3: Effects of different pre-training schemes on human dataset. The red and blue colors indicate the first and second best scores, respectively.
fine-tuning tani-tuning data
pre-training data
3DPW PA-MPIPEL
H36M PA-MPJPEL
MuPaTS 3DPCK1
pre-training method
random init.
56.37
52.72
67.12
ImageNel
classification
55.65
48.36
67.76
(labelod)
PeCLR
Spurr
2021
57.27
49.84
66.88
MSCOCO (unlabeled)
H
MoCo v2
Chen ct al.
2020c
56.35
50.06
66.95
MSCOCO
SwAV
Caron al.
2020
58.20
56.50
66.77
(100%)
AGORA
56.77
51.96
67.80
3DHPSE
SURREAL
56.45
52.51
67.65
part segmenetation
54.37
48.25
68.55
MSCOCO
DonsePosc est.
54.16
49.18
68.43
pose est.
44.89
69.04
random init.
73.37
67.59
57.32
ImageNet
classification
63.29
58.79
62.75
(labeled)
PeCLR
(Spurr al.
2021
71.24
66.58
60.16
MSCOCO (unlabeled)
MoCo
Chen et al.
2020c
66.93
62.06
61.36
H36M+
MSCOCO
SwAV
al.
2020
75.98
73.83
58.40
(10%)
AGORA
64,71
63.17
65.73
3DHPSE
SURREAL
59.10
64.50
segmenctation est.
56.80
54.10
67.69
MSCOCO
DensePuse est.
57.33
54.46
68.02
pose
56.92
55.31
67.12
clasaltcation
classiicttio
PNCLR
PeCLi
McCo 92
MoCo
IPSE (AGORA)
(AGORA)
Training (int--Dningg #ooche
Training (fina-buiin]
Figure 4: Learning curves of PA-MPIPE on 3DPW in the fine-tuning stage when using full fine- tuning data (Left) and 10% of fine-tuning data (Right). The backbone is initialized with different weights pre-trained on a human dataset by different pre-training methods.
We think the results of SSL on human data support the statement in Section 4.2 that representa- tions learned by SSL are hard to embed high-level information related to humans. Synthetic data pre-training has potential to benefit from rich pose and appearance diversity, but a domain gap prob- lem from highly different image appearances between synthetic and real images seems to remain. Compared with SSL and synthetic data pre-training, 2D innotation-basel pre-training appears to ef- fectively transfer high-level priors of humans, such as body articulation, to 3DHPSE, Well-aligned 3D body pose and robustness occlusion in Figure verify the effectiveness of 2D annotation- based pre-training. Figure 6 visualizes each human part's 's feature activation of PARE (Kocabas) al. 202I) to further inspect how representations learned by each pre-training approach affect the human mesh regressor's understanding of human geometry.
5 DISCUSSION
Our empirical observations propose that we should explore 2D cmmotation-based pre-training instead of SSL. for 3DHPSE. However, SSL may still have advantage over 2D annotation-based pre-
8

---

Published as a conference paper at ICLR 2023
unlabeled MoCo v2
synthetic 3DHPSE
unlabeled MoCo v2
syntheticc 3DHPSE
20 pose
2D pose est.
Figure 5: Qualitative comparison among different pre-training schemes. We highlighted their repre- sentative failure cases with red circles.
"attention on head"
"attention olbow" unlabeled MoCo v2
unlabeled synthetic 3D> MoCo v2
roal 20 2D est.
synthetic 3D
real 20 2D est.
3DHPSE
3DHPSE
Figure 6: Feature attention visualization of PARE (Kocabas et al. 2021 for different human parts.
training in the aspect of labeling costs. In this regard, we answer two questions that could help future 3DHPSE researchers to choose their pre- pre-training strategies.
(i) Does SSL entail zero cost? No. It is not true in the aspect of data collection. Collecting, cleaning, and curating data for SSL demand resources (Russakovsky
al. 2015 He et al. 2019
Kotar et al. 2021). More importantly, the current SSL methods for 3DHPSE (Spurr et al. 2021
Zimmermann et al. 2021) assume human to be centered in an input image, which an unrealistic
setting for in-the-wild images without bounding box labels. Thus, we should consider the labeling
cost of bounding boxes when using SSL (Purushwalkam & Guptal 2020 Goyal et al. 2021
Nouby et al. 2021). Further discussion is provided in Section B of Appendix.
(ii) SSL efficient than 2D annotation-based pre-training? No. SSL pre-trains on massive un- labeled data, which involves high computational costs. It often results in order of magnitude more computation than the supervised counterparts (Hénaff et al., 202T). In addition, due to double fea- ture encoders and complex feature contrasting mechanism, SSL takes more time than 2D annotation- based pre-training to pre-train on the same amount of pre-training data. Pre-training ResNet-50 (He) 2016) with PeCLR (Spurr al. 2021), MoCo v2 (Chen al. (020c), and the 2D pose estimator on MSCOCO took 110, 226, and 14 hours respectively, with the four RTX 2080Ti GPUs.
6 CONCLUSION
We have investigated different approaches for pre-training a 3DHPSE backbone. SSL, which has recently become the major trend in the community, has been thoroughly inspected. 2D annotation- based pre-training and synthetic data pre-training have also been experimented, since they share the similar motivation of reducing labeling costs and transferring useful representations 3DHPSE. We experimented with multiple methods of each approach on multiple benchmarks to not draw conclusion valid limited setting. Please also refer to Section D to confirm that our observations on the three pre-training approaches are preserved regardless of 3DHPSE mesh regressor. Our empirical results show that 1) SSL is yet to replace the de facto paradigm of the ImageNet classifi- cation pre-training, and 2) 2D annotation- based pre-training can effectively transfer the knowledge to 3DHPSE, despite the least amount of pre-tree pre-training data. We believe our findings, analysis, and discussion will significantly influence future research pre-training for 3DHPSE.

---

Published as a conference paper at ICLR 2023
Acknowledgements. This work supported in part by the IITP grants [No.2021-0-01343, Ar- tificial Intelligence Graduate School Program (Scoul National University), No.2022-0-00156, No. 2021-0-02063 and No. No.2002-00000156], and the NRF grant [No. 2021M3A9E4000782) funded by the Korea government (MSIT), and AIRS Company in Hyundai Motor Company Kia Corporation through HMC/KIA-SNU AT Consortium Fund.
REFERENCES
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estima- tion: New benchmark and state of the art analysis. In CVPR, 2014.
Fabien Baradel, Thibault Groueix, Philippe Weinzaepfel, Romain Brégier, Yannis Kalantidis, and Grégory Rogez. Leveraging mocap data for human mesh recovery. In 3DV, 2021.
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael Black. Keep it SMPL: Automatic estimation of 3D human pose and shape from single image. In ECCV, 2016.
Zhongang Cai, Mingyuan Zhang, Jiawei Ren, Chen Wei, Daxuan Ren, Jiatong Li, Zhengyu Lin, Haiyu Zhao, Shuai Yi, Lei Yang, et al. Playing for 3d human recovery. arXiv, 2021.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. CVPR, 2017.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NewrlPS, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. In NeurlPS, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:201 1, 10566, 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020c.
Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2Mesh: Graph convolutional network for 3D human pose and mesh recovery from a 2D human pose. In ECCV, 2020.
Hongsuk Choi, Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee. Beyond static features for temporally consistent 3d human pose and shape from video. In CVPR, 2021.
Hongsuk Choi, Gyeongsik Moon, JoonKyu Park, and Kyoung Mu Lee. Learning to estimate robust 3d human mesh from in-the-wild crowded scenes. In CVPR, 2022.
Mickael Cormier, Fabian Röpke, Thomas Golda, and Jürgen Beyerer. Interactive labeling for human pose estimation in surveillance videos. In ICCV workshop. 2021.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint 1rXiv:2112.10740, 2021.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hicrarchies for accu- rate object detection and semantic segmentation. In CVPR, 2014.
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux. Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2 103.01988 2021.
10

---

Published as a conference paper at ICLR 2023
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. In NeurlPS, 2020.
Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In CVPR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In ICCV, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Olivier J Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, and João Carreira. Efficient visual pretraining with contrastive detection. In ICCV, 2021.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m 6.6:: Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI, 2014.
Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC, 2010.
Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi. Exemplar fine-tuning for 3d human model fitting towards in- in-the-wild 3d human pose estimation. In 3DV, 2021.
Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018.
Angjoo Kanazawa, Jason Zhang, Panna Felsen, and Jitendra Malik, Learning 3d human dynamics from video. In CVPR, 2019.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NeurlPS, 2020.
Muhammed Kocabas, Nikos Athanasiou, and Michael Black. VIBE: Video inference for human body pose and shape estimation. In CVPR, 2020.
Muhammed Kocabas, Chun-Hao P Huang, Otmar Hilliges, and Michael J Black. Pare: Part attention regressor for 3d human body estimation. In ICCV, 2021.
Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis. Learning to recon- struct 3D human pose and shape via model-fitting in the loop. In ICCV, 2019a.
Nikos Kolotouros, Georgios Pavlakos, and Kostas Daniilidis. Convolutional mesh regression for single-image human shape reconstruction. CVPR, 2019b.
Klemen Kotar, Gabriel Ilharco, Ludwig Schmidt, Kiana Ehsani, and Roozbeh Mottaghi. Contrasting contrastive self- supersuper representation learning pipelines. In ICCV, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. CACM, 2017.
Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In CVPR, 2021.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In 2014
Buyu Liu and Vittorio Ferrari. Active learning for human pose estimation. 2017.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. SMPL: A skinned multi-person linear model. ACM TOG, 2015.
11

---

Published as a conference paper at ICLR 2023
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining, In ECCV, 2018.
Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 3DV, 2017.
Dushyant Mchta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, and Christian Theobalt. Single-shot multi-person 3d pose estimation from monocular rgb. In 3DV, 2018.
Gyeongsik Moon and Kyoung Mu Lee. I2L-MesNNet: Image-to-Lixel prediction network for accu- rate 3D human pose and mesh estimation from single RGB image. In ECCV, 2020.
Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Accurate 3d hand pose estimation for whole-body human mesh estimation. In CVPR workshop, 2022a.
Gyeongsik Moon, Hongsuk Choi, and Kyoung Mu Lee. Neuralannot: Neural annotator for 3d human mesh training sets. In CVPR workshop, 2022b.
Hui En Pang, Zhongang Cai, Lei Yang, Tianwei Zhang, and Ziwei Liu. Benchmarking and analyzing 3d human pose and shape estimation beyond algorithms. In NeurIPS, 2022.
Dim P Papadopoulos, Jasper RR Uijlings, Frank Keller, and Vittorio Ferrari. Extreme clicking for efficient object annotation. In ICCV, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Detito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurlPS, 2017.
Priyanka Patel, Cau P Huang, Joachim Tesch, David T Hoffmann, Shashank Tripathi, and Michael Black. Agora: Avatars in geography optimized for regression analysis. CVPR, 2021.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from single image. In CVPR, 2019.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: In- variances, augmentations and dataset biases. In NeurIPS, 2020.
Helge Rhodin, Mathieu Salzmann, and Pascal Fua. Unsupervised geometry-aware representation
for 3d human pose estimation. In ECCV, 2018.
Helge Rhodin, Victor Constantin, Isinsu Katircioglu, Mathieu Salzmann, and Pascal Fua. Neural scene decomposition for multi-person motion capture. In CVPR, 2019.
Yu Rong, Ziwei Liu, Cheng Li, Kaidi Cao, and Chen Change Loy. Delving deep into hybrid anno- tations for 3d human recovery in the wild. In ICCV, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Adrian Spurr, Aneesh Dahiya, Xi Wang, Xucong Zhang, and Otmar Hilliges. Self-supervised 3d hand pose estimation from monocular rgb via contrastive learning. In ICCV, 2021.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
Atharva Tendle and Mohammad Rashedul Hasan. A study of the generalizability of self-supervised representations. MLA, 2021.
Gül Varol, Javier Romero. Xavier Martin, Naureen Mahmood, Michael Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR, 2017.
12

---

Published as a conference paper at ICLR 2023
Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and moving camera. In ECCV, 2018.
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In ECCV, 2018.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. ICML workshop, 2015.
Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In ICCV,, 2021a.
Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. PyMAF: 3D human pose and shape regression with pyramidal mesh alignment feedback loop. In ICCV, 2021b.
Christian Zimmermann, Duygu Ceylan, Jimei Yang. Bryan Russell, Max Argus, and Thomas Brox. Freihand: dataset for markerless capture of hand pose and shape from single rgh images. In ICCV, 2019.
Christian Zimmermann. Max Argus, and Thomas Brox. Contrastive representation learning for hand shape estimation. GCPR, 2021.
13

---

Published as a conference paper at ICLR 2023
APPENDIX
In this supplementary material, we provide more experiments and discussions that could not be included the main text due to the lack of pages. The contents are summarized below:
Effects of SSL and 2D annotation-based pre-training on different amounts of data; Insta Va- riety(246K) (Kanazawa al. 2019) and MPII(25K) (Andriluka al. 2014) are experimented in addition to MSCOCO(150K) (Lin et al. 2014).
Discussion about labeling costs of SSL and 2D annotation-based pre-training
C - Effects of different pre-training approaches, when fine-tuned on different sizes of human data (30% and 60% of (Human3.6M (Ionescu et al. 2014) + MSCOCO (Lin al. 2014
D Effects of different human mesh regressors.
A EFFECTS OF PRE-TRAINING ON DIFFERENT AMOUNTS OF DATA
SSL is known to require large-scale pre-training data be effective (He et al.] 2020). In this regard, we apply PeCLR (Spurr et al. 2021) and MoCo v2 (Chen et al. 2020c) on Insta Vari- ety(246K) Kanazawa al., 2019), which is 1.64x bigger than MSCOCO (55K) (Lin al. 2014). Interestingly, pre-trainig on Insta Variety consistently shows worse accuracy in all benchmarks as shown in Table 4 If we use less fine-tuning data, the accuracy gap increased, except PeCLR on Human3.6n 66. We think the seemingly contradictory results come from the noisy bounding box used to crop a human from an image. The bounding box of a human in Insta Variety is calculated by mea- suring minimum and maximum æ, y locations of OpenPose (Cao al. 2017 's 2D pose estimation. Since the estimated values inevitably have errors, the bounding box obtained from them may not produce nunan-centered image. Thus, the input distribution could be severely different during pre-training and fine-tuning tuning, which could lead to counter-intuitive results.
We also provide the results of 2D pose estimation pre-training MPII(25K) (Andriluka et al., 2014). Despite much less pre-training data, 2D pose estimation pre-training achieves comparable accuracy to that of pre-training MSCOCO. This shows the cost-effectiveness of 21D pose estimation pre-training, which is further discussed in the next section.
Table Effects of pre-training on different amounts of data. Insta Variettyr))) Kanazawa al,, 2019). MSCOC0(150K) (Lin et al.) 2014), and MPII(25K) (Andriluka et al. 2014) are used for pre-training. The red and blue colors indicate the first and second best scores, respectively.
fine fine-tuning data
pre- treitraining
3DPW
H36M
MuPoTS 3DPCK1
pre-training method
PA-MPIPE,
PA-MPIPEL
InstaVariets Varicty
PeCIR Spurr et al.
2021
57.45
51.17
66.77
(unlabeled)
MoCo v2
Chen et al.
2020c
57.28
51.14
67.53
H336M
MSCOCO
PeCLR
Spurr et al.
2021
57.27
49.84
66.88
MSCOCO
(100%)
(unlabeled)
MoCo
Chen al.
2020c
56.35
50.06
66.95
MSCOCO
2D pose est.
53.34
44.89
69.04
MPII
2D pose
54.71
46.12
67.61
Instavariety (unlaheled)
PeCLR Spurr et al.
2021
74.65
64.77
56.51
MoCo v2
Chen al.
2020c
76.49
70.42
58.72
H36M+ MSCOCO (10%)
MSCOCO
PeCLR
Spurr et
2021
71.24
66.58
60.16
(unlabeled)
MoCo v2
Chen et al.
2020c
66.93
62.06
61.36
MSCOCO
2D posc
56.92
55.31
67.12
MPII
pose est.
62.71
58.39
62.57
14

---

Published as a conference paper at ICLR 2023
B DISCUSSION ABOUT LABELING COSTS OF SSL AND 2D ANNOTATION- PRE- TRAININI
Some could value SSL on bounding box-labeled human data more than 2D annotation-basee pre- training, despite the experimental results of the main text. For example, assuming less fine-tuning data like the main text's Table 6 MoCo v2 (Chen et al.) (020c) may be more attractive option than the 2D pose estimation pre-training, considering that bounding box annotations take less cost than 2D pose annotations. In this context, analyze the cost-effectiveness of different pre-training approaches by comparing the annotation cost in terms of annotation time.
Table supports that the 2D pose estimation pre-training is much similar annotation time. The state-of-the-art object annotation paper (Papadopoulos et al. 2017) reported 7 seconds per one bounding box annotation. We assume that the same time would take for human bounding box. Liu et al. (Liu & Ferrari, (017)) reported 1.5 seconds per one human key point. Since a person MPII (Andriluka et al., 2014) images have 14 key points, we can assume that the annotation time is 21 seconds per person. Cormier et al. (Cormier al. (022] reported 42.8 seconds per bounding box and pose of one person. take the number of Cormier al. (Cormier et al. 2021 in Table
cost-effective, given the
Table 5: Comparison between cost-efficctiveness of SSL on bounding box-labeled data and 2D pose estimation pre-training.
pre-training method
pre-training data
3DPW PA-MPJPEL
H36M PA-MPIPEL
MuPoTS 3DPCK1
annotation time
MoCo v2 Chen et al., 2020c)
MSCOCO (150K) (unlabeled)
150Kx7s=10500
56.35
50.06
66.95
2D pose est.
MPII (25K)
25Kx42.8s= 1070Ks
53.34
44.89
69.04
C FINE-TUNING ON DIFFERENT SIZES OF HUMAN DATA
We explore the effects of different pre-training approaches in different semi-supervised settings by varying sizes of fine-tuning data. Experimental results using 10%, 30%, 60%, and 100% fine- tuning data of (Human3.66 66M (Ionescu al. 2014) MSCOCO (Lin et al. 2014)) are shown in Table and Figure 7 SSL starts to surpass the random initialization baseline when fine-tuning data reduced 30%. Synthetic data pre-training also shows a similar tendency. Only 2D annotation-
based pre-training consistently outperforms the random initialization baseline, and the accuracy margin enlarged as the fine-tuning data reduces.
15

---

Published as a conference paper at ICLR 2023
Table 6: 3DHPSE evaluation results on pre-training schemes with different sizes of human data. We report the best accuracy of each scheme on 3DPW, Human3.6M and MuPoTS-3D dataset. The red and blue colors indicate the first and second best scores, respectively.
fine-tuning data
pre-training data
3DPW PA-MPIPEL
H36M PA-MPJPEJ
MuPoTS 3DPCK
pre-training method
random init.
56.37
52.72
67.12
ImageNet
classification
55.65
48.36
67.76
(labeled)
H36M+
MSCOCO
PeCLR Spurr et
2021
57.27
49.84
66.88
MSCOCO
(unlabeled)
MoCo v2
Chen et al.
2020c
56.35
50.06
66.95
(100%)
AGÖRA
56.77
51.96
67.80
3DHPSE
SURREAL
56.45
52.51
67.65
DensePose est.
54.16
49.18
68.43
MSCOCO
2D pose
53.34
44.89
69.04
random init.
57.31
50.61
67.30
ImageNet (labeled)
classification
55.63
51.87
67.34
H36M+ MSCOCO
MSCOCO (unlabeled)
PeCLR Spurr et al.
2021
59.17
52.14
65.80
MoCo v2
Chen et al.
2020c
58.57
52.34
66.63
(60%)
AGŌRA
57.23
$3.99
68.87
3DHPSE
SURREAL
57.69
53.10
66.47
DensePose est.
54.80
49.38
68.44
MSCOCO
2D pose random init.
54.41
$2.03
67.91
62.40
56.49
63.20
ImageNet (labeled)
classification
57.85
53.02
65.74
H36M+ MSCOCO
MSCOCO
PeCLR
Spurr et al.
2021
63.43
55.90
63.66
(unlabeled)
MoCo v2
Chen et al.
2020c
60.95
55.09
64.96
(30%)
AGŌRA
60.50
56.05
67.43
3DHPSE
SURREAL
57.92
53.93
68.74
DensePose est.
55.44
52.37
68.40
MSCOCO
2D pose
53.94
47.39
68.56
random init.
73.37
67.59
57.32
ImageNet
classification
63.29
58.79
62.75
(labeled)
H36M+ MSCOCO
MSCOCO
PeCLR
Spurr et al.
2021
71.24
66.58
60.16
(unlabeled)
MoCo v2
Chen al.
2020c
66.93
62.06
61.36
(10%)
AGORA
64.71
63.17
65.73
3DHPSE
SURREAL
62.51
59.10
64.50
DensePose est.
57.33
54.46
68.02
MSCOCO
2D pose est.
56.92
55.31
67.12
16

---

Published as a conference paper at ICLR 2023
102
10% >
cossiiiction
30%
Cassifiatioo
PeCUA
PECLR
MECO V2
MaCo
JOHPSE (ACONAL
JCHPSE (AGORA)
57-
SOHISE (SURRCAL)
JOMPSE (SUUUEAA))
DensePose
20 pose est.
Traiiing iffne tuntegi spects
Traniin tuninal
22
60%
Clave aliiiin
100%
classiication
PeCLR
FeCLR
Mcto V2
MoCo
TOHPSE (AGORA)
30HPSE (AGORA)
30HPPS (SURREALI
SOMPSE (SURNEAL)
DensePose
20 pose
Tradking (fiee- (fine-Tuning) Tuningi #pects
350
120
Figure 7: Learning curves of PA-MPJPE on 3DPW in the fine-tuning stage with different weights pre-trained on different sizes of human data by different pre-training methods.
D EFFECTS OF DIFFERENT HUMAN MESH REGRESSORS
In Table and Table g we experiment with different human mesh regressors (Kolotouros et al., 2019a Zhang al., 202 Moon al. 2022a Moon & Lee 2020) that have distinet architectures
and achieved state-of the art accuracy recently. The best pest-performing methods of SSL, synthetic data pre-training, and 2D amnotation-biin based pre-training are used compare. Overall, the same tendency is observed. SSL and synthetic data pre-training underperform the classification baseline, while 2D annotation based pre-training outperforms in multiple 3DHPSE benchmarks. This shows that our observations are not limited to single 3DHPSE method but can generalize to different methods. The notable result that 12L-MeshNet (Moon & Lee 2020) produces overall the same accuracy regardless of the pre-training approaches. We conjecture that the different tendency of 12L- MeshNet comes from its architecture and target representation. While other human mesh regressors estimate the SMPL (Loper al.) 2015) parameters with MLP layers the end of their networks, 121-MeshNet directly regresses 2.5D locations of mesh vertices in a fully convolutional manner. The 2.5D representation expresses the xy locations of mesh vertices in the image pixel space and the z location as root relative depth. As a result, the accuracy of 12L-Meshie PL-MeshNet depends on the segmentation of human from cropped human-centris image. This partially posits the problem of 3DHPSE to the foreground estimation, which may less demand on learning complex priors about humans, such as 3D human body articulation.
17

---

Published as a conference paper at ICLR 2023
Table 7: Effects of different human mesh regressors. We utilize SPIN (Kolotouros et al. 2019a) PyMAF Zhang et al. (202Tb), Pose2Pose (Moon et al. 2022a), and 12L-MeshNet (Moon Leep 2020) as regressors. We use full data (Human3 6M and MSCOCO) data for fine-tuning. The red and blue colors indicate the first and second best scores, respectively.
pre-training data
3DPW PA-MPJPEU
H36M PA-MPIPEL
MuPoTS 3DPCK
regressor
pre- pre-training method
ImageNet
classification
64.73
51.21
66.32
MSCOCO
MoCo v2 Chen et al.
2020c
64.98
57.04
63.98
SPIN
AGORA
3DHPSE 2D pose est.
66.00
55,51
64.99
MSCOCO
59.78
51,29
66.75
ImageNet
classification
60.37
67.23
68.19
MSCOCO
MoCo v2 (Chen et al.
2020c
64.09
69.49
65.49
PyMAF
AGORA
3DHPSE 2D posc est.
65.60
71.63
66.72
MSCOCO
59.04
69.53
70.46
ImageNet
classification
57.07
44.32
67.80
MSCOCO
MoCo v2 CCen et 2020c
62.11
53.91
65.33
Pose2Pose
AGORA
3DHPSE 2D pose classification MoCo v2 Chen al. 2020c
61.00
51.28
66.51
MSCOCO
56.78
41.79
68.54
ImageNet
60.74
39,42
70.76
12L-MeshNet (lixel stage)
MSCOCO
61.11
38.34
71.50
AGORA
3DHPSE
60.48
39.62
71.73
MSCOCO
2D posc est.
60.49
38.99
71.88
Table 8: Effects of different human mesh regressors in semi-supervised setting, We utilize SPIN (Kolotouros et al., (019a)) PyMAF (Zhang et al.. (021b), Pose2Pose (Moon al., 2022a), and 12L-MeshNet (Moon & Lee 2020) as regressors. We use 10% of (Human3.6M) and MSCOCO) data fine-tuning. The red and blue colors indicate the first and second best scores, respectively.
pre-training data
3DPW
H36M
MuPoTS 3DPCK*
regressor
pre-training method
PA-MPIPEL
PA-MPJPEL
ImageNet
classification
78.99
75.97
55.00
MSCOCO
MoCo v2
Chen et al.
2020c
86.85
81.63
48.47
SPIN
AGORA
3DHPSE 2D posc est. classification
83.73
82.76
54.04
MSCOCO
64.57
64.96
65.51
ImageNet
67.64
73.62
63.76
MSCOCO
MoCo v2 CCen et al.
2020c
71.90
80.43
58.69
PyMAF
AGORA
3DHPSE pose est. classification
69.65
81.71
62.04
MSCOCO
59.41
73.84
68.91
ImageNct
66.85
61.06
64.79
MSCOCO
MoCo v2 Chen 2020c
74.25
73.33
59.61
Pose2Pose
AGORA
3DHPSE
69.66
67.04
63.02
MSCOCO
2D pose est. classification
57.97
54.91
68.45
ImageNet
61.47
50.74
71.97
12L-MeshNet (lixel stage)
MSCOCO
MoCo v2 (Chen al. 2020c
62.38
51.88
72.28
AGORA
3DHPSE
61.43
51.72
72.15
MSCOCO
2D pose est.
62.05
51.21
71.50
18


Reviews:
[{"review_id": "z8kyuFQov5", "reviewer": "Reviewer_jtxR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides empirical study of how to pre-train the vision backbone for human pose and mesh estimation. The authors conducted many experiments with different pre-training data. The authors observed using real images with 2D human pose labels (such as COCO) is the most effective one. ", "summary_of_the_review": "I have a mixed feeling about this paper. The paper presents a very useful exploration of pre-training vision backbone for human pose and mesh reconstruction. All the discussions and analysis are valuable. It will be very helpful to future researchers. This is good as a technical report. But when considering it as a research paper, the novelty of pre-training technique is weak.\n\nProbably because the considered SSL pre-training methods don't work well, the paper consists of several negative observations for SSL. The only positive signal comes from pre-training with real images & pose labels (such as COCO dataset). This is more like a revisit, as some recent works have pre-trained a PoseNet in a similar way for pose-to-mesh pipeline. That is, there is no novel pre-training technique introduced. This weakens the contribution of this submission.\n\nHowever, considering the valuable analysis, discussions, and the great efforts of conducting all these experiments, this paper still provides useful guidance to the community. ", "strength_and_weaknesses": "Strengths:\n1. The paper is easy to follow and well-written. The authors described their experiments and different analysis settings clearly.\n2. The authors conducted very thorough experiments to validate which data is useful for pre-training the backbone.\n3. The paper provides several interesting discussions and insightful findings. It could be a useful reference for future research.\n\nWeaknesses:\n1. Prior works have been discovered that 2D re-projection loss is very critical to exiting methods. The main message of this paper seems to double confirm the importance of 2D pose data. While the authors show that pre-training the backbone is important, there is no other new pre-training technique introduced to effectively boost the SOTA performance. \n2. I appreciate the efforts of experimenting with SSL methods. However, the results seem not very positive. Since SSL might be a bit more difficult to train, I wonder if the SSL pre-training needs more training tricks and more diverse data to enhance feature representation generalizability.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "rOH-2edtLU", "reviewer": "Reviewer_C9uS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work provides a thorough evaluation of various backbone pre-training methods such as SSL, with human datasets, and semi-supervised setting. ", "summary_of_the_review": "Despite I deeply appreciate the investigation in pretraining backbone with SSL for human pose and shape estimation, I find it difficult to justify the significance as the finding seems to suggest that SSL does not work in this case. More importantly, there are relevant works on backbone training strategies and synthetic data, showing much more promising results. Therefore, I find the paper in its current form does not meet the bar of ICLR.", "strength_and_weaknesses": "\n1. An in-depth investigation is presented in the paper to cover various pre-training methods.\n\n2. A wide variety of base models have also been tested.\n\n\nWeaknesses:\n1. Although I appreciate this work as an investigation into using SSL for human pose and shape estimation, it might be difficult to justify the significance. Recent works have provided some studies on the impact of backbone selection for human pose and shape estimation [A, B], and training with 2D pose estimation as a proxy task has been shown to be effective for human pose and shape estimation.\n\n2. It is unconventional to use backbone pretraining + head fine-tuning for human pose and shape estimation as fully supervised training has been widely adopted in the field. I agree that it is interesting if pretraining has shown promising results, but there is a significant gap between the evaluated pretraining methods and the SoTA performance. E.g. PARE achieves easily <50 mm PA-MPJPE on 3DPW, but struggles at ~55 mm PA-MPJPE in Table 1. The practical use of this investigation might be questionable.\n\n3. The conclusion on synthetic data might need some adjustments. It has been shown in [C, B] that synthetic data, if used properly, improve baseline models such as HMR, SPIN, VIBE and PARE on real datasets.\n\n4. The semi-supervised learning setting might not be very relevant in today's context as synthetic data incurs almost no cost to scale up. In addition, semi-supervised learning results in a performance gap that is too significant to easily justify its usefulness in real-life applications.\n\n5. There seems to be a slight contradiction in Sec 4.2. It is stated that \"a backbone network is preferred to learn more about human features\" so SSL does not work, but it is also stated that \"high-level semantics ... could be beneficial transferred to inference on humans\". I think it may be helpful to provide a more clear-cut explanation that separates classification pretraining and SSL.\n\n[A] Pang et. al., Benchmarking and Analyzing 3D Human Pose and Shape Estimation Beyond Algorithms, NeurIPS (Dataset Track) 2022\n\n[B] Cai et. al., Playing for 3D Human Recovery, arXiv 2021\n\n[C] Patel et. al. AGORA: Avatars in Geography Optimized for Regression Analysis, CVPR 2021\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BPxgQyIoI3v", "reviewer": "Reviewer_s7cS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an exhaustive empirical study into the effect on performance, for 3D human pose and shape estimation, of different types of pre-training of the backbone network using different training datasets. The different types of pre-training investigated are supervised classification, SSL, and semi-supervised learning and the datasets used vary from large and diverse (ImageNet) to human centric (MSCOCO)  and synthetic. \n\nThe main conclusions from the empirical results are for 3D human pose and shape estimation:\n1) supervised ImageNet pre-training dominates by quite a large margin the standard SSL ImageNet pre-training approaches\n2) Pre-training on smaller but more domain specific tasks (2d keypoint estimation) and data (images of people) can be as effective as pre-training on larger but more diverse datasets.\n\nContributions:\n\n* The paper presents a very thorough and extensive  set of sensible experiments exploring if SSL pre-training should replace the standard practice of using a backbone pre-trained on ImageNet in a supervised fashion for 3D human pose and shape estimation and deciding it should definitely not as this point in time!", "summary_of_the_review": "The empirical results in this paper would be of interest to many. But perhaps, because of the lack of insightful analysis and probing into the reason for the poor performance of SLL pre-training this would cause some hesitation in acceptance at ICLR.\n\nAfter the authors' response:\n\nI appreciate the authors' detailed response to the review and the explanation for the \"relative\" poor performance of SSL pre-training. After reading the response I have increased my rating to an accept. ", "strength_and_weaknesses": "Strengths:\n\n- The paper presents a very nice set of experiments with many settings which I think would be very much appreciated by those working on the problem of 3D human pose estimation.\n\nWeaknesses:\n\n- There is no real investigation as to why SSL does not work as well for pre-training. There is some conjecture in section 4.2 but not much probing experimentally or qualitatively beyond the pre-training conditions and the final performance result. But there is no probing or analyzing the representations learnt in the different set-ups. For example networks trained on ImageNet in a supervised fashion are great at recognising textures whereas self-supervised representations are not good at recognising color. Are such issues coming into play? Is the problem the data-augmentations being used as opposed to SSL? Which augmentations could be problematic and why?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "16o4u7gd9u3", "reviewer": "Reviewer_EdAX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducted an empirical study on the effectiveness of the pre-training of a backbone for the task of 3D human pose and shape estimation (3DHPSE). In the experiments, the authors compared different pretraining strategies, including purely unlabeled self-supervised pre-training, 2D annotation-based pre-training, and synthetic data pre-training. The paper concludes that the self-supervised pre-training on unlabeled images is not effective for 3DHPSE and the 2D annotation-based pre-training brings the largest improvements.\n", "summary_of_the_review": "This paper provides a comprehensive study on the effectiveness of the pre-training for the task of 3DHPSE, which is important to facilitate the research of the 3DHPSE task. The experiments are comprehensive and support the claims of the paper. However, the originality of this work is somewhat limited as parts of the claims have actually been adopted by recent state-of-the-art methods. Given the overall quality and originality, I currently rate this paper as marginally above acceptance.", "strength_and_weaknesses": "Strength:\n\n+ This paper is overall well-written and well-organized. The experimental results on different pre-training settings are convicing and support the claim of the paper.\n+ A comprehensive study and analysis on the effectiveness of the pre-training for the 3DHPSE task is appreciated. The experiments on different self-supervised pre-training settings are new to the community, which is the aspect to support the acceptance of this paper.\n\nWeaknesses:\n- The main concerns regarding this submission are about its practical effects on the 3DHPSE community as the conclusion is actually in line with the practices of recent state-of-the-art methods. More specifically, recent approaches such as Pose2Pose and PARE have used the 2D pose pre-trained backbone in their training strategies. Different pretraining settings of the backbone are also discussed in [Zhang et al.].\nZhang et al., Learning 3D Human Shape and Pose From Dense Body Parts, TPAMI 2020\n\n- The authors only investigate contrastive learning-based methods such as PeCLR for pose estimation. How about self-supervised learning methods on video data, for example [Sun et al. ECCV20, CVPR22]? \nSun et al., Self-Supervised Keypoint Discovery in Behavioral Videos, CVPR 2022\nSun et al., View-Invariant Probabilistic Embedding for Human Pose, ECCV 2020\n\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster