LiftedCL: Lifting Contrastive Learning for Human-Centric Perception
Authors: Ziwei Chen
Qiang Li
Xiaofeng Wang
Wankou Yang
Venue: ICLR 2023 poster (2023)
Keywords: contrastive learning
human-centric perception

Abstract: Human-centric perception targets for understanding human body pose, shape and segmentation. Pre-training the model on large-scale datasets and fine-tuning it on specific tasks has become a well-established paradigm in human-centric perception. Recently, self-supervised learning methods have re-investigated contrastive learning to achieve superior performance on various downstream tasks. When handling human-centric perception, there still remains untapped potential since 3D human structure information is neglected during the task-agnostic pre-training. In this paper, we propose the Lifting Contrastive Learning (LiftedCL) to obtain 3D-aware human-centric representations which absorb 3D human structure information. In particular, to induce the learning process, a set of 3D skeletons is randomly sampled by resorting to 3D human kinematic prior. With this set of generic 3D samples, 3D human structure information can be learned into 3D-aware representations through adversarial learning. Empirical results demonstrate that LiftedCL outperforms state-of-the-art self-supervised methods on four human-centric downstream tasks, including 2D and 3D human pose estimation (0.4% mAP and 1.8 mm MPJPE improvement on COCO 2D pose estimation and Human3.6M 3D pose estimation), human shape recovery and human parsing.

Published as a conference paper at ICLR 2023
LIFTEDCL: LIFTING CONTRASTIVE LEARNING FOR HUMAN-CENTRIC PERCEPTION
Ziwei Chen! Qiang Xiaofeng Wang', Wankou Yang!
School of Automation, Southeast University Key Lab of Mcasurement and Control of Complex Systems of Engineering, Ministry of Education, Southeast University, Nanjing, China 3 Y-tech, Kuaishou Technology Institute of Automation, Chinese Academy of Sciences (r trichard_chheee chen, wkyang ) @seu. edu. liqiang0 (akang00@kua wawkaaaaiuuhuuu.. shou. com wanggxiacfeeg22200i.co
RSB8
20
humen
Human
Downstream task model
Backbone
Head Network
LiftedCL pre-trained
Figure I: The backbone network pre-trained by LiftedCL can be transferred to various human-centric downstream tasks including human pose estimation, human shape recovery and human parsing. The first column shows two examples from the Human3.66 6M (lonescu al. (013) dataset. The second and third columns present the estimated 2D and 3D pose. The last two columns demonstrate the reconstructed human mesh and the estimated human semantic parts.
ABSTRACT
Human-centric perception targets for understanding human body pose, shape and segmentation. Pre-training the model on large-scale datasets and fine-tuning it on specific tasks has become rell-established paradigm in human- centric perception. Recently, self-supervised learning methods have re-investigate contrastive learning to achieve superior performance various downstream tasks. When handling human-centric perception, there still remains untapped potential since 3D human structure information is neglected during the task-agnostic pre-training. In this paper, we propose the Lifting Contrastive Learning (LiftedCL) to obtain 3D-aware human-centric representations which absorb 3D human structure information. In particular, to induce the learning process, a set of 3D skeletons is randomly sampled by resorting to 3D human kinematic prior. With this set of generic 3D samples, 3D human structure information can be learned into 3D-aware representations through adversarial learning. Empirical results demonstrate that LiftedCL outperforms state-of of ttthhe-r art self-supervised methods on four human-centrr downstream tasks, including 2D and 3D human pose estimation (0.4% mAP and 1.8 mm MPJPE improvement on COCO 2D pose estimation and Human3.6M 3D pose estimation), human shape recovery and human parsing.
INTRODUCTION
Human-centrie perception, such as human pose estimation (iiao et al. 2018 Sun et al. 2019 Pav
al. 2019 Gong al. (02T), human shape recovery (Kanazawa et al. 20188] Choi al. 2020 Xu al. 2021 and human parsing (Yang et al.) 2019 Li et al. 2020 Gong al. 2018). has received significant attention in computer vision. Similar to other computer vision tasks, pre-training the
"Corresponding authors.

---

Published as a conference paper at ICLR 2023
model has become widely-used paradigm in human-centric perception. Generally, models are first pre-trained on large-scale datasets (c.g., ImageNet (Deng et al. (2009) and then fine-tuned on specific human-centric downstream task.
For human-centric perception, leveraging 3D human structure information on fine-tuning stage has been demonstrated effective improve the performance. For instance, in the task of 3D pose estimation. RepNet (Wandt& & Rosenhahn 2019) adds a KCS (Wandt et al. 2018) layer into an adversary to better represent bone lengths and joint angles of pose, which achieves more accurate 3D pose reconstruction results. HMR (Kanazawa et al. 2018) employs prior human body model parameterized by shape and 3D joint angles and shows competitive results on 3d pose estimation and part segmentation. In (Qiu et al. 2019), a penalty is be added when the estimated 3D pose has unreasonable limb lengths according to the human body structure prior. Such methods show that leveraging 3D human kinematic prior on fine-tuning stage contributes to the performance. We argue that models can also benefit from 3D human kinematic prior on pre-training stage.
Concurrently, powered by contrastive representation learning, recent self-supervised pre-training methods (Chen et al. 2020a He et al.] 2020; Grill et al.] 2020; Caron et al., (0188) have broken the dominance of supervised ImageNet pre-training on various downstream tasks, including image classification, object detection, semantic segmentation, etc. These self-supervised learning methods mainly adapt the image-level instance discrimination formulation as a pretext task to learn transferable representations. Besides, some methods (Wang et al. 2021a] Xie et al., 2021 Xiao et al. 2021 Wang et al. 2021b) extend the image-level contrastive learning framework to dense paradigm, achieving superior performance on dense prediction tasks.
However, existing contrastive representation learning methods are mainly designed for image classi- fication, object detection and semantic segmentation, rather than human-centric perception. Better results in these tasks may not guarantee superior performance in human-centric perception (see Table 4.. Moreover, most of these works do not link 3D prior to 2D representation learning. In human-centrie perception, there exist some challenging cases, for example, invisible joints, self- occluded keypoints, in which 3D human kinematic prior can be utilized to help better understand the relationship between body parts. Thus, it is still desirable to have pre-training approach for human-centric tasks. Our goal is to improve contrastive learning by leveraging 3D human structure information for human-centric pre-training in simple yet effective way.
To this end, we propose novel contrastive learning framework termed LiftedCL to exploit 3D human structure information for human-centric pre-training. Firstly, we generalize the conventional InfoNCE loss (Oord et al.) 2018) to equivariant paradigm. Based on this, image-level invariant and pixel-level equivariant contrastive learning are applied to the projected feature vectors and maps respectively. Meanwhile, the representations the hidden human structure information. In particular, a set of 3D skeletons is randomly sampled by resorting to 3D human kinematic prior. With this set of real 3D samples, an adversary is adopted to induce the learning of 3D-aware human-centric representations.
transformed into 3D human skeleton to better reveal
We demonstrate the effectiveness of our proposed LiftedCL by pre-training using MS COCO (Lin cet al. 2014) human images and fine-tuning on specific target dataset. Compared to the state-of-the-un method PixPro (Xie et al., 202T), LiftedCL achieves significant improvements on various human- centric downstream tasks, including COCO 2D human pose estimation (+0.4% mAP), MPII 2D human pose estimation (+0.3% PCKh@0.5), Human3.66 6.6M 2D human pose estimation (+0.9% JDR), Human3.66 6. 3D human pose estimation (1 (1.8mm MPJPE), 3DPW human shape recovery (1.7mm reconst. error) and LIP human parsing (+0.5% mloU).
Our main contributions are summarized as follows:
We propose the Lifting Contrastive Learning (LiftedCL) for human-centric pre-training in simple yet effective way.
We demonstrate feasible approach to learn 3D-aware representations via lifting and adversarial learning only using single-view images.
LiftedCL significantly outperforms state-of- oftte-of-th-arr the-art self-supervised learning methods on four human-centric downstream tasks, including 2D and 3D human pose estimation (0.4% mAP and 1.8 mm MPJPE improvement on COCO 2D pose estimation and Human3.66 3.. 3D pose estimation), human shape recovery and human parsing.
2

---

Published as a conference paper at ICLR 2023
Encoder
features
Contrastive Loss:
Projector
2 nnv equiv
Liav + Lequiv
Backbone
3D skeleton
Lifting Network
3D 3D-wwa representations
X
Randomly sampled 3D skeletons Y
Discriminator
real/fake 3D skeleton
Adversarial Loss: ladv
Figure 2: Overall architecture of the proposed LiftedCL. The LiftedCL consists of an encoder and a discriminator. The encoder takes input view to generate invariant feature vector Lives equivariant feature maps Legu and 3D human skeleton X. Generated 3D human skeleton fed into the discriminator to tell whether it is real. Finally, the encoder is optimized by combination of contrastive loss and adversarial loss.
2 RELATED WORKS
Centric Perception. With the development of deep learning, human-centric perception has achieved great progress in recent years. Pose estimation, a fundamental yet challenging problem in computer vision, is to localize human anatomical keypoints (e.g., head, shoulder, wrist, etc.) or parts in given image 3D space. Lots of work have achieved accurate and robust performance in 2D pose estimation (Xiao et al. 2018 [Sun et al., 2019)) Yang et al. 202T). For 3D pose estimation (Pavllo be roughly grouped into two categories: one-stage approaches which directly learn the 3D poses from images and two-stage approaches which first estimate 2D poses and then lift them to 3D poses. In this work, focus on the first setting because existing self-supervised learning approaches learn representations from images rather than 2D poses. For 3D human shape recovery (Xu al. 2021, Choi et 2020 Kanazawal et al. 2018). 3D human body is usually represented by statistical model (e.g. SMPL (Loper et al. 2015)) and the task to estimate the parameters of the statistical model. Human parsing (Yang et al. 2019 Li et al. 2020) Gong et al. 2018) is to assign each image pixel from the human body to semantic category. Region analysis and spatial correlation are usually considered to segment body parts. Our goal is to learn human-centric representations for downstream tasks transfer.
al. 2019 Gong al. 2021 Wandt & Rosenhahn (019), approaches
Self-supervised representation learning. Self-supervised representation learning, a kind unsu- pervised learning, has driven significant progress in recent deep learning research. It aims to learn informative and transferable representations for various downstream tasks. Early self-supervised learning approaches explore wide range of pretext tasks to learn good representation. including spatial jigsaw puzzles (Noroozi & Favaro, 2016)), rotation prediction (Gidaris et al.] 2018 Chen al. 2019), colorization (Zhang al.) (016)) and
(Doersch et al. 20155) Pathak et al. 2016).
However, these approaches achieve very limited success in computer vision and the rest-performance approach on a specific task would be sub-optimal on another downstream task.
Recently, contrastive learning has been demonstrated to show incredible promise in computer vision. SimCLR (Chen et al. 2020a) is one breakthrough approach, which adopts the instance discrimination formulation as its pretext task. By applying a diverse set of data augmentations to training images, generated views from the same input are considered as positive pairs while views from different input images as dissimilar pairs It maximize the similarity in latent space between positive pairs and repelling dissimilar pairs. Besides, MoCo views the contrastive learning as dictionary look-up and introduces momentum update mechanism to ensure the consistency of negative samples. Moreover, some methods (Caron et al. 2018 2020 2019) extend instance- discrimination to predict the cluster assignment. Nonetheless, these approaches are mainly designed for image classification and can be sub-optimal for dense prediction tasks.
To fill the gap between image-level and pixel-level prediction, several approaches (Wang al. 2001aa Xie et al., 220T Wang et al.) 20216; O Pinheiro et al. 2020) are proposed to explore dense contrastive representation learning. DenseCl. (Wang al. 202Ta) extends and generalizes the
3

---

Published as a conference paper at ICLR 2023
Invariant Projector
Invariant Contrastive Loss
(4))
Equivariant Projector
Equivariant Contrastive Loss
momentum update
Invariant Projector
4
mxx
Backbone
(t,);
Equivariant Projector
I
f
F2
Figure 3:
Contrastive representation learning in our LiftedCL. We perform invariant and equivari- ant contrastive learning 3D-aware representations generated by backbone network.
existing MoCo framework to dense paradigm. VADeR Pinheiro et al. 2020 and PixPro (Xie iet al.) 2021) map corresponding pixel-wise features in each view to their associated features according to the parameters of the affine transformation. SetSim (Wang al.. (0211b explore set similarity across views for more robust representation learning. Prior3D (Hou et al. 2021) leverages multi-view geometry information augment contrastive learning. While these methods achieve promising results on image classification, object detection and semantic segmentation, there still remains untapped potential since 3D human body structure information is neglected during pre- pre-training. HCMoCo (Hong et al., (022) propose versatile pre-train model to leverage sparse human body structure priors for comann-centrie tasks, however, multi-modal data is required during training which expensive and difficult to collect. In this work, we introduce LiftedCL., a simple yet effective approach for human-centric pre-training which only requires single-view data such as COCO human images.
METHOD
3.1
OVERALL ARCHITECTURE
The overall architecture of our proposed LiftedCL, is shown in figure 2 it is composed of an encoder and discriminator. The encoder consists of a backbone, two projectors and a lifting network. Firstly, given input view, the representations are extracted by the backbone network, e.g., ResNet (He et al.) 2016), and are fed into two projection heads to generate invariant feature vectors Zivv and equivariant feature maps Fequiv Cheng et al. (2021) 1); Feige (2019). Upon them, an invariant contrastive loss function Line (He et al. 2020) and an equivariant contrastive loss function Lequie are adopted for representation learning.
Parallel to the projectors, we attach a lifting network to transform the representations into 3D human skeleton. To incorporate 3D human structure information into the representations, improving them to become 3D-aware, a discriminator trained alternatively to distinguish between lifted 3D skeletons and real 3D skeletons. Note that real 3D skeletons be randomly sampled using human kinematic priors without any annotated data (see appendix B). In all, the whole encoder can be trained by optimizing a combination of contrastive similarity loss and adversarial loss as below.
L= A (Line + Leguiv) + Ag LaLd:
(1)
where X and A controls the relative importance of each loss function. We set A and X2 to 0.5.
3.2 CONTRASTIVE REPRESENTATIONS
As illustrated in Figure 3 we perform contrastive learning on 3D-aware representations generated by backbone network to obtain good generic representations. The contrastive learning framework be divided into two parts, invariant contrastive learning and equivariant contrastive learning.
For image-level invariant contrastive learning, follow (He et al. 2020 to build the learning framework. As shown in Figure 3 given an image I, two augmented views {T.. I&} can be generated by applying various augmentations I, where I (T),), Tt (T), by E T. Each view is first fed
4

---

Published as a conference paper at ICLR 2023
(a)
Pooling
m blocks
Residual
Residual
Vector (1,
Layer
Block
(b)
Concal
FC
Real Not real
skeleton (4.
Skip
Resicual
Block
Rully layor
Figure 4: (a) The architecture of lifting network. 3D-aware representations are first vectorized by convolutional and pooling layers and then lifted 3D by residual blocks and fully connected layers. (b) Network architecture of the discriminator, consisting of KCS layer and several residual blocks.
into a backbone network f to extract representations and then fed into the invariant projection head to generate feature vector 2. Here, { } be encoded as {zy. 2k Rd by encoder and its comeetuu-uudatted one. Feature vectors generated from two views the same input are considered positive pairs, while negatives encoded from different images. A standard contrastive loss function is employed to pull bg close to its positive key 2k, while pushing it away from negative keys,
explzz Zk 2TT
Line -log
explzn Ak (///T) + E=n. exp(zy Zh /T)
where 7 is the temperature hyper-parameter and z comes from queue.
For equivariant contrastive learning, feature map Fg E Cx xS.. and its positive pair F& are generated from two views Iy. J& of the same input image by convolutional projection head, then inverse geometric augmentation (44)²² 1, (lx) 1 is applied separately. Here (Lg)s is the geometric transformation of ty and the other is the appearance transformation (fg)a- Let define inverse geometric augmented feature maps F, (Lz) Fe (I+) and they be as S₁ x Sw feature vectors. Let 7 denotes the 5th out of S₁ x Sw, encoded vectors from F, and Fx,, and here we use ST - Sw for simpler illustration. If inverse geometric augmentation is not applied before calculating the loss function, the model needs to be invariant to all geometric transformations T. However,
perception requires equivariance with respect to transformations as these change human poses in augmented views. Moreover, positive pairs can be easily found when applying inverse geometric augmentation as they share the same spatial location. Thus, we define the equivariant loss function as
Lequiv
1
Σ
-log
log
exp(f" /2/T) exp(F= h + Ee e F2/T)*
(3)
S2
where 7e is the pooled feature vector of F5 of view from different image.
3.3 LIFTING WITH ADVERSARIAL LEARNING
Our LiftedCL leverages 3D human body structure information. As shown in Figure 4 lifting network employed to transfer representations into 3D human skeleton. The lifting network mainly composed of several residual blocks consisting of fully connected layers. For given image R 3x where stands for the number joints. Adversarial training used to guide the lifting network outputting correct 3D skeleton, thus encoding the 3D human structure information into the representations, making it 3D-aware.
I, the lifting network extracts 3D skeleton X
The architecture of the discriminator is similar to the lifting network, consisting of fully connected layers, shown in Figure 4 Moreover, to better detect properties of human skeletons such as kinematic chains, bone lengths and joint angle limits, kinematic chain space (KCS) Wandt & Rosenhahn (2019): Wandt et al. (2018) layer added into our discriminator. KCS can represent joint angles and bone lengths of human skeleton. We define bone by as the vector between the r-th and t-th joint.
by
Pr Pt
X

---

Published as a conference paper at ICLR 2023
where
(0,, .0...... 0, ..,0....,,,,,..........
(5)
with at position r and -1 position Then matrix B gaxb can calculated by extending to all bones.
B = (61 bg..... bb) = -XC,
(6)
here C e Rxxb is the number of bones. By calculating B" B, KCS matrix can be obtained
12
V B B-
Each element its diagonal represents the bone length and other elements represent the relation between two joint vectors. KCS layer helps the discriminator detect 3D human structure information more effective way. The complete network trained alternatively with the diseriminator. Let E represent the encoder including the backbone network, projection heads and the lifting network, and D represent the discriminator, then the loss function
Ladv E. Eys-paacets) [D()) + E, Ex- (P)((T)) 1)
By limiting bone length and joint angle, we randomly sample several rational skeletons y. So the whole network can be trained in an unsupervised way under the adversarial learning paradigm.
EXPERIMENTS
4.1 PRE-TRAINING SETUP
Our pre-training experiments are conducted on MS COCO (Lin et al. 2014) and only the training set used for pre-training. In human-centric perception, human bounding boxes are usually provided, so we crop human regions to better learn representations, resulting 150K images for pre-training. We adopt SGD as the optimizer with initial learning rate of 0.03 and we set its weight decay and momentum 0.0001 and 0.9. Each pre-training model optimized on 4 GPUs with a cosine learning rate decay schedule and mini-batch size of 128 for 200 epochs (details in appendix(C).
However, most existing sate-of-the--ne art self- superviedd learning methods are pre-trained on ImageNet (Deng al. (2009), pre-trained on ImageNet. Stage The encoder is then pre-trained using COCO human images. For efficiency, download their official 200-epoch ImageNet pre-trained weights to initialize the encoder as stage I pre- vveetraining. For our LiftedCL, we adopt the MoCo-v2 ImageNet 200-epoch pre-trained weights for encoder initialization. Note that MoCo-v2 ImageNet 200-epoch pre-trained weights show inferior performance compared to other SoTA methods, while our LiftedCL. still surpasses other methods using it initialization..
conduct two-stage pre-training for fair comparison: Stage I: The encoder
4.2
MAIN RESULTS AND DISCUSSION
We compare our LiftedCL with other state-of-the-art self-supervised learning methods on four human- centric tasks, including 2D pose estimation, 3D pose estimation, 3D human shape recovery and human parsing. See more details appendix
2D Human Pose Estimation. We fine-tune SimplePose (Xiao et al. 2018 detector with ResNet50- C4 backbone on MS COCO (Lin al., 2014). MPIT (Andriluka al. 2014) and Human3.66 3.6M [Ionescu et al.) 2013). As illustrated in Table our LiftedCL outperforms the state-of-the-art method PixPro by 0.4% mAP on COCO human pose estimation. Table 2 shows that our LiftedCL surpasses PixPro by 0.4% P.Kh@0.5. Fine-tuning results Human3.6m 3.6M 2D pose estimation is shown in Table our LiftedCL obtains significant improvement by 0.9% JDR than PixPro.
3D Human Pose Estimation. The 3D human pose estimation results Human3.6M are reported in Table We follow the settings in (Li et al. 202I) to fine-tune a ResNet50 based model. Our LiftedCL outperforms PixPro by significant improvements, 1.8mm and 0.8mm at MPJPE and PA-MPIPE.

---

Published as a conference paper at ICLR 2023
Table I: 2D Pose Estimation fine-tuned MS COCO. A ResNet-50 based model Xiao al. (2018) adopt for all methods. 'IN' indicates the model pre-trained on ImageNet. 'IN+CC' indicates two-stage pre-training on ImageNet and COCO. *** indicates using PixPro ImageNet pre-trained weights as stage pre-training. Each model is pre-trained for 200 epochs.
pre-train
mAPP
AP30
AP73
APM
APL
AR
Random init.
69.1
90.5
77.1
66.7
73.5
72.4
Super. IN
70.4
88.6
78.3
67.1
77.2
76.3
MoCo-v2 IN
Chen et al.
2020b)
70.3 (-0.1)
88.6
77.9
67.0
77.1
76.2
DenseCL IN
Wang
a
70.6 (+0.2)
88.7
78.2
67.4
77.3
76.4
ReSim-C4 IN Xıao ct al.
(2021)
70.4 (+0.0)
88.5
77.9
67.1
77.2
76.3
PixPro IN Xie
(202]
70.7 (+0.3)
88.7
78.2
67.2
77.7
76.6
MoCo-v2 IN+CC
70.5 (+0.1)
88.8
78.2
67.1
77.5
76.5
PixPro IN+CC
70.7 (+0.3)
89.2
78.2
67.4
77.5
76.5
Ours IN+CC
70.9 (+0.5)
88.9
78.1
67.5
77.7
76.8
Ours IN+CC
71.1 (+0.7)
89.0
78.3
67.6
77.8
76.8
Table 2D Pose Estimation fine-tuned on MPI. (PCKh@0.5)
pre-train
Hea
Sho
Elb
Wri
Hip
Kne
Ank
Total
Random init.
96.3
94.4
87.0
80.5
86.8
81.2
76.4
86.7
Super. IN
96.5
95.4
88.9
83.5
88.1
89.7
79.1
88.5
MoCo-v2 IN
96.4
95.2
89.0
83.3
87.5
83.3
79.1
88.3 (-0.2)
DenseCL IN
96.8
95.4
89.0
83.7
88.1
83.9
79.6
88.6 (+0.1)
ReSim- C4 IN
96.7
95.4
89.1
83.7
87.9
84.1
79.0
88.6 (+0.1)
PixPro IN
96.7
95.6
89.4
83.8
88.3
84.6
80.4
88.9 (+0.4)
MoCo-v2 IN+CC
96.6
95.4
88.9
84.1
88.0
83.8
79.3
88.6 (+0.2)
PixPro IN+CC
96.6
95.6
89.3
83.8
88.5
84.7
81.0
89.0 (+0.5)
Ours IN+CC
96.8
95.6
89.4
83.9
88.5
84.2
79.6
88.9 (+0.4)
Ours* IN+CC
96.7
95.5
89.3
83.9
88.4
84.4
79.9
89.3 (+0.8)
2021) to fine-tune a ResNet50 based model on a mixed dataset. The 3D human shape recovery results evaluated on 3DPW (von Marcard et al. 2018) are reported in Table MoCo-v2 baseline has no advantage over supervised ImageNet pre-traiming. Our LiftedCL outperforms state-of-thee thetttr-rrt method PixPro by 1.7mm reconstruction error.
3D Human Shape Recovery. We follow the settings in (Xu et al.
2018 are reported in Table 6 Human Parsing. The human parsing results on LIP (Liang et
ResNet50 based model with self- ceff-correction purification strategy (Li et al 2020) is fine-tuned. Our LiftedCL outperforms PixPro by 0. 1% mACC and 0.5%mloU.
Discussion. LiftedCL. achieves significant improvements, especially on Human3.6N 3D pose estimation. When pre-trained on ImageNet, existing self-supervised learning methods do not show superior performance than supervised pre-training except PixPro. Concurrently, when adapting two- stage pre-training our LiftedCL surpasses PixPro by a large margin, which strongly demonstrates that combining contrastive learning with 3D human prior is beneficial to human-centric tasks.
Besides, self-supervised learning methods bring different performance improvements on downstream tasks, which can be related to the diversity and quantity of fine-tuning data. COCO and MPII consists of large number of images taken from wide-range of real-world activities, fine-tuning a pre-trained model on them only achieves limited improvements. For other datasets like Human3.6M 3DPW, which is collected a controlled scene or contains a limited number of human subjects, fine-tuning results benefit lot from pre-training, especially our LiftedCL.
4.3 ABLATION STUDY AND VISUALIZATION
We conduct ablation experiments on Human3.6M, 2D and 3D pose estimation to show how LiftedCL works. SimplePose (Xiao et al.) 2018) and Joint Detection Rate (JDR) is used for 2D human pose estimation, RI.EPose (Li 2021 and PA-MPJPR is used for 3D human pose estimation.
7

---

Published as a conference paper at ICLR 2023
Table 3: 2D Pose Estimation fine-tuned on Human3.6M. (JDR)
Table 4: 3D Pose Estimation fine-tuned Human3.6M, (MPJPE and PA-MPIPE mm)
pre-train
root
rhip
Total f
pre-train
MPJPE +
PA-MPJPE +
Random init.
97.1
86.4
78.7
Random init.
70.8
52.8
Super. IN
94.3
89.7
85.0
Super. IN
60.0
44.7
MoCo-v2 IN
95.8
89.6
85.6 (+0.6)
MoCo-v2 IN
61.0
46.3 (+1.6)
DenseCL IN
93.5
86.0
85.7 (+0.7)
DenseCL IN
65.1 (45.1)
46.7 (+2.0)
ReSim-Co IN
96.1
90,9
86.0 (+1.0)
ReSim C4 IN
63.1 (43.1)
47.0 (+2.3)
PixPro IN
96.0
91.3
86.4 (+1.4)
PixPro IN
58.9 (-1.1)
43.7 (-1.0)
MoCo-v2 IN+CC
96.2
91.8
86.2 (+1.2)
MoCo-v2 IN+CC
61.2 (+1.2)
45.4 (+0.7)
PixPro IN+CC
95.9
92.0
87.1 (+2.1)
PixPro IN+CC
58.7 (-1.3)
43.3 (-1.4)
Ours IN+CC
95.5
91.9
88.0
(+3.0)
Ours IN+CC
56.9 (-3.1)
42.5 (-2.2)
Table Human shape recovery fine-tuned on 3DPW.
Table Human parsing fine-tuned on LIP.
pre-train
Reconst, error +
pre-train
m.CCC f
mIoU
Random init.
116.3
Random init.
63.5
50.3
Super. IN
106.1
Super. IN
63.7
51.8
MoCo-v2 IN+CC
111.5 (45.4)
MoCo-v2 IN+CC
64,9 (+1.2)
53.0 (+1.2)
PixPro IN+CC
104.5 (-1.6)
PixPro IN+CC
67.5 (+3.8)
54.6 (+2.8)
Ours IN+CC
102.8 (-3.3)
Ours IN+CC
67.6 (43.9)
55.1 (43.3)
Pre-train dataset. In Table 7 we show the results of different pre-training datasets. When only using Human3.6M, LiftedCL improves the performance by 2.7% JDR and 4.9 mm PA-MPJPE compared to random initialization. Using COCO brings 6.9% JDR and 8.4 mm PA-MPJPE improvements, which is superior than supervised ImageNet pre- preitirin Note that COCO pre-training only uses 1/7 images and less iterations. As for the results two-stage pre-training, surpasses the ImageNet supervised pre-training method by 3.0% JDR and 2.2 mm PA-MPJPE.
Loss function. Our loss function composed three parts: Tines Leguiv and Tradv- To show how each component contributes to LiftedCI., we pre-train the backbone using different loss combinations. We report the results Table We refer to the ImageNet pre-training method as our baseline. shows that invariant contrastive loss improves the baseline by 1.2% JDR and -0.7 mm PA-MPJPE. Adding the equivariant loss brings another 0.9% JDR and 1.1 mm PA-MPJPE gains. When all three losses are used, the pose estimation performance improves for 3.0% JDR and 2.2 mm PA-MPJPE compared to the baseline. Meanwhile, directly applying the lifting network does not bring notable improvements. It may be because lifting network would collapse without good representations learned by contrastive learning.
Table 7: Ablation study of pre-training dataset.
Pre-train method
Pre-train dataset
epoch
JDR 1
PA-MPIPE
Random init.
78.7
52.8
Supervise
ImageNet
100
85.0
44.7
MoCo-v2
H36M
200
79.8 (-5.2)
49.6 (+4.9)
MoCo-v2
COCO
200
(-1.4)
45.9 (+1.2)
MoCo-v2
ImageNet
200
85.6 (+0.6)
46.3 (+1.6)
MoCo-v2
ImageNet COCO
200
86.2 (+1.2)
45.4 (+0.7)
Ours
H36M
200
(-3.6)
47.9 (+3.2)
Ours
COCO
200
85.6 (40.6)
44.4 (-0.3)
Ours
ImageNet COCO
200
88.0 (+3.0)
42.5 (-2.2)
8

---

Method Published as a conference paper at ICLR 2023
Table Ablation study of different loss fune- tions.
Table 9: Ablation study of fine-tuning w/o 3D prior.
JDR
PA-MPIPE
Method
JDR
PA-MPJPE &
Supervise
85.0
44.7
Supervise IN
85.0
44.7
Inv.
86.2
45,4
+ prior
86.1
44.2
Inv. + Equiv.
86.9
44.3
Ours
88.0
42.5
Lifting
85.2
44.8
+ prior
88.7
42.3
Inv. + Equiv. + Lifting
88.0
42.5
Input image
Lifted rep.
Input image
Lifted rep.
Input image
Lifted rep.
Figure 5: visualization of 3D skeletons generated by lifting network.
3D prior. We further investigate the effect of 3D prior. When fine-tuning with 3D human kinematic prior, we additionally train a lifting network to estimate 3D pose. Based on the estimated 3D pose, the bone length and joint ankles be calculated and if they not within the reasonable range, an extra loss will be added to penalize the incorrect estimation. As shown in Table 9 fine-tuning with 3D human kinetic prior improves the JDR 1.1 % and PA-MPJPE by 0.5 mm, demonstrating the effectiveness of 3D human kinetic prior. However, even when fine-tuning without 3D prior, our LiftedCL surpassed that by 1.9% JDR and 1.8 mm PA-MPJPE, which shows that our LiftedCL can leverage 3D human kinetic prior more effectively. Moreover, our LiftedCL can also benefit from 3D prior when fine-tuning, which is in accordance with our intention that LiftedCL is designed for pre-training and 3D prior can help on both pre-traine pre-training and fine-tuning stage.
3D skeleton visualization. We visualize the 3D skeletons generated by lifting network in Figure 5 Given an input view, generated 3D skeletons can roughly reveal the human skeleton. It is in accordance with our intention that human body structure information learned during pre-training. Mcanwhile, the generated 3D skeletons is not very accurate annotated data is used and only an adversarial loss is applied for 3D-aware human-centric representation learning.
CONCLUSION
In this work, we proposed a simple yet effective self-supervised representation learning framework LiftedCL for 3D-aware human-centric pre-training. By lifting 2D contrastive representations to 3D human skeleton and adapt adversarial learning. LiftedCL encodes 3D human structure information into the learned representations. Besides, with 3D human kinematic prior, real 3D skeletons be generated by randomly sampling reasonable 3D human skeletons, which alleviates the problem of expensive and time-consuming data collection. Our method shows promising improvements in a serious of human-centri tasks, and more importantly, shows the great potential of improving contrastive learning with 3D prior through adversarial learning. Our limitation that cropped human images are required during pre-training. In the future, we will further investigate how to embed 3D prior into representations with aigle-view input on object detection.

---

Published as a conference paper at ICLR 2023
6 ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Nos. 62276061.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pp. 3686-3693, 2014.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Piotr Bojanowski. Julien Mairal, and Armand Joulin. Unsupervised pre- tr-iiiniiig of image features non-curated data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2959-2968, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances Neural Information Processing Systems, 33:9912-9924, 2020.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12154-12163, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.00 04297, 2020b.
Zezhou Cheng, Jong-Chyi Su, and Subhransu Maji. On equivariant and invariant learning of object landmark representations. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9897-9906, 2021.
Hongsuk Choi, Gyeongsik Moon, and Kyoung Mu Lee. Pose2mesh: Graph convolutional network for 3d human pose and mesh recovery from 2d human pose. In European Conference on Computer Vision, pp. 769-787. Springer, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. Proceedings of the IEEE international conference on computer vision, pp- 1430, 2015.
Ilya Feige. Invariant-equivariiat representation learning for multi-class data. In International Conference on Machine Learning, pp. 1882-1891. PMLR, 2019.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 2018
Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang. and Liang Lin. Instance-level human parsing via part grouping network. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 770-785, 2018.
Kehong Gong, Jianfeng Zhang, and Jiashi Feng. Poseaug: A differentiable pose augmentation framework for 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2575-8584 2021.
Jean-Basee Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to supervieed learning. Advances in Neural Information Processing Systems, 33:21271-2122 2020.

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-97227 2020.
Fangzhou Hong. Liang Pan, Zhongang Cai, and Ziwei Liu. Versatile multi-modal pre-training for human-centric perception. arXiv preprint arXiv:2203. 13811 2022.
Hou, Saining Xie, Benjamin Graham, Angela Dai, and Matthias Nießner. Pri3d: Can 3d priors help 2d representation learning? In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 3693-5702, 2021.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m 6.6: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and muchine intelligence, ((677):1325-333)) 2013.
Angjoo Kanazawa, Michael Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7122-712 7131, 2018.
Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu. Human pose regression with residual ligeliieldhood estimation. In ICCV, 2021.
Peike Li, Yunqiu Xu, Yunchao Wei, and Yi Yang. Self-correction for human parsing. IEEE Transactions Putlern Analysis and Machine Intelligence, 2020.
Xiaodan Liang, Ke Gong, Xiaohui Shen, and Liang Lin. Look into person: Joint body parsing & pose estimation network and a new benchmark. IEEE transactions on pattern analysis and machine intelligence, 41(4):871-88) 885, 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference computer vision, pp. 69-84. Springer, 2016.
Pedro O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsupervised learning of dense visual representations. Advances Neural Information Processing Systems, 13:4489-45000 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807. 03748, 2018
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2536-2544, 2016.
Dario Pavilo, Christoph Feichtenhofer, David Grangier, and Michael Auli 3d human pose estimation in video with temporal convolutions and semi-supervised training. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7753-7762, 2019.
Haibo Qiu, Chunyu Wang, Jingdong Wang, Naiyan Wang, and Wenjun Zeng. Cross view fusion for 3d human pose estimation. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 4342 4351, 2019.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pp. 5693-5703 2019.
12

---

Published as a conference paper at ICLR 2023
Timo von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll Recovering accurate 3d human pose in the wild using imus and a moving camera. Proceedings of the European Conference Computer Vision (ECCV). pp. 601-617, 2018.
Bastian Wandt and Bodo Rosenhahn. Repnet: Weakly supervised training of adversarial repro- jection network for 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7782-7791, 2019.
Bastian Wandt, Hanno Ackermann, and Bodo Rosenhahn. A kinematic chain space for monocu- lar motion capture. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for sule-ssee visual pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3024-3033, 2021a.
Zhaoqing Wang, Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang, Mingming Gong, and Tongliang Liu. Exploring set similarity for dense self self- super-ueee representation learning. arXiv preprint arXiv:2107.08712: 2021b.
Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV), pp. 466-481, 2018
Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 10539-10548, 2021.
Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16684-16693, 2021.
Xiangyu Xu, Hao Chen, Francese Nogune,Noguer, Laszlo A Jeni, and Fernando De la Torre. 3d human pose, shape and texture from low-resolution images and videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Lu Yang, Qing Song, Zhihui Wang, and Ming Jiang. Parsing r-enn for instance-level human analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 364-373 2019.
Sen Yang. Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Keypoint localization via transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11802-11812, 2021.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pp. 649-666. Springer, 2016.
13

---

Published as a conference paper at ICLR 2023
A GENERATION OF COCO HUMAN IMAGES
Our pre-training experiments on COCO only use human images, which are generated by cropping human regions according the annotations. For each human bounding box, the box is first extended to fixed aspect ratio, i.e., height: width 4 : 3, and then we crop from images without distorting human bounding boxes' aspect ratio. Finally, resize the cropped image fixed size, 256 192. Figure 6 shows some examples of our cropped human images.
TOP
Figure 6: Cropped human images from MS COCO.
B SAMPLING OF REAL 3D SKELETONS
We generate a set of real 3D samples for adversarial learning by randomly sampling reasonable 3D human skeletons satisfying 3D human kinematic prior without any annotated data. We define a 3D human skeleton
P (Jroot: : Bright pelvis Iright ... Jie fte (wrist) 3
where J stands for the number of joints and we use 17 joint human skeleton. We first set root joint Jroot (0, 0, 0)7 and then any other joint be generated using its parent joint and bone vector.
jehaid Sporent R bparent_ehiil-
(10)
Jehita and jparent represent the child joint and its parent joint separately. bpc E 3 is the default bone vector related to these two joints. We first define a template 3D skeleton. In this way, each default bone vector 6pe can be obtained according to that. is the random length ratio and R is the random rotation matrix to change the bone length and direction. We set a E [0.9, 1.1] for all bones and set different rotation ranges for different bones according the human kinematic prior. For example, for the knee joint, Dpelerss_neee = (0, 45, 0)T. a E [0.9, 1.1] and R culer(a, b, c), a e [-902, 909), d [-100 180"), c [-90P, 90%
The 3D human skeleton can generated by uniformly sampling a and R in their ranges. For the generated 3DD skeleton, it is scaled by dividing it by its Frobenius norm. To simulate different camera
14

---

Published as a conference paper at ICLR 2023
42
-1A
62
34 -22
36
44
00
24
92
14
98
-44
W
60
24
Figure 7: Randomly sampled 3D human skeletons.
views, generated 3D skeletons are randomly rotated around Z axis. We generate a set of 10K 3D human skeletons real 3D samples for adversarial learning in LiftedCL. Figure shows some examples of our randomly sampled 3D skeletons.
C IMPLEMENTATION DETAILS OF LIFTEDCL
default backbone, is worth noting that approach can be employed to other networks, e.g. HRNet (Sun et al. 2019). Following (Chen et al. 22000a] He et 2020), invariant projection head consists of global average pooling layer and MLP, which takes the feature maps as input and generates a global feature vector for each view. For equivariant projection head, it outputs features maps having the same size of the input consisting of two 1 x 1 convolution layers with ReLU layer between them. The first convolution layer's 's dimension is 2048, and the final output dimension is 128. For both the invariant and equivariant contrastive learning, the dictionary size is set to 16384. The momentum is set to 0.999, Shuffling BN (He et al. 2020) is used during training. The temperature 7 in contrastive loss set to 0.2.
We adopt ResNet50 (He al. 2016)
layer
The data augmentation pipeline consists of 256 x 256-pixel random resized cropping, random color jittering, random geay-seale conversion, gaussian blurring and random horizontal flip following (Chen) al. 2020b). Rotation is not used for fair comparison with previous self supervisdd methods. Inverse geometry transformations are applied in equivariant contrastive loss, the output feature maps will be flipped horizontally if the input view area in two views' feature maps F,, F1 will be cropped and resized to 16 x 16 resolution as inverse geometric augmented feature maps, then equivariant contrastive loss can be calculated. The training code of our framework is modified from the official Py Torch implementation of MoCo
flipped before. For random resized cropping, shared
15

---

Published as a conference paper at ICLR 2023
D FINE-TUNING DETAILS
2D Human Pose Estimation. We tune SimplePose (Xiao et al. 2018) detector with ResNet50- 2014) and Human3.6M (Ionescu et al. 2013). The learning schedule follows the setting (Xiao et al. 2018).
C4 backbone on MS COCO (Lin al. 2014), MPII (Andriluka
For 2D human pose estimation on COCO, adapt the official codes of SimplePose The fine- tuning is conducted on COCO train2017 dataset for 140 epochs with a mini-batch size of 128, including 57K images and 150K person instances. The evaluation is conducted on the va/2017 set, containing 5000 images. The standard evaluation metric is based on Object Keypoint Similarity (OKS). We follow the evaluation metric (Sun al. 2019) and report standard average precision and recall scores: AP50 (AP at OKS = 0.50), APrs, AP (the mean of AP scores at 10 positions, OKS 0.50, 0.55. 0.90, 0.95), APM for medium objects, APL for large objects, and AR at OKS 0.50, 0.55, o.., 0.90, 0.95.
When fine-tuning SimplePose on MPII Human Pose dataset, we also use its official codes. MPII consists of 28K subjects for training and 12K subjects for testing. The standard metric PCKh@0.5 (a = 0.5) score nermalzormllized probability of correct keypoint) score used.
For Human3.0 6M 2D pose estimation, we refer the official codes in QQuu et al. 2019 and follow its default training schedule. Human3.66 3.. dataset contains ~ 63K images for training and ~ 9K images for testing. The pose estimation accuracy is measured by Joint Detection Rate (JDR), which means the percentage of the successfully detected joints. The estimated joint is regarded as successfully detected if the distance between it and the groundtruth location is smaller than a half of the head size.
3D Human Pose Estimation, For 3D human pose estimation Human3.6M, we fine-tune a model with the ResNet50 backbone (Li al. 2021) using default training schedule" on subjects S1, 5, 7, 8 and evaluate on subjects S9 and SI1. We use two evaluation metries: Mcan Per Joint Position Error (MPJPE) millimeters and MPJPE over aligned predictions with GT 3D poses by rigid transformation (PA-MPJPE).
3D Human Shape Recovery. We follow the training schedule and settings in (Xu al. 2021
to fine-tune a ResNet50 based model on mixed dataset. The 3D human shape recovery results are evaluated on 3DPW (von Marcard et al. 2018 and reconstruction error is used to evaluate the performance.
Human Parsing. A ResNet50 based model with self-correction purification strategy (Li al. 2 is fine-tuned LIP dataset. LIP dataset contains 50462 images with pixel-wise annotations with semantic human part labels. We fine-tune pre-trained models on Irain set and evaluate the performance validation set, consisting 30462 images and 10000 images respectively.
E ADDITIONAL VISUALIZATION OF LIFTED 3D SKELETONS
Additional visualization of lifted 3D skeletons by lifting network is shown in Figure g
"htrp:://githubb.. ttans://gtuub.com//fff www.://defiiiiuioooooooooooooeoo
Cororeeioo-oooooo----o Human-Parsiiee
16

---

Published as a conference paper at ICLR 2023
Input image
Lifted rep.
Input image
Lifted rep.
Input image
Lifted rep.
ON SNN LEGON 1000
Figure 8: Additional visualization of lifted 3D skeletons.
17


Reviews:
[{"review_id": "PVQ616sWoL", "reviewer": "Reviewer_1MfS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper proposed a novel architecture for self-learning of human-centric perception problems: 2D/3D human pose estimation, human shape recovery and human parsing. The proposed method is simple and effective -- exceeding SOTA for several benchmarks. The encoder subnetwork uses Contrastive Learning and Lifting network to generate the 3D skeleton of the input; this is then combined with Adversarial loss from Discriminator sub-network to improve the encoder. \n\n", "summary_of_the_review": "Overall, paper is a good piece of work which should be accepted for ICLR. There are aspects of the work which are incremental, e.g. using of Contrastive Loss, 3D structure representation of human. However, the scientific contributions of it are still sufficiently significant, especially in view of its strong empirical experimental results.\n\n", "strength_and_weaknesses": "Strengths\n1. Paper's motivations and methods are well presented and supported by experimental results.\n2. Experimental setups are comprehensive and strongly supports the core claim of the paper: self-supervised learning network with contrastive approach to generate 3D representation of human images is superior to other self-supervised approaches.\n3. Ablation studies are thorough and well-done.\n\nWeaknesses\n1. Incremental novelty as proposed method is similar to other approaches for CV tasks, including image classification, object detection, semantic segmentation, etc\n2. Experimental results only compare with other self-supervised methods. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NyfMRsapxs", "reviewer": "Reviewer_5VTe", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the Lifting Contrastive Learning (LiftedCL) workflow to train robust human-centric representations that are useful for downstream 2D and 3D human tasks such as pose estimation, shape prediction, and parsing. There are two core components. The contrastive learning part generates features that are invariant and equivariant (under inverse transforms). The \"lifting\" part generates 3D skeletons from the human-centric representations and is trained with a discriminator along with randomly sampled real 3D skeletons. Experiments show that the proposed training produces more useful features for downstream tasks than competitive alternatives such as MoCo-v2 and PixPro. ", "summary_of_the_review": "Overall, I like this paper because it studies an interesting and important problem of encoding 3D-aware representations in human-centric tasks and provides a viable solution. The design components in this workflow could inspire future research in this direction. The experiment results are solid on multiple human-centric tasks. The paper could be further strengthened by adding the 2D baseline mentioned above, but it is a good contribution to the community in its current form.", "strength_and_weaknesses": "### Strength\n- The intuition of learning a robust representation that encodes 3D human information makes a lot of sense to me.\n\n- The designed pipeline is straightforward. It could inspire further development in this direction.\n\n- The invariance and equivariance training is novel in the context of 3D human feature learning tasks. \n\n- I like the idea of having unpaired randomly sampled 3D skeletons as auxiliary training data.\n\n- Experiments show the solid performance of the proposed method when trained on ImageNet + COCO: downstream tasks have significant improvement. That validates the idea of encoding 3D human structure information is useful for human-centric tasks.\n\n### Weakness\n- I think the paper could use a strong baseline in which all the 3D elements are replaced by 2D counterparts. For example, instead of lifting 2D features to 3D skeletons, the network will output 2D skeletons and a discriminator will compare those with randomly sampled real 2D skeletons. Other parts of the network will be kept as much as possible. If the proposed LiftedCL outperforms this 2D alternative on 2D human tasks such as pose estimation and parsing, then it is clear that 3D is essential to both 2D and 3D human tasks. I agree with the authors that 3D is important but it will be more convincing with such an experiment.\n\n- I might have missed it, but is there an ablation study on the effectiveness of the KCS layer?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "E20_ZQhQ0f", "reviewer": "Reviewer_3nTM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose, Lifting Contrastive Learning (LiftedCL), a feature representation learning approach based on contrastive learning with primary focus on the task of 2D/3D human pose estimation from a single image. Authors further use kinematic chain space (KCS) layer and a discriminator to regularise the skeleton. The key idea is to decompose the information into two, invariant and equivariant features. \n\nAuthors provide key experiments on MSCOCO, MPII and H36m datasets. Ablation study shows that the unsupervised pre-training with invariant and equivariant features improves performance over vanilla supervised training.", "summary_of_the_review": "I'm currently on the fence for this work. The key ideas seem to be coming from other works (see weakness section) and I've seen better numbers on 2D/3D pose estimation task. Although these SOTA works (including the best performing version of SimplePose, whose smaller version authors use as baseline) use much bigger networks than Res50 so they are not directly comparable.\nI don't see a very compelling reason to accept this work. I'm open to changing my mind if authors can clarify the limited technical novelty aspect.", "strength_and_weaknesses": "Strengths:\n+ Contrastive learning for 3D tasks is quite useful because acquiring annotated 3D data is very hard for for a lot of tasks. Unsupervised feature learning approaches make a lot of sense.\n+ I like the ablation study to motivate the choice of using invariant and equivariant feature learning.\n+ The approach is straightforward and easy to follow.\n\nWeakness:\n- The idea of using equivariance and invariance for feature learning has been proposed in Cheng et al. ICCV'21 (not cited). Eg: They also use contrastive learning on augmented images and use that augmentation of the same image are positive pairs and different images serve as negative pairs. How is the proposed formulation different?\n\n- How is KCS + adversarial learning (Sec 3.3) different than Critic Network in RepNet? Eq. 4,5,6,7 here are same as Eq. 3,4,5,6,7 in RepNet.\n\n- Sec 3.2: To learn equivariant features, since the inverse transformation is known, why do we need to use contrastive learning? We have direct supervision that features of positive pairs should be the same after the inverse transform. Can the authors comment on this?\n\n- Datasets like H36m are a bit saturated for pose estimation as they are a bit synthetic and controlled. 3DPW is more relevant dataset these days. Authors use it for shape recovery but not pose estimation evaluation. Can the authors comment on this?\n \nClarifications and minor suggestions:\n- Sec 3.2: What augmentations are used?\n- Add citation to Wandt et al. ECCVW'18 and RepNet, CVPR'19 in Sec 3.3. Authors mention this in related work but it is better to add the citation here as well, otherwise KCS looks like a contribution of this work.\n- PixPro is just an unsupervised feature representation method. How do the authors use it for 2D pose prediction in Sec 4.2? Can the authors provide more details?\n- Add a bullet list of key contributions. It clearly distinguishes the contributions of this work from the other works. This is quite useful as the final approach often uses components from other works.\n- The ideas of equivariance and invariance are quite well known in the 3D vision community. Authors might be interested in checking some of these works out. eg: ART, Zhou et al. 3DV'22.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster