Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes the Lifting Contrastive Learning (LiftedCL) workflow to train robust human-centric representations that are useful for downstream 2D and 3D human tasks such as pose estimation, shape prediction, and parsing. There are two core components. The contrastive learning part generates features that are invariant and equivariant (under inverse transforms). The "lifting" part generates 3D skeletons from the human-centric representations and is trained with a discriminator along with randomly sampled real 3D skeletons. Experiments show that the proposed training produces more useful features for downstream tasks than competitive alternatives such as MoCo-v2 and PixPro. 
strength_and_weaknesses: ### Strength
- The intuition of learning a robust representation that encodes 3D human information makes a lot of sense to me.

- The designed pipeline is straightforward. It could inspire further development in this direction.

- The invariance and equivariance training is novel in the context of 3D human feature learning tasks. 

- I like the idea of having unpaired randomly sampled 3D skeletons as auxiliary training data.

- Experiments show the solid performance of the proposed method when trained on ImageNet + COCO: downstream tasks have significant improvement. That validates the idea of encoding 3D human structure information is useful for human-centric tasks.

### Weakness
- I think the paper could use a strong baseline in which all the 3D elements are replaced by 2D counterparts. For example, instead of lifting 2D features to 3D skeletons, the network will output 2D skeletons and a discriminator will compare those with randomly sampled real 2D skeletons. Other parts of the network will be kept as much as possible. If the proposed LiftedCL outperforms this 2D alternative on 2D human tasks such as pose estimation and parsing, then it is clear that 3D is essential to both 2D and 3D human tasks. I agree with the authors that 3D is important but it will be more convincing with such an experiment.

- I might have missed it, but is there an ablation study on the effectiveness of the KCS layer?

clarity,_quality,_novelty_and_reproducibility: ### Clarity
The paper is straightforward and easy to follow. 

### Quality
The quality of the work is good.

### Novelty
I think the idea of encoding 3D-aware representations in human-centric tasks via lifting-to-3D contrastive learning is novel and it can inspire future studies in this direction.

### Reproducibility
There seems to be a sufficient amount of detail in the paper for reproducibility. It will be better if the authors plan to release their code. It is not currently mentioned in the submission.
summary_of_the_review: Overall, I like this paper because it studies an interesting and important problem of encoding 3D-aware representations in human-centric tasks and provides a viable solution. The design components in this workflow could inspire future research in this direction. The experiment results are solid on multiple human-centric tasks. The paper could be further strengthened by adding the 2D baseline mentioned above, but it is a good contribution to the community in its current form.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose, Lifting Contrastive Learning (LiftedCL), a feature representation learning approach based on contrastive learning with primary focus on the task of 2D/3D human pose estimation from a single image. Authors further use kinematic chain space (KCS) layer and a discriminator to regularise the skeleton. The key idea is to decompose the information into two, invariant and equivariant features. 

Authors provide key experiments on MSCOCO, MPII and H36m datasets. Ablation study shows that the unsupervised pre-training with invariant and equivariant features improves performance over vanilla supervised training.
strength_and_weaknesses: Strengths:
+ Contrastive learning for 3D tasks is quite useful because acquiring annotated 3D data is very hard for for a lot of tasks. Unsupervised feature learning approaches make a lot of sense.
+ I like the ablation study to motivate the choice of using invariant and equivariant feature learning.
+ The approach is straightforward and easy to follow.

Weakness:
- The idea of using equivariance and invariance for feature learning has been proposed in Cheng et al. ICCV'21 (not cited). Eg: They also use contrastive learning on augmented images and use that augmentation of the same image are positive pairs and different images serve as negative pairs. How is the proposed formulation different?

- How is KCS + adversarial learning (Sec 3.3) different than Critic Network in RepNet? Eq. 4,5,6,7 here are same as Eq. 3,4,5,6,7 in RepNet.

- Sec 3.2: To learn equivariant features, since the inverse transformation is known, why do we need to use contrastive learning? We have direct supervision that features of positive pairs should be the same after the inverse transform. Can the authors comment on this?

- Datasets like H36m are a bit saturated for pose estimation as they are a bit synthetic and controlled. 3DPW is more relevant dataset these days. Authors use it for shape recovery but not pose estimation evaluation. Can the authors comment on this?
 
Clarifications and minor suggestions:
- Sec 3.2: What augmentations are used?
- Add citation to Wandt et al. ECCVW'18 and RepNet, CVPR'19 in Sec 3.3. Authors mention this in related work but it is better to add the citation here as well, otherwise KCS looks like a contribution of this work.
- PixPro is just an unsupervised feature representation method. How do the authors use it for 2D pose prediction in Sec 4.2? Can the authors provide more details?
- Add a bullet list of key contributions. It clearly distinguishes the contributions of this work from the other works. This is quite useful as the final approach often uses components from other works.
- The ideas of equivariance and invariance are quite well known in the 3D vision community. Authors might be interested in checking some of these works out. eg: ART, Zhou et al. 3DV'22.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. I have concerns about originality as the key contributions of this work seem a bit derivative of existing works, equivariance and invariance  formulation is very similar to Cheng et al. ICCV'21 and lifting to 3D using KCS is similar to RepNet, CVPR'19. Can the author clarify why the current approach is not a combination of these two?
summary_of_the_review: I'm currently on the fence for this work. The key ideas seem to be coming from other works (see weakness section) and I've seen better numbers on 2D/3D pose estimation task. Although these SOTA works (including the best performing version of SimplePose, whose smaller version authors use as baseline) use much bigger networks than Res50 so they are not directly comparable.
I don't see a very compelling reason to accept this work. I'm open to changing my mind if authors can clarify the limited technical novelty aspect.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Paper proposed a novel architecture for self-learning of human-centric perception problems: 2D/3D human pose estimation, human shape recovery and human parsing. The proposed method is simple and effective -- exceeding SOTA for several benchmarks. The encoder subnetwork uses Contrastive Learning and Lifting network to generate the 3D skeleton of the input; this is then combined with Adversarial loss from Discriminator sub-network to improve the encoder. 


strength_and_weaknesses: Strengths
1. Paper's motivations and methods are well presented and supported by experimental results.
2. Experimental setups are comprehensive and strongly supports the core claim of the paper: self-supervised learning network with contrastive approach to generate 3D representation of human images is superior to other self-supervised approaches.
3. Ablation studies are thorough and well-done.

Weaknesses
1. Incremental novelty as proposed method is similar to other approaches for CV tasks, including image classification, object detection, semantic segmentation, etc
2. Experimental results only compare with other self-supervised methods. 
clarity,_quality,_novelty_and_reproducibility: 1. Clarity is excellent. Paper is well-written and clearly presented.
2. Quality is good. All sections are well organized. Figures, Tables etc. are also properly labelled and easy to understand.
3. Novelty is incremental. See Weaknesses above.
4. Reproducibility is high as many details are given. However, it's not clear if source code will be shared after acceptance.
summary_of_the_review: Overall, paper is a good piece of work which should be accepted for ICLR. There are aspects of the work which are incremental, e.g. using of Contrastive Loss, 3D structure representation of human. However, the scientific contributions of it are still sufficiently significant, especially in view of its strong empirical experimental results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by three experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to ICLR 2023. All the reviewers acknowledged the good execution of leveraging invariance and equivariance for learning 3D-aware human-centric representations. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper, e.g., adding stronger baselines in which all the 3D elements are replaced by 2D counterparts. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!

justification_for_why_not_higher_score: The key ideas of this work seem to be coming from other works and might be incremental, e.g., the usage of contrastive loss and 3D structure representation of humans.

justification_for_why_not_lower_score: This paper shows strong empirical experimental results, which could be helpful to further development in the community.
Decision: Accept: poster