Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an exhaustive empirical study into the effect on performance, for 3D human pose and shape estimation, of different types of pre-training of the backbone network using different training datasets. The different types of pre-training investigated are supervised classification, SSL, and semi-supervised learning and the datasets used vary from large and diverse (ImageNet) to human centric (MSCOCO)  and synthetic. 

The main conclusions from the empirical results are for 3D human pose and shape estimation:
1) supervised ImageNet pre-training dominates by quite a large margin the standard SSL ImageNet pre-training approaches
2) Pre-training on smaller but more domain specific tasks (2d keypoint estimation) and data (images of people) can be as effective as pre-training on larger but more diverse datasets.

Contributions:

* The paper presents a very thorough and extensive  set of sensible experiments exploring if SSL pre-training should replace the standard practice of using a backbone pre-trained on ImageNet in a supervised fashion for 3D human pose and shape estimation and deciding it should definitely not as this point in time!
strength_and_weaknesses: Strengths:

- The paper presents a very nice set of experiments with many settings which I think would be very much appreciated by those working on the problem of 3D human pose estimation.

Weaknesses:

- There is no real investigation as to why SSL does not work as well for pre-training. There is some conjecture in section 4.2 but not much probing experimentally or qualitatively beyond the pre-training conditions and the final performance result. But there is no probing or analyzing the representations learnt in the different set-ups. For example networks trained on ImageNet in a supervised fashion are great at recognising textures whereas self-supervised representations are not good at recognising color. Are such issues coming into play? Is the problem the data-augmentations being used as opposed to SSL? Which augmentations could be problematic and why?
clarity,_quality,_novelty_and_reproducibility: In general the paper is well-written and easy to read with the main findings of the paper clearly presented and articulated. 

One point of confusion for me is:

- The semi-supervised setting is referred to many times in the main text and forms one of the main talking points in section 4.2. I may have missed it, but it is not entirely clear how semi-supervised learning is implemented in this paper.
summary_of_the_review: The empirical results in this paper would be of interest to many. But perhaps, because of the lack of insightful analysis and probing into the reason for the poor performance of SLL pre-training this would cause some hesitation in acceptance at ICLR.

After the authors' response:

I appreciate the authors' detailed response to the review and the explanation for the "relative" poor performance of SSL pre-training. After reading the response I have increased my rating to an accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides empirical study of how to pre-train the vision backbone for human pose and mesh estimation. The authors conducted many experiments with different pre-training data. The authors observed using real images with 2D human pose labels (such as COCO) is the most effective one. 
strength_and_weaknesses: Strengths:
1. The paper is easy to follow and well-written. The authors described their experiments and different analysis settings clearly.
2. The authors conducted very thorough experiments to validate which data is useful for pre-training the backbone.
3. The paper provides several interesting discussions and insightful findings. It could be a useful reference for future research.

Weaknesses:
1. Prior works have been discovered that 2D re-projection loss is very critical to exiting methods. The main message of this paper seems to double confirm the importance of 2D pose data. While the authors show that pre-training the backbone is important, there is no other new pre-training technique introduced to effectively boost the SOTA performance. 
2. I appreciate the efforts of experimenting with SSL methods. However, the results seem not very positive. Since SSL might be a bit more difficult to train, I wonder if the SSL pre-training needs more training tricks and more diverse data to enhance feature representation generalizability.
clarity,_quality,_novelty_and_reproducibility: The presentation and experiment design are good and clear.
summary_of_the_review: I have a mixed feeling about this paper. The paper presents a very useful exploration of pre-training vision backbone for human pose and mesh reconstruction. All the discussions and analysis are valuable. It will be very helpful to future researchers. This is good as a technical report. But when considering it as a research paper, the novelty of pre-training technique is weak.

Probably because the considered SSL pre-training methods don't work well, the paper consists of several negative observations for SSL. The only positive signal comes from pre-training with real images & pose labels (such as COCO dataset). This is more like a revisit, as some recent works have pre-trained a PoseNet in a similar way for pose-to-mesh pipeline. That is, there is no novel pre-training technique introduced. This weakens the contribution of this submission.

However, considering the valuable analysis, discussions, and the great efforts of conducting all these experiments, this paper still provides useful guidance to the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work provides a thorough evaluation of various backbone pre-training methods such as SSL, with human datasets, and semi-supervised setting. 
strength_and_weaknesses: 
1. An in-depth investigation is presented in the paper to cover various pre-training methods.

2. A wide variety of base models have also been tested.


Weaknesses:
1. Although I appreciate this work as an investigation into using SSL for human pose and shape estimation, it might be difficult to justify the significance. Recent works have provided some studies on the impact of backbone selection for human pose and shape estimation [A, B], and training with 2D pose estimation as a proxy task has been shown to be effective for human pose and shape estimation.

2. It is unconventional to use backbone pretraining + head fine-tuning for human pose and shape estimation as fully supervised training has been widely adopted in the field. I agree that it is interesting if pretraining has shown promising results, but there is a significant gap between the evaluated pretraining methods and the SoTA performance. E.g. PARE achieves easily <50 mm PA-MPJPE on 3DPW, but struggles at ~55 mm PA-MPJPE in Table 1. The practical use of this investigation might be questionable.

3. The conclusion on synthetic data might need some adjustments. It has been shown in [C, B] that synthetic data, if used properly, improve baseline models such as HMR, SPIN, VIBE and PARE on real datasets.

4. The semi-supervised learning setting might not be very relevant in today's context as synthetic data incurs almost no cost to scale up. In addition, semi-supervised learning results in a performance gap that is too significant to easily justify its usefulness in real-life applications.

5. There seems to be a slight contradiction in Sec 4.2. It is stated that "a backbone network is preferred to learn more about human features" so SSL does not work, but it is also stated that "high-level semantics ... could be beneficial transferred to inference on humans". I think it may be helpful to provide a more clear-cut explanation that separates classification pretraining and SSL.

[A] Pang et. al., Benchmarking and Analyzing 3D Human Pose and Shape Estimation Beyond Algorithms, NeurIPS (Dataset Track) 2022

[B] Cai et. al., Playing for 3D Human Recovery, arXiv 2021

[C] Patel et. al. AGORA: Avatars in Geography Optimized for Regression Analysis, CVPR 2021

clarity,_quality,_novelty_and_reproducibility: The paper is clearly presented, and the investigation in SSL for human pose and shape estimation is new.
summary_of_the_review: Despite I deeply appreciate the investigation in pretraining backbone with SSL for human pose and shape estimation, I find it difficult to justify the significance as the finding seems to suggest that SSL does not work in this case. More importantly, there are relevant works on backbone training strategies and synthetic data, showing much more promising results. Therefore, I find the paper in its current form does not meet the bar of ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper conducted an empirical study on the effectiveness of the pre-training of a backbone for the task of 3D human pose and shape estimation (3DHPSE). In the experiments, the authors compared different pretraining strategies, including purely unlabeled self-supervised pre-training, 2D annotation-based pre-training, and synthetic data pre-training. The paper concludes that the self-supervised pre-training on unlabeled images is not effective for 3DHPSE and the 2D annotation-based pre-training brings the largest improvements.

strength_and_weaknesses: Strength:

+ This paper is overall well-written and well-organized. The experimental results on different pre-training settings are convicing and support the claim of the paper.
+ A comprehensive study and analysis on the effectiveness of the pre-training for the 3DHPSE task is appreciated. The experiments on different self-supervised pre-training settings are new to the community, which is the aspect to support the acceptance of this paper.

Weaknesses:
- The main concerns regarding this submission are about its practical effects on the 3DHPSE community as the conclusion is actually in line with the practices of recent state-of-the-art methods. More specifically, recent approaches such as Pose2Pose and PARE have used the 2D pose pre-trained backbone in their training strategies. Different pretraining settings of the backbone are also discussed in [Zhang et al.].
Zhang et al., Learning 3D Human Shape and Pose From Dense Body Parts, TPAMI 2020

- The authors only investigate contrastive learning-based methods such as PeCLR for pose estimation. How about self-supervised learning methods on video data, for example [Sun et al. ECCV20, CVPR22]? 
Sun et al., Self-Supervised Keypoint Discovery in Behavioral Videos, CVPR 2022
Sun et al., View-Invariant Probabilistic Embedding for Human Pose, ECCV 2020





clarity,_quality,_novelty_and_reproducibility: Overall, the clarity and quality of this paper are good. The experiments are also reproducible. As the conclusions of this paper bring limited effects on the community, the novelty is the main issue of this work.
summary_of_the_review: This paper provides a comprehensive study on the effectiveness of the pre-training for the task of 3DHPSE, which is important to facilitate the research of the 3DHPSE task. The experiments are comprehensive and support the claims of the paper. However, the originality of this work is somewhat limited as parts of the claims have actually been adopted by recent state-of-the-art methods. Given the overall quality and originality, I currently rate this paper as marginally above acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper conducts an empirical study on how to pre-train visual representation for better human pose and mesh estimation. All the reviewers agree that the paper provides thorough experiments and the findings are interesting and insightful. All the reviewers agree to accept the paper and the AC agrees.
justification_for_why_not_higher_score: The paper provides extensive experiments and achieves some insightful conclusions. It is interesting and beneficial for the community, but at the same time there remains concerns on the significance of the result.
Decision: Accept: poster