Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The manuscript presents a comprehensive benchmarking study in the terms of datasets, network backbone and training strategy. The authors analyzed the database characteristics, data enhancement strategy, network structure and loss function design by conducting vast experiments, and improved the accuracy of human body reconstruction with proper training configurations. After testing many aspects, the authors believe that the conducted experiments evaluate the existing methods more fairly, and the recommendation configuration can be used as a new baseline for the human mesh recovery problem.
strengths: + The motivation is clear. I quite agree with the author's view that the current HMR research lacks the analysis of training data sets, and the benchmarking study in this paper is a good attempt to solve this problem.
+ The author has done a lot of research and experiments, and comprehensively analyze the various data sets for training. The tested method covers the mainstream methods, with many interesting and significant recommendations made.
+ The overall paper is well written and easy to follow.
weaknesses: - The benchmark is still based on previous metrics. The methodology and dataset contribution for the benchmark is limited.
- The coverage of the methods is not thorough, as some network models are missing, like PyMAF [79]. I also understand that it is difficult to incorporate all approaches into an HMR-based framework, and I would like to see the discussion about how to evaluate approaches that differ widely from the HMR-based framework.
- The effectiveness of training methods is closely related to the selected network backbones, and this paper can only test training methods on the most mainstream network models. So I believe the recommendation configuration obtained has certain limitations.
correctness: I think the claims made in the submission are generally correct. The evaluation methods and experiment design are appropriate while minor concerns are raised in the 'Weaknesses'.
clarity: The overall paper is well written and easy to follow.
relation to prior work: The comprehensive evaluation of data sets and training methods are presented and not involved in previous work. This difference is discussed in the manuscript.
additional feedback: As mentioned in the 'Weakness' section, I think the authors should try to compose a more fair data set and corresponding benchmarking metrics based on the analysis. In addition, authors are encouraged to expand the pool of the network models to cover more approaches.
review_title: A comprehensive evaluation of datasets, network models and training strategies for HMR problem

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors tackle the problem of extensively evaluating human mesh recovery model and training parameters to find the optimal parameters for a strong baseline. They perform extensive ablation studies on many components of the model, including data type, data augmentation, model choice, model initialization, and training strategy. The authors achieve improved results on 3DPW.
strengths: 1. The problem is properly motivated and studies a well-known field of human mesh recovery
2. Experiments are thorough and reasonable
3. The recommendations provided by the authors via the benchmarking study will be helpful for future researchers to build strong models
4. Strong baselines will provide a better baseline for fair future comparisons
weaknesses: 1. The authors use HMR as the baseline. Many improved models have come out since then, which could have been better choices.
2. While the paper claims to "provide strong baselines for fair comparisons of algorithms," baselines are only done for one model (HMR). 
3. The paper does not present a dataset or a benchmark, but rather proposes stronger baselines on existing benchmarks, and recommendations for training strategies - it's not clear that this is in scope for the D&B track. 

correctness: The paper discusses multiple components of HMR training, each of which is supported with experiments. The experimental design is appropriate and performed correctly, with extensive experiments providing convincing support for the recommendations made by the authors.
clarity: The paper is well written, without obvious errors.
relation to prior work: Yes, this paper is the first large-scale benchmarking study of human mesh recovery.
additional feedback: Since your goal is to provide a "strong baselines for fair comparisons of algorithms" I think it would be reasonable to provide more than 1 baseline (HMR), which is currently the case. The paper gradually builds towards finding the optimal dataset, model parameters, and training method, so as a final conclusion, I would have liked to seen a set of baselines on many models besides HMR.
review_title: Review for 3D Human Pose and Shape Estimation Benchmarking Study 

final_score: rating: 2: Strong rejection
review_content: summary and contributions: This paper experimentally studies the task of 3D human pose and shape estimation from three aspect: dataset, backbone, and training strategy. Specifically, this paper provides a detailed analysis of 31 datasets on their data characteristics, as well as how they affect the performance when used for training. The authors experiment with 10 different neural network architectures as backbones and compare their performance. The dataset used for pre-training the backbone is also experimentally shown to be important. Moreover, training strategies including data augmentation and training loss are discussed. Although the time and effort spent on the experiments is significant, no meaningful insight is given by the authors.
strengths: This paper provides extensive experimental results for the task of 3D human pose and shape estimation, which are well organized and easy to follow. This can save time for other researchers who are curious about this research field from a practical perspective.
weaknesses: 1. Although the time and effort spent by the authors on all the experiments is impressive, the contribution of this paper is questionable. All the remarks are obvious to the researchers in this field even without any of the experiments. It is good to know the numbers and rankings given by the experiments, but that is all. There is nothing more meaningful that can be learnt from this paper. 
2. It is helpful to put all the datasets together and show some statistics, but all the datasets are kept as is, and nothing new is introduced. The experiment of different training set is strongly biased by the HMR method, as well as the 3DPW and H36M test set. The conclusions drawn from this experiment is ad-hoc.
3. The backbone architecture and pre-training is another problem, which has nothing specific to do with the task of 3D human pose and shape estimation.
4. Data augmentation is a common practice when there is limited training data. There is nothing new about data augmentation in this paper.
5. Training loss selection depends on the method and the data. The difference between MSE and L1 loss is already known. No additional insight is provided in this paper.

correctness: Yes. The claims made in the submission are correct. The evaluation methods and experiment design are appropriate and performed correctly.
clarity: Yes. The paper is well written and easy to follow.
relation to prior work: Yes. The difference of this work from previous works is clearly discussed.
additional feedback: I admire the time and effort that the authors spent on all the experiments. The experiments may be helpful to someone, but not to the serious researchers in this field.
review_title: A comprehensive benchmark for 3D human pose and shape estimation

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper studies various components of 3D pose and shape estimation, that are usually overlooked or heuristically addressed by relevant work. A large-scale experimental study is performed that raises interesting insights on the impact of dataset mixing, importance of pretraining, effectiveness of various backbones, need for augmentation strategies, robustness to noise and others. Based on their finding, authors manage to train a strong baseline on 3DPW with a rather simple model. Multiple good practises for 3D pose and shape estimation are identified.
strengths: -Most of the focus in 3D mesh recovery is on algorithmic development with less effort on establishing proper evaluation protocols. Authors demonstrate how commonly overlooked factors like dataset mixing, augmentation strategy and model initialisation can effect reported performance leading to unfair comparison between different methods.

-The investigation performed in this work can serve as a valuable set of best practises for developing mesh recovery models. Multiple factors are explored like optimal dataset mix, value of pretraining, best augmentation strategies and most suitable losses, e.t.c.

-Various interesting insights are raised. One interesting example is that difference in performance cannot always be explained by the indoor-outdoor domain gap hypothesis and more carful analysis of underlying factors is needed.
weaknesses: -I am slightly concerned regarding how relevant this work is to NeurIPS Datasets and Benchmarks Track. Obviously, this is not a Dataset paper. Insights on important dataset attributes for 3D mesh recovery are only given through UMAP visualisations. Simple experimentation on optimal dataset mixtures and weighting schemes is also performed. Even though this work does audits on existing datasets (that is inside the score of NeurIPS Datasets and Benchmarks Track), the main focus is rather on identifying overlooked aspects of common pipelines for 3D mesh recovery (pretraining, augmentation, robustness to noise, appropriate losses).

-Section 3.3 provides an interesting analysis of the robustness of common architectures in the injection of controlled noise. The main motivation for this, is that many algorithms use pseudo-annotations during training. Such an analysis might not be representative of corruption in real-world pseudo-annotations. For example, an inaccurate body part will rarely be on a random location (as shown in Fig of the appendix) and inaccurate pseudo-labels are more likely on difficult samples (highly articulated poses or large occlusion). 

-Experimental setting in section 3.3 might be a bit unclear. What is the x-axis on Fig2.b? Does the scale and location caption refer to the axis? Also, based on Fig2 how slight noise can help training?

-What do authors mean in line250, stating that transferring knowledge is not necessary to further finetune the backbone with ETF-COCO and how this is shown in Table 7?

-It’s interesting how weighting schemes can lead to increased performance. On the other hand, in the presented form this is a heuristic approach (manually selecting weights for multiple datasets) that increases method complexity.
correctness: I do not have any concerns about the correctness of this work.
clarity: The paper is well written.
relation to prior work: Related work introducing the various components explored in this work is well presented.
additional feedback: This work provides an interesting insight into overlook aspects of 3D human pose and shape estimation. I am slightly concerned about the relevance to NeurIPS Datasets and Benchmarks Track.
review_title: Interesting insights and best practises into overlook aspects of 3D human pose and shape estimation

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper systematically varies several dataset, architecture, and training choices while keeping evaluations consistent to provide a well justified set of best practices, along with specific recommendations when keeping other factors equal.  While many of the general findings are unsurprising (the mixture of datasets can make a big difference, it is useful for datasets to have more diversity), some of the findings aren’t obvious a priori so the evidence provided here is useful (e.g., models tend to be more robust to noise in SMPL parameters than 3D points). Moreover, it is quite useful to have the specific details of datasets and choices in one place, even those that are not surprising. The paper is correct that, since each publication makes different choices in all of the dimensions, it is difficult to make good evidence-based decisions, so this paper makes a significant contribution in reducing some of the ambiguities when datasets, architectures, and training choices aren’t matched.

strengths: This paper makes several overall contributions on several axes. Here is a very brief attempt at outlining the takeaways though it is not necessarily comprehensive.

* Dataset:
    * Diversity in various attributes is useful
    * Having occlusion and SMPL fittings in the ground truth is useful
    * More datasets is not better, it can even be harmful
    * Careful configuration of different contributions (weightings) for different datasets can bring a large improvement.
    * Directly altering the partitions is more effective than reweighting the samples. 
* Noise:
    * Some noise in SMPL is ok
    * Noise in keypoints is bad 
    * Replacing a percentage of pose parameters (i.e., for some body parts) with random noise (not just adding some noise to good values) is also bad.
* Architecture:
    * Increased capacity in CNNs helps (at a power cost of course)
    * Transformers are better than CNNs at comparable FLOPs
* Training:
    * Directly supervising to SMPL parameters is better than supervising to keypoints or other features.
    * It is helpful to pre-train backbone for a mesh estimation model with a good pose estimation model.
    * Most augmentations help, self-mixing being one of the most useful.
    * Datasets with a lot of diversity don’t need as much augmentation (or it can be harmful), notably when it comes to self-occlusion.
    * L1 loss is better than L2 loss


weaknesses: Most of the findings are not surprising to a practitioner in the field, although the specific details for the specific datasets and architectures are still quite good to have.  It might be useful to have a summary table at the beginning or end as spelled out in the “Results” blocks throughout the paper.  It would also be good enumerate more of the resulting open questions raised by these findings, especially those that are somewhat surprising (or at least, not obvious).  For instance, why is directly altering the partitions more effective than reweighting the samples?  Why is L1 loss better than L2?  Some explanations are occasionally offered, so the main ask is to offer explanations more consistently.

correctness: Did not check, but the methodology seems sound so I believe it is correct.

clarity: Quite clear. The “Remarks” blocks are helpful.  Might be useful to put these into a single summary somewhere.

relation to prior work: Does a good comparison to prior work, indeed the goal is to put more prior work on the same playing-field.

additional feedback: The paper could try to provide more explanations for why some choices are better, or at least state these as outstanding questions for what would help advance the field.  For instance, a main finding is that adding more datasets is not better, and can even be harmful; moreover careful configuration of different contributions (weightings) for different datasets can bring a large improvement.  Of course it’s good to see the specific details about how specific combinations of public datasets impacts algorithms. Yet the paper didn’t link these findings back to the specific diversities in these datasets discussed in the previous section.  One has to do some work to see the contributions of different datasets *by name* and then map that to the contributions of different amounts of *attributes* of these datasets. Even with this, keeping the unit of analysis as an entire dataset is a bit cumbersome. Ideally (hopefully this will come in future work) we would like these findings to lead to a recipe for constructing a training set with ideal combinations of characteristics (whether through merging parts of existing datasets or creating a new one). A similar desire applies to other findings as mentioned above.
review_title: A good benchmark and anaylsis paper.  Most findings are not surprising but good to have them all in one place for the pose and shape estimation domain.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper gives a detailed analysis of 3D human recovery in datasets, backbones, and training policies. Inspiring and detailed discussions and sound-designed experiments are conducted. Besides, a project page with detailed documentation is also provided which would help the community for future work. Overall, I think this is a solid and good work for 3D human-related works.
strengths: + The discussions and experiments are inspiring and make solid contributions.

+ The results of the large-scale experiments reveal several important discoveries, e.g., the mixed training of multiple datasets, the differences between various backbones, whether the noisy reconstructions affect the training, etc.

+ Thorough details and settings of the experiments are given in the suppl and main text, which would help the readers to understand this work better.

+ The project page is organized well with detailed documentation, open models, and implementations which are very useful for the community.
weaknesses: - Contributions to dataset/benchmark: though I think this is a solid work, the last two contributions may be kind of off the point of the NeurIPS dataset and benchmark track. The backbone and training strategies analysis are more like the method contribution. But I think this is not a big deal considering the solid works.

- Lacking possible advice on how to build a better test set for this direction. Though the discussed factors are important, the relations between the train and test data also matter a lot. Existing test sets also have many problems and possible aspects to improve. Please give some discussions about this point which would make this work more solid and inspiring.

- There is no comparison between different methods given the same set of training datasets to clearly indicate how they perform in a fair comparison. If possible, I think this would make the first contribution better and more beneficial.
correctness: Yes, easy to follow.
clarity: Good.
relation to prior work: Yes, and sound.
additional feedback: 1. In some tables like Tab. 1, 2, and 4, maybe some bold scores can make the comparison clear like Tab. 6.

2. L165, advanced algorithm. I suggest changing this to a more sophisticated algorithm or other adjectives as no free lunch.

3. I like the augmentation analysis. But the two sets of results give the opposite conclusion. Though I agree that different datasets have different properties thus will give different reactions to the augmentations. However, could the authors give some more instructive advice for future works?
review_title: A solid contribution to 3D human recovering

Meta-review: metareview: This paper presents a benchmark (in the sense of a large scale ablation and evaluation of existing works/datasets), considering the effects of dataset, "backbone" and training strategies for the problem of human pose and shape estimation. There is a consensus among reviewers that this paper's main limitation is the (over)emphasis on the human mesh recovery (HMR) framework. The forum discussions provide convincing arguments that despite this, the paper still offers with a major contribution to this albeit limited view of the problem. Another common issue raised by reviewers is whether this paper is suitable for the NeurIPS D&B track. I agree with the authors (and Reviewer BCuk) that this paper's premise is indeed well suited for this track, as described in the call for papers and FAQ. Therefore, I dismiss this as a limitation. Finally, there is a strong outlier review, whose exceptionally low score is obliterating this paper's average score. I have read the discussion in detail and while I find that this reviewer is making interesting points, I do not agree that any of them are serious enough to withhold publication of this work. I am content to see the reviewer's points as providing pointers for other future works that may be more influential than the presented one. The possible existence of superior future studies is not a reason to reject this paper. In conclusion, backed by the strong majority of reviewers, I recommend accepting this paper to the NeurIPS 2022 Datasets and Benchmarks program.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept