Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper reveals an interesting finding that minMPJPE metric would result in miscalibrated distribution in a 2D-to-3D lifting problem, which is particularly unsuitable for using MPJPE to evaluate those distribution-based methods that generate multiple hypotheses to solve the ambiguity. It provides a toy example that shows optimizing minMPJPE would have a larger ECE than NLL and lead to overconfident distribution, while a good NLL may not guarantee excellent MPJPE, but helps to calibrate the distribution. To this end, they propose cGNFs and train it by likelihood maximization to learn the conditional 3D human pose distribution. Together with optimizing minMPJPE, their method would achieve comparable performance to SOTA. From experimental results, their method has better ECE performance on H36M, and an obvious boost in MPJPE can be seen on occluded samples.  
strength_and_weaknesses: Strength
* This paper starts with good motivation and provides a toy example to provide a possible explanation about the miscalibration caused by minMPJPE.
* The performance of the proposed method is impressing on occluded samples by both MPJPE and ECE metrics.

Weaknesses
* The introduction in Sec.3 is hard to follow. I think this section is significant for helping construct a basic understanding of how ECE is used in 3D pose regression. But I fail to read it when I came to Sec.3.1. The authors should explain why using median is reasonable. Why not use other statistics, such as mean? If I understand correctly, this cumulative distribution function should give the probability of being less than a certain distance. However, I think the probability of the distance between the ground truth pose and the median is what we need. In Figure 1, why the model is underconfident if the optimal variance is larger than 0.5?   

* The comparison with Biggs et al. (2020), Kolotouros et al. (2021), Oikarinen et al., (2020) should be added. Biggs et al. (2020) and Kolotouros et al. (2021) share a similar idea with this paper that simultaneously optimizes likelihood and joint distance. Oikarinen et al., (2020) use 2D pose as input and train with GNN. I would suggest the authors add more discussions with these papers to highlight the differences and explain the advantages of cGNFs. These paper also publicize their code with accessible pretrained models, it would be better if the authors provide experimental comparisons.

* Although the good performance could be seen on ECE metric, the loss on MPJPE metric is also noticeable. Through the measurement of the article design, it seems that this method has made a great contribution to improving the credibility of the model. However, the results also show that certain gains on ECE would result in a loss on MPJPE. It’s hard to say whether such consequences are acceptable to us, especially when the loss is not subtle (7.7mm increase on MPJPE comparing cGNF w L_sample and Wehrbein et al. (2021)).

[1] Biggs, Benjamin, et al. 3D Multi-bodies: Fitting Sets of Plausible 3D Human Models to Ambiguous Image Data, 2020

[2] Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic modeling
for human mesh recovery, 2021

[3] Tuomas P. Oikarinen, Daniel C. Hannah, and Sohrob Kazerounian. Graphmdn: Leveraging graph
structure and deep learning to solve inverse problems, 2020
clarity,_quality,_novelty_and_reproducibility: Quality
* This paper lacks sufficient details on Sec.3, which lowers its quality.

Clarity
* There are some theoretical analyses that need to be enhanced. The whole motivation and experiment are clear.

Originality
* The novelty is that it questions the plausibility of one widely used metric for evaluating 3D pose models, which has not been seen in other works. 

summary_of_the_review: This paper starts with a nice motivation, but the writing and presentation are suboptimal. This paper claims the necessity of using ECE to evaluate the regression model, but I think Sec.3 is not solid enough to let me believe the way they formulate ECE is correct, which further affects the credibility of comparison results in Sec.5. I would consider changing my rating If the above comments are addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper raises an interesting problem, which is the miscalibrated distributions caused by using $minMPJPE$ to choose the best estimates in multi-hypothesis 3D pose estimation. To resolve the miscalibration, the paper uses the existing ECE metric proposed by Naeini et al., and proposes cGNF to learn the conditional distribution $p(x|c)$.
strength_and_weaknesses: Pros

- The topic and the perspective of this paper are interesting. 
- The analysis and the plots demonstrate the miscalibration issue to some extent.

Cons

- Except for the $minMPJPE$ metric, how about other factors, such as the model itself? For example, the distribution of the multiple hypotheses is depending on the design of the normalizing flow model, therefore, the miscalibration of the variance may also originate from the capacity of the model itself. 
- The logic is not so clear. I'm agreed that one of the miscalibration reasons is using $minMPJPE$. But how could cGNF solve this and does it have a theoretical guarantee? And there are already some works using normalizing flow as mentioned in the paper, what's the difference between cGNF and [1][2]?
- Experiments are too few. The goal of multi-hypothesis is to solve the ambiguity in monocular 3D pose estimation by approaching the real distribution, which is expected to improve the ability to generalize to a new scene. However, all the experiments are conducted on H36M, which is a quite restricted and constrained scenario. Therefore, could the method also solve the miscalibration issue on the unseen 3DHP dataset (which is a common practice to evaluate the generalization ability of the model in this field)?
- The authors also mention that the uncertainty could not be corrected captured due to the miscalibration, but no experiments about the uncertainty are presented using the cGNF. Could cGNF provide correct uncertainty? Quantitative or quality results may be needed. 
- Writing is not so clear and consistent, some notations are not clearly explained, and some even have typos. E.g. the $H_c$ in Fig. 2 caption, is actually $\bf{H}^{c}$ in the main text of Sec. 4, right? If wrong, please correct me.

[1] Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic modeling for human mesh recovery. In ICCV, 2021
[2] Tom Wehrbein, Marco Rudolph, Bodo Rosenhahn, and Bastian Wandt. Probabilistic monocular 3d
human pose estimation with normalizing flows. In International Conference on Computer Vision
(ICCV), October 2021
clarity,_quality,_novelty_and_reproducibility: Writing is not so clear. Some notations are inconsistent and not explained well.

The quality and novelty are good but lack more quantitative results to prove, and more theoretical proofs of cGNF.

Code is not provided.

summary_of_the_review: This paper is novel and interesting, but is not ready for publication at ICLR at this stage due to limited analysis and experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work focuses on two main aspects of lifting 2D pose into 3D: 1) proving that the popular MPJPE metric is causing models to be overconfident and not accounting for uncertainty, and proposing a new metric in pose estimation that also measures calibration, ECE; 2) proposing a new method, cGNF conditioned Graph Normalizing Flows, that performs comparably in the MPJPE spectrum while outperforming other methods in ECE.
strength_and_weaknesses: ## Strength

- Analysis of calibration and the metric minMPJPE: this work conducts a thorough analysis of the issues with the popular minMPJPE metric. Namely, probabilistic methods trained to minimize MPJPE tend to collapse to the mean and not account for the uncertainty (overconfident). By approximating the uncertainty of the pose by using model prediction error and uncertainty of the samples by using deviation from the median, the analysis shows that directly utilizing MPJPE as a loss/metric will cause the methods to produce a distribution that has high calibration error.
    - The authors conducted a series of well-thought-out toy examples to showcase this phenomenon.
- The proposed cGNF model achieves comparable results against SOTA method in terms of MPJPE while also producing well-calibrated pose distribution that captures the uncertainty of the input 2D pose.
- This work conducts a comprehensive experimental analysis of state-of-the-art methods and showcased the strength of the proposed method. The paper is also clear and well-written.

## Weakness

- While the proposed shortcoming of MPJPE and ECE metric makes intuitive sense, I find the proposed method is quite disconnected from the main motivation. It is hard for me to find how the design choices made for cGNF relate to a better measure of the underlying distribution. The proposed “training using subset of observation” is close to a masking strategy and the training loss is a fairly standard NLL loss for normalizing flow in pose estimation [2]. As a result, I could not make a connection between the objective of obtaining a better-clibrated model and the actual proposed method (is the innovation in the architecture? I could not make a connection there either). While I find the analysis interesting and well-designed and quantifies a known pose estimation issue well (that SOTA methods often do not measure uncertainty well), the method does not seem to draw insight from it.
    - The claimed to estimate both conditional (which I see) and marginal (which I do not see) using the cGNF model needs to be further explained.
    - I also do not see how this is a “zero-shot density estimation problem” and how randomly using a subset to train can lead to this. (It could be me not understanding it and if the authors could further elaborate on this I could consider raising the score.).
- While the paper focuses on quantifying uncertainty and occlusion in pose estimation, few examples and results were actually shown showcasing the strength of the model. It would significantly strengthen the claim if **extensive** visual examples could be shown the benefit of the model (e.g. uncertain 2D keypoints actually correspond to the more spread-out hypothesis, and, equally important, that 2D keypoints with little ambiguity leads to a model that is closer to the mean).

## Question to authors

- During the analysis of the miscalibration behavior in using minMPJPE, the models’ samples’ deviation from the median is used to construct the error distribution. The difference between the ground truth and the median is then used to approximate the ground truth error.
    - I am not sure how the second part approximates the actual uncertainty in the ground truth samples. Each ground truth sample $m$ has a unique uncertainty associated with it. For instance, occluded poses lead to more significant errors. In equation (3) the summation term lumps all of them together and forms a distribution. This amounts to measuring the uncertainty at a per-joint level ($\epsilon_{m, k}$) and not per sample level. On the other hand, the uncertainty of the model is done at a per-sample model. I understand that it would be difficult to measure the uncertainty at a per-sample model for ground truth data, but the current model seems questionable.

[1] Wehrbein, T., Rudolph, M., Rosenhahn, B., & Wandt, B. (2021). Probabilistic Monocular 3D Human Pose Estimation with Normalizing Flows. *2021 IEEE/CVF International Conference on Computer Vision (ICCV)*, 11179-11188.

[2] Kolotouros, N., Pavlakos, G., Jayaraman, D., & Daniilidis, K. (2021). Probabilistic Modeling for Human Mesh Recovery. *2021 IEEE/CVF International Conference on Computer Vision (ICCV)*
, 11585-11594.

Small error: right above page 3 equation (3), should it be $\epsilon_{m, k}^*$?
clarity,_quality,_novelty_and_reproducibility: This paper is well-written and clear. I find the analysis of the minMPJPE metric novel and well-motivated, though I find it hard to make a connection between the proposed model and the analysis. 
summary_of_the_review: I find the analysis presented in the paper novel and a good contribution to the community, while the proposed model is less so. There are a few remaining questions that I would like to discuss the authors and am open to more clarifications.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work proposes a method to lift 2D human body keypoints to 3D by estimating the distribution of 3D joint locations. Special care is taken to ensure that, apart from the correct estimation of the mean, the estimated distributions are also well calibrated: It is argued and experimentally shown with toy examples that sample-based metrics for multi-hypothesis generation lead to overconfident distribution with variance lower than the true one. To mitigate this, a normalizing-flow based approach is proposed that takes advantage of the natural graph structure of the human kinematics. The resulting approach regresses the mode of the estimated distribution with small accuracy loss compared to current SoTA methods, while being better calibrated.
strength_and_weaknesses: This work is generally well presented, and the results are convincing.

One issue regarding the motivation of the work is the actual need for a well-calibrated method, especially if there is a trade-off with respect to accuracy: One may favor over-confident and more accurate models over (slightly) less accurate ones that are well-calibrated. It is perhaps useful to "know when you don't know", but arguably it is more useful to have more accurate estimations on average.

Another issue that may not be as important is the use of isotropic distributions: for the problem at hand, arguably the uncertainty of depth is possibly higher than that for the other two dimensions. This in turn may skew the estimation of the variance when assuming isotropic behavior.
clarity,_quality,_novelty_and_reproducibility: Within the space limitations of the conference, I think the clarity of presentation is fine. The quality and novelty of this work are also without major issues, despite the fact that normalizing flows have been used in other problems previously. Finally, the manuscript is clear enough to fully enable reproducibility of the work to the best of my understanding.
summary_of_the_review: After the issues raised by the other reviewers and the response of the authors, I think this work is not yet ready for acceptance in ICLR. Although there is a motivation for the problem, it is not clear whether the proposed approach tackles it explicitly.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper raises an interesting and overlooked problem for multi-hypothesis 3D human pose estimation.  Specifically, existing approaches tend to yield and favour miscalibrated distributions due to the use of the minMPJPE metric.  To mitigate the mis-calibration, the paper proposes adopting an existing metric (ECE) and couples it with a normalising flow model to learn conditional pose distributions.

All the reviewers found the idea interesting and appreciate the analysis.  However, each reviewer had concerns over the clarity and questions.  Many of the clarity issues were resolved throughout an active discussion period between the reviewers and authors.  However, at the end of the discussion period, some concerns still remained for the reviewers.  The biggest point of contention shared by several reviewers was the soundness of the proposed ECE metric and at the end of the review period, 3 of the 4 authors recommend reject, while a fourth reviewer is borderline.

Having read through the paper and exchange between the reviewers and authors, the AC concurs that the paper needs further work before it is ready for publication.
justification_for_why_not_higher_score: - soundness of propose ECE metric not clear, does not seem qualified 
justification_for_why_not_lower_score: n/a
Decision: Reject