CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators

Authors: Harry Zhang, Luca Carlone
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Human pose estimation, 3d vision, uncertainty quantification

Abstract:
We introduce CHAMP, a novel method for learning sequence-to-sequence, multi-hypothesis 3D human poses from 2D keypoints by leveraging a conditional distribution with a diffusion model. To predict a single output 3D pose sequence, we generate and aggregate multiple 3D pose hypotheses. For better aggregation results, we develop a method to score these hypotheses during training, effectively integrating conformal prediction into the learning process. This process results in a differentiable conformal predictor that is trained end-to-end with the 3D pose estimator. Post-training, the learned scoring model is used as the conformity score, and the 3D pose estimator is combined with a conformal predictor to select the most accurate hypotheses for downstream aggregation. Our results indicate that using a simple mean aggregation on the conformal prediction-filtered hypotheses set yields competitive results. When integrated with more sophisticated aggregation techniques, our method achieves state-of-the-art performance across various metrics and datasets while inheriting the probabilistic guarantees of conformal prediction.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1729059235653

thread_type:
official_review

<thread_object>
creator: reviewer_83jw

<content>
summary:
This paper proposes using a diffusion model to generate multiple hypotheses for temporal 2D to 3D pose estimation and introduces Conformal Prediction (CP) to select more plausible hypotheses and aggregate them. The method achieves competitive results on two pose estimation benchmarks.

strengths:
1. The paper introduces the novel idea of using Conformal Prediction to select more plausible results from multiple hypotheses, which is a fresh perspective.
2. The quantitative experiments demonstrate that the hypotheses selected and aggregated using this strategy outperform the results without filtering.

weaknesses:
1. The paper's contribution is to be discussed. First, the ambiguity in temporal 2D-to-3D pose estimation is not as significant as in the single-frame, single-view case. Additionally, generating multiple 3D hypotheses using diffusion models is already a widely used technique (DiffuPose (Choi et al., 2023) DiffPose (Gong et al., 2023) DiffPose (Feng et al., 2023) D3DP (Shan et al., 2023)). If the only contribution is the use of CP, its novelty might be limited. If the authors could show that the CP strategy also performs well in single-RGB input cases or demonstrates its effectiveness in single-image mesh estimation, it would significantly enhance the paper's contribution.
2. The quantitative results are not very solid. The improvement on 3DHP is quite marginal, making it unclear whether the improvement is due to random seed variation. These results do not convincingly support the method's effectiveness.
3. The visual results (e.g., Figure 10) do not show cases where severe occlusion leads to incorrect 2D estimates or cases where 3D depth errors occur. It’s important to demonstrate that CP can filter out incorrect hypotheses in such challenging cases, where multi-hypothesis generation is truly necessary. This ties back to the first weakness, as the temporal nature of the data may already provide additional constraints, reducing ambiguity. Therefore, the authors should explore more varied tasks to prove CP's efficacy, such as monocular pose estimation.

questions:
Regarding the method, it is unclear how the approach handles situations where the 2D estimates are incorrect or when the 2D pose is truncated beyond the image boundaries.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640063695

thread_type:
official_review

<thread_object>
creator: reviewer_aay1

<content>
summary:
This paper focuses on 3D pose estimation from 2D pose inputs in a multi-hypothesis sequence-to-sequence way. Following previous methods, they also adopt a diffusion model to regress the 3D pose hypotheses from sequential 2D pose inputs. Different from previous work, this paper proposes to learn a score model along with the regression model to predict the confidence of multiple regressed hypotheses. The scores of multiple hypotheses could be used to filter out the outliers to refine prediction set.   
The eperiments are mostly conducted on two benchmarks, Human3.6M and MPI-INF-3DHP. Compared with directly averaging all multi-hypothesis predictions, averaging filtered predictions with the proposed score model reduces 1.5mm MPJPE on Human3.6M, while selecting the 3D pose whose 2D projection is more consistent with the 2D pose further reduces 1.5mm MPJPE on Human3.6M.

strengths:
1. Learning the uncertainty of multi-hypothesis 3D human pose regression seems to be reasonable. Aside from building the distrubution of 3D human poses, how to select the best ones is an interesting research topic.   
2. Designs in scoring the hypotheses via conformal prediction are subtle. Detailed ablation study further reveals the influence of different parameters.   
3. The description of method is clear. Especially, Fig. 2 provides a clear overview of the relationship between different designs.

weaknesses:
1. Experiments.   
Tab. 1 and Tab. 2 show the improvement, compared with CHAMP-Naive, via introducing comformal prediction (CHAMP) and further selecting the 3D pose via measuring the 2D projection and 2D pose (CHAMP-Agg).  However, one important experiment seems to be missing here, CHAMP-Naive + -Agg. How well it performs if the proposed comformal prediction is not used?   
This experiment will clearly show how important the proposed comformal prediction is.   
Besides, the quantative eperiments are only conducted on indoor benchmarks. The qualitative results are relatively limited.  Performance on in-the-wild benchmarks, such as 3DPW, is not clear.   

2. The performance improvement is not significant compared with the baseline SOTAs, which might effect the value of this work.

questions:
See the experiments mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665217225

thread_type:
official_review

<thread_object>
creator: reviewer_rtwb

<content>
summary:
This work tackles the task of 2D-to-3D lifting of temporal human pose sequences through a diffusion-based multi-hypothesis approach. The novel aspect is applying the tools of conformal prediction to this task. That is, the authors propose to learn a conformity score function in an adversarial setup and simulate conformal prediction during training, directly minimizing the conformal set size (the inefficiency). Quantitative evaluation is performed on the well-known Human3.6M and MPI-INF-3DHP datasets, with additional qualitative evaluation on in-the-wild videos. Results show that filtering according to conformity scores and aggregating the results improves the metrics and achieves state-of-the-art results. Ablations also show that the end-to-end training of the scoring function is beneficial.

strengths:
The application of conformal prediction to 3D human pose estimation is an interesting and original aspect. Multi-hypothesis 3D pose estimation has been a relevant research area due to ambiguities in the task, but practical methods are often not formulated in a proper probabilistic manner. Using conformal prediction techniques is a promising direction and taking a first step in this direction is relevant to the community.
The overview of the related works is extensive, and the comparison of them helps the reader to place this work in relation to them. The text is well-structured and easy to follow.
The authors use already existing techniques in good ways, for example using the already proven MixSTE model for denoising, incorporating adversarial training and conformal calibration.
Results on Human3.6M and MPI-INF-3DHP both demonstrate the benefit of the method.
Ablations and hyperparameter studies confirm the design choices.

weaknesses:
The method is only evaluated on studio datasets, but not on outdoor ones, such as 3DPW and EMDB. Evaluation on such less restricted videos would be more convincing.
It is unclear whether CHAMP's learned conformity scoring performs better than the prior established 2D-joint projection based version (see question below).

questions:
I don't fully understand the CHAMP-Agg method. Is this using J-Agg on the full set of hypotheses, or does it also use the learned scoring function in some way? How should we interpret the fact that "CHAMP-Agg" is better than "CHAMP"? Does it mean that the proposed learned conformity scoring does not perform better than the prior method of aggregation proposed by Shan et al. (2023)?

To the paragraph after Eq. 5: How can we interpret the "true value to be included in" the conformal set in case of continuous values like 3D coordinates? In case of classification this makes more sense, but perhaps the language has to adjusted here.
What 2D pose estimator is used in the main experiments, or is the ground truth used?
Why are values missing from Table 3?

Smaller issues:
GAN is cited as the CACM 2020 paper, but perhaps the original NIPS 2014 version is better suited for citation.
In Eq 9, perhaps Temperature could be denoted by some other symbol because mathcal{T} and tau look very similar visually (except for size). Similarly I would not recommend relying on the subtle difference between C and mathcal{C} for important differences (as noted in the footnote of page 6), a hat, tilde or similar feature may be more visible.

Page 7: Human3.6M is no longer the "largest indoor dataset for 3D human pose estimation". There are many larger ones such as DNA-Rendering.
It should be stated explicitly that E2E refers to "end-to-end".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730792244483

thread_type:
official_review

<thread_object>
creator: reviewer_dk3y

<content>
summary:
CHAMP presents a conformalized multi-hypothesis pose estimation pipeline. Based on diffusion model, the multiple pose hypothesis are refined by a differentiable comformal prediction module with an inefficiency loss. Experiments show that CHAMP achieves state-of-the-art results on several benchmarks.

strengths:
1. The proposed conformal prediction is simple yet effective. 
2. This paper is well-written and easy to understand.

weaknesses:
1. CP removes the predictions with low confidence and visualisations also show after CP the predictions become more certain and accurate. However, the results are mainly shown with easy cases where occlusions barely present. I am wondering how CP performs on hard cases where occlusions, truncations exist?
2. It will be better if this method can be extended to image-to-3d pipelines.

questions:
This paper is well-presented and validated. The only thing I concern is its performance on occluded or truncations where diverse multiple hypothesis are preferred.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732483884152

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Meta Rebuttal - Addressing Common Questions and Brief Summary of Changes

comment:
Once again, we sincerely appreciate the reviewers’ efforts to review our work. We have addressed the comments and suggestions and ran new experiments to address the points raised by the reviewers. We have **updated the manuscript PDF and highlighted the new changes** to the content. Some common questions raised by the reviewers were summarized in this section, and individual reviewers’ questions were addressed in the comments below.

### **Common Questions**:
- **CHAMP performance under “difficult” cases such as truncation and occlusion (raised by Reviewer dk3Y and Reviewer 83JW) in 2D inputs**: We clarify that CHAMP assumes 2D keypoints are provided and improving the quality of 2D keypoints is not in the scope of our work. However, it would still be interesting to see how CHAMP behaves under ambiguous 2D keypoints (i.e., 2D keypoints with large uncertainty from truncation and occlusion).  We selected both synthetically-truncated RGB inputs and self-occluded RGB inputs. To make the task more difficult, as suggested by Reviewer 83JW, we trained another model using $N=1$ (frame number = 1) to eliminate the constraints/disambiguation from the temporal effects. Interestingly, CHAMP performs well in these cases and learns to discern occluded/wrong 2D keypoints from the good ones. Some interesting behaviors are observed —CHAMP tends to output more diverse proposals for ambiguous keypoints. (Appendix K)
- **CHAMP’s performance on harder datasets such as 3DPW (raised by Reviewer rtwB and aAY1)**: We were able to test our method on 3DPW after the submission period. Following previous work, we test on 3DPW as an out-of-distribution generalization task, by training jointly on Human 3.6M and MPII. CHAMP performs well in this task and beats baseline methods. (Appendix L).
- **CHAMP’s performance compared to current SOTAs (raised by Reviewer aAY1 and Reviewer 83JW)**: While the performance increase over SOTAs is not huge, the improvement is quite noticeable going from D3DP (CHAMP’s human reconstruction backbone) to CHAMP. Our main argument in this work is that Conformal Prediction with an end-to-end trained conformity score (uncertainty) is able to improve performance without substantial changes to the pose prediction backbone. We further provided **a new baseline, CHAMP-Naive-Agg**, indicating that the CP-based aggregation improvement is a lot more significant than using joint-based aggregation.
- **CHAMP's novelty (raised by Reviewer rtwB and 83JW)**: We maintain that the end-to-end conformity score in 3D HPE is a key novel contribution in that it is able to improve the 3D HPE performance without substantial backbone design changes. We added an important component on top of current SOTA methods, namely, the uncertainty quantification part ---**by checking the set membership using the learned uncertainty score, one could decide if the results can be trusted with probabilistic guarantees**. Moreover, the added CP component based on ConTr was initially designed for classification. To our knowledge, we were the first work that modified and applied it for a regression setting. Moreover, our method did achieve **reliable uncertainty quantification (empirical coverage), which is a trait that no other baseline method could achieve**.

### **Main Changes to the Manuscript**:
- **Section 4**: Notation and citation changes as suggested by Reviewer rtwB.
- **Section 5**: Added new CHAMP-Naive-Agg experiment suggested by Reviewer aAY1.
- **Section 5**: E2E clarification and missing baseline performance suggested by Reviewer rtwB.
- **Appendix J**: Single-frame variation of CHAMP suggested by Reviewer 83JW
- **Appendix K**: Performance when large uncertainty in 2D keypoints is present, suggested by Reviewer dk3Y and Reviewer 83JW.
- **Appendix L**: 3DPW performance requested by Reviewer rtwB and aAY1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1741161055362

thread_type:
official_comment

<thread_object>
creator: reviewer_83jw

<content>
title:
Concern Regarding the Omission of Previously Discussed Points in the Final Version

comment:
It is concerning that the authors have not incorporated the key points we discussed during the discussion phase into the final version, despite their explicit commitment to doing so. Given the strong connections between CHAMP and prior works, our discussion emphasized the need for a more thorough comparison and clearer positioning. These clarifications were essential for distinguishing CHAMP’s contributions and ensuring that its novelty is properly articulated. 

Since the authors had explicitly acknowledged the importance of these discussions and committed to incorporating them, it is disappointing that they are not reflected in the final version. Addressing these points would have significantly improved the clarity and impact of the paper.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses the task of 2D-to-3D lifting of temporal human pose sequences through a diffusion-based multi-hypothesis approach. It introduces Conformal Prediction (CP) to select more plausible hypotheses and aggregate them.  The method learns a conformity score function in an adversarial setup and simulates conformal prediction during training, directly minimizing the conformal set size (the inefficiency).  Applying conformal prediction to 3D human pose estimation is an interesting; in particular, in the context of selecting more plausible results from multiple hypotheses.  The proposed uncertainty quantification component will be beneficial to the community. One the other hand, the reviewers raised concerns regarding insufficient validation, unclear explanation of the proposed method, and lacking comprehensive discussion on the positioning and contribution of the presented work against prior works. To demonstrate the generalization capability of the method, evaluation on in-the-wild datasets (where occlusion, truncations exist) such as 3DPW and EMDB is required.  It is also suggested that the effectiveness of the CP strategy in single-RGB input cases should be demonstrated to enhance the contribution. The authors have provided additional experiments to address the concerns on experiments in the rebuttal with arguing in-depth analysis of the results. The authors’ rebuttal and following discussion between the authors and the reviewers have resolved the raised concerns, leading to clear contribution of the paper with convincing validation.  The reviewers are all positive for the paper.  This paper should be accepted, accordingly.