CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators
Authors: Harry Zhang
Luca Carlone
Venue: ICLR 2025 Poster (2025)
Keywords: Human pose estimation
3d vision
uncertainty quantification
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A novel method for estimating 3D human pose from RGB videos with uncertainty certification

Abstract: We introduce CHAMP, a novel method for learning sequence-to-sequence, multi-hypothesis 3D human poses from 2D keypoints by leveraging a conditional distribution with a diffusion model. To predict a single output 3D pose sequence, we generate and aggregate multiple 3D pose hypotheses. For better aggregation results, we develop a method to score these hypotheses during training, effectively integrating conformal prediction into the learning process. This process results in a differentiable conformal predictor that is trained end-to-end with the 3D pose estimator. Post-training, the learned scoring model is used as the conformity score, and the 3D pose estimator is combined with a conformal predictor to select the most accurate hypotheses for downstream aggregation. Our results indicate that using a simple mean aggregation on the conformal prediction-filtered hypotheses set yields competitive results. When integrated with more sophisticated aggregation techniques, our method achieves state-of-the-art performance across various metrics and datasets while inheriting the probabilistic guarantees of conformal prediction.

Published as a conference paper at ICLR 2025
CHAMP: CONFORMALIZED 3D HUMAN MULTI- HYPOTHESIS POSE ESTIMATORS
Harry Zhang, Luca Carlone
Massachusetts Institute of Technology Cambridge, MA 02139, USA [harryz, Icarlone| @mi t. edu
ABSTRACT
We introduce CHAMP, novel method for learning sequence-to-sequene multi- hypothesis 3D human poses from 2D keypoints by leveraging a conditional dis- tribution with diffusion model. To predict single output 3D pose sequence, we generate and aggregate multiple 3D pose hypotheses. For better aggregation results, develop method to score these hypotheses during training. effectively integrating conformal prediction into the learning process. This process results differentiable conformal predictor that is trained end-to- mod-to-end with the 3D pose es- timator. Post-training, the learned scoring model is used as the conformity score, and the 3D pose estimator is combined with conformal predictor to select the most accurate hypotheses for downstream aggregation. Our results indicate that using simple mean aggregation on the conformal prediction-filtest hypotheses set yields competitive results. When integrated with more sophisticated aggrega- tion techniques, our method achieves stateoofthe--an performance across various metrics and datasets while inheriting the probabilistic guarantees of conformal prediction. Interactive 3D visualization, code, and data will be available at this website,
INTRODUCTION
Learning to estimate 3D human poses from videos is important task in computer vision and robotics. Typical 3D human pose estimators learn to predict single 3D human pose from an RGB image, either in an end-to-end manner or by relying pre-existing 2D keypoint detection methods applied to the RGB images. Recent advances in sequence-to-sequenee modeling enable learning 3D human poses in sequence by using RGB videos, which significantly improves the flexibility and efficiency of such estimators (Zhang et al. 2022b]a). a). While promising, such approaches only focus on single-hon hypothhsysss predictions, which is an inherently ill posed problem when trying to reconstruct a 3D human pose given inputs collected from one viewpoint.
Thus,
recent line of work focuses on learning multi-hypothesis 3D) human pose estimators from 2D inputs. Instead of learning one deterministic target, such methods model the 2D-to-3D pose learning problem as learning conditional distribution, which describes the distribution of the 3D poses given 2D inputs. This change in problem formulation prompts the use of generative models such GANs, VAES, and Diffusion models (Li & Lee 2020 Sharma et al. 2019 Shan al. 2023), which can propose multiple hypotheses of 3D poses given single 2D input. For practical uses, one should also consider aggregating the hypotheses generate one single output prediction. However, due to the imperfection of the trained generative models, using all hypotheses when aggregating could be inefficient and suboptimal if some are exceedingly inaccurate.
To counter the aforementioned problems, we take inspiration from an important tool in statistical learning, conformal prediction (CP) (Shafer & Vovk 2008 Angelopoulos & Bates 202I), which uses post-training calibration step to guarantee user-specified coverage: by allowing to predict confidence sets C(X), CP guarantees the true value Y to be included with confidence level a, when the calibration examples (X,, YY) Ieal are drawn exchangeably from the test distribution. There are typically two steps involved in the CP process: In the calibration step, the conformity scores on the calibration set ranked to determine cut-off threshold for the
iec P(Y C(X) 2

---

Published as a conference paper at ICLR 2025
predicted values guaranteeing coverage - a, usually via quantile computation. In the prediction step, conformity scores, which measure the conformity between the output and possible ground-truth values, are computed to construct the confidence sets ((X) by using the calibrated threshold т. CP
Figure 1: CHAMP sample results obtained wild videos collected from TikTok. Having observed 2D keypoints, CHAMP proposes multiple hypotheses of the 3D human skeleton poses, and then conformal predictor trained end-to-ena with the pose estimator refines confidence set by filtering out low-conformity- score hypotheses. This smaller set will used downstream aggregation for single output prediction.
is highly flexible as it can be applied to any machine learning model. However, since it is applied post-training, learning the model parameters is not informed about the post-hoc conformalization step: the model is not tuned towards any specific CP objective such as reducing expected confidence set size (inefficiency). To bias the training towards lower inefficiency, ConfTr (Stutz et al., 2021) proposes fully differentiable CP operation for classification tasks, which is applied end-to-end with the prediction model and optimizes differentiable inefficiency loss tandem with the original class loss for the classifier. This operation allows the model to learn parameters to reduce the inefficiency of the confidence set size during conformalization. Inspired by this, we wish to learn a scoring model that scores the "quality" of each hypothesis by measuring its conformity to the ground truth. We can then use this scoring model to simulate CP and optimize for the inefficiency in differentiable manner, as done in (Stutz al. 2021). During test time, propose a large number of hypotheses and do regular CP to filter out the low-score ones before feeding into the downstream aggregation steps. The learned CP procedure can be applied to any multi-hypor pytthesis estimator training process to refine the hypotheses confidence set size during test time for better aggregation results. With this learned CP wrapper, we present CHAMP, Conformalized 3D HumAn Multi- Multi-Hypothheess Pose Estimator. To summarize, the contributions of this work include:
haltiisyp 3D human pose estimator from 2D keypoints. A novel method to conformalize 3D human pose estimates during training by learning a score function to rank the quality the proposed hypotheses.
novel gequence-to-sqquenee
A novel method to aggregate the multiple hypotheses from the estimator output using confor- malization based on learned score function.
Quantitative and qualitative results that demonstrate the tfate-oftthe-e results of our method on a variety real-world datasets.
RELATED WORK
Diffusion Models are a family of generative models that gradually corrupt data by adding noise,
and then learn to recover the original data by reversing this process (Sohl-Dickstein et al. 2015 Ho
jct al. 2020 Song & Ermon| 2020 Song et al.) (2020b a). They have achieved success in generating
samples in various applications such image generation (Ho et al.) 2022 Saharia
et al.
2022a Nichol et al. 2021 Batzolis et al. 2021 Rombach et al. 2022; Sahani et al. 2022b),
multi-modal learning (Lev [eekkoiteh et al. 2022 Kim et al. 2022 Huang al. 2022 Avrahami al.
2022 Fan et al. 2023) and pose estimations in 3D vision problems (Gong et al.
2023 Wu et al.
2023
We use Diffusion models to learn 3D human poses from 2D keypoints.
2

---

Published as a conference paper at ICLR 2025
3D Human Pose Estimation is an important problem in computer vision and robotics. With deep learning, various end-to-end approaches have been proposed (Tekin et al. 2016; Pavlakos et al.
2017 Sun et al. 2018). However, with the maturity of 2D human keypoints detection, (Ho et al. 2022]. more robust approaches focus on uplifting 2D keypoints to 31), resulting in better perfor- mance (Xu & Takano 2021 Zhao al.] 2019 Ma al. (02T), where one typically deals with frame-to-frame problem, or
soaieencett-sqqeene problem. Being able to predict 3D keypoints sequence directly from a 2D keypoint sequence is highly desirable as it has higher efficiency and flexibility. In this scheme, deterministic methods learn to predict one single 3D output from the 2D input (Zhan et al., 2022: Zhang et al. 2022b] [Zhao et al. 2023). However, with a single viewpoint, such a formulation is ill-posed because there could be many possible 3D poses satisfying the same 2D keypoint configuration. Thus, it is desirable to model the problem as a conditional probabil- ity distribution. Deep generative models have shown great potential in modeling such distributions.
Specifically, mixed-density network (Li & Lee 2019), VAE (Sharma et al. 2019). normalizing flows (Wehrbein et al. 2021). GAN (Li & Lee 2019) and Diffusion models (H (Holmquist & Wandi [023]
Shan et al., (023) have all been applied to modeling such conditional distribution. In our work, we
use (Zang et al. 2022b) as the backbone for generating 3D keypoints sequences from 2D keypoints
sequences. To infer multiple hypotheses, we follow the frameworks in (Shan et al., 2023 Gong jet al.
2023 Zhou et al. 2023) to learn recover 3D pose hypotheses from Gaussian noises. Ci
ct al. (2023) uses an innovative approach to learning probabilistic human poses via gradient flow
diffusion. Zhu et al. (2023) proposes a novel framework for scaling up 2D-3D human pose learning.
Mchraban et al. (2024) models local dependencies inherent in human pose sequences, outperforming transformers backbones. Xu et al. (2024) proposes multiple hypotheses with Diffusion and learns to
score them end-to-end, but it does not consider the inefficiency of the prediction set during training.
Conformal Prediction (CP) is a powerful and flexible distribution- free uncertainty quantification technique that can be applied to any machine learning model (Angelopoulos & Bates 2021 Shafer & Vovk 2008). Assuming the exchangeability of the calibration data, CP has desirable coverage guarantees. Thus. it has been applied to many fields such as robotics (Huang et al.) 2023 Lindemann
cct 2023). pose estimation (Yang & Pavone (023), and image regression Teneggi et al. 2023
Angelopoulos et al. 2022). More sophisticated CP paradigms have been also proposed tackle
distribution shift and online learning problems (Gibbs & Candès) 2022 Bhatnagar et al.; 2023
Since CP is applied post-training, the learned model is not aware of such conformalization during training. To gain better control over the confidence set of CP, learning conformal predictors and nonconformity score end-to-end with the model has been proposed in (Stutz et al., 2021 Fisch et al., 2021] Bai et al. 2022). More closely related to our proposal is the work of (Stutz et al.) 2021), but instead of using raw logits as the conformity score for classification, we learn an extra scoring model as the score to rank the hypotheses and simulate CP during training.
3 PROBLEM FORMULATION
We are interested in the problem of learning a sequence of 3D human poses from sequence of 2D human pose keypoints. We assume the 2D keypoints are available to us, which could be detected from the RGB images using well-established methods such (Li et al. 2022b). Formally, given the input 2D keypoints sequence x = fp221n = 1,..., N), where pa E Jx2 our goal is to learn conditional distribution for the possible corresponding 3D positions of all joints po pg((a)), where the sequence y (podlm In 1,...,N! N) and p E IR Xx3 Here, N represents the number of input and output frames and J is the number of human joints in each frame. With the learned distribution Pe(y) æ). we are able to do inference and generate hypotheses of 3D poses Hy E " Jx3 where H is the number of the hypotheses. We then conformalize the hypotheses to select the higher-quality ones and aggregate the latter in order to obtain a single estimate ÿ as the final output.
4 METHODS
We use Diffusion Model (Ho et al. 2020) to learn the conditional distribution po(y (z) due to its capability of modeling complex distributions and producing high-quality samples. With the trained Diffusion Model, we able generate many hypotheses sequences and aggregate them to produce a robust single prediction sequence for the final output. To achieve this, we learn an extra scoring model to rank the hypotheses and optimize the size of the hypotheses confidence set, thresholded by
3

---

Published as a conference paper at ICLR 2025
Keypoints
Keypoints Hy potheses
Refined Keypoints Hypotheses
Diffusion
Differentiable
Model
CP
Forward Pass
Backward Pass
Pose Loss
Ineffieiency
Loss
Figure 2: CHAMP Overview. CHAMP takes input sequence of 2D keypoints detected series of input RGB video frames. The 2D keypoints sequence gets fed into Diffusion Model to produce 3D keypoints hypotheses for the sequence. The output of the Diffusion Model is supervised via Pose Loss. Then we apply differentiable CP end-to-end during training the hypotheses sequences, resulting in smaller confidence set. The confidence set is used calculate Inefficiency Loss during training. Note that show frame in the sequence and hard assignment for confidence set during training for better interpretability.
the 90% quantile score, effectively simulating conformal prediction during training, as illustrated in Fig. 2
4.1 LEARNING 3D HUMAN POSES WITH DIFFUSION MODEL
The use of Diffusion Models 3D human pose estimation has been shown effective various previous works (Feng et al., 2023 Gong al.) 2023] Shan et al. 2023 Choi al.) 2023). and adapt the same paradigm in modeling the forward and the reverse process for the Diffusion Model.
Forward Process. In the forward process, the Diffusion Model takes input ground-truth 3D human pose sequence Yo & R x3 and a sampled time step ~ Unif(O, 7), where T is the maximum number of diffusion steps. Then the forward process gradually diffuses the input by adding independent Gaussian noises NN0 I)) at each step. The nice property reported in (Ho al. 2020) shows that the process can be succinetly written as:
g (yo) = Vā,yo + VI-
where āg is a constant depending on the cosine variance schedule.
Reverse Process. To train such Diffusion Model, in the reverse process, we denoise the corrupted 3D pose sequence yt- While the derivation in (Ho et al. (0200) simplifies the ELBO objective to min- imizing the distance between the injected noise E, and learned noise Eg (v/āyo + VI ãg t), we frame the problem slightly differently, following (Shan et al. (023). Instead of training net- work to learn the injected noise, we train a denoiser model Do that outputs the predicted pose yo directly:
Do = Do (3: x.,
where the denoiser model takes input the corrupted 3D pose sequence, the input 2D keypoints quence, and the diffusion step to recover the uncorrupted input pose sequence Jo. We then supervise the predicted sequence with the ground-truth sequence using the mean per-joint MSE:
llvo Doll
Cpose
(3)
N
Denoiser Model. We build upon the MixSTE model (Zhang et al. 2022b) as the denoiser model in the reverse process of the Diffusion Model. MixSTE uses two separate attention mechanisms, effectively learning spatial and temporal relationships of the keypoints sequence in modular way by combining several spatial and temporal attention blocks. To condition the MixSTE network on the corrupted 3D keypoints sequence, we change the network input by concatenating 2D keypoints and noisy 3D poses and we add a diffusion timestep embedding after the input embedding in the first attention block. A detailed description of the denoiser model is given in Appendix B and
Generating Hypotheses. Using the denoiser model. which models the conditional distribution po(y) ar), able to generate number of hypotheses during inference. Following the reverse
4

---

Published as a conference paper at ICLR 2025
process of the Diffusion Model, obtain the hypotheses set H, {ÿ', y ..... ................................ 8"} of size H by sampling standard normal noises y ~ I)) E RV/333 where the superseript h [1.2..... H] indexes the hypotheses, and feeding them into the denoiser model Dg:
J' Dolyt, a. T)
(4)
4.2 LEARNING CONFORMALIZATTION FOR THE HYPOTHESES CONFIDENCE SET
While we are able to generate arbitrarily large numbers of hypotheses during test time with the distribution learned by the Diffusion Model, we still need to aggregate the proposed hypotheses into single prediction. The correct way aggregate the hypotheses remains open problem. Taking a naive average would be suboptimal as the existence of outliers in the proposed hypotheses might skew the average. Other approaches improve the aggregation step by selecting the hypotheses that result in closer distances to 2D keypoints after projection via camera matrices, but still retain large set of hypotheses before the aggregation step (Shan al. 2023).
Conformal Prediction. A possible solution is to conformalize the hypotheses using conformal prediction (CP). Conformal prediction produces set of predictions which covers the ground truth with high probability: given a trained prediction model fe. conformal prediction first calibrates a cutoff value by ranking calibration set Laa given each component's 's conformity score ((yeat) and setting the cutoff value as the user-specified (1 + (cc-qannilll of the calibration set scores. After the calibration step (conditioned on the trained model), the conformal predictor constructs the conformal prediction set as follows:
Co(or, = {y (1) 2 +}
(5)
With a properly selected conformity score function ø. as well as a user-specified coverage parame- ter a, the post-training conformal prediction guarantees the true value's conformity score has a set membership in Ce(2, T) with confidence level a, i.c., P(yeT E Co(x, T)) 2 a. A key metric for a conformal predictor is the size of the confidence set Co(az, T), or inefficiency: a good confidence set should be large enough to cover the ground truth, and small enough to be informative (low ineffi- ciency). A very large confidence set could cover the ground truth with high probability, but it might not be necessarily useful as it is inefficient to consider lot of possible values (high inefficiency). Given H hypotheses of 3D human pose sequences, we wish to use the threshold conformal predictor above to refine the hypotheses set, resulting in H' "good" hypotheses, where H' H. Essentially, we wish to have a -nefifffice conformal predictor, which selects a smaller number of clite hy- potheses from the set for downstream aggregation. CP is intended to be used as "wrapper" around the prediction model, and we wish to achieve better, fine-grained control over the inefficiency of the CP wrapper for downstream aggregation, conditioned on the input 2D and output 3D human pose sequence data. Thus, inspired by (Stutz et al. 202I), we integrate CP into the training, optimizing for the inefficiency of the confidence set online by simulating CP during training in a differentiable manner.
Learning CP via Inefficiency Optimization. We would like to make the model itself aware of the post-hoe hypotheses aggregation step so that the aggregation step does not get skewed by low- quality hypotheses. Thus, there should be an extra component in the model to shape the hypotheses confidence set. We build on top of the ConfTr paradigm proposed in (Stutz et al. (021). We first design a conformity score function for poses. Unlike (Yang & Pavone, 2023). where the score function & is hand-designed, we wish to learn conformity score function de that measures the distance between the proposed hypothesis and the ground truth. During training, we propose Huain predicted output sequences as the hypotheses for each input 2D keypoints sequence in the mini- batch. To implement the score function, we reuse the input embedding layer of the modified MixSTE and further project the embeddings (conditioned with an extra MI.P that predicts a value E [0, 1] representing the probability that the hypothesis embedding belongs the manifold of plausible 3D human pose embeddings. The conformity score function is implemented as a discriminator-style scoring function that measures the quality of the generated 3D keypoints (Kocabas et al. 2020). Thus, higher score function output means the hypothesis is more likely (more realistic) to be from ground-truth 3D pose sequence distribution in the embedding space. Formally, the scoring function optimizes the following loss:
input 2D keypoints æt) into two scalar values
Ls E [(So(OeT) 1)²] E (0*)))
(6)
5

---

Published as a conference paper at ICLR 2025
where Our and @ are the input embeddings of ground-truth 3D poses and generated 3D poses con- ditioned on the 2D keypoints a, and S9 is the scoring function MLP. We also add an adversarial loss that will be back-prue prop-ppope into the denoiser model, done in Kocabas et al. (2020); Goodfellow et al. 2014), which encourages the 31) pose prediction model to output more realistic samples:
Ladv = E [[So - 1)²]
Lastly, during training, we use the output of So the the conformity score between the hypothesis 3D pose sequence y" and the ground-truth 3D pose sequence yor:
dh (y"" (O
(8)
With the conformity score model output, we perform differentiable CP on each mini-batch of size B during training. Similar to (Stutz et al., (02T), we split each set of Huain hypotheses of the mini- batch in half: the first half, Heal, is used for calibration, and the second half, H for prediction and loss computation. On Heal, we calibrate r by computing the (1 + 1/ caal )a-quantile of the conformity scores in a differentiable manner. On Hped we calculate the soft inefficiency score by using the soft assignment of hypothesis g being in the prediction set given the quantile threshold
(o( T) := (00(3")
17
where a here is the sigmoid function and 17 is the temperature hyperparameter. When 7 + 0, we recover the hard assignment and (1, 7) = 1 if (br 2 r and 0 otherwise Thus, we are able to measure the size of the CP set of the hypotheses by Σn (a, On Hpred. we compute Cz r)
and then calculate the differentiable inefficiency score:
Hgrall
Ω (Ca( (Z) max
Σ Co(aeT) - k,
(10)
where is hyperparameter that avoids penalizing singletons. With the differentiable inefficiency score defined, we are able to optimize the expected prediction set size across batches during training:
Saize = log E (C r))]
(11)
Combined with the adversarial loss, we have the "inefficieney" loss:
Lineff
Lsize
Ladv.
(12)
Full Training Objective. The computational graph of this loss involves the denoiser model input embedding, and since every operation is made differentiable by construction, we are able to back- propagate the inefficiency loss into the model, further shaping the model on top of the original pose loss. The final training objective of our model is thus:
C = Ppsse + XCineff
(13)
where is the weight for combined the inefficiency loss.
4.3 CONFORMALIZED INFERENCE
During inference, any Conformal Prediction method can be applied to re-calibrate on a held-out calibration set Leal as usual, i.c. the thresholds obtained during training is not kept during test time. Given trained pose estimation denoiser model Do and the conformity scoring model do trained end-to-end with Do. for each testing 2D keypoints sequence, æ, we wish to construct hypotheses confidence set for the downstream aggregation as follows:
Ca(x, = (g: de (g) 7
(14)
To generate candidate hypotheses for the hypotheses confidence set, sample from the learned distribution po(yla) (x) H times using (4) and refine them using (14), resulting in a smaller hypotheses set of size H' We then aggregate the set Co(æ,, r) by using weighted average based the conformity score output values to obtain a single output for practical use:
Egecu(m.r) (0())) y
Youtput
(15)
EgeCo Φo (g)
Note that we disambiguate prediction sets from hard and soft assignments respectively using Co and Co.

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
To evaluate our method, we train and test on standard human pose estimation datasets and provide quantitative results. We also provide qualitative results on in-the-wild videos.
Human3.6M (lonescu al. 2013) is the standard indoor dataset for 3D human pose estimation.
The dataset collects videos from II actors engaging in 15 activities and the pose sequence videos are captured by 4 synchronized and calibrated cameras 50Hz. Similar to (Shan al.) 2023 Zhang et al. 2022b), we train on 5 actors (S1, S5, S6, S7, S8) and evaluate on 2 actors (S9, S11). Quantitatively, following the standard evaluation scheme, we report the mean per joint position error (MPJPE), which is often referred to as Protocol #1, which computes the mean Euclidean distance between estimated and ground truth 3D joint positions in millimeters. We also provide Protocol #2
results in Appendix F
MPI-INF-3DHP (Mehta al. 2017) a more challenging dataset consisting of indoor and outdoor activities, from 14 camera views which cover greater diversity of poses. The training set contains activitics, and the test set contains 6. We preprocess the dataset with the valid frames provided by the authors for testing, following (Zhang et al., 2022b) [haan al., 2023 Gong al. 2023) Quantitatively, we report MPJPE, the percentage of correct keypoints (PCK), which deseribes the percentage of keypoints with Euclidean error less than 150mm, as well as the AUC score for this percentage.
For both of the datasets, in our setting, hold 2% of the training dataset for conformal cal- ibration during test time. The held-out calibration set is not seen by the network during training. We compare 4 variations of our method. The backbone (weights) of the variations and the training process remain the same, but the key difference lies in the aggregation step during test time:
Trained with inefficiency loss, but for multi-hypottheis scenarios, during test time, we do not refine the hypotheses set with CP, and we use all hypotheses for aggregation. In single-hypothesis scenarios, we only propose hypothesis the output.
CHAMP-Naive:
CHAMP-Naive-Agge CHAMP-Naive but aggregated with the J-Agg method from (Shan et al., 2023) instead of taking simple average.
CHAMP: Same backbone as CHAMP-NE -Nive,, but only aggregates refined hypotheses confidence
set in eq. (14) via weighted mean aggregation.
the J-Agg method from (Shan al. 2023) on the set in eq. (14). which uses known or estimated intrinsic camera parameters to reproject 3D hypotheses to the 2D camera plane and selecting the joint hypotheses with the minimum reprojection error. CHAMP-Best: We J-Best method in (Shan al. 2023) the set in eq. [4), which selects the joint hypothesis that is closest to the ground truth, and then combines the selected joints into the final 3D pose. This the upper bound of J-Agg performance.
CHAMP-AgE Instead of taking the average,
reprojectro
5.1 RESULTS HUMAN-3.6M
We discuss the quantitative results on the Human-3.6M dataset. In the single-hypothesis setting, set = to compare with other deterministic methods. While our method primarily focuses on multi-hypothesis scenarios, we propose 1 hypothesis and use the CHAMP-Naive variation, without using the conformal prediction pipeline. As the results suggest in Table | our method achieves performance on par with the current state-of-the-art methods the single-hypothesis case.
In the multi-hyyptthss setting, we set H 80. CHAMP variants achieve SOTA results, especially when combined with more sophisticated aggregations proposed in D3DP (Shan et al., (003)) On av- erage, without the conformal prediction pipeline. CHAMP-Nan error and improves by 2.3mm when using conformal prediction and mean aggregation. Using J-Agg and J-Best, CHAMP gets further improved by 2. Im and 4. 4.0mm respectively.
is able to achieve 39.2mm MPJPE
5.2 RESULTS ON MPI-INF-3DIE
We discuss the results on the MPI-INF-31D dataset in Table 2 In the CHAMP-Naive achieves results on par with other deterministic methods for the three metrics.
setting,

---

Published as a conference paper at ICLR 2025
Mean Per-Jeint Pustion Kiroe (Prolocol # II
AYG.
Dir
Dise.
Ext
Leest
Phone
Photo
Fose
Par
SH
SILX
Smake
Wal
Wasss
Single Hypothesis
Kayst> Zhnn ec 100
29.7
14.7
85.7
45.7
15.4
51.0
59.9
35.8
65.9
58.7
61.7
50.2
-5.4
51.5
35.6
21.8
STE 13RT 31 20123
63.6
39.9
43.4
40.0
40.9
46.4
50.6
42.1
59.8
55.8
61.6
44.9
43.3
44.9
29.9
30.3
P-STMOT Shinee 1e 2222
42.8
38.9
42.7
40.4
44.1
45.5
69.7
63.9
59.9
55.5
59.4
44.9
42.2
42.7
29.0
25.4
MixSTE PAiii 63 AMAA
41.0
37.9
80.1
37.5
39.4
43.3
50.0
39.8
59.9
52.5
56.6
42.4
40.1
40.5
27.6
27.7
DUB Ainserrin SI 20r44
40.6
37.9
41.9
26.5
39.5
40.8
29.2
40.1
40.7
47.9
55.5
40.2
41.1
40.3
30.8
28.6
D3DF SSnn 21 2012
80.0
37.7
79.9
35.7
58.2
41.9
48.8
79.5
38.3
50.5
55.9
41.6
79.4
39.00
27.4
23.5
CIIA MrP-NNive (1-11
19.7
37.8
90.0
35.7
58,1
41.5
65.7
39.2
35.2
50.1
53.3
41.5
38.9
39.7
26.7
27.2
Mulippe R6 Hypolliess
[ X723
43.0
19.3
23.1
46:
40.9
44.9
51.2
4016
41.3
53.5
60.3
43.7
411
43.8
20.8
30.6
GopAMEn
(hkannan 2011
61.3
51.9
56.1
55.
58.0
63.5
75.1
53.3
56.5
69.4
92.7
60.3
58.0
65.5
49.8
51.6
DillaPose
LLiee 2003
40.4
43.4
50.7
45.5
50.2
49.5
53.4
48.6
45.01
56/9
70.77
47.8
48.2
51.3
43.1
43.4
DinPose
36,9
33.2
36.6
33.0
35.6
57..6
45.1
35.7
35.5
46.4
49.9
57.3
35.6
36.5
24.4
24.1
DitPPse
43.3
38.1
43.3
35.3
43.1
46.5
48.2
39.0
37.6
$1.9
59.3
41.7
47.6
45.5
37.4
36.0
DADP
90.5
17.3
30.5
35.6
37.8
41.4
48.2
90.1
37.6
49.9
52.8
41.3
39
58.4
27.2
27.1
CHAMI--NINV
90.2
17.1
39.1
35.1
37.1
41.4
48.1
90.0
36.9
49.4
52.0
40.6
36.9
58.9
27.3
27.1
CHAMP-Naivv-AAg
38.7
36.8
36,9
34,4
36.4
41.0
47.2
36.7
36.5
48.5
51.2
40.2
38.4
58.2
27.3
26.5
CHAMP
36,9
36.2
38.4
32.5
33.2
39,1
43.9
36.2
34.5
44.9
48.7
39.1
37.4
51.7
25.3
26.2
CHAMP-Ag
34.8
34.9
36.5
31.1
31,1
57.8
41.3
33.5
32.4
41.8
45.
$7.2
34.9
36.1
24.4
24.1
DSDI-Road (Shnn 2025
35.4
350
54.8
31.7
33.1
37.5
43.7
84.8
33.6
45.7
478
37.0
35.0
35.0
34.3
24.1
CHASTE-RSS
32.8
32.
33.2
28.9
20.6
35.7
41.4
32.3
31.4
39.7
43.0
36.1
329
52.8
22.8
22.8
Table 1: MPJPE (L) results on Human-3.66 3.6. 6M dataset. Red: lowest Blue: second lowest error. In the single- hypothesis setting, method without CP. In the multi-hyyothesis setting, compare four different variants CHAMP discussed the main text. "Upper-bound performance, uses ground truth.
In the multi-hyypothesis setting, we again set H = 80. CHAMP-Naive again yields fairly competitive results and when combined with conformal prediction and mean, J-Agg, and J-Best aggregation techniques, the performance continuously gets improved across the three metrics, with the CHAMP-Agg metric being the most competitive one, achieving SOTA results. These results here on this dataset follow the same trend as those on the Human3.6M. 6M dataset.
MPPPE
Single Rypotheis
P-STMD Stan 2t.l
75.8
MixSTE
75.8
DAMP
77.8
76.0
Mulappe Ily Hlynohheses
MitFormct Li
2022h
95.8
63.3
58.0
DifPPss
Coli 24
R1743
98.0
73.9
29.1
DullPose
TEdis & mir
A3743
90.6
62.8
64.6
DSDP Agr SStan et
97.3
73.2
29.7
CHAASEP-Nav
73.1
29.9
CHAMMP-Navee Nair
78.2
29.5
CHAMP
73.4
20.1
CHAMP-AQC
78.7
DXD2 Dest SSaa et 2021
9830
75.
28.1
CHANWi--er
98.2
79.2
28.0
5.3 IN-THE- WILD VIDEOS
Table 2: Results on the 3DHP dataset.
To show the generalizability of our method to in-the- wild videos, we collect videos from YouTuube and Tik- Tok. To construct 2D keypoints input, we use Cascaded Pyramid Network (Chen et al. 2018) fine- tuned on the Human3.66 6M dataset as the default weights are trained on MSCOCO (Lin et al. [2014). which has slightly different skeleton structure. We directly apply the CHAMP model trained on the Human3.6n 6M dataset to test on in-the-wild videos. Sample results are shown Fig. where the input videos collected from Tok. Qualitative results suggest that CHAMP is able filter out outlier hypotheses using the conformity score function trained and-to-end with the pose estimation model. More results shown in Appendix G For 3D visualization, please refer this anonymized website to interact with CHAMP's predictions in 3D.
5.4 ABLATION STUDIES WITH HUMAN3.CR 66 DATASET
MPIPE Traiiig Hypotteses
MPIPE vs Ineffiinncy Loss Strmgth
10
MPJPE Nambers
E2E
Scp.
Fix.
E2E
Sep.
Fix.
MPJPE
32.8
35.1
35.3
Vilae A
Figure 4: Comparison of confor- mity scores. Top: filtered hy- potheses of two joints using the three scoring functions. Bottom: MPJPE (mm) values.
Figure Comparison thy- potheses in training and inference with 4 variants of CHAMP. Red: Naive, Yellow; CHAMP, Brown: Agg, Green: Best.
Figure 6: Comparison
in
loss definition L (Eq. [3} with variants of CHAMP Red: Naive, Yellow: CHAMP, Brown: Agg.
Green: Best.
8

---

Published as a conference paper at ICLR 2025
Choice of the Conformity Function. We compare our proposed end-to-end learned conformity score function with a separately trained conformity score function (Corso et al. 2022). and hand- designed conformity score function. We compare the average performance across all actions on the Human3.6W dataset. For the separately trained conformity score function, we first only train the pose estimation backhone and generate 100 hypotheses per input sequence over small subset of the training dataset. We then train the score function to predict if the proposed hypothesis conditioned on the input 2D keypoints results in an MPJPE of less than 25mm and use the ground-truth pose sequence as supervision. For the hand-designed conformity score, we use Ppea from (Yang & Pavone 2023), measuring the maximum MPIPE with respect to the ground truth across output frames. Note Φpeak measures "nonconformity". so we negate such values to fit our setup. All three variants use the J-Best aggregation method. From Fig. 4 end-to-end (E2E) score function yields the best performance and the two learned variants supersede the hand-designed version. In the two examples shown in Fig. 4 the filtered hypotheses seem to be more concentrated for learned score functions. It's 's worth noting that all three variants result in competitive performance, demonstrating the importance of CP.
Number of Hypotheses. Another interesting ablation study conduct is to ablate on the number of hypotheses during training (Hainn) and inference (H). We compare the effects of Huain. H in Fig. [] on the four variants of CHAMP. To compare the effect of Hurin, we fix H 80 and train a model for each value of hypotheses proposed during differentiable CP. Similarly, to compare the effect of H, we fix Huain = 20 during training and use the same model with different H during inference for CP. Results suggest that a good performance can be achieved with Huain = 20 during training, while any number higher than 20 brings marginal improvement and requires more GPU memory. Moreover, H = 80 during inference is sufficient for all four variants.
Strength of the Inefficiency Loss. Finally, we ablate on the value of À in the overall training objective L. This is an important ablation in that we can find suitable strength of the inefficiency loss Lineft to make sure does not conflict fpese- Results in Fig. 6 suggest that = 0.6 is the most efficient strength across all values, as smaller À does not train the scoring model sufficiently and higher A conflicts with the pose loss. This corroborates the smallcr-scale hyperparameter sweep experiments we conducted before training the models.
Correlation between Conformity Score and Prediction Error. We also investigate if the leareed conformity score is correlated with prediction error (MPJPE). To achieve this, we collect 500 random test samples and generate hy- potheses as well as the predicted conformity score values. We plot these values in Fig. 7 Experiment data suggest that the predicted conformity score has a Pearson corre- lation value of 10.41 with the MPJPE value. We further fit an Ordinary Least Squares (OLS) regression line by re- gressing the predicted score onto the prediction error. The resulting OLS fit achieves an R2 of 0.26, demonstrating that the conformity score has a fairly strong explanatory power to explain the prediction error.
Conformity Score Prediction Error
Erce& & Score
0.0
0.2
0.5 Confonmity Score
0.4
0.8
1.0
Figure 7: Correlation and OLS fit between MPJPE and predicted conformity score.
Enpertainty-Augmentee Human Pose Estimation Methods.
Other
We now compare different implementations for uncer- tainty estimation in human pose estimation. Several prior works have explored uncertainty in 3D human pose/shape
Baseline
MPJIPE (mm))
3DMB (Biggs et
20201
58.2
reconstruction (Rempe et al. 2021 Dwivedi al. 2024 Biggs et al. 2020; Zhang et al.; 2023] 2024). The cited
HuMOR (Rempe
2021
49.9
POCO DDiiedd et al.
2024
47.7
RoHM (Ziang cl al.
2024
41.6
works all simultancously recover shape and pose and we
KNOWN //han a.
20133
40.5
only compare with the pose metrics. While the meth-
D3DP Shan al.
20231
39.5
ods all incorporate uncertainty in the pose estimation pro- cess, sampling with the (learned) uncertainty remains non- trivial due to the lack of flexibility. Moreover, multiple- dypothesis-basee methods from probabilistic generative models (Shan et al., |023] Gong et al.) (202)) significantly outperform other models. CHAMP builds on top of this and added extra layer of uncertainty
Diff el al.
2023
36.9
CHXMP[Our)
34.8
Table 3: Comparison with other uncertainty design choices.
9

---

Published as a conference paper at ICLR 2025
learning, which further improves the generative model's performance. We also provide detailed design choice comparisons with the aforementioned methods in the Appendix
5.5 EXCHANGEABILITY AND CP GUARANTEE
The inherent assumption of CP requires the calibration dataset to be exchangeable, which is weaker than asking them to be independent (Angelopoulos Bates] 202T). This assumption typically fails when the calibration set is single video sequence, where the image frames are temporally cor- related. We learn sequence-to-sequee of image frames from different videos, instead of single frames from the same video. This would drastically decrease the temporal correlation across the calibration dataset. Moreover, as shown in Yann & Pavone, 2023), the video frames tend to be more independent if the videos are taken by multiple evenly spaced cameras, which is the case for our datasets of interest, We also investigate the empirical coverage of method in Human 3.6M test set in Appendix A Results suggest that we are able to inherit the coverage guarantee from CP with the learned conformity score even if the dataset is not fully exchangeable.
model, where the input and output are long sequences
5.6 IMPLEMENTATION AND TRAINING DETAILS
CHAMP's denoiser model uses Adam optimizer with a weight decay parameter of 0.1 and momen- tum parameters 32 0.99. For the training objective eq. (13), we use À 0.6. We train the CHAMP model using an NVIDIA V100 GPU, where the training process consumes an amortized GPU memory of 30GB, for 300 epochs with a batch size of 8 and a learning rate of 5e-5 and reduce it on plateau with factor of 0.5. Following previous work (Zhang et al. 2022b [Shan et al. [021] 2023). we use input pose sequence of 243 frames (N = 243) of Human3.66 3.6 universal skeleton format. During training, the number of hypotheses is 20, and #DDIM iterations is set to 1. During inference, they are set to 80 and 10, The maximum number of diffusion steps is = 999.
6 LIMITATIONS
While our method shows a promising improvement over the current 3D human pose estimation methods, it does have several limitations. First, many hypotheses need to be proposed during training to improve the learned conformity score, which consumes a lot more GPU memory. Second, a lot of computation is needed for training the scoring function, making the whole training process fairly slow. Lastly, CHAMP only learns single-huma 3D skeleton estimation, without considering human shape estimation.
CONCLUSION
This work presents CHAMP, novel method for learning multi-hyypotheess 3D human poses with learned conformal predictor from 2D keypoints. We empirically show that CHAMP achieves com- petitive results across multiple metrics and datasets, and when combined with more sophisticated downstream aggregation methods, it achieves state-of-the-an performance. Future work includes using more recent CP techniques that relax the exchangcability assumption (Barber et al. (023)), using more efficient the pipeline up dense human pose-shape joint prediction scenarios (Loper al. 2023
models such as Mamba (Gu & Dao (023), and scaling
ETHICS STATEMENT
We take ethics very seriously and our research conforms to the ICLR Code of Ethics. Human pose estimation is a well-established research area, and this paper inherits all the impacts of the research area, including potential for dual use of the technology in both civilian and military applications. We believe that the work does not impose a high risk for misuse. Furthermore, the paper does not involve crowdsourcing or research with human subjects.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
Our paper makes use of publicly available open-source datasets, ensuring that the data required for reproducing our results is accessible all researchers. We have thoroughly documented all aspects of our model's training, including the architecture, hyperparameters, optimizer settings, learning rate schedules, and any other implementation details for achieving the reported results. These details are provided in Section and Appendix C D of our paper. Additionally, we specify the hardware and software configurations used for our experiments to facilitate replication. We anticipate that should not be challenging for other researchers to reproduce the results and findings presented in this paper.
ACKNOWLEDGEMENTS
This work was partially funded by Amazon Robotics under the "Safe Autonomy Leveraging Intelli- gent Optimization" program, by the ARL DCIST program, and by the ONR RAPID program.
REFERENCES
Anastasios N Angelopoulos and Stephen Bates. A gentle introduction conformal prediction and distribution-fiet uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
Anastasios N Angelopoulos, Amit Pal Kohli, Stephen Bates, Michael Jordan, Jitendra Malik,
Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution-free uncertainty quantification and applications in imaging. In International Confer- ence on Machine Learning, pp. 717-730. PMLR, 2022.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Yu Bai, Song Mei, Huan Wang, Yingho Zhou, and Caiming Xiong. Efficient and differentiable conformal prediction with general function classes. arXiv preprint arXiv:2202. /091, 2022.
Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan Tibshirani. prediction beyond exchangeability. The Annals of Statistics, 51(2): (12)):16--455 646,6,45420 2023.
Conformal
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. Conditional image generation with score-based diffusion models. arXiv preprint urXiv:2111. 13606, 2021.
Aadyot Bhatnagar, Huan Wang, Caiming Xiong, and Yu Bai. Improved online conformal prediction via strongly adaptive online learning. In International Conference Machine Learning, pp- 2337-2363. PMLR, 2023.
Benjamin Biggs, David Novotny, Sebastien Ehrhardt, Hanbyul Joo, Ben Graham, and Andrea Vedaldi. 3d multi-bodies: Fitting sets of plausible 3d human models to ambiguous image data. Advances in neural information processing systems, 33:20496-2088 2020.
Yilun Chen, Zhicheng Wang, Yuxiang Peng. Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7103-7112, 2018.
Jeongjun Choi, Dongseok Shim, and H Jin Kim. Diffupose: Monocular 3d human pose estima- tion via denoising diffusion probabilistic model. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3773-3780, IEEE, 2023.
Hai Ci, Mingdong Wu, Wentao Zhu, Xiaoxuan Ma, Hao Dong, Fangwei Zhong and Yizhou Wang- Gfpose: Learning 3d human pose prior with gradient fields. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 4800-4810, 2023.
Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Diffdock: Dif- fusion steps, twists, and turns for molecular docking. arXiv preprint 10..172210.01776, 2022.
11

---

Published as a conference paper at ICLR 2025
Sai Kumar Dwivedi, Cordelia Schmid, Hongwei Yi, Michael J Black, and Dimitrios Tzionas. Poco: 3d pose and shape estimation with confidence. In 2024 International Conference on 3D Vision (3DV), pp. 85 95. IEEE, 2024.
Wyn-Cyuan Fan, Yen-Chun Chen, DongDong Chen, Yu Cheng, Lu Yuan, and Yu-Chiang Frank Wang- Frido: Feature pyramid diffusion for complex scene image synthesis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 579-587, 2023.
Runyang Feng, Yixing Gao, Tze Ho Elden Tse, Xueqing Ma. and Hyung Jin Chang. Diffpose: Spatiotemporal diffusion model for video-based human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14861-14872, 2023.
Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Few-shot conformal prediction with auxiliary tasks. International Conference on Machine Learning, pp. 332993339. PMLR, 2021.
Isaac Gibbs and Emmanuel Candès. Conformal inference for online prediction with arbitrary distri- bution shifts. arXiv preprint arXiv:2208. 08401, 2022.
Jia Gong, Lin Geng Foo, Zhipeng Fan, Qiuhong Ke, Hossein Rahmani, and Jun Liu. Diffpose: To- ward more reliable 3d pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13041-13051 2023.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint K3l::2312.00752 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47)11333 2022.
Karl Holmquist and Bastian Wandt. Diffpose: Multi-hypothesis human pose estimation using diffu- sion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 15977-15987, 2023.
Huang Huang, Satvik Sharma, Antonio Loquercio, Anastasios Angelopoulos, Ken Goldberg, and Jitendra Malik. Conformal policy learning for sensorimotor control under distribution shifts. arXiv preprint arXiv:2311.0145: 2023.
Rongjie Huang. Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. Prodiff: Progressive the 30th ACM International
fast diffusion model for high-quality text-to-speech, In Proceedings the Conference on Multimedia, pp. 2595-2605, 2022.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, ((75):1325-139 2013.
Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-tts: A diffusion model for text-to-speech via classifier guidance. International Conference on Machine Learning, pp. 11119-11133. PMLR, 2022.
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5253-5263, 2020.
Alon Levkovitch, Eliya Nachmani, and Lior Wolf. Zero-shot voice conditioning for denoising dif- fusion uts models. arXiv preprint arXiv:2206.02246 2022.
12

---

Published as a conference paper at ICLR 2025
Chen Li and Gim Hee Lee. Generating multiple hypotheses for 3d human pose estimation with mixture density network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9887-9895, 2019.
Chen Li and Gim Hee Lee. Weakly supervised generative network for multiple 3d human pose hypotheses. arXiv preprint 2000::::00000077000, 2020.
Wenhao Li, Hong Liu, Runwei Ding, Mengyuan Liu, Pichao Wang, and Wenming Yang. Exploiting temporal contexts with strided transformer for 3d human pose estimation. IEEE Transactions Multimedia, 25: 15:11222-299 1293, 2022a.
Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, and Luc Van Gool. Mhformer: Multi-hypothenr transformer for human pose estimation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 13147-13156, 2022b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Lars Lindemann, Matthew Cleaveland, Gihyun Shim, and George Pappas. Safe planning in dy- namic environments using conformal prediction. IEEE Roboties and Automation Letters, 2023.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851-866. 2023.
Xiaoxuan Ma, Jiajun Su, Chunyu Wang, Hai Ci, and Yizhou Wang. Context modeling in 3d human pose estimation: A unified perspective. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6238-6247, 2021.
Soroush Mehraban, Vida Adeli, and Babak Taati. Motionagformer. Enhancing 3d human pose estimation with transformer rransformergenformme network. In Proceedings of the IEEE/CVF winter ference on applications of computer vision, pp. 6920-6930, 2024.
Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 2017 international conference on 3D vision (3DV), pp. 506-51661 516. IEEE, 2017.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint
2021.
Tuomas Oikarinen, Daniel Hannah, and Sohrob Kazerounian. Graphmdn: Leveraging graph struc- ture and deep learning to solve inverse problems. In 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1-9. IEEE, 2021.
Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. Tax-pose: Task-specifi cross-pose estimation for robot manipulation. In Conference on Robot Learning, pp. 1783-1792 PMLR, 2023.
Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Danillidis. Coarse-to fo-niee volumetric prediction for single-image 3d human pose. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 7025-7034, 2017.
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas J Guibas. Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11888-1 1499, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
13

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: sagge-toiimage diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions paltern anal- ysis and machine intelligence, 45(4):4713-4726, 2022b.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
Wenkang Shan, Haopeng Lu, Shanshe Wang, Xinfeng Zhang, and Wen Gao. Improving robustness and accuracy via relative information encoding in 3d human pose estimation. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1444-4454, 2021.
Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. P-stmo: Pre-trained spatial temporal many-to-one model for 3d human pose estimation. In European Conference on Computer Vision, pp. 461 461-478. Springer, 2022.
Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Zhao Wang, Kai Han, Shanshe Wang, Siwei Ma, and Wen Gao. Diffusion-basse based 3d human pose estimation with multi-hypothesis aggregation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14761-14771, 2023.
Saurabh Sharma, Pavan Teja Varigonda, Prashast Bindal, Abhishek Sharma, and Arjun Jain. Monoc- ular 3d human pose estimation by generation and ordinal ranking. In Proceedings of the IEEE/CVF international conference computer vision, pp.
2019.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song and Stefano Ermon. Improved techniques for training booee-baedd generative models. Advances in neural information processing systems, 33: 13:12838-12448, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13445 2020b.
David Stutz, Ali Taylan Cemgil, Arnad Doucet, et al. Learning optimal conformal classifiers. arXiv preprint arXiv:2110.09192. 2021.
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5693-5703, 2019.
Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In Proceedings of the European conference computer vision (ECCV), pp. 529-545, 2018.
Bugra Tekin, Artem Rozantsev, Vincent Lepetit, and Pascal Fua. Direct prediction of 3d body poses from motion compensated sequences. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 991-1000, 2016.
Jacopo Teneggi, Matthew Tivnan, Web Stayman, and Jeremias Sulam. How to trust your diffusion model: A convex optimization approach conformal risk control. In International Conference on Machine Learning. pp. 33940-33960, PMLR, 2023.
Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference computer vision (ECCV), pp. 601-617, 2018.
14

---

Published as a conference paper at ICLR 2025
Tom Wehrbein, Marco Rudolph, Bodo Rosenhahn, and Bastian Wandt. Probabilistic monocular 3d human pose estimation with normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11119-11222 11208, 2021.
Yue Wu, Yongzhe Yuan, Xiaolong Fan, Xiaoshui Huang, Maoguo Gong, and Qiguang Miao.
Perdiffusion: Diffusion probabilistic models for point cloud registration. arXiv preprint arXiv:2312.06063 2023.
Tianhan Xu and Wataru Takano. Graph stacked hourglass networks for 3d human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp-
16105-161 14, 2021.
Yuan Xu, Xiaoxuan Ma, Jiajun Su, Wentao Zhu, Yu Qiao, and Yizhou Wang. Scorchypo: Probabilis- tic human mesh estimation with hypothesis scoring. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 979-989, 2024.
Cheng-Yes Yen Yang. Jiajia Luo, Lu Xia, Yuyin Sun, Nan Qiao, Ke Zhang, Zhongyu Jiang, Jenq-Neng Hwang, and Cheng- Hao Kuo. Camerapose: supervise monocular 3d human pose esti- mation by leveraging in-the-wild 2d annotations. In Proceedings of the IEEE/CVF Winter Con- ference on Applications of Computer Vision, pp. 2924-2933, 2023.
Heng Yang and Marco Pavone. Object pose estimation with statistical guarantees: Conformal key- point detection and geometric uncertainty propagation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 8947-8958, 2023.
Yu Zhan, Fenghai Li, Renliang Weng, and Wongun Choi. Ray3d: ray-based 3d human pose esti- mation for monocular absolute 3d localization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 131125,,,,, 2022.
Jinlu Zhang, Yujin Chen, and Zhigang Tu. Uncertainty-aware 3d human pose estimation from monocular video. In Proceedings of the 30th ACM International Conference on Multimedia, pp- 5102-5113 5113, 2022a.
Jinlu Zhang, Zhigang Tu, Jianyu Yang. Yujin Chen, and Junsong Yuan. Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13222-1322,, 2022b.
Siwei Zhang, Bharat Lal Bhatnagar, Yuanlu Xu, Alexander Winkler, Petr Kadlecek, Siyu Tang, and Federica Bogo. Rohm: Robust human motion reconstruction via diffusion. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14606-14617, 2024.
Yufei Zhang, Hanjing Wang, Jeffrey Kephart, and Qiang Ji. Body knowledge and uncertainty modeling for monocular 3d human body reconstruction. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 9030-9032, 2023.
Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N Metaxas. Semantic graph convo- lutional networks for 3d human pose regression. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3425-3435, 2019.
Qitao Zhao, Ce Zheng, Mengyuan Liu, Pichao Wang, and Chen Chen. Poseformerv2: Explor- ing frequency domain for efficient and robust 3d human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8877-8886, 2023.
Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding. 3d human pose estimation with spatial and temporal transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 11656-11665 2021.
Jieming Zhou, Tong Zhang. Zeeshan Hayder, Lars Petersson, and Mehrtash Harandi. Diff3dhpe: A diffusion model for 3d human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2092-2102, 2023.
Wentao Zhu, Xiaoxuan Ma, Zhaoyang Liu, Libin Liu, Wayne Wu, and Yizhou Wang. Motionbert: A unified perspective on learning human motion representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15085-15099, 2023.
15

---

Published as a conference paper at ICLR 2025
A COVERAGE GUARANTEE AND EMPIRICAL COVERAGE
Please refer to the standard proof shown in (Angelopoulos & Bates 2021 Shafer & Vovk 2008 for the coverage guarantees of conformal prediction.
To evaluate the empirical coverage of CHAMP, we calculate the mean coverage percentage across test data. Specifically, formed by the conformity function on the test data:
eq. (14) to check if the ground truth belongs the confidence
Σ lest Mwi
(UOT r))
Σ (0o(UGT) 2 2)
I I
We compare the three choices of conformity scores for the empirical coverage calculation. We use the test set from Human 3.6M and calculate the empirical coverage values across all activities. During training and testing, keep 0.1 for CP. From Fig. g the empirical coverage is
100
1:2
Separate
95
Fixed
Empirical Coverage
90
85
80
75
70
AVG.
Dir
Dise.
Eat
Greet
Phone
Photo
Pose
Pur.
Sit
SirD
Smoke
Wait
WatDD
Walk
WalkT
Figure 8: Empirical coverage comparison across three different conformity functions. We keep the best À 0.6 for comparison.
around 90% + 5% for all activities, which remains close to 1 a, and in some cases, it exceeds this value. This encouraging given the exchangeability assumption is not fully satisfied in our dataset.
B DENOISER IN TRAINING VS INFERENCE
We first revisit DDIM (Song & Ermon 2020). We start with the reverse process derivation and rewrite g(yr-1 yy, yo) to be parameterized by a desired standard deviation o;:
9:-1 1ot-190
04-16--1
Vac-130
ar-1 cfE-1 OfE
Vac-1
31 VI C3Ep
(1) (2:) + V - ar 07E0 (g:) + o,E
(1)
=aa-11
y1 VI
c (1) (21) + v
-a--1 -0,c(00 (y:) + d
16

---

Published as a conference paper at ICLR 2025
where the model Eg therefore we have:
6 (-) predicts the E, from yr. Since g(y--1 1y:. NN 2.1,
Ūt-1 50 1 Czt
B. =
Let of = 3t where is a hyperparameter that controls the sampling stochasticity. During generation, don't have follow the whole chain f 1,..., ,,T, but only subset of steps. Denote t' < as two steps in this accelerated trajectory. The DDIM update step is defined as follows:
(2) qe & ( We yo) = N (xe; -N Var (v. -0.-VI- ared (g))) .f.1
Following D3DP (Shan et al. 2023), the denoiser model Da uses DDIM sample denoised poses from the corrupted ones. During training, we run DDIM for only step for the sake of efficiency:
y' = Do(yt: x. T), v ~ I)..) vh f1. ... H
During inference, we rum DDIM for K = 10 times, and each step is defined as:
1= T = T (k + 1) 1)/K)), = {0, (0,.... ...., ,K - 11
y = Do th x, wh = , H}
1 av. 3/" + v ār of E + OrE
t t+t t'
C TRAINING-CALIDRTTINN-TTTT
DATA SPLIT
We discuss the split of training, calibration, and testing data. We first split the testing data the same way as (Zhang et 2022b) [Shan et al. 2023] Gong et al. 2023) to ensure the fairness of results. We then further split the training set into the actual training dataset and calibration dataset before inference. Specifically, we split the training dataset by uniformly sampling 2% subset as the calibration dataset, and CHAMP is only trained the remaining 98%. Note here that this 2% calibration dataset is not seen during training by any means. During training, we split each mini- batch evenly into Bpeed and Beal-
D ARCHITECTURE DETAILS
We provide more detailed version of Fig. We illustrate the detailed architecture of the denoiser model Do as well as the conformity scoring model so in Fig. 9 MixSTE (Zhang et al. 20226) is used the backbone of the denoiser. MixSTE combines 16 alternative spatial and temporal atten- tion blocks. In the implementation. the channel size is set to 512. Similar to (Shan et al. 2023 Gong al. 2023), we concatenate 2D keypoints and noisy 3D poses as inputs and add diffusion timestep embedding after the input embedding using sinusoidal function. For the conformity scoring model, we concatenate 2D keypoints and predicted hypotheses 3D poses as inputs, reuse the input embed- ding layer, and use an extra MI. to project the input embeddings into score in the range [0, 1]. In training. CP is made differentiable by using soft ranking and soft assignment, resulting in a differen- tiable inefficiency loss Lineft. This loss is combined with the pose loss Lpose when backpropagating into the network. During inference (light cyan shaded area), we refine the hypotheses confidence set with regular CP held-out calibration set, resulting in H' < H hypotheses. We then aggregate the refined set, resulting in a single final prediction.
E PROCRUSTES-MPIPE PERFORMANCE
The Procrustes MPJPE (P-MPJPE) metric, which is often referred to as Protocol #2, computes MPJPE after the estimated poses align to the ground truth using rigid transformation. It is another
17

---

Published as a conference paper at ICLR 2025
2D Keypoints
Input Embed ding
Artn.
Spatial
Atto.
Temporal
16X
Atto.
Spatial
Attn.
Temporal
Pred. 3D
Pose Loss
Keypoints
Noisy 3D Keypoints
(R, (H,
(H, N; 3)
Emnbed.
Emaeed
Training
2D Keypoints
Embedding
(H, 1)
(N, J, 2)
Input
MLP
Score
Conformity Scores
Differentäable
Inefficiency Loss
Pred. 3D Keypoints
Regular
Refined
Final
(H. J.
CP
Hypotheses
Aggregation
Pred
N,
(N, 3)
Inference
Figure 9: Detailed architecture of Da and Sp, the denoiser and conformity score models during training and inference. During training, the construction of refined hypotheses after differentiable CP is done via soft assignment. During inference, regular CP used with the trained conformity score function.
standard metric for measuring a 3D pose estimator's performance (Pan et al. 2023 We report the performance of the four variants of CHAMP against other baselines on the Human3.0m 3.6M dataset in Table Quantitative results suggest our method achieves SOTA performance on this metric as well.
Procrtalrs Por- Joint Fosition Ermor [Prolccol & 21-
mI Sit
AVG.
Dir
Disc.
Est
Ceeet
Phone
Phoso
Fiss
Fur.
SRD.
Smokke
Wait
WaIRD:
Walk
WaITT
Stndl Hypothess
STFllaam bylday
35.3
12.7
35.5
13.5
35.4
35.9
41.6
31.0
31.9
25.1
50.1
364
33.5
35.1
23.9
25.0
P.STMOE Sese 2013
34.4
31.3
35.2
33.9
53.9
55.4
39.3
32.5
31.5
44.6
48.2
36.3
53.9
34.4
23.8
23.9
MixSTE
32.6
30.8
33.1
30.3
31.8
33.1
39.1
30.5
42.5
44.5
34.0
30.8
32.7
22.9
DUE Vunnerrs
32.5
30.3
$4.6
29.5
34.7
31.6
38.9
31.8
31.9
39.2
42.8
32.1
32.6
31.4
25.
23.8
D1D
31.7
30.6
32.5
29.1
31.0
$1.9
37.6
30.3
29.4
40.6
43.6
33.3
30.5
31.4
21.5
22.4
CRAMIP-NIIV
31.0
30.7
32.0
20.1
30.6
$1.1
35.8
29.1
78.4
39.7
41.4
33.0
50.1
31.3
20.8
23.3
Muitiple Hyypotheses
Milseme
3433
32.8
45.7
35.4
54.5
34.4
GuphMDNN
Sheamneree
39.7
54.5
69.0
42.4
58.0
Din
28.8
DilPosu
44.6
DSDP-Ao
30.4
CHAMP-ARE
27.7
27.4
Table P-MPIPE (L) results Human-3.6M4 dataset. "Upper-bound performance, needs ground truth.
F COMPARISON TO OTHER UNCERTAINTY DESIGN CHOICES
We now discuss different design choices for uncertainty estimation in human pose estimation. Sev- eral prior works have explored uncertainty in 3D human pose/shape reconstruction (Rempe et al. 2021 Dwivedi et 2024 Biggs et al. 2020 Zhang et al.) 2023 2024). While methods such as Dwiivedi et al., (024 also learn an explicit confidence value for occlusion, using the confidence to sample good poses is non-trivial. Moreover, motion-
methods such as (Zhang et al. 2024 Rempe al.] 10 use physical contacts and trajectory consistency to make the uncertain estimates more robust, which does not explicitly model the "quality" of the samples. Zhang et al. (2023) use explicit anatomy constraints and also shows that uncertainty modeling can indeed improve model performance. Lastly, Biggs et al. (2020) generates fixed number of hypotheses and learns to choose the best ones, which is similar in flavor to CHAMP but lacks the probabilistic flexibility and the erage guarantee from CP. While all aforementioned works are valid design choices for uncertainty in human pose learning, CHAMP lends itself particularly well probabilistic pose estimation meth-
18

---

Published as a conference paper at ICLR 2025
Figure 10: Results of running CHAMP on in-the-wild videos collected from TikTook and YouTube.
ods, which can then be augmented with an explicit uncertainty modeling function learned in tandem with the pose estimation model.
G MORE QUALITATIVE RESULTS ON IN-TE--WILD VIDEOS
We showcase the performance CHAMP on more in-the-wild videos in this section. We collect various YouTube and TikTok videos and detect 2D keypoints with Cascaded Pyramid Network. Note that the official weights provided by CPN are trained in COCO format, so we used a fine-tuned version of the weights for Human3.6M universal skeleton format. With such 2D keypoints, are able to directly use the model trained on Human3.6M, dataset and apply to real-world videos without any fine-tuning. In Fig. 10 for each row, show examples. For every three images,
19

---

Published as a conference paper at ICLR 2025
the leftmost image is the RGB observation, and the middle image is all the hypotheses proposed by CHAMP's pose estimation backbone, and the rightmost image is the hypotheses set refined by the learned conformal predictor.
H LEARNED CP IN CHAMP
We provide more demonstrations of the learned conformal predictor in CHAMP. Specifically, in Fig. 11 we provide more examples of predicted hypotheses before and after the conformal predictor powered by the learned score function.
Before CP
After CP
Figure 1:: Examples of CP in CHAMP filtering out bad hypotheses during inference. Qualitatively, the Iearned score function filters out outlier predictions.
I CONFORMITY SCORE VISUALIZATION
We also show some qualitative visualization of the learned conformity score. We heatmap color-code different hypotheses.
20

---

Published as a conference paper at ICLR 2025
Figure 12: Failure case and the predicted conformity score. While we see that hypotheses further away from the ground truth get lower weight, most of the hypotheses here are off, significantly skewing the final output. Moreover, we see that the variance of the estimate of the most incorrect joint (i.e. the right elbow) does appear a lot higher than others, looking spread out. This shows that higher uncertainty tends to be associated with higher error.
Figure Learned conformity score prediction on 3DPW data. see that both cases, the conformity score higher in the more concentrated areas and vice versa.
21

---

Published as a conference paper at ICLR 2025
J SINGLE-FRAMD VARIATION
When demonstrating the effectiveness the proposed method, one problem that the ambiguity in video 3D pose estimation where temporal information is abundant is not significant in the single-frame case. Thus, we are interested in if the proposed CP pipeline could be beneficial for single-frame image-based 3D pose estimation. Note that this requires minimal change to our method pipeline as we can adjust the number of frames (N) to from 243. We thus train our method as well as the baseline methods with . and test on the same dataset.
Single Franne LV
IJ Meun Per- Per-Joiat Position Error (Protocol
1 Min
AVG.
Dir
Disc.
Extt
Greet
Phone
Photo
Fose
Fur.
SILD.
Smoke
Wait
WaRRD.
Walk
WalkT
28.9
Smat
42.1
30.5
29.1
41.9
30.8
41.8
40.9
39.6
19.8
28.9
MEAve-Naive-Aag
40.7
33.9
29,9
39.4
40.4
42.8
68.9
39.8
50.8
55.2
41.2
39.1
38.8
29.3
27.7
CHAMP
38.5
38.2
39.4
34.5
36.3
40.1
4622
37.1
36.4
46.9
50.9
40.4
39.2
39.1
27:2
27.7
CHAMP-Age
37.1
36.5
18.1
33.1
34,2
38.4
44.1
35.1
33.9
44.9
49.2
40.2
37.9
37.8
26.5
27.2
Table 5: MPJPE (L) results on Human-3.6N. dataset. Note that all baselines shown are trained with input frames number N 1.
As expected, with fewer frames available, all tested baselines should result in worse performance than with multiple frames. Interestingly, as we can see Table CHAMP-Agg is able to achieve more significant edge over the previous SOTA (DiffPose) in single- figmefframe setting vs in multi- frame (3.4mm vs 1mm). Moreover, CHAMP-Age achieves more improvement over the backbone D3DP in the single-frame setting. Thus, with less temporal information available (i.e. when the ambiguity is more severe), CHAMP achieves even more competitive performance compared to baselines.
K DIVERSE HYPOTHESES UNDER LARGE UNCERTAINTY
Another interesting study to conduct is test the proposed method's behaviors under large uncer- tainty. Here, we specify large uncertainty as three scenarios:
Truncation in 2D RGB input.
Wrong 2D keypoints prediction.
Self-occlusion in 2D RGB input.
While first two types large uncertainty, truncation in 2D RGB input (which gives inaccurate 2D keypoints) and wrong 2D keypoints, are not the focus of our work (since we assume 2D keypoints from some off-the-shelf method), it would interesting to see how CHAMP behaves when such uncertainty occurs. To test this, we use the single-frame model trained in the previous section tths allows us to isolate the temporal effects and focus on uncertainty in 2D RGB inputs only, as the temporal nature of the data may already provide additional constraints, reducing ambiguity.
K. TRUNCATED RGB INPUT
When the RGB image input is truncated, meaning that some keypoints are not fully visible, it might be possible that the 2D keypoints detector can still predict decent 2D keypoints. As shown in CPN Chen et al. (2018) and HRNet (un et al. (2019), the 2D keypoints detector does have some robustness truncation (external occlusion). We synthetically create truncated (externally occluded) images and test CHAMP on them, shown in Fig. 14] In the first case, the two feet are truncated and in the second case, one hand is truncated. In both cases, the 2D keypoints detector was able predict the truncated 2D keypoints. Since our method takes as input 2D keypoints, so long as all keypoints are available, will run as usual, agnostic to the truncation.
K.2 WRONG 2D KEYPOINTS
It is also likely that the 2D keypoints detector makes mistakes due to noisy background, imprecise segmentation, etc. Under this circumstance, the 2D keypoints are off to begin with. Granted, when the 2D keypoints extremely wrong, the 3D) keypoints output will also be off -this constitutes
22

---

Published as a conference paper at ICLR 2025
48
2D Keypoints
All Hypotheses
After Refinement
Another View
Figure 14: CHAMP Performance Truncated Images. As long the keypoints detected, the pipeline output keypoints as usual. This relies on 2D keypoint detector's robustness to external occlusion.
one key failure mode of CHAMP. However, we are interested to see if the uncertainty score in CHAMP is able "correct" the output by proposing more diverse samples. In Fig. 15 we select two images where the 2D keypoint detector makes mistakes. In the first image, the leg keypoints completely off. Interestingly, the proposed 3D keypoints than other joints. After the CP refinement step, the left leg joints are still more diverse, covering a off. We
lot more diverse for the left leg
wider region than other joints outputs do. In the second image, the two hands keypoints again see a much noisier proposal set for the two joints. After the refinement, the two hands outputs noticeably more diverse. One possible reason for this interesting phenomenon could be that the 2D keypoints input itself are seen as "unnatural" compared to the training data, forcing the network output more diverse 3D poses.
K.3 SELF-OCCLUDED RGB INPUTS
Lastly, another source of large uncertainty in RGB inputs is self-occlusion. Note that self-occlusion typically happens when the human subject itself occludes some joints, and this is different from truncation, where the source of occlusion is external. We are interested in testing CHAMP when self-occlusion present and checking if any interesting behavior occurs. Fig. 16 we show two images where the human subject occludes itself, obstructing the visibility of some joints. In the first image, the dancer's left arm is occluded by the right arm entirely. While the 2D detector is able to differentiate the two joints and output two different 2D keypoints, they very close and the 3D locations are ambiguous. CHAMP outputs noisier proposals for the occluded arm joints (elbow and hand). Interestingly, after the CP refinement, the elbow joint pose outputs almost "bimodal" and the hand pose outputs are still a lot noisier than other joints. For the second image, the dancer's right leg is occluded by the left leg. Similarly, CHAMP outputs noisier proposals for the right knee and right foot both before and after the CP refinement. One possible reason for this interesting phenomenon that the network intrinsically learned output different poses when the 2D keypoints are close to each other, suggesting possible self-occlusion.
PERFORMANCE ON 3DPW
The 3DPW dataset Von Marcard et al. 2018) is a more challenging dataset collected in outdoor environment using IMU (inertial measurement unit) sensors with mobile phone lens. To ensure
23

---

Published as a conference paper at ICLR 2025
22
14
14
18
1.
1.4
0.
2D Keypoints
All Hypotheses
After Refinement
Another View
Figure 15: CHAMP Performance Wrong 2D Keypoints. Interestingly, CHAMP outputs a lot more diverse proposals for the joints that are off more correct joints.
3.8
6.
2D Keypoints
All Hypotheses
After Refinement
Another View
Figure 16: CHAMP Performance Self Occluded RGB Inputs. Interestingly, CHAMP outputs lot more diverse proposals for the joints that are occluded than for non-occluded joints.
fair comparison, following Yang et al. (023). we compare CHAMP with other baselines by taking
as input ground-truth 2D keypoints. We compare CHAMP (N = 1), D3DP (N 1) Shan et al. 2003),, DiftPose (N 1) Gong al. (2023), and CameraPose Yang et al. (2023) trained jointly
on Human 3.6M and MPII without using 3DPW during training and test on 3DPW using the PA-
MPJPE (aligned after rigid transformation in Table Note that in Shan et al. (2023) did not provide
quantitative results on 3DPW so we retrained D3DP jointly on Human 3.6M and MPII and tested on 3DPW.
24

---

Published as a conference paper at ICLR 2025
Baseline
CHAMP-Naive 64.2
CHAMP-NaNveAAgE
CHAMP
CHAMP-ARE
D3DP
CameraPose
DifiPose
PA-MPJPE (mm)
03.7
61.5
60.8
64.3
63.3
64.0
Table 6: Baseline Methods Trained on Human3. 6M and MPII and Tested on 3DPW with PA-MPJPE Metric
As we can see, CHAMP outperforms other baselines by a noticeable margin. Even though 3DPW was not seen during training, our method achieves competitive performance. Moreover, we see similar trend that J-Agg's edge is less than CP refinement: from CHAMP-Naive to CHAMP-Naive- Agg, the error is reduced by 0.5 mm, but it gets reduced by 2.7mm going from CHAMP-Naive to CHAMP, where CP refinement is added. This again indicates the importance of using CP to aggregate the hypotheses.
M MODULAR DESIGN
CHAMP uses the idea of attention mechanism to encode both spatial and temporal information. This is inherited from the usage the MixSTE D3DP backbone, which uses a Transformer- -based encoder. While prior work such as Zheng al. (2021 Zhao al. (2023) do consider multi-
hypothesis settings or uncertainty quantification, switching the transformer in D3DP backbone in (2023) will result in a similar multi- holti-hyyottesis model that is able to quantify uncertainty via end-to-end CP.
CHAMP with Zheng al. (2021) or Zhao
Similarly, Ci al. (2023) uses an innovative approach to learning probabilistic human poses via gradient flow diffusion. The key difference lies in the formulation of the diffusion process to generate multiple hypotheses. We show that CHAMP's framework is modular enough we are able to replace the diffusion backbone with GFPose's 's gradient- based diffusion. Mchraban et al. (2024) uses GCNFormer to improve the modeling of local dependencies inherent in human pose sequences, which is shown to outperform transformers. The key difference is again the choice of backbone. We replace the human pose backbone in CHAMP with GCNFormer and show the improvement gained using the conformity score.
Thus, for the aforementioned baselines, we finetune them with the end-to-end score function pro- posed in our work and compare how much the MPJPE gets reduced. We show the improvement brought by using the end-to-end conformal score function on the Human 3.6M dataset below in Table 7
Baseline
PoscFormer 1.5
PoseFormer V2
GFPPse 1.7
MotionAGFormer 1.8
CHAMP 2.3
MPJPE Reduction (mm)
1.2
Table 7: Using CHAMP's End-to-End Score Function Different Backbones
25


Reviews:
[{"review_id": "VThp8lQtBj", "reviewer": "Reviewer_dk3Y", "summary": "CHAMP presents a conformalized multi-hypothesis pose estimation pipeline. Based on diffusion model, the multiple pose hypothesis are refined by a differentiable comformal prediction module with an inefficiency loss. Experiments show that CHAMP achieves state-of-the-art results on several benchmarks.", "questions": "This paper is well-presented and validated. The only thing I concern is its performance on occluded or truncations where diverse multiple hypothesis are preferred.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed conformal prediction is simple yet effective. \n2. This paper is well-written and easy to understand.", "weaknesses": "1. CP removes the predictions with low confidence and visualisations also show after CP the predictions become more certain and accurate. However, the results are mainly shown with easy cases where occlusions barely present. I am wondering how CP performs on hard cases where occlusions, truncations exist?\n2. It will be better if this method can be extended to image-to-3d pipelines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v5hJAaUu80", "reviewer": "Reviewer_rtwB", "summary": "This work tackles the task of 2D-to-3D lifting of temporal human pose sequences through a diffusion-based multi-hypothesis approach. The novel aspect is applying the tools of conformal prediction to this task. That is, the authors propose to learn a conformity score function in an adversarial setup and simulate conformal prediction during training, directly minimizing the conformal set size (the inefficiency). Quantitative evaluation is performed on the well-known Human3.6M and MPI-INF-3DHP datasets, with additional qualitative evaluation on in-the-wild videos. Results show that filtering according to conformity scores and aggregating the results improves the metrics and achieves state-of-the-art results. Ablations also show that the end-to-end training of the scoring function is beneficial.", "questions": "I don't fully understand the CHAMP-Agg method. Is this using J-Agg on the full set of hypotheses, or does it also use the learned scoring function in some way? How should we interpret the fact that \"CHAMP-Agg\" is better than \"CHAMP\"? Does it mean that the proposed learned conformity scoring does not perform better than the prior method of aggregation proposed by Shan et al. (2023)?\n\nTo the paragraph after Eq. 5: How can we interpret the \"true value to be included in\" the conformal set in case of continuous values like 3D coordinates? In case of classification this makes more sense, but perhaps the language has to adjusted here.\nWhat 2D pose estimator is used in the main experiments, or is the ground truth used?\nWhy are values missing from Table 3?\n\nSmaller issues:\nGAN is cited as the CACM 2020 paper, but perhaps the original NIPS 2014 version is better suited for citation.\nIn Eq 9, perhaps Temperature could be denoted by some other symbol because mathcal{T} and tau look very similar visually (except for size). Similarly I would not recommend relying on the subtle difference between C and mathcal{C} for important differences (as noted in the footnote of page 6), a hat, tilde or similar feature may be more visible.\n\nPage 7: Human3.6M is no longer the \"largest indoor dataset for 3D human pose estimation\". There are many larger ones such as DNA-Rendering.\nIt should be stated explicitly that E2E refers to \"end-to-end\".", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The application of conformal prediction to 3D human pose estimation is an interesting and original aspect. Multi-hypothesis 3D pose estimation has been a relevant research area due to ambiguities in the task, but practical methods are often not formulated in a proper probabilistic manner. Using conformal prediction techniques is a promising direction and taking a first step in this direction is relevant to the community.\nThe overview of the related works is extensive, and the comparison of them helps the reader to place this work in relation to them. The text is well-structured and easy to follow.\nThe authors use already existing techniques in good ways, for example using the already proven MixSTE model for denoising, incorporating adversarial training and conformal calibration.\nResults on Human3.6M and MPI-INF-3DHP both demonstrate the benefit of the method.\nAblations and hyperparameter studies confirm the design choices.", "weaknesses": "The method is only evaluated on studio datasets, but not on outdoor ones, such as 3DPW and EMDB. Evaluation on such less restricted videos would be more convincing.\nIt is unclear whether CHAMP's learned conformity scoring performs better than the prior established 2D-joint projection based version (see question below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e0p1YmnkHS", "reviewer": "Reviewer_aAY1", "summary": "This paper focuses on 3D pose estimation from 2D pose inputs in a multi-hypothesis sequence-to-sequence way. Following previous methods, they also adopt a diffusion model to regress the 3D pose hypotheses from sequential 2D pose inputs. Different from previous work, this paper proposes to learn a score model along with the regression model to predict the confidence of multiple regressed hypotheses. The scores of multiple hypotheses could be used to filter out the outliers to refine prediction set.   \nThe eperiments are mostly conducted on two benchmarks, Human3.6M and MPI-INF-3DHP. Compared with directly averaging all multi-hypothesis predictions, averaging filtered predictions with the proposed score model reduces 1.5mm MPJPE on Human3.6M, while selecting the 3D pose whose 2D projection is more consistent with the 2D pose further reduces 1.5mm MPJPE on Human3.6M.", "questions": "See the experiments mentioned above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Learning the uncertainty of multi-hypothesis 3D human pose regression seems to be reasonable. Aside from building the distrubution of 3D human poses, how to select the best ones is an interesting research topic.   \n2. Designs in scoring the hypotheses via conformal prediction are subtle. Detailed ablation study further reveals the influence of different parameters.   \n3. The description of method is clear. Especially, Fig. 2 provides a clear overview of the relationship between different designs.", "weaknesses": "1. Experiments.   \nTab. 1 and Tab. 2 show the improvement, compared with CHAMP-Naive, via introducing comformal prediction (CHAMP) and further selecting the 3D pose via measuring the 2D projection and 2D pose (CHAMP-Agg).  However, one important experiment seems to be missing here, CHAMP-Naive + -Agg. How well it performs if the proposed comformal prediction is not used?   \nThis experiment will clearly show how important the proposed comformal prediction is.   \nBesides, the quantative eperiments are only conducted on indoor benchmarks. The qualitative results are relatively limited.  Performance on in-the-wild benchmarks, such as 3DPW, is not clear.   \n\n2. The performance improvement is not significant compared with the baseline SOTAs, which might effect the value of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UdRs27wgTK", "reviewer": "Reviewer_83JW", "summary": "This paper proposes using a diffusion model to generate multiple hypotheses for temporal 2D to 3D pose estimation and introduces Conformal Prediction (CP) to select more plausible hypotheses and aggregate them. The method achieves competitive results on two pose estimation benchmarks.", "questions": "Regarding the method, it is unclear how the approach handles situations where the 2D estimates are incorrect or when the 2D pose is truncated beyond the image boundaries.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces the novel idea of using Conformal Prediction to select more plausible results from multiple hypotheses, which is a fresh perspective.\n2. The quantitative experiments demonstrate that the hypotheses selected and aggregated using this strategy outperform the results without filtering.", "weaknesses": "1. The paper's contribution is to be discussed. First, the ambiguity in temporal 2D-to-3D pose estimation is not as significant as in the single-frame, single-view case. Additionally, generating multiple 3D hypotheses using diffusion models is already a widely used technique (DiffuPose (Choi et al., 2023) DiffPose (Gong et al., 2023) DiffPose (Feng et al., 2023) D3DP (Shan et al., 2023)). If the only contribution is the use of CP, its novelty might be limited. If the authors could show that the CP strategy also performs well in single-RGB input cases or demonstrates its effectiveness in single-image mesh estimation, it would significantly enhance the paper's contribution.\n2. The quantitative results are not very solid. The improvement on 3DHP is quite marginal, making it unclear whether the improvement is due to random seed variation. These results do not convincingly support the method's effectiveness.\n3. The visual results (e.g., Figure 10) do not show cases where severe occlusion leads to incorrect 2D estimates or cases where 3D depth errors occur. It’s important to demonstrate that CP can filter out incorrect hypotheses in such challenging cases, where multi-hypothesis generation is truly necessary. This ties back to the first weakness, as the temporal nature of the data may already provide additional constraints, reducing ambiguity. Therefore, the authors should explore more varied tasks to prove CP's efficacy, such as monocular pose estimation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)