CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators

## Abstract

Abstract We introduce CHAMP, a novel method for learning sequence-to-sequence, multi-hypothesis 3D human poses from 2D keypoints by leveraging a conditional distribution with a diffusion model. To predict a single output 3D pose sequence, we generate and aggregate multiple 3D pose hypotheses. For better aggregation results, we develop a method to score these hypotheses during training, effectively integrating conformal prediction into the learning process. This process results in a differentiable conformal predictor that is trained end-to-end with the 3D pose estimator. Post-training, the learned scoring model is used as the conformity score, and the 3D pose estimator is combined with a conformal predictor to select the most accurate hypotheses for downstream aggregation. Our results indicate that using a simple mean aggregation on the conformal prediction-filtered hypotheses set yields competitive results. When integrated with more sophisticated aggregation techniques, our method achieves state-of-the-art performance across various metrics and datasets while inheriting the probabilistic guarantees of conformal prediction.

## 1 Introduction

Learning to estimate 3D human poses from videos is an important task in computer vision and robotics. Typical 3D human pose estimators learn to predict a single 3D human pose from an RGB image, either in an end-to-end manner or by relying on pre-existing 2D keypoint detection methods applied to the RGB images. Recent advances in sequence-to-sequence modeling enable learning 3D human poses in sequence by using RGB videos, which significantly improves the flexibility and efficiency of such estimators (Zhang et al., , ). While promising, such approaches only focus on single-hypothesis predictions, which is an inherently ill-posed problem when trying to reconstruct a 3D human pose given inputs collected from one viewpoint.

Thus, a more recent line of work focuses on learning multi-hypothesis 3D human pose estimators from 2D inputs. Instead of learning one deterministic target, such methods model the 2D-to-3D pose learning problem as learning a conditional distribution, which describes the distribution of the 3D poses given 2D inputs. This change in problem formulation prompts the use of generative models such as GANs, VAEs, and Diffusion models (Li & Lee, ; Sharma et al., ; Shan et al., ), which can propose multiple hypotheses of 3D poses given a single 2D input. For practical uses, one should also consider aggregating the hypotheses to generate one single output prediction. However, due to the imperfection of the trained generative models, using all hypotheses when aggregating could be inefficient and suboptimal if some are exceedingly inaccurate.

To counter the aforementioned problems, we take inspiration from an important tool in statistical learning, conformal prediction (CP) (Shafer & Vovk, ; Angelopoulos & Bates, ), which uses a post-training calibration step to guarantee a user-specified coverage: by allowing to predict confidence sets $C(X)$, CP guarantees the true value $Y$ to be included with confidence level $\alpha$, i.e. $P(Y\in C(X))\geq 1-\alpha$ when the calibration examples $(X_{i},Y_{i})\in I_{\text{cal}}$ are drawn exchangeably
from the test distribution. There are typically two steps involved in the CP process: In the calibration step, the conformity scores on the calibration set are
ranked to determine a cut-off threshold $\tau$ for the predicted values guaranteeing coverage $1-\alpha$, usually via quantile computation. In the prediction step,
conformity scores, which measure the conformity between the output and possible ground-truth values, are computed to construct the confidence sets $C(X)$ by using the calibrated threshold $\tau$.

Figure: Figure 1: CHAMP sample results obtained on in-the-wild videos collected from TikTok. Having observed 2D keypoints, CHAMP proposes multiple hypotheses of the 3D human skeleton poses, and then a conformal predictor trained end-to-end with the pose estimator refines the confidence set by filtering out low-conformity-score hypotheses. This smaller set will be used in downstream aggregation for a single output prediction.
Refer to caption: x1.png

CP is highly flexible as it can be applied to any machine learning model. However, since it is applied post-training, learning the model parameters is not informed about the post-hoc conformalization step: the model is not tuned towards any specific CP objective such as reducing expected confidence set size (inefficiency). To bias the training towards lower inefficiency, ConfTr (Stutz et al., ) proposes a fully differentiable CP operation for classification tasks, which is applied end-to-end with the prediction model and optimizes a differentiable inefficiency loss in tandem with the original class loss for the classifier. This operation allows the model to learn parameters to reduce the inefficiency of the confidence set size during conformalization.
Inspired by this, we wish to learn a scoring model that scores the “quality” of each hypothesis by measuring its conformity to the ground truth. We can then use this scoring model to simulate CP and optimize for the inefficiency in a differentiable manner, as done in (Stutz et al., ). During test time, we propose a large number of hypotheses and do regular CP to filter out the low-score ones before feeding into the downstream aggregation steps. The learned CP procedure can be applied to any multi-hypothesis estimator training process to refine the hypotheses confidence set size during test time for better aggregation results. With this learned CP wrapper, we present CHAMP, a Conformalized 3D HumAn Multi-Hypotheses Pose Estimator. To summarize, the contributions of this work include:

- •
A novel sequence-to-sequence, multi-hypothesis 3D human pose estimator from 2D keypoints.
- •
A novel method to conformalize 3D human pose estimates during training by learning a score function to rank the quality of the proposed hypotheses.
- •
A novel method to aggregate the multiple hypotheses from the estimator output using conformalization based on a learned score function.
- •
Quantitative and qualitative results that demonstrate the state-of-the-art results of our method on a variety of real-world datasets.

## 2 Related Work

Diffusion Models are a family of generative models
that gradually corrupt data by adding noise,
and then learn to recover the original data by reversing this process (Sohl-Dickstein et al., ; Ho et al., ; Song & Ermon, ; Song et al., ). They have achieved success in generating high-fidelity samples in various applications such as image generation (Ho et al., ; Saharia et al., ; Nichol et al., ; Batzolis et al., ; Rombach et al., ; Zhang et al., ), multi-modal learning (Levkovitch et al., ; Kim et al., ; Huang et al., ; Avrahami et al., ; Zhang et al., ) and pose estimations in 3D vision problems (Gong et al., ; Wu et al., ; Devgon et al., ). We use Diffusion models to learn 3D human poses from 2D keypoints.

3D Human Pose Estimation is an important problem in computer vision and robotics. With deep learning, various end-to-end approaches have been proposed (Tekin et al., ; Pavlakos et al., ; Elmquist et al., ). However, with the maturity of 2D human keypoints detection, (Ho et al., ; Cao et al., ; Ma et al., ; Zhang, ; Sim et al., ), more robust approaches focus on uplifting 2D keypoints to 3D, resulting in better performance (Martinez et al., ; Zhou et al., ; Xu & Takano, ; Zhao et al., ; Lim et al., , ), where one typically deals with a frame-to-frame problem, or a sequence-to-sequence problem. Being able to predict a 3D keypoints sequence directly from a 2D keypoint sequence is highly desirable as it has higher efficiency and flexibility. In this scheme, deterministic methods learn to predict one single 3D output from the 2D input (Pavllo et al., ; Zeng et al., ; Zheng et al., ; Shan et al., ; Jin et al., ; Hu et al., ; Zhan et al., ; Eisner et al., ). However, with a single viewpoint, such a formulation is ill-posed because there could be many possible 3D poses satisfying the same 2D keypoint configuration. Thus, it is desirable to model the problem as a conditional probability distribution. Deep generative models have shown great potential in modeling such distributions. Specifically, mixed-density network (Li & Lee, ; Pan et al., , ), VAE (Sharma et al., ; Zhang et al., ), normalizing flows (Wehrbein et al., ), GAN (Li & Lee, ), and Diffusion models (Feng et al., ; Choi et al., ; Rommel et al., ; Avigal et al., ; Yao et al., ; Shen et al., ) have all been applied to modeling such conditional distribution. In our work, we use (Zhang et al., ) as the backbone for generating 3D keypoints sequences from 2D keypoints sequences. To infer multiple hypotheses, we follow the frameworks in (Shan et al., ; Gong et al., ; Zhou et al., ) to learn to recover 3D pose hypotheses from Gaussian noises.

Conformal Prediction (CP) is a powerful and flexible distribution-free uncertainty quantification technique that can be applied to any machine learning model (Angelopoulos & Bates, ; Shafer & Vovk, ). Assuming the exchangeability of the calibration data, CP has desirable coverage guarantees. Thus, it has been applied to many fields such as robotics (Huang et al., ; Avigal et al., ; Sun et al., ), pose estimation (Yang & Pavone, ), and image regression (Teneggi et al., ; Angelopoulos et al., ). More sophisticated CP paradigms have been also proposed to tackle distribution shift and online learning problems (Angelopoulos et al., ; Gibbs & Candès, ; Bhatnagar et al., ; Feldman et al., ). Since CP is applied post-training, the learned model is not aware of such conformalization during training. To gain better control over the confidence set of CP, learning conformal predictors and nonconformity score end-to-end with the model has been proposed in (Stutz et al., ; Fisch et al., ; Bai et al., ). More closely related to our proposal is the work of (Stutz et al., ), but instead of using raw logits as the conformity score for classification, we learn an extra scoring model as the score to rank the hypotheses and simulate CP during training.

## 3 Problem Formulation

We are interested in the problem of learning a sequence of 3D human poses from a sequence of 2D human pose keypoints. We assume the 2D keypoints are available to us, which could be detected from the RGB images using well-established methods such as (Li et al., ). Formally, given the input 2D keypoints sequence $\bm{x}=\{\bm{p}^{2d}_{n}|n=1,\dots,N\}$, where $\bm{p}^{2d}_{n}\in\mathbb{R}^{J\times 2}$, our goal is to learn a conditional distribution for the possible corresponding 3D positions of all joints $p_{\theta}(\bm{y}|\bm{x})$, where the sequence $\bm{y}=\{\bm{p}^{3d}_{n}|n=1,\dots,N\}$ and $\bm{p}^{3d}_{n}\in\mathbb{R}^{J\times 3}$. Here, $N$ represents the number of input and output frames and $J$ is the number of human joints in each frame. With the learned distribution $p_{\theta}(\bm{y}|\bm{x})$, we are able to do inference and generate hypotheses of 3D poses $\bm{H}_{\bm{y}}\in\mathbb{R}^{H\times N\times J\times 3}$, where $H$ is the number of the hypotheses. We then conformalize the hypotheses to select the higher-quality ones and aggregate the latter in order to obtain a single estimate $\tilde{\bm{y}}$ as the final output.

## 4 Methods

We use a Diffusion model (Ho et al., ) to learn the conditional distribution $p_{\theta}(\bm{y}|\bm{x})$ due to its capability of modeling complex distributions and producing high-quality samples. With the trained diffusion model, we are able to generate many hypotheses sequences and aggregate them to produce a robust single prediction sequence for the final output. Moreover, we would like to make the model itself aware of the post-hoc hypotheses aggregation step so that the aggregation step does not get skewed by low-quality hypotheses. Thus, there should be an extra component in the model to shape the hypotheses confidence set. To achieve this, we learn an extra scoring model to rank the hypotheses and optimize the size of the hypotheses confidence set thresholded by the 90% quantile score, effectively simulating conformal prediction during training, illustrated in Fig. 2.

Figure: Figure 2: CHAMP Overview. CHAMP takes as input a sequence of 2D keypoints detected on a series of input RGB video frames. The 2D keypoints sequence gets fed into a Diffusion model to produce 3D keypoints hypotheses for the sequence. The output of the Diffusion model is supervised via a Pose Loss. Then we apply differentiable CP end-to-end during training on the hypotheses sequences, resulting in a smaller confidence set. The confidence set is used to calculate an Inefficiency Loss during training. Note that we show one frame in the sequence and hard assignment for the confidence set during training for better interpretability.
Refer to caption: x2.png

### 4.1 Learning 3D Human Poses with a Diffusion Model

The use of Diffusion models in 3D human pose estimation has been shown effective in various previous works (Feng et al., ; Gong et al., ; Shan et al., ; Choi et al., ), and we adapt the same paradigm in modeling the forward and the reverse process for the Diffusion model.

Forward Process. In the forward process, the Diffusion model takes as input a ground-truth 3D human pose sequence $\bm{y}_{0}\in\mathbb{R}^{N\times J\times 3}$ and a sampled time step $t\sim\text{Unif}(0,T)$, where $T$ is the maximum number of diffusion steps. Then the forward process gradually diffuses the input by adding independent Gaussian noises $\bm{\epsilon}\sim\mathcal{N}(0,\bm{I})$ at each step. The nice property reported in (Ho et al., ) shows that the process can be succinctly written as:

$$ $q(\bm{y}_{t}|\bm{y}_{0})=\sqrt{\bar{\alpha}_{t}}\bm{y}_{0}+\sqrt{1-\bar{\alpha }_{t}}\bm{\epsilon}$ (1) $$

where $\bar{\alpha}_{t}$ is a constant depending on the cosine variance schedule.

Reverse Process. To train such a Diffusion model, in the reverse process, we denoise the corrupted 3D pose sequence $\bm{y}_{t}$. While the derivation in (Ho et al., ) simplifies the ELBO objective to minimizing the distance between the injected noise $\bm{\epsilon_{t}}$ and a learned noise $\bm{\epsilon}_{\theta}(\sqrt{\bar{\alpha}_{t}}\bm{y}_{0}+\sqrt{1-\bar{\alpha}_
{t}}\bm{\epsilon}_{t},t)$, we frame the problem slightly differently, following (Shan et al., ). Instead of training a network to learn the injected noise, we train a denoiser model $D_{\theta}$ that outputs the predicted pose $\bar{\bm{y}}_{0}$ directly:

$$ $\bar{\bm{y}}_{0}=D_{\theta}(\bm{y}_{t},\bm{x},t)$ (2) $$

where the denoiser model takes as input the corrupted 3D pose sequence, the input 2D keypoints sequence, and the diffusion step to recover the uncorrupted input pose sequence $\bar{\bm{y}}_{0}$. We then supervise the predicted sequence with the ground-truth sequence using the mean per-joint MSE:

$$ $\mathcal{L}_{\text{pose}}=\frac{||\bm{y}_{0}-\bar{\bm{y}}_{0}||^{2}_{2}}{N \cdot J}$ (3) $$

Denoiser Model. We build upon the MixSTE model (Zhang et al., ) as the denoiser model in the reverse process of the Diffusion model. MixSTE uses two separate attention mechanisms, effectively learning spatial and temporal relationships of the keypoints sequence in a modular way by combining several spatial and temporal attention blocks. To condition the MixSTE network on the corrupted 3D keypoints sequence, we change the network input by concatenating 2D keypoints and noisy 3D poses and we add a diffusion timestep embedding after the input embedding in the first attention block. A detailed description of the denoiser model is given in Appendix B and D.

Generating Hypotheses. Using the denoiser model, which models the conditional distribution $p_{\theta}(\bm{y}|\bm{x})$, we are able to generate a number of hypotheses during inference. Following the reverse process of the Diffusion model, we obtain the hypotheses set $\bm{H}_{\bm{y}}=\{\bar{\bm{y}}^{1},\bar{\bm{y}}^{2},\dots,\bar{\bm{y}}^{H}\}$ of size ${H}$ by sampling standard normal noises ${\bm{y}}^{h}_{T}\sim\mathcal{N}(0,\bm{I})\in\mathbf{R}^{N\times J\times 3}$, where the superscript $h=[1,2,\dots,H]$ indexes the hypotheses, and feeding them into the denoiser model $D_{\theta}$:

$$ $\bar{\bm{y}}^{h}=D_{\theta}({\bm{y}}^{h}_{T},\bm{x},T)$ (4) $$

### 4.2 Learning Conformalization for the Hypotheses Confidence Set

While we are able to generate arbitrarily large numbers of hypotheses during test time with the distribution learned by the Diffusion model, we still need to aggregate the proposed hypotheses into a single prediction. The correct way to aggregate the hypotheses remains an open problem. Taking a naive average would be suboptimal as the existence of outliers in the proposed hypotheses might skew the average. Other approaches improve the aggregation step by selecting the hypotheses that result in closer distances to 2D keypoints after projection via camera matrices, but still retain a large set of hypotheses before the aggregation step (Shan et al., ).

Conformal Prediction. A possible solution is to conformalize the hypotheses using conformal prediction (CP). Conformal prediction produces a set of predictions which covers the ground truth with high probability: given a trained prediction model $f_{\theta}$, conformal prediction first calibrates a cutoff value $\tau$ by ranking a calibration set $I_{\text{cal}}$ given each component’s conformity score $\phi(\bm{y}_{\text{cal}},\bm{y}_{\text{GT}})$ and setting the cutoff value as the user-specified $(1+1/|I_{\text{cal}}|)\alpha$-quantile of the calibration set scores. After the calibration step, the generic threshold conformal predictor constructs the conformal prediction set as follows:

$$ $C(\bm{x},\tau)=\{\bm{y}:\phi(\bm{y},f_{\theta}(\bm{x}))\geq\tau\}$ (5) $$

With a properly selected conformity score function $\phi$, as well as a user-specified coverage parameter $\alpha$, the post-training conformal prediction guarantees the true value to be included in $C(\bm{x},\tau)$ with confidence level $\alpha$, i.e. $P(\bm{y}_{\text{GT}}\in C(\bm{x},\tau))\geq\alpha$. A key metric for a conformal predictor is the size of the confidence set $C_{\theta}(\bm{x},\tau)$, or *inefficiency*: a good confidence set should be large enough to cover the ground truth, and small enough to be informative (low inefficiency). A very large confidence set could cover the ground truth with high probability, but it might not be necessarily useful as it is inefficient to consider a lot of possible values (high inefficiency). Given ${H}$ hypotheses of 3D human pose sequences, we wish to use the threshold conformal predictor above to refine the hypotheses set, resulting in ${H}^{\prime}$ “good” hypotheses, where ${H}^{\prime}\leq H$. Essentially, we wish to have a low-inefficiency conformal predictor, which selects a smaller number of elite hypotheses from the set for downstream aggregation. CP is intended to be used as a “wrapper” around the prediction model, and we wish to achieve better, fine-grained control over the inefficiency of the CP wrapper for downstream aggregation, conditioned on the input 2D and output 3D human pose sequence data. Thus, inspired by (Stutz et al., ), we integrate CP into the training, optimizing for the inefficiency of the confidence set online by simulating CP during training in a differentiable manner.

Learning CP via Inefficiency Optimization. We build on top of the ConfTr paradigm proposed in (Stutz et al., ). We first design a conformity score function for poses. Unlike (Yang & Pavone, ), where the score function $\phi$ is hand-designed, we wish to learn a conformity score function $\phi_{\theta}$ that measures the distance between the proposed hypothesis and the ground truth. During training, we propose ${H}_{\text{train}}$ predicted output sequences as the hypotheses for each input 2D keypoints sequence in the mini-batch. To implement the score function, we reuse the input embedding layer of the modified MixSTE and further project the embeddings into two lower-dimensional tensors with an extra MLP for the hypothesis and the ground-truth 3D pose sequence respectively and then calculate the cosine similarity between the two embeddings. Thus, a higher score function output means the hypothesis is more similar to the ground-truth 3D pose sequence in the embedding space. Formally, the conformity score between the hypothesis 3D pose sequence $\bar{\bm{y}}^{h}$ and the ground-truth 3D pose sequence $\bm{y}_{\text{GT}}$ is measured as:

$$ $\phi_{\theta}(\bar{\bm{y}}^{h},\bm{y}_{\text{GT}})=\frac{s_{\theta}\left(\text {IE}_{\theta}(\bar{\bm{y}}^{h})\right)^{\top}s_{\theta}(\text{IE}_{\theta}(\bm {y}_{\text{GT}}))}{||s_{\theta}\left(\text{IE}_{\theta}(\bar{\bm{y}}^{h}))|| \cdot||s_{\theta}(\text{IE}_{\theta}(\bm{y}_{\text{GT}})\right)||}$ (6) $$

where $s_{\theta}$ is the extra MLP in the conformity score model and $\text{IE}_{\theta}$ is the input embedding layer of the denoiser model.
With the conformity score model output, we perform differentiable CP on each mini-batch of size $B$ during training. Similar to (Stutz et al., ), we split each set of $H_{\text{train}}$ hypotheses of the mini-batch in half: the first half, $\bm{H}_{\text{cal}}$, is used for calibration, and the second half, $\bm{H}_{\text{pred}}$, for prediction and loss computation. On $\bm{H}_{\text{cal}}$, we calibrate $\tau$ by computing the $(1+1/|\bm{H}_{\text{cal}}|)\alpha$-quantile of the conformity scores in a differentiable manner. This is done using a smooth sorting operation, which has been widely studied in (Blondel et al., ) and (Cuturi et al., ). On $\bm{H}_{\text{pred}}$, we calculate the soft inefficiency score by using the soft assignment of hypothesis $\bar{\bm{y}}^{h}$ being in the prediction set given the quantile threshold $\tau$:

$$ $C_{\theta}(\bar{\bm{y}}^{h},\tau):=\sigma\left(\frac{\phi_{\theta}(\bar{\bm{y} }^{h},\bm{y}_{\text{GT}})-\tau}{\mathcal{T}}\right)$ (7) $$

where $\sigma$ here is the sigmoid function and $\mathcal{T}$ is the temperature hyperparameter. When $\mathcal{T}\rightarrow 0$, we recover the hard assignment and $C_{\theta}(\bar{\bm{y}}^{h},\tau)=1$ if $\phi_{\theta}(\bar{\bm{y}}^{h},\bm{y}_{\text{GT}})\geq\tau$ and 0 otherwise. Thus, we are able to measure the size of the CP set of the hypotheses by $\sum_{h}C_{\theta}(\bar{\bm{y}}^{h},\tau)$.
On $\bm{H}_{\text{pred}}$, we compute $C_{\theta}(\bar{\bm{y}}^{h},\tau)$ and then calculate the differentiable inefficiency score:

$$ $\Omega\left(C_{\theta}(\bm{H}_{\text{pred}},\tau)\right)=\max\left(\sum_{h=1}^ {|\bm{H}_{\text{pred}}|}C_{\theta}(\bar{\bm{y}}^{h},\tau)-\kappa,0\right)$ (8) $$

where $\kappa$ is a hyperparameter that avoids penalizing singletons. With the differentiable inefficiency score defined, we are able to optimize for the expected inefficiency across batches during training:

$$ $\mathcal{L}_{\text{ineff}}=\log\mathbb{E}[\Omega(C_{\theta}(\bm{H}_{\text{pred }},\tau))]$ (9) $$

Full Training Objective. The computational graph of this loss involves the denoiser model input embedding, and since every operation is made differentiable by construction, we are able to backpropagate the inefficiency loss into the model, further shaping the model on top of the original pose loss. The final training objective of our model is thus:

$$ $\mathcal{L}=\mathcal{L}_{\text{pose}}+\lambda\mathcal{L}_{\text{ineff}}$ (10) $$

where $\lambda$ is a hyperparameter representing the weight of the inefficiency loss.

### 4.3 Conformalized Inference

During inference, to retain the coverage guarantee of CP, any CP method can be applied to re-calibrate $\tau$ on a held-out calibration set $I_{\text{cal}}$ as usual, i.e., the thresholds $\tau$ obtained during training is not kept during test time. Given a trained pose estimation denoiser model $D_{\theta}$ and the conformity scoring model $\phi_{\theta}$ trained end-to-end with $D_{\theta}$, for each testing 2D keypoints sequence, $\bm{x}$, we wish to construct a hypotheses confidence set for the downstream aggregation as follows:

$$ $C_{\theta}(\bm{y},\tau)=\{\bm{y}:\phi_{\theta}(\tilde{\bm{y}},\bm{y})\geq\tau\}$ (11) $$

where $\tilde{\bm{y}}$ is obtained via eq. (4) and $\bm{y}_{T}\sim\mathcal{N}(0,\bm{I})$. In practice, we take the mean of 20 $\tilde{\bm{y}}$. To generate candidate hypotheses for the hypotheses confidence set, we sample from the learned distribution $p_{\theta}(\bm{y}|\bm{x})$ $H$ times using (4) and refine them using (11), resulting in a smaller hypotheses set $\bm{H}_{\bm{y}}$ of size $H^{\prime}$. We then aggregate the set $\bm{H}_{\bm{y}}$ to obtain a single output for practical use.

## 5 Experiments

To evaluate our method, we train and test on standard human pose estimation datasets and provide quantitative results. We also provide qualitative results on in-the-wild videos.

Human3.6M (Ionescu et al., ) is the standard and the largest indoor dataset for 3D human pose estimation. The dataset collects videos from 11 actors engaging in 15 activities and the pose sequence videos are captured by 4 synchronized and calibrated cameras at 50Hz. Similar to (Shan et al., ; Zhang et al., ), we train on 5 actors (S1, S5, S6, S7, S8) and evaluate on 2 actors (S9, S11). Quantitatively, following the standard evaluation scheme, we report the mean per joint position error (MPJPE), which is often referred to as Protocol #1, which computes the mean Euclidean distance between estimated and ground truth 3D joint positions in millimeters. We also provide Protocol #2 results in Appendix E.

MPI-INF-3DHP (Mehta et al., ) is a more challenging dataset consisting of indoor and outdoor activities, from 14 camera views which cover a greater diversity of poses. The training set contains 8 activities, and the test set contains 6. We preprocess the dataset with the valid frames provided by the authors for testing, following (Zhang et al., ; Shan et al., ; Gong et al., ). Quantitatively, we report MPJPE, the percentage of correct keypoints (PCK), which describes the percentage of keypoints with Euclidean error less than 150mm, as well as the AUC score for this percentage.

For both of the datasets, in our setting, we hold out 2% of the training dataset for conformal calibration during test time. The held-out calibration set is not seen by the network during training. We compare 4 variations of our method. The backbone (weights) of the variations and the training process remain the same, but the key difference lies in the aggregation step during test time:

- •
CHAMP-Naive: Trained with inefficiency loss, but for multi-hypothesis scenarios, during test time, we do not refine the hypotheses set with CP, and we use all hypotheses with a mean aggregation. In single-hypothesis scenarios, we only propose 1 hypothesis as the output.
- •
CHAMP: Same backbone as CHAMP-Naive, but only aggregates refined hypotheses confidence set in eq. (11) via mean aggregation.
- •
CHAMP-Agg: Instead of taking the average, we use the J-Agg method from (Shan et al., ) on the set in eq. (11), which uses known or estimated intrinsic camera parameters to reproject 3D hypotheses to the 2D camera plane and selecting the joint hypotheses with the minimum reprojection error.
- •
CHAMP-Best: We use J-Best method in (Shan et al., ) on the set in eq. (11), which selects the joint hypothesis that is closest to the ground truth, and then combines the selected joints into the final 3D pose. This is the upper bound of J-Agg performance.

### 5.1 Results on Human-3.6M

We discuss the quantitative results on the Human-3.6M dataset. In the single-hypothesis setting, we set $H=1$ to compare with other deterministic methods. While our method primarily focuses on multi-hypothesis scenarios, we propose 1 hypothesis and use the CHAMP-Naive variation, without using the conformal prediction pipeline. As the results suggest in Table 1, our method achieves performance on par with the current state-of-the-art methods in the single-hypothesis case.

In the multi-hypothesis setting, we set $H=80$. CHAMP variants achieve SOTA results, especially when combined with more sophisticated aggregations proposed in D3DP (Shan et al., ). On average, without the conformal prediction pipeline, CHAMP-Naive is able to achieve 40mm MPJPE error and improves by 1.5mm when using conformal prediction and mean aggregation. Using J-Agg and J-Best, CHAMP gets further improved by 1.7mm and 3.6mm respectively.

**Table 1: MPJPE ($\downarrow$) results on Human-3.6M dataset. Red: lowest error. Blue: second lowest error. In the single-hypothesis setting, our method runs without CP. In the multi-hypothesis setting, we compare four different variants of CHAMP as discussed in the main text. ^∗Upper-bound performance, uses ground truth.**
| Mean Per-Joint Position Error (Protocol # 1) - mm |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | AVG. | Dir | Disc. | Eat | Greet | Phone | Photo | Pose | Pur. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT. |
| Single Hypothesis |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Ray3D (Zhan et al., ) | 49.7 | 44.7 | 48.7 | 48.7 | 48.4 | 51.0 | 59.9 | 46.8 | 46.9 | 58.7 | 61.7 | 50.2 | 46.4 | 51.5 | 38.6 | 41.8 |
| STE (Li et al., ) | 43.6 | 39.9 | 43.4 | 40.0 | 40.9 | 46.4 | 50.6 | 42.1 | 39.8 | 55.8 | 61.6 | 44.9 | 43.3 | 44.9 | 29.9 | 30.3 |
| P-STMO (Shan et al., ) | 42.8 | 38.9 | 42.7 | 40.4 | 41.1 | 45.6 | 49.7 | 40.9 | 39.9 | 55.5 | 59.4 | 44.9 | 42.2 | 42.7 | 29.4 | 29.4 |
| MixSTE (Zhang et al., ) | 41.0 | 37.9 | 40.1 | 37.5 | 39.4 | 43.3 | 50.0 | 39.8 | 39.9 | 52.5 | 56.6 | 42.4 | 40.1 | 40.5 | 27.6 | 27.7 |
| DUE (Zhang et al., ) | 40.6 | 37.9 | 41.9 | 36.8 | 39.5 | 40.8 | 49.2 | 40.1 | 40.7 | 47.9 | 53.5 | 40.2 | 41.1 | 40.3 | 30.8 | 28.6 |
| D3DP (Shan et al., ) | 40.0 | 37.7 | 39.9 | 35.7 | 38.2 | 41.9 | 48.8 | 39.5 | 38.3 | 50.5 | 53.9 | 41.6 | 39.4 | 39.8 | 27.4 | 27.5 |
| CHAMP-Naive (H=1) | 40.2 | 39.5 | 40.3 | 35.8 | 38.7 | 42.2 | 48.7 | 40.2 | 38.2 | 50.3 | 53.2 | 41.5 | 39.2 | 40.2 | 26.9 | 27.3 |
| Multiple Hypotheses |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MHFormer (Li et al., ) | 43.0 | 39.2 | 43.1 | 40.1 | 40.9 | 44.9 | 51.2 | 40.6 | 41.3 | 53.5 | 60.3 | 43.7 | 41.1 | 43.8 | 29.8 | 30.6 |
| GraphMDN (Oikarinen et al., ) | 61.3 | 51.9 | 56.1 | 55.3 | 58.0 | 63.5 | 75.1 | 53.3 | 56.5 | 69.4 | 92.7 | 60.1 | 58.0 | 65.5 | 49.8 | 53.6 |
| DiffuPose (Choi et al., ) | 49.4 | 43.4 | 50.7 | 45.5 | 50.2 | 49.6 | 53.4 | 48.6 | 45.0 | 56/9 | 70.7 | 47.8 | 48.2 | 51.3 | 43.1 | 43.4 |
| DiffPose (Gong et al., ) | 36.9 | 33.2 | 36.6 | 33.0 | 35.6 | 37.6 | 45.1 | 35.7 | 35.5 | 46.4 | 49.9 | 37.3 | 35.6 | 36.5 | 24.4 | 24.1 |
| DiffPose (Feng et al., ) | 43.3 | 38.1 | 43.3 | 35.3 | 43.1 | 46.6 | 48.2 | 39.0 | 37.6 | 51.9 | 59.3 | 41.7 | 47.6 | 45.5 | 37.4 | 36.0 |
| D3DP-Agg (Shan et al., ) | 39.5 | 37.3 | 39.5 | 35.6 | 37.8 | 41.4 | 48.2 | 39.1 | 37.6 | 49.9 | 52.8 | 41.2 | 39.2 | 39.4 | 27.2 | 27.1 |
| CHAMP-Naive | 40.0 | 39.4 | 40.1 | 35.4 | 38.2 | 42.0 | 48.5 | 40.0 | 38.1 | 50.1 | 52.9 | 43.4 | 39.1 | 39.9 | 27.7 | 28.2 |
| CHAMP | 38.5 | 36.5 | 38.9 | 34.2 | 37.1 | 40.1 | 46.9 | 38.2 | 36.5 | 47.9 | 50.7 | 40.1 | 37.6 | 37.6 | 25.7 | 26.4 |
| CHAMP-Agg | 36.0 | 36.2 | 36.1 | 30.9 | 31.8 | 39.5 | 43.6 | 35.1 | 33.9 | 43.1 | 47.2 | 39.9 | 34.9 | 36.1 | 25.1 | 25.9 |
| D3DP-Best^∗ (Shan et al., ) | 35.4 | 33.0 | 34.8 | 31.7 | 33.1 | 37.5 | 43.7 | 34.8 | 33.6 | 45.7 | 47.8 | 37.0 | 35.0 | 35.0 | 24.3 | 24.1 |
| CHAMP-Best^∗ | 34.9 | 35.9 | 35.2 | 30.3 | 31.4 | 37.4 | 42.9 | 34.4 | 33.5 | 41.5 | 46.9 | 37.9 | 34.8 | 34.8 | 24.5 | 24.6 |

**Table 2: Results on the 3DHP dataset.**
|  | PCK $\uparrow$ | AUC $\uparrow$ | MPJPE $\downarrow$ |
| --- | --- | --- | --- |
| Single Hypothesis |  |  |  |
| P-STMO (Shan et al., ) | 97.9 | 75.8 | 32.2 |
| MixSTE (Zhang et al., ) | 96.9 | 75.8 | 35.4 |
| D3DP (Shan et al., ) | 97.7 | 77.8 | 30.2 |
| CHAMP-Naive | 97.5 | 77.7 | 30.2 |
| Multiple Hypotheses |  |  |  |
| MHFormer (Li et al., ) | 93.8 | 63.3 | 58.0 |
| DiffPose (Gong et al., ) | 98.0 | 75.9 | 29.1 |
| DiffPose (Feng et al., ) | 94.6 | 62.8 | 64.6 |
| D3DP-Agg (Shan et al., ) | 97.7 | 78.2 | 29.7 |
| CHAMP-Naive | 97.5 | 78.1 | 29.9 |
| CHAMP | 97.9 | 78.4 | 29.1 |
| CHAMP-Agg | 98.0 | 78.6 | 28.6 |
| D3DP-Best (Shan et al., ) | 98.0 | 79.1 | 28.1 |
| CHAMP-Best | 98.2 | 79.1 | 28.0 |

### 5.2 Results on MPI-INF-3DHP

We discuss the results on the MPI-INF-3DHP dataset in Table 2. In the single-hypothesis setting, CHAMP-Naive achieves results on par with other deterministic methods for the three metrics.

In the multi-hypothesis setting, we again set $H=80$. CHAMP-Naive again yields fairly competitive results and when combined with conformal prediction and mean, J-Agg, and J-Best aggregation techniques, the performance continuously gets improved across the three metrics, with the CHAMP-Agg metric being the most competitive one, achieving SOTA results. These results here on this dataset follow the same trend as those on the Human3.6M dataset.

### 5.3 In-the-Wild Videos

To show the generalizability of our method to in-the-wild videos, we collect videos from YouTube, TikTok, and 3DPW dataset (Von Marcard et al., ). To construct 2D keypoints input, we use Cascaded Pyramid Network (Chen et al., ) fine-tuned on the Human3.6M dataset as the default weights are trained on MSCOCO (Lin et al., ), which has a slightly different skeleton structure. We directly apply the CHAMP model trained on the Human3.6M dataset to test on in-the-wild videos. Sample results are shown in Fig. 1, where the input videos are collected from TikTok. Qualitative results suggest that CHAMP is able to filter out outlier hypotheses using the conformity score function trained end-to-end with the pose estimation model. More results are shown in Appendix F.

### 5.4 Ablation Studies with Human3.6M Dataset

Figure: Figure 3: Your Figure Caption Here
Refer to caption: x3.png

Choice of the Conformity Function.
We compare our proposed end-to-end learned conformity score function with a separately trained conformity score function (Corso et al., ), and a hand-designed conformity score function. We compare the average performance across all actions on the Human3.6M dataset. For the separately trained conformity score function, we first only train the pose estimation backbone and generate 100 hypotheses per input sequence over a small subset of the training dataset. We then train the score function to predict if the proposed hypothesis conditioned on the input 2D keypoints results in an MPJPE of less than 25mm and use the ground-truth pose sequence as supervision. For the hand-designed conformity score, we use $\phi_{\text{peak}}$ from (Yang & Pavone, ), measuring the maximum MPJPE with respect to the ground truth across output frames. Note $\phi_{\text{peak}}$ measures “nonconformity”, so we negate such values to fit our setup. All three variants use the J-Best aggregation method. From Fig. 6, E2E score function yields the best performance and the two learned variants supersede the hand-designed version. In the two examples shown in Fig. 6, the filtered hypotheses seem to be more concentrated for learned score functions. It’s worth noting that all three variants result in competitive performance, demonstrating the importance of CP.

Number of Hypotheses. Another interesting ablation study to conduct is to ablate on the number of hypotheses during training ($H_{\text{train}}$) and inference ($H$). We compare the effects of $H_{\text{train}}$, $H$ in Fig. 6, on the four variants of CHAMP. To compare the effect of $H_{\text{train}}$, we fix $H=80$ and train a model for each value of hypotheses proposed during differentiable CP. Similarly, to compare the effect of $H$, we fix $H_{\text{train}}=20$ during training and use the same model with different $H$ during inference for CP. Results suggest that a good performance can be achieved with $H_{\text{train}}=20$ during training, while any number higher than 20 brings marginal improvement and requires more GPU memory. Moreover, $H=80$ during inference is sufficient for all four variants.

Strength of the Inefficiency Loss. Finally, we ablate on the value of $\lambda$ in the overall training objective $\mathcal{L}$. This is an important ablation in that we can find suitable strength of the inefficiency loss $\mathcal{L}_{\text{ineff}}$ to make sure it does not conflict $\mathcal{L}_{\text{pose}}$. Results in Fig. 6 suggest that $\lambda=0.05$ is the most efficient strength across all values, as smaller $\lambda$ does not train the scoring model sufficiently and higher $\lambda$ conflicts with the pose loss. This corroborates the smaller-scale hyperparameter sweep experiments we conducted before training the models.

### 5.5 Exchangeability and CP Guarantee

The inherent assumption of CP requires the calibration dataset to be exchangeable, which is weaker than
asking them to be independent (Angelopoulos & Bates, ). This assumption
typically fails when the calibration set is a single video sequence, where the image frames are temporally correlated. We learn a sequence-to-sequence model, where the input and output are long sequences of image frames from different videos, instead of single frames from the same video. This would drastically decrease the temporal correlation across the calibration dataset. Moreover, as shown in (Yang & Pavone, ), the video frames tend to be more independent if the videos are taken by multiple evenly spaced cameras, which is the case for our datasets of interest. We also investigate the empirical coverage of our method in Human 3.6M test set in Appendix A. Results suggest that we are able to inherit the coverage guarantee from CP with the learned conformity score even if the dataset is not fully exchangeable.

### 5.6 Implementation and Training Details

CHAMP’s denoiser model uses Adam optimizer with a weight decay parameter of 0.1 and momentum parameters $\beta_{1}=\beta_{2}=0.99$. For the training objective in eq. (10), we use $\lambda=0.05$. We
train the CHAMP model using an NVIDIA V100 GPU, where the training process consumes an amortized GPU memory of 18GB, for 300 epochs with a batch size of 8 and a learning rate of 5e-5 and reduce it on plateau with a factor of 0.5. Following previous work (Zhang et al., ; Shan et al., , ), we use input pose sequence of 243 frames ($N=243$) of Human3.6M universal skeleton format. During training, the number of hypotheses is 20, and #DDIM iterations is set to 1. During inference, they are set to 80 and 10. The maximum number of diffusion steps is $T=999$.

## 6 Limitations

While our method shows a promising improvement over the current 3D human pose estimation methods, it does have several limitations. First, the theoretical coverage guarantee of CP cannot be fully justified due to the input videos’ temporal correlation. Moreover, many hypotheses need to be proposed during training to improve the learned conformity score, which consumes a lot more GPU memory. Lastly, CHAMP only learns single-human 3D skeleton estimation, without considering dense, multi-human scenarios.

## 7 Conclusion

This work presents CHAMP, a novel method for learning multi-hypotheses 3D human poses with a learned conformal predictor from 2D keypoints. We empirically show that CHAMP achieves competitive results across multiple metrics and datasets, and when combined with more sophisticated downstream aggregation methods, it achieves state-of-the-art performance. Future work includes using more recent CP techniques that relax the exchangeability assumption (Barber et al., ), using more efficient sequence-to-sequence models such as Mamba (Gu & Dao, ), and scaling the pipeline up to dense, multi-human pose-shape joint prediction scenarios (Loper et al., ).

## Appendix A Coverage Guarantee and Empirical Coverage

Please refer to the standard proof shown in (Angelopoulos & Bates, ; Shafer & Vovk, ) for the coverage guarantees of conformal prediction.

To evaluate the empirical coverage of CHAMP, we calculate the mean coverage percentage across test data. Specifically, we use eq. (11) to check if the ground truth belongs in the confidence set formed by the conformity function on the test data:

$$ $\displaystyle\bar{\mathcal{C}}$ $\displaystyle=\frac{1}{|{I}_{\text{test}}|}\sum_{\bm{y}_{\text{GT}}\sim{I}_{ \text{test}}}\mathbbm{1}\left(\bm{y}_{\text{GT}}\in C_{\theta}(\bm{y},\tau)\right)$ $\displaystyle=\frac{1}{|{I}_{\text{test}}|}\sum_{\bm{y}_{\text{GT}}\sim{I}_{ \text{test}}}\mathbbm{1}\left(\phi_{\theta}(\tilde{\bm{y}},\bm{y}_{\text{GT}}) \geq\tau\right)$ $$

We compare the three choices of conformity scores for the empirical coverage calculation. We use the test set from Human 3.6M and calculate the empirical coverage values across all activities.

Figure: Figure 7: Empirical coverage comparison across three different conformity functions. We keep the best $\lambda=0.05$ for comparison.
Refer to caption: extracted/5622292/figs/app1.png

During training and testing, we keep $\alpha=0.1$ for CP. From Fig. 7, we see the empirical coverage is around $90\%\pm 5\%$ for all activities, which remains close to $1-\alpha$, and in some cases, it exceeds this value. This is encouraging given the exchangeability assumption is not fully satisfied in our dataset.

## Appendix B Denoiser in Training vs Inference

We first revisit DDIM (Song & Ermon, ). We start with the reverse process derivation and rewrite $q(\bm{y}_{t-1}|\bm{y}_{t},\bm{y}_{0})$ to be parameterized by a desired standard deviation $\sigma_{t}$:

$$ $\displaystyle\bm{y}_{t-1}$ $\displaystyle=\sqrt{\alpha_{t-1}}\bm{y}_{0}+\sqrt{1-\alpha_{t-1}}\bm{\epsilon} _{t-1}$ $\displaystyle=\sqrt{\alpha_{t-1}}\bm{y}_{0}+\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2 }}\bm{\epsilon}_{t-1}+\sigma_{t}\bm{\epsilon}$ $\displaystyle=\sqrt{\alpha_{t-1}}\left(\bm{y}_{t}-\sqrt{1-\alpha_{t}}\bm{ \epsilon}^{(t)}_{\theta}(\bm{y}_{t})\right)+\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2 }}\bm{\epsilon}^{(t)}_{\theta}(\bm{y}_{t})+\sigma_{t}\bm{\epsilon}$ $\displaystyle=\sqrt{\alpha_{t-1}}\left(\bm{y}_{t}-\sqrt{1-\alpha_{t}}\bm{ \epsilon}^{(t)}_{\theta}(\bm{y}_{t})\right)+\sqrt{1-\alpha_{t-1}-\sigma_{t}^{2 }}\bm{\epsilon}^{(t)}_{\theta}(\bm{y}_{t})+\sigma_{t}\bm{\epsilon}$ $$

where the model $\bm{\epsilon}^{(t)}_{\theta}(\cdot)$ predicts the $\bm{\epsilon}_{t}$ from $\bm{y}_{t}$. Since $q(\bm{y}_{t-1}|\bm{y}_{t},\bm{y}_{0})=\mathcal{N}(\bm{y}_{t-1};\mu(\bm{y}_{t},
\bm{y}_{0}),\beta_{t}\mathbf{I})$, therefore we have:

$$ $\tilde{\beta}_{t}=\sigma_{t}^{2}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t }}\beta_{t}$ $$

Let $\sigma_{t}^{2}=\eta\cdot\beta_{t}$ where $\eta$ is a hyperparameter that controls the sampling stochasticity. During generation, we don’t have to follow the whole chain $t=1,\ldots,T$, but only a subset of steps. Denote $t^{\prime}<t$ as two steps in this accelerated trajectory. The DDIM update step is defined as follows:

$$ $q_{t^{\prime},\theta}(\bm{y}_{t^{\prime}}|\bm{y}_{t},\bm{y}_{0})=\mathcal{N} \left(\bm{x}_{t^{\prime}};\sqrt{\alpha_{t^{\prime}}}\left(\bm{y}_{t}-\sqrt{1- \alpha_{t}}\bm{\epsilon}^{(t)}_{\theta}(\bm{y}_{t})\right),\sigma_{t}^{2} \mathbf{I}\right)$ $$

Following D3DP (Shan et al., ), the denoiser model $D_{\theta}$ uses DDIM to sample denoised poses from the corrupted ones. During training, we run DDIM for only 1 step for the sake of efficiency:

$$ $\bar{\bm{y}}^{h}=D_{\theta}({\bm{y}}^{h}_{T},\bm{x},T),\;{\bm{y}}^{h}_{T}\sim \mathcal{N}(0,\mathbf{I})\quad\forall h=\{1,\cdots,H\}$ $$

During inference, we run DDIM for $K=10$ times, and each step is defined as:

$$ $\displaystyle t=T\cdot(1-k/K),\;\;t^{\prime}$ $\displaystyle=T\cdot(1-(k+1)/K)),\;\;k=\{0,\cdots,K-1\}$ $\displaystyle\bar{\bm{y}}^{h}$ $\displaystyle=D_{\theta}(\bar{\bm{y}}^{h}_{t},\bm{x},t),\quad\forall h=\{1, \cdots,H\}$ $\displaystyle\bar{\bm{y}}^{h}_{t^{\prime}}$ $\displaystyle=\sqrt{\overline{\alpha}_{t^{\prime}}}\cdot\bar{\bm{y}}^{h}+\sqrt {1-\overline{\alpha}_{t^{\prime}}-\sigma_{t}^{2}}\cdot\bm{\epsilon}_{t}+\sigma _{t}\bm{\epsilon}$ $\displaystyle t$ $\displaystyle\leftarrow t^{\prime}$ $$

## Appendix C Training-Calibration-Testing Data Split

We discuss the split of training, calibration, and testing data. We first split the testing data the same way as (Zhang et al., ; Shan et al., ; Gong et al., ) to ensure the fairness of results. We then further split the training set into the actual training dataset and a calibration dataset before inference. Specifically, we split the training dataset by uniformly sampling a 2% subset as the calibration dataset, and CHAMP is only trained on the remaining 98%. Note here that this 2% calibration dataset is not seen during training by any means. During training, we split each mini-batch evenly into $B_{\text{pred}}$ and $B_{\text{cal}}$.

## Appendix D Architecture Details

We provide a more detailed version of Fig. 2. We illustrate the detailed architecture of the denoiser model $D_{\theta}$ as well as the conformity scoring model $s_{\theta}$ in Fig. 8. MixSTE (Zhang et al., ) is used as the backbone of the denoiser. MixSTE
combines 16 alternative spatial and temporal attention blocks. In the implementation, the channel size is set to 512. Similar to (Shan et al., ; Gong et al., ), we concatenate 2D keypoints and noisy 3D poses as inputs and add a diffusion timestep embedding after the input embedding using a sinusoidal function. For the conformity scoring model, we concatenate 2D keypoints and predicted hypotheses 3D poses as inputs, reuse the input embedding layer, and use an extra MLP layer to project the input embeddings into a lower dimension of 3. In training, CP is made differentiable by using soft ranking and soft assignment, resulting in a differentiable inefficiency loss $\mathcal{L}_{\text{ineff}}$. This loss is combined with the pose loss $\mathcal{L}_{\text{pose}}$ when backpropagating into the network. During inference (light cyan shaded area), we refine the hypotheses confidence set with regular CP on a held-out calibration set, resulting in $H^{\prime}<H$ hypotheses. We then aggregate the refined set, resulting in a single final prediction.

Figure: Figure 8: Detailed architecture of $D_{\theta}$ and $s_{\theta}$, the denoiser and conformity score models during training and inference. During training, the construction of refined hypotheses set after differentiable CP is done via soft assignment. During inference, regular CP is used with the trained conformity score function.
Refer to caption: x4.png

## Appendix E Procrustes-MPJPE Performance

The Procrustes MPJPE (P-MPJPE) metric, which is often referred to as Protocol #2, computes MPJPE after the estimated poses align to the ground truth using a rigid transformation. It is another standard metric for measuring a 3D pose estimator’s performance (Pan et al., ). We report the performance of the four variants of CHAMP against other baselines on the Human3.6M dataset in Table 3. Quantitative results suggest our method achieves SOTA performance on this metric as well.

**Table 3: P-MPJPE ($\downarrow$) results on Human-3.6M dataset. ^∗Upper-bound performance, needs ground truth.**
| Procrustes Per-Joint Position Error (Protocol # 2) - mm |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | AVG. | Dir | Disc. | Eat | Greet | Phone | Photo | Pose | Pur. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT. |
| Single Hypothesis |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| STE Li et al. () | 35.2 | 32.7 | 35.5 | 32.5 | 35.4 | 35.9 | 41.6 | 33.0 | 31.9 | 45.1 | 50.1 | 36.4 | 33.5 | 35.1 | 23.9 | 25.0 |
| P-STMO Shan et al. () | 34.4 | 31.3 | 35.2 | 32.9 | 33.9 | 35.4 | 39.3 | 32.5 | 31.5 | 44.6 | 48.2 | 36.3 | 32.9 | 34.4 | 23.8 | 23.9 |
| MixSTE Zhang et al. () | 32.6 | 30.8 | 33.1 | 30.3 | 31.8 | 33.1 | 39.1 | 31.1 | 30.5 | 42.5 | 44.5 | 34.0 | 30.8 | 32.7 | 22.1 | 22.9 |
| DUE Zhang et al. () | 32.5 | 30.3 | 34.6 | 29.6 | 31.7 | 31.6 | 38.9 | 31.8 | 31.9 | 39.2 | 42.8 | 32.1 | 32.6 | 31.4 | 25.1 | 23.8 |
| D3DP Shan et al. () | 31.7 | 30.6 | 32.5 | 29.1 | 31.0 | 31.9 | 37.6 | 30.3 | 29.4 | 40.6 | 43.6 | 33.3 | 30.5 | 31.4 | 21.5 | 22.4 |
| CHAMP-Naive | 31.2 | 30.9 | 32.1 | 29.4 | 30.9 | 31.2 | 35.9 | 29.4 | 28.9 | 40.1 | 41.9 | 33.1 | 30.0 | 30.8 | 20.9 | 22.4 |
| Multiple Hypotheses |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MHFormer Li et al. () | 34.4 | 34.9 | 32.8 | 33.6 | 35.3 | 39.6 | 32.0 | 32.2 | 43.5 | 48.7 | 36.4 | 32.6 | 34.3 | 23.9 | 25.1 | 34.4 |
| GraphMDN Oikarinen et al. () | 46.9 | 39.7 | 43.4 | 44.0 | 46.2 | 48.8 | 54.5 | 39.4 | 41.1 | 55.0 | 69.0 | 49.0 | 43.7 | 49.6 | 38.4 | 42.4 |
| DiffuPose Choi et al. () | 39.9 | 35.9 | 40.3 | 36.7 | 41.4 | 39.8 | 43.4 | 37.1 | 35.5 | 46.2 | 59.7 | 39.9 | 38.0 | 41.9 | 32.9 | 34.2 |
| DiffPose Gong et al. () | 28.7 | 26.3 | 29.0 | 26.1 | 27.8 | 28.4 | 34.6 | 26.9 | 26.5 | 36.8 | 39.2 | 29.4 | 26.8 | 28.4 | 18.6 | 19.2 |
| DiffPose Feng et al. () | 32.0 | 28.1 | 31.5 | 28.0 | 30.8 | 33.6 | 35.3 | 28.5 | 27.6 | 40.8 | 44.6 | 31.8 | 32.1 | 32.6 | 28.1 | 26.8 |
| D3DP-Agg Shan et al. () | 31.6 | 30.6 | 32.4 | 29.2 | 30.9 | 31.9 | 37.4 | 30.2 | 29.3 | 40.4 | 43.2 | 33.2 | 30.4 | 31.3 | 21.5 | 22.3 |
| CHAMP-Naive | 32.0 | 31.9 | 33.1 | 30.8 | 31.1 | 32.2 | 38.1 | 29.2 | 28.3 | 41.2 | 43.1 | 34.5 | 31.2 | 30.4 | 22.6 | 23.9 |
| CHAMP | 31.1 | 30.8 | 32.4 | 29.9 | 30.9 | 31.4 | 37.2 | 27.8 | 26.9 | 40.2 | 42.7 | 32.9 | 30.6 | 29.8 | 21.4 | 22.8 |
| CHAMP-Agg | 30.2 | 30.1 | 31.7 | 28.6 | 30.1 | 30.2 | 36.1 | 26.9 | 25.4 | 39.1 | 41.4 | 31.9 | 29.8 | 28.9 | 20.9 | 21.9 |
| D3DP-Best^∗ Shan et al. () | 28.7 | 27.5 | 29.4 | 26.6 | 27.7 | 29.2 | 34.3 | 27.5 | 26.2 | 37.3 | 39.0 | 30.3 | 27.7 | 28.2 | 19.6 | 20.3 |
| CHAMP-Best^∗ | 28.5 | 28.9 | 30.1 | 27.2 | 27.3 | 28.2 | 34.4 | 24.7 | 24.1 | 37.2 | 38.9 | 29.9 | 28.1 | 27.1 | 20.1 | 19.9 |

## Appendix F More Qualitative Results on In-the-Wild Videos

Figure: Figure 9: Results of running CHAMP on in-the-wild videos collected from TikTok and YouTube.
Refer to caption: x5.png

We showcase the performance of CHAMP on more in-the-wild videos in this section. We collect various YouTube and TikTok videos and detect 2D keypoints with Cascaded Pyramid Network. Note that the official weights provided by CPN are trained in COCO format, so we used a fine-tuned version of the weights for Human3.6M universal skeleton format. With such 2D keypoints, we are able to directly use the model trained on Human3.6M dataset and apply it to real-world videos without any fine-tuning. In Fig. 9, for each row, we show two examples. For every three images, the leftmost image is the RGB observation, and the middle image is all the hypotheses proposed by CHAMP’s pose estimation backbone, and the rightmost image is the hypotheses set refined by the learned conformal predictor.

## Appendix G Learned CP in CHAMP

We provide more demonstrations on the learned conformal predictor in CHAMP. Specifically, in Fig. 10, we provide more examples of predicted hypotheses before and after the conformal predictor powered by the learned score function.

Figure: Figure 10: Examples of CP in CHAMP filtering out bad hypotheses during inference. Qualitatively, the learned score function filters out outlier predictions.
Refer to caption: x6.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: This paper is well-presented and validated. The only thing I concern is its performance on occluded or truncations where diverse multiple hypothesis are preferred.
soundness: 3
strengths: 1. The proposed conformal prediction is simple yet effective. 
2. This paper is well-written and easy to understand.
summary: CHAMP presents a conformalized multi-hypothesis pose estimation pipeline. Based on diffusion model, the multiple pose hypothesis are refined by a differentiable comformal prediction module with an inefficiency loss. Experiments show that CHAMP achieves state-of-the-art results on several benchmarks.
weaknesses: 1. CP removes the predictions with low confidence and visualisations also show after CP the predictions become more certain and accurate. However, the results are mainly shown with easy cases where occlusions barely present. I am wondering how CP performs on hard cases where occlusions, truncations exist?
2. It will be better if this method can be extended to image-to-3d pipelines.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: I don't fully understand the CHAMP-Agg method. Is this using J-Agg on the full set of hypotheses, or does it also use the learned scoring function in some way? How should we interpret the fact that "CHAMP-Agg" is better than "CHAMP"? Does it mean that the proposed learned conformity scoring does not perform better than the prior method of aggregation proposed by Shan et al. (2023)?

To the paragraph after Eq. 5: How can we interpret the "true value to be included in" the conformal set in case of continuous values like 3D coordinates? In case of classification this makes more sense, but perhaps the language has to adjusted here.
What 2D pose estimator is used in the main experiments, or is the ground truth used?
Why are values missing from Table 3?

Smaller issues:
GAN is cited as the CACM 2020 paper, but perhaps the original NIPS 2014 version is better suited for citation.
In Eq 9, perhaps Temperature could be denoted by some other symbol because mathcal{T} and tau look very similar visually (except for size). Similarly I would not recommend relying on the subtle difference between C and mathcal{C} for important differences (as noted in the footnote of page 6), a hat, tilde or similar feature may be more visible.

Page 7: Human3.6M is no longer the "largest indoor dataset for 3D human pose estimation". There are many larger ones such as DNA-Rendering.
It should be stated explicitly that E2E refers to "end-to-end".
soundness: 3
strengths: The application of conformal prediction to 3D human pose estimation is an interesting and original aspect. Multi-hypothesis 3D pose estimation has been a relevant research area due to ambiguities in the task, but practical methods are often not formulated in a proper probabilistic manner. Using conformal prediction techniques is a promising direction and taking a first step in this direction is relevant to the community.
The overview of the related works is extensive, and the comparison of them helps the reader to place this work in relation to them. The text is well-structured and easy to follow.
The authors use already existing techniques in good ways, for example using the already proven MixSTE model for denoising, incorporating adversarial training and conformal calibration.
Results on Human3.6M and MPI-INF-3DHP both demonstrate the benefit of the method.
Ablations and hyperparameter studies confirm the design choices.
summary: This work tackles the task of 2D-to-3D lifting of temporal human pose sequences through a diffusion-based multi-hypothesis approach. The novel aspect is applying the tools of conformal prediction to this task. That is, the authors propose to learn a conformity score function in an adversarial setup and simulate conformal prediction during training, directly minimizing the conformal set size (the inefficiency). Quantitative evaluation is performed on the well-known Human3.6M and MPI-INF-3DHP datasets, with additional qualitative evaluation on in-the-wild videos. Results show that filtering according to conformity scores and aggregating the results improves the metrics and achieves state-of-the-art results. Ablations also show that the end-to-end training of the scoring function is beneficial.
weaknesses: The method is only evaluated on studio datasets, but not on outdoor ones, such as 3DPW and EMDB. Evaluation on such less restricted videos would be more convincing.
It is unclear whether CHAMP's learned conformity scoring performs better than the prior established 2D-joint projection based version (see question below).

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: See the experiments mentioned above.
soundness: 3
strengths: 1. Learning the uncertainty of multi-hypothesis 3D human pose regression seems to be reasonable. Aside from building the distrubution of 3D human poses, how to select the best ones is an interesting research topic.   
2. Designs in scoring the hypotheses via conformal prediction are subtle. Detailed ablation study further reveals the influence of different parameters.   
3. The description of method is clear. Especially, Fig. 2 provides a clear overview of the relationship between different designs.
summary: This paper focuses on 3D pose estimation from 2D pose inputs in a multi-hypothesis sequence-to-sequence way. Following previous methods, they also adopt a diffusion model to regress the 3D pose hypotheses from sequential 2D pose inputs. Different from previous work, this paper proposes to learn a score model along with the regression model to predict the confidence of multiple regressed hypotheses. The scores of multiple hypotheses could be used to filter out the outliers to refine prediction set.   
The eperiments are mostly conducted on two benchmarks, Human3.6M and MPI-INF-3DHP. Compared with directly averaging all multi-hypothesis predictions, averaging filtered predictions with the proposed score model reduces 1.5mm MPJPE on Human3.6M, while selecting the 3D pose whose 2D projection is more consistent with the 2D pose further reduces 1.5mm MPJPE on Human3.6M.
weaknesses: 1. Experiments.   
Tab. 1 and Tab. 2 show the improvement, compared with CHAMP-Naive, via introducing comformal prediction (CHAMP) and further selecting the 3D pose via measuring the 2D projection and 2D pose (CHAMP-Agg).  However, one important experiment seems to be missing here, CHAMP-Naive + -Agg. How well it performs if the proposed comformal prediction is not used?   
This experiment will clearly show how important the proposed comformal prediction is.   
Besides, the quantative eperiments are only conducted on indoor benchmarks. The qualitative results are relatively limited.  Performance on in-the-wild benchmarks, such as 3DPW, is not clear.   

2. The performance improvement is not significant compared with the baseline SOTAs, which might effect the value of this work.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: Regarding the method, it is unclear how the approach handles situations where the 2D estimates are incorrect or when the 2D pose is truncated beyond the image boundaries.
soundness: 2
strengths: 1. The paper introduces the novel idea of using Conformal Prediction to select more plausible results from multiple hypotheses, which is a fresh perspective.
2. The quantitative experiments demonstrate that the hypotheses selected and aggregated using this strategy outperform the results without filtering.
summary: This paper proposes using a diffusion model to generate multiple hypotheses for temporal 2D to 3D pose estimation and introduces Conformal Prediction (CP) to select more plausible hypotheses and aggregate them. The method achieves competitive results on two pose estimation benchmarks.
weaknesses: 1. The paper's contribution is to be discussed. First, the ambiguity in temporal 2D-to-3D pose estimation is not as significant as in the single-frame, single-view case. Additionally, generating multiple 3D hypotheses using diffusion models is already a widely used technique (DiffuPose (Choi et al., 2023) DiffPose (Gong et al., 2023) DiffPose (Feng et al., 2023) D3DP (Shan et al., 2023)). If the only contribution is the use of CP, its novelty might be limited. If the authors could show that the CP strategy also performs well in single-RGB input cases or demonstrates its effectiveness in single-image mesh estimation, it would significantly enhance the paper's contribution.
2. The quantitative results are not very solid. The improvement on 3DHP is quite marginal, making it unclear whether the improvement is due to random seed variation. These results do not convincingly support the method's effectiveness.
3. The visual results (e.g., Figure 10) do not show cases where severe occlusion leads to incorrect 2D estimates or cases where 3D depth errors occur. It’s important to demonstrate that CP can filter out incorrect hypotheses in such challenging cases, where multi-hypothesis generation is truly necessary. This ties back to the first weakness, as the temporal nature of the data may already provide additional constraints, reducing ambiguity. Therefore, the authors should explore more varied tasks to prove CP's efficacy, such as monocular pose estimation.

Meta-review: This paper addresses the task of 2D-to-3D lifting of temporal human pose sequences through a diffusion-based multi-hypothesis approach. It introduces Conformal Prediction (CP) to select more plausible hypotheses and aggregate them.  The method learns a conformity score function in an adversarial setup and simulates conformal prediction during training, directly minimizing the conformal set size (the inefficiency).  Applying conformal prediction to 3D human pose estimation is an interesting; in particular, in the context of selecting more plausible results from multiple hypotheses.  The proposed uncertainty quantification component will be beneficial to the community. One the other hand, the reviewers raised concerns regarding insufficient validation, unclear explanation of the proposed method, and lacking comprehensive discussion on the positioning and contribution of the presented work against prior works. To demonstrate the generalization capability of the method, evaluation on in-the-wild datasets (where occlusion, truncations exist) such as 3DPW and EMDB is required.  It is also suggested that the effectiveness of the CP strategy in single-RGB input cases should be demonstrated to enhance the contribution. The authors have provided additional experiments to address the concerns on experiments in the rebuttal with arguing in-depth analysis of the results. The authors’ rebuttal and following discussion between the authors and the reviewers have resolved the raised concerns, leading to clear contribution of the paper with convincing validation.  The reviewers are all positive for the paper.  This paper should be accepted, accordingly.

Decision: Accept (Poster)