CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
Authors: Harry Zhang
Luca Carlone
Venue: ICML 2025 poster (2025)
Keywords: 3D Human Pose-Shape Learning with Uncertainty
Primary area: applications->computer_vision

Abstract: We introduce CUPS, a novel method for learning sequence-to-sequence 3D human shapes and poses from RGB videos with uncertainty quantification. To improve on top of prior work, we develop a method to {generate and score multiple hypotheses during training}, effectively integrating uncertainty quantification into the learning process. This process results in a deep uncertainty function that is trained end-to-end with the 3D pose estimator. Post-training, the learned deep uncertainty model is used as the conformity score, which can be used to calibrate a conformal predictor in order to {assess} the quality of the output prediction. Since the data in human pose-shape learning is not fully exchangeable, we also {present} two practical bounds for the coverage gap in conformal prediction, developing theoretical backing for the uncertainty bound of our model. Our results indicate that
by taking advantage of deep uncertainty with conformal prediction, our method achieves state-of-the-art performance across various
metrics and datasets while inheriting the probabilistic guarantees of conformal prediction. Interactive 3D visualization, code, and data will be available at https://sites.google.com/view/champpp.

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
Harry Zhang'
Luca Carlone
Abstract
We introduce CUPS, a novel method for learn- ing sequence-to-squenne 3D human shapes and poses from RGB videos with uncertainty quan- tification. To improve top of prior work, we develop a method to generate and score multi- ple hypotheses during training, effectively inte- grating uncertainty quantification into the learn- ing process. This process results in a deep un- certainty function that is trained end-to-end with the 3D pose estimator. Post-training, the learned deep uncertainty model is used the conformity score, which can be used to calibrate confor- mal predictor in order to assess the quality of the output prediction. Since the data in human pose-shape learning is not fully exchangeable, we also present two practical bounds for the cov- erage gap conformal prediction, developing theoretical backing for the uncertainty bound of model. Our results indicate that by taking advantage of deep uncertainty with conformal prediction, our method achieves state-of-the-art performance across various metrics and datasets while inheriting the guarantees of conformal pre- diction. 3D visualization, code, and data will be available this website.
Input
Single Prediction
Set Prediction
Figure 1: CUPS sample results obtained in-the-wild videos. Given RGB frames, CUPS reconstructs sequence 3D meshes, and then a conformal predictor calibrated using deep uncer- tainty function - trained end-to-end with the human pose-shape estimator quantifies the uncertainty of the output SMPL.
parametrized human models, such as SMPL (Kanazawa et al., 2018), which represent human body shape and pose with well-defined joint and structure parameters, made it possible for modern deep learning models to efficiently learn to predict human poses and shapes in a systematic way by directly regressing SMPL parameters from video inputs. At the same time, safety-critical applications, in- cluding robotics and autonomous vehicles, demand com- puter vision algorithms that are able to quantify the uncer- tainty in their estimates and possibly provide performance guarantees (Yang & Pavone, 2023).
1. Introduction
Recovering sequence of human meshes (i.e., shapes and poses) from monocular video is fundamental chal- lenge with wide-ranging applications in computer vision, robotics, AR/VR, and computer graphics. Such technol- ogy has the potential to minimize reliance on traditional motion capture systems or labor-intensive facilitating the generation of human motion templates for tasks such as animating 3D avatars. The emergence of
annotations,
Massachusetts Institute of Technology. Correspondence
Few existing works in human reconstruction have explored the direction of uncertainty-awarc human pose and shape prediction due to two challenges. First, it difficult to ensure the predicted pose and shape are close to the groundtruth under out-o--dist data heavy occlu-
Harry Zhang <harryz& mit.edu?.
mit.duzmi....u........
Proceedings of the 1and International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
sions. Second, an efficient human shape and pose predic- tion model takes video frames input, where the data is not fully exchangeable. Such non-exchangeability makes it difficult for uncertainty quantification methods such as conformal prediction (Angelopoulos & Bates, ; Shafer & Vovk, 2008) to provide a formal statistical error bound between the estimation and the groundtruth. While a more recent line of work focuses on learning multiple outputs or learning variances as uncertainty (Zhang & Carlone, 2024), none has addressed the problem of providing reliable sta- tistical error bound when the data is not exchangeable. For practical uses in safety-critical scenarios, one should be able to tell when to trust the human reconstruction model.
weighted quantiles to tackle data distribution shifts.
To bring CP into human reconstruction, we design methodology that learns a deep uncertainty score of human reconstruction output in an end-to-end manner by predict- ing multiple hypotheses of human shapes and poses dur- ing training. The deep uncertainty value is then used in the calibration step by incorporating the theoretical toolkit provided in (Barber et al., 2023), retaining statistical guar- antees even when exchangeability is violated. Our results indicate that taking advantage of deep uncertainty with conformal prediction, our method achieves statc-of-the-art performance across various metrics and datasets. Using the probabilistic guarantee of correctness inherited from CP, we also provide theoretical lower bounds of perfor- mance for human mesh reconstruction when the data is not exchangeable. The result is CUPS, Conformalized Uncertainty human Pose-Shape estimator. To sum- marize, our contributions include:
To counter the aforementioned challenges, we first take in- spiration from an important tool in statistical learning, con- formal prediction (CP) (Shafer & Vovk, 2008; Angelopou- los & Bates, 2021), which uses a post-truining calibra- tion step to guarantee a user-specified coverage. Assume that we want to predict an output Y (e.g,, the true pose and shape of a human) from inputs X (e.g., a sequence of frames). By allowing to predict confidence sets C(X) (e.g.. a set of human reconstructions), CP guarantees the true value Y to be included in C(X) with confidence level a, i.e., P(Y C(X)) 2 1 a, given set of calibration examples Y3) E Teat that are exchangeable with the test distribution. There are typically two steps involved in CP. In the calibration step, the conformity scores of the ex- amples in the calibration set are ranked to determine a cut- off threshold Qi-a- via quantile computation. In the pre- diction step, the conformity score measures the conformity between the output and the unknown ground-truth value, which is used-in conjunction with the threshold Qi- to construct the confidence sets C(X). By construction. the set ((X) provides a quantification of uncertainty: each prediction in C(X) has a conformity score above a1 quantile and hence a plausible estimate for Y.
An uncertainty-aware 3D human shape-pose estimator from 2D RGB videos (Section 4.1 & Section 4.2). A novel method to conformalize 3D human estimates during training by learning a score function to rank the uncertainty of the proposed estimates (Section 4.2).
A novel uncertainty quantifier for human reconstruc- tion outputs using non -on-exchange conformal predic- tion and deep uncertainty function (Section 4.3).
A theoretical analysis of the uncertainty when the data is not fully exchangeable and two practical lower bounds one completely new and one adapted from (Barber al., 2023) for the coverage perfor- mance (Section 4.3).
Quantitative and qualitative results that demonstrate the state-of-the-art results of our method on a variety of real-world datasets (Section 5).
2. Related Work
While CP is a flexible tool that can be applied to any ma- chine learning model, it assumes the data to be exchange- able, i.e., that the dataset distribution remains invariant un- der permutation. Such an assumption breaks in many real- world applications. For example, if the dataset comes from video frames, the constructed dataset is obviously not ex- changeable since permuting sequences of frames changes the underlying distribution. While there have been tech- niques that aim to increase the exchangeability of video data by taking long video sequences (Zhang & Carlone, 2024) or observations from evenly-spaced cameras (Yang & Pavone, 2023), the theoretical guarantee of CP cannot be fully justified if the data is not exchangeable. To cope with the lack of exchangeability, we leverage a recent ex- tension by Barber et al. (2023) that allows dealing with datasets where the exchengeability assumption no longer holds. The key idea in Barber et al. (2023) is to use
3D Human Shape and Pose Estimation. End-to-end approaches for human pose estimation include (Pavlakos et al., 2017; Sun et al., 2018). With the maturity of 2D hu- man keypoints detection (Ho et al., 2022; Ma et al., 2022), more robust approaches focus on lifting 2D keypoints to 3D, resulting in better performance (Xu & Takano, 2021; Ma et al., 2021; Ci et al., 2019). In this scheme, determin- istic methods learn to predict one single 3D output from the 2D input (Zhan et al., 2022; Zhang et al., 2022). In many applications, it is desirable to also recover the shape of humans beyond a skeleton of keypoints. (Loper et al., 2023; Kanazawa et al., 2018) propose SMPL, a univer- sal parametrization for human pose and shape. MEVA (Luo et al., 2020) utilizes VAE encode the motion se- quence and generate coarse human mesh sequences which are then refined via residual correction. VIBE (Kocabas
2

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
al., 2020). TCMR (Choi et al., 2021) and MPS-Net (Wei et al., 2022) encode representations of three different input lengths and then learns the mid-frame of the sequence with either a recurrent network or an attention module. GLoT (Shen et al., 2023) is a new model that decouples long- term and short-term correlations. We incorporate human mesh with uncertainty learning during training, forcing the network to output higher-quality meshes.
spectively. We wish to learn a human reconstruction func- tion fo(X) that approximates Y. We are also interested in learning a deep uncertainty function ( Y), which measures the inherent uncertainty of the human reconstruc- tion function So when taking as input X and outputting Y. Such a learned uncertainty function will be used as a con- formity score our method, and enables the construction of a prediction set via conformal prediction.
Uncertainty in 3D Human Reconstruction. Due un- certainty such as occlusion in RGB inputs, deep genera- tive models have been used in modeling conditional dis- tributions for such problems. Mixed-density network (Li
4. Method
We use transformer- architecture (Shen et al., 2023) to predict SMPL parameters from 2D video sequences. We also learn an uncertainty scoring model together with the reconstruction model. At test time, we use the uncertainty scoring model for conformal prediction. This results in CUPS, Conformalized Uncertainty- Shape estimator. The pipeline of CUPS is shown in Fig- ure 2.
& Lee, 2019), VAE (Sharma et al., 2019), normalizing flows (Wehrbein et al., 2021), GAN (Li & Lee, 2019), and Diffusion models (Holmquist & Wandt, 2023; Shan et al., 2023) have all been applied to modeling such conditional distribution. Dwivedi et al. (2024) learn an explicit confi- dence value for occlusions. Motion-based methods such as (Zhang et al., 2024; Rempe et al., 2021) use physi- cal contacts and trajectory consistency to make the uncer- tain estimates more robust. Zhang et al. (2023) use ex- plicit anatomy constraints to improve model performance. Lastly, Biggs et al. (2020) generate fixed number of hy- potheses and learn to choose the best one. We learn un- certainty score by augmenting training outputs and use the uncertainty score to probabilistically certify the outputs.
human Pose-
4.1. GLoT Human Reconstruction Model
The human reconstruction model in CUPS is based on the Global-to-Local Transformer (GLoT) architecture pro- posed by (Shen et al., 2023), which robustly leverages in- formation learned with deep networks as well as human prior structures while decoupling the short-term and long- term modeling processes. We summarize the key compo- nents proposed in (Shen et al., 2023) below.
Conformal Prediction. CP is a powerful and flexible distribution-fie uncertainty quantification technique that can be applied to any machine learning model (Angelopou- los & Bates, 2021; Shafer & Vovk, 2008) under the as- sumption of exchangeable data. Assuming the exchange- ability of the calibration data, CP has desirable coverage guarantees. Thus, it has been applied to many fields such as robotics (Sun et al., 2024), pose estimation (Yang & Pavone, 2023), and image regression (Angelopoulos et al., 2022). More sophisticated CP paradigms have been also proposed to tackle distribution shift and online learning problems (Angelopoulos et al., 2024). More recently, the- oretical grounding for conformal prediction beyond ex- changeability assumption (Barber et al., 2023) has been proposed, which provides analysis tools for non-fully- exchangeable datasets such as videos in ML problems.
Global Motion Modeling. First, a pretrained ResNet-50 extracts features from individual frames, resulting in static tokens referred to as S = {81, ,88 E TRTx2015, The global motion modeling step begins by randomly mask- ing subset of static tokens along the temporal dimension, denoted as S9 E (11-))p'x2ua) where p represents the mask ratio. The unmasked tokens are then passed through global encoder. During the global decoder phase, the mean SMPL parameters encoded by an MLP (SMPL tokens) are padded into the masked positions and the entire sequence is fed into the global decoder, which generates a long-term representation. Subsequently, the global motion modeling step applies an iterative regressor (Kanazawa et al., 201 8; Kocabas et al., 2020) to obtain the global initial SMPL se- quence, denoted by Y {09, J.).
3. Problem Formulation
Local Parameter Correction. A local parameter correc- tion step refines the SMPL parameters outputted by the global modeling step. The local correction step consists of a local transformer and a Hierarchical Spatial Correla- tion Regressor. The local transformer captures short-term local details in neighboring frames: nearby frames' static tokens are selected for short-term modeling, denoted as St = { selected neighborhood. When decoding, cross-attention is
We are interested in the problem of learning sequence of 3D human shapes and poses from a sequence of 2D RGB images. Formally, given the input 2D video sequence X E xxxxXx3XX where H, W are the dimension of each frame and T is the length of the input sequence, our goal to learn to output human shapes and poses, described by SMPL parameters Y (0,3), where 0 E RR21X6TT and ( E I 10xT model the joint 6D pose and mesh shape, re-
++ where w represents the length of the
3

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
Calibration Dataset
Global Modion Modoling
Global Encoder
Global Decoder
Global Regressor
1
Σ Rins
Deep Uncertainty Function Sa
Lecal Poramette Carrection
Glebal- Local Repe.
Co (x
Local Encoder
Local Decoder
Local
Deep Uncertainty Conformal Set
Globa--
Φg1 (X)
Input X
Global-Local Human Reconstructor
Output Y
Figure 2: CUPS Overview. CUPS takes as input sequence of input RGB video frames. The RGB video frames get encoded and fed into a global-looe local transformer human reconstruction model to produce SMPL parameters representing the human pose and shape in 3D as well as a decoupled global-local embedding. The output of the human reconstructor is supervised via SMPL loss. While training, we also learn a deep uncertainty function that learns to rank the uncertainty of the produced output sequence. Then after training, this deep uncertainty function used the conformity score for constructing conformal for conformal prediction.
applied to the query of the mid-frame token and key and value of the global encoder, capturing global human motion consistency and local, fineggrained human mesh structures.
(Shafer & Vovk, 2008), raw logits (Stutz et al., 2021), 1), or hand-designed functions (Yang Pavone, 2023). A more recent line of work has demonstrated the benefits of using learned nonconformity score function (Zhang & Carlone, 2024), where the score is learned in end-to-end manner with the machine learning model. In CUPS, we also learn the score end-to-end with the human reconstruction model, Moreover, as the nonconformity score measures how "un- usual" the datapoint relative to the calibration set, it pro- vides an inherent uncertainty measure for the model: the lower the nonconformity score, the less uncertain the model is about the datapoint, and vice versa.
Global-Local Representation. In the hierarchical spatial correlation regressor step, the model incorporates both the global prediction and decoupled global-local representa- tion into the regressor. The model learns joint correlations within the kinematic structure by modeling the local intra- frame human mesh structure, outputting correction term Y (0', 3 }. where 0' is obtained using an MLP applied on the decoupled global-locat representation 6 (X) and global output 8", and B' using (X) and (3.. The hu- man reconstruction model's final output SMPL values are obtained by adding the initial global prediction and the lo- cal correction output: = (0.3) = Y Y
"0',3')
Formally, we wish to learn function So(X, Y) E [0 as the nonconformity score, which we refer to as the Deep Uncertainty Function.
Definition (Deep Uncertainty Function). The Deep Un- certainty Function takes input the decoupled global- local representation & Φb well as the corrected SMPL parameters Y and oulputs value between O and 1 using an MLP:
Training Objective. We follow previous works and apply standard 1.2 loss the SMPL parameters and 3D/2D joints location (Kanazawa et al., 2018). We also follow the veloc- ity loss on 3D/2D joint location proposed in (Shen et al., 2023) to learn consistency and capture the long-range de- pendency. refer the combined loss as Lg.
(Spred)
[
(1)
where &pred ML (X), 0, the output from a multi- multi-layer-perceproon and the sigmoid function.
4.2. Deep Uncertainty Function
Conformal predictors are calibrated using nonconformity score function (Angelopoulos & Bates, 2021). Intuitively, this function measures to which extent datapoint un- usual relative calibration dataset. The most common nonconformity functions either simple residual terms
Training Time Ensemble Augmentation. To better learn the deep uncertainty function, we augment the model out- put with randomness such that the function learns to rank different samples. We achieve this by utilizing the intrin- sie randomness in the human reconstruction model. Note

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
that the global statie tokens 8 (and thus the global-local representation Φ (X)) is obtained by randomly mask- ing some portion of the input video frames. We augment each training step by randomly masking the video frames for Huain times, effectively simulating Hman hypotheses given the same input data. Thus, for each input video se- quence X, we get multiple hypotheses of SMPL parame- ters prediction: {r 1ET and global-local representation (X)) Mir. The proposed samples are used to train So.
changeability holds (formally defined in Appendix A), the CP calibration is done by choosing threshold using the (I a)th quantile of the conformity scores calculated on
the calibration set. Succinetly, we can define this threshold t on the calibration dataset as follows:
= Ci-a ( dss Eosoxx.r))
(4)
where oa represents the point mass at point a and Q rep- resents the quantile calculation. However, when the cali- bration is no longer exchangeable, such a threshold would not yield the desirable coverage guarantee of standard CP. Barber al. (2023) propose incorporate weighting terms in the CP calibration. Specifically, the new threshold r" is now calculated with prespecified weights w::
Training Objective. The uncertainty function is imple- mented as a discriminator-style scoring function that mea- sures the quality of the generated SMPI. parameters condi- tioned on the input sequence, similar to the discriminator loss in (Kocabas et al., 2020). Thus, a lower score fune- tion output means the output is more likely (more realistic) to be from the ground-truth distribution in the embedding space. Formally, the deep uncertainty function optimizes the following loss:
T = -Q1aa Σ iq - -SSAX..)s)
(5)
Ls- - EE [X, Y =T + E [(1 (X, ))
(2)
where i;; E [0, 1] denotes a prespecified weight placed on data point i. The values of w; is a design choice and it should intuitively be large for data with low non- conformity in the calibration. We discuss some practical design choices for the weights in the sections below.
where Yor is the groundtruth SMPL parameters. Intu- itively, this loss makes sure that samples close to the ground truth get higher conformity and vice versa, encouraging the scoring model to discriminate the prediction from the ground truth. Moreover, we also add an adversarial loss that will be back-propagated into the denoiser model, done in (Kocabas et al., 2020; Zhang & Carlone, 2024; Goodfellow et al., 2020), which encourages the prediction model to output more realistic samples, adversarially con- fusing the discriminator:
SMPL Conformal Prediction. Once the threshold value T" is calibrated, we are able to do conformal prediction. Using T" defined in Equation (5) and the deep uncertainty function, for a datapoint X, we define the Deep Uncer- tainty Conformal Set (DUCS) as the conformal prediction set.
Definition 2 (DUCS). The deep uncertainty conformal prediction set is the set of input-oupuut pairs X,Y Y such that the deep uncertainty value So(X, Y, is below the cal- ibrated threshold value T
Las = E [S (,( Y)
(3)
The overall training loss is sum of SMPL and score loss: Cost La + XLs + fadv), where is a hyperparameter.
-o Y)((
(6)
4.3. Conformal Human Reconstruction
Next we use the deep uncertainty function to quantify the uncertainty of the predicted human SMPL output. We first leverage the theoretical toolkit provided by (Barber et al., 2023), combining it with CUPS' deep uncertainty function, and then build on top of the coverage guarantee for non- exchangeable conformal prediction in (Barber et al., 2023) and provide two practical error bounds based on the char- acteristic of the video dataset and the design choice of the CUPS architecture.
For test video sequence X and predicted SMPL parame- ters Y, it is straightforward to check its set membership in DUCS. More importantly, as we will show in Section 5.3, we can explicitly make a set prediction by using Monte Carlo Dropout during test time.
Under the framework of nonexchangeable conformal pre- diction (Barber al., 2023), we analyze some theoreti- cal properties of DUCS, which are amenable to our hu- man reconstruction pipeline. We first define the tuple Z, = (X, Y)), which denotes the i-th example in our cal- ibration dataset. We then construct the following sequence by combining the tuples Z;: = (Z,, Zz,... Zn) and define
SMPL Conformal Calibration. We introduce the calibra- tion step post-training by using the score in Definition 1. For any prediction Y from the human reconstruction model and its corresponding input video sequence X, SX,, Y) measures its "similarity" to an existing dataset- - the non- conformity score for conformal calibration. When ex-
Z' =Zi,... Z 1, Zn, ... Zn-1, Zi), (7)
5

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
which represents Z sequence after swapping the last point with the i-th calibration datapoint. Now, we define the weights W; needed for calibration (cf. (5)). As mentioned above, the exact weight formulation is a design choice, but to facilitate our analysis, we choose the following design based on the Euclidean distance of learned features.
where p is a decay hyperparameter and π ((w)) maps w, to its ranked position E [n]] among all weights. Then the normalized weights are W; = W. ",,"; Assuming that the most recent chungepoint the video datasel occurred time steps ago -such that DTv DTv(Zi (Z, || Zn, 0 for i > - k and could be arbitrarity large otherwise we have the following bound:
Definition 3 (Feature Distance Weight), For our SMPL conformal calibration, the weight is defined based on the feature distance between the predicted SMPI. S and the ground SMPL feature:
E Drv (So(Z) || So(Z)) s spp:.
(11)
I1o
11
W:: exp
(8)
This bound suggests that the miscoverage gap remains small as long as k is large.
The second bound is novel and models the deep uncertainty output on the calibration set S Z) using beta distribu- tion. In practice, this is achieved at test time using Monte Carlo Dropout (Gal & Ghahramani, 2016). The bound depends on the beta distributions formed by So(Z) and (z)))
where T is the temperature hyperparameter, Dpred
(L (Xx,) (X 3,) (.) is the predicted embedding in Def-
inition 1 and o.5T ground-truth embedding.
(X:)) " is the
Then the quantile weights w; is the normalized version of wy. The exact normalization technique is again design choice that we describe below in Equation (11).
Theorem (Miscoverage under Beta Distribution). As- sume the deep uncertainty values of the calibration set of size follow Beta distributions: Sa(Z)
3(a1, n (1), So(Z') ~ B(u2, 02). If we assume that the difference between parameters an and as is bounded by k. we gel the following bound without any assumption on the weights:
We now formally state the coverage guarantee of DUCS using Theorem 2 in (Barber et al., 2023). Without assum- ing exchangcability, DUCS is designed to be robust against distribution shifts.
Theorem 1 (Nonexchangeable Coverage (Barber et al., 2023, Thm. 2))- Under possibly noheexchaneeable dataset distribution, the conformal prediction set defined in Defini- tion? 2 satisfies the following coverage guarantee:
2k Σ w. DIv (So(Z) S $12- 20-2(1--
212-2
k.
(12)
This bound is stronger when is smaller. Intuitively, the two distributions formed by swapping are similar then the miscoverage gap will be small. We refer the reader to Appendix for the proof of Theorem 2 and 3.
P(Y E (6(
w DTv (5 So(Z))
(9)
where W; is the normalized weight obtained via Defini- tion? 3, DIv (- -I-) represents the total variation distance, So ( So (Z.)) and similarly for SolZ))
5. Experiments
We provide a proof in Appendix B. following the out- line in (Barber et al., 2023). The extra term Σ. w; DTv (Z)(Z) || 5 (Z)) referred to as the miscoverage gap. Next, we present two practical bounds for the miscov- erage gap by leveraging the structure of the datasets and architectures we use in our mesh estimation problems.
We provide a quantitative evaluation of our model against sfate-of-the-art baselines. We also provide ablation stud- ies and qualitative results to support our design choices. We follow previous baselines (Shen al., 2023; Kanazawa et al., 2018; 2019; Dwivedi al., 2024; Choi et al., 2021) and report several intra-frame metries, including Mean Per Joint Position Error (MPJPE). Procrustes-aligned MPJPE (PA-MPJPE), and Mean Per Vertex Position Error (MPVPE). Following (Shen et al., 2023; Choi et al., 2021), we also provide a result for the second-order accelcration error (Accel) for the inter-frame smoothness.
The first bound is borrowed from (Barber et al., 2023). We leverage the video dataset characteristics, assuming the dis- tribution shift happens periodically. Weights are designed to account for this periodic change.
Theorem 2 (Miscoverage under Periodic Change (Barber al., 2023, 4.4)). Using w; in Definition 3, we define the auxiliary weight w/;
5.1. Baselines Comparisons
We follow the same dataset split and setup as done in previous works and evaluated on 3DPW (Von Marcard
w/ = p" m+w-s)(w)),
(10)
6

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
3DPW
(Von Marcard et al,, 2018)
MPLINF- (Wei et al., 2022)
Human3.6M (lonescu al. 2013)
Methud
PA APIIPE 1
MPIPE + MPVPE &
Accel
PA-MPIPE 1 MPIPE 1 Accel
PA-MPIPE 1
MPIPE 5
Accel
Frames
VIBE (Kocabas et al, 2020)
57.6
91.9
25.4
68.9
103.9
27.3
53.3
78.0
27.3
16
MEVA (Lao et 2020)
54.7
86.9
11.6
65.4
96.4
11.1
53.2
76.0
15.3
TCMR (Choi et al., 2021)
52.7
86.5
102.9
7.1
63.5
97.3
8.5
52.0
73.6
3.9
16
MPS-Net (Wei al., 2022)
52.1
84.3
99.7
7.4
62.8
96.7
9.6
47.4
69.4
3.6
POCO (Dwivedi et al., 2024)
50.5
80.5
96.5
6.7
62.1
93.7
8.1
46.4
68.
3.5
GLoT (Sben all, 2023)
50.6
80.7
96.3
6.6
61.5
93.9
7.9
46.3
67.0
3.6
16
CUPS (Ours)
48.7
76.2
91.7
6.9
61.3
92.8
7.2
44.0
6338
3.5
16
Table I:
Multiple errors (L) results on 3DPW. MPI-INF-DDHP, and Human3.66 6.M.....M..... All methods use 3DPW training set for training, Comparisons show that CUPS outperforms other baseline methods in the vast majority of metries.
et al., 2018), Human3.6N 3.6 (Lonescu et al., 2013), and MPII- 3DHP (Mehta et al., 2017). More details on the construc- tion of training dataset are Appendix D. As shown in Table 1, our model outperforms state-of-the-ari baseline methods. On 3DPW, for example, we outperform GLoT's PA-MPJPE by 1.9mm, MPJPE by 4.5mm, and MPVPE by 4.6mm. While Accel performance was slightly worse off on 3DPW, on the other two datasets, our method sur- passes baselines on all metrics. Moreover, our method out- performs GLoT by a noticeable margin, indicating that the deep uncertainty function So important in that forces the human reconstruction output higher-quality samples during training. In Table 2. following previous works (Choi et al., 2021; Dwivedi et al., 2024; Shen et al., 2023), we measure our method's generalizability to unseen datasets. In this set of experiments, none of the methods uses 3DPW dataset during training. Again, our method outperforms baselines by a noticeable margin.
Choice of Conformity Score Function, We compare our proposed deep uncertainty function (DUF) trained using adversarial loss with several different losses in Figure 4: score function augmented with inefficiency loss (Ineff.) (Stutz et al., 2021) during training and classifier-style conformity score function (Class.). From Figure 4, we see that DUF and ceeffiiinncy-uugeentee DUF result in similar performance quantitatively. While the classifier-style loss performs better than without any scoring function quantita- tively, the predicted mesh shape on the videos is less real- istic. Please refer to Appendix E for more details.
Strength of the Uncertainty Loss. Finally, we ablate the hyperparameter of the training loss for Sg (X, Y), X, in the overall training objective Lu This is an important abla- tion in that we can find a suitable scale of the loss for the deep uncertainty function to make sure it does not conffict with the pose loss optimization. Results in Figure suggest that 0.6 the most efficient strength across all values, as a smaller scale does not train the scoring model sufficiently and a higher scale conflicts with the pose loss.
3DPW MPIPE MPYPE 1 Accel
Method
PA-MPIPE J
HMR (Kamuaawa et ol., 2018)
76.7
130.0
37,4
3DMB (Riges al,, 2020)
74.9
1211.8
5.3. Monte Carlo Dropout
SPIN (Kolotouras ol., 2019)
59.2
116.4
29.8
VIBE (Kocalas al., 20200
HMMR (Kunuzawa al., 2019)
72.6
56.5
116.5
93.5
139.3
113.4
27.1
15.2
One interesting byproduct of learning the deep uncertainty function S,(X, (Y) is that we construct the DUCS Co(X) explicitly by sampling the output SMPL parame- ters multiple times, just like during training time ensem- ble augmentation. While the model itself is not exactly probabilistic, we can emulate its stochasticity during infer- ence time with Monte Carlo Dropout which lends itself to modeling the model uncertainty in a Bayesian way (Gal & Ghahramani, 2016). This procedure effectively enables us to make multi-hypothesti predictions during test time, and by checking the set membership of each hypothesis, we are able to explicitly construct the DUCS (x(X) with mini- mal changes to the model. Prediction sets from using MC Dropout are shown in Figure
TCMR (Choi 2021)
55.8
95.0
111.5
7.0
(Wei 2022)
54.0
POCD (Dwivedl all, 2014)
54.7
89.3
108.4
GLoT (Shen et all, 2023)
53.5
89.9
107.8
6.7
CUPS (Ours)
33.0
85.7
103.6
Table Multiple errors (L) results on 3DPW. None of the meth- ods use 3DPW for training. CUPS outperforms all baselines.
5.2. Ablation Studies
Training Time Ensemble Augmentation. We augment the training dataset online to better train the deep un- certainty function by leveraging the intrinsic stochasticity (e.g., frame masking) of the human reconstruction model. We compare results as function of the number of sam- ples (H). When H = 1, there is augmentation and we are just running the forward pass once. We train several models using a range of H values and evaluate the MPJPE using these models in Figure 3. Results suggest that using more proposed samples during training reduces test error overall and the improvement saturates after 30.
5.4. In-the- Wild Videos
To test the generalizability of our method to in-the-wild videos, we collect videos from YouTube and TikTok. We directly apply the CUPS model trained on the 3DPW
7

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
SDPW МPЛРЕ HH Training Ensembble
MPJPE Uncertainty Strength
78.5
77.5
MPPI:
ME3PL
77.0
DUF
Ineff.
Class.
76.5
Number of Proposed Samples
10
20
30
MPJPE
76.2
76.2
77.4
Scale of Deep Uncertainey Timction Loss
0.2
Figure 3: Comparison of nr. of samples proposed during training time ensemble.
Figure 4: Conformity scores choices on 3DPW (bottom) and internet videos (top).
Figure 5: Comparison of strength of certainty loss the total training loss.
88% + 3% for all three datasets, which remains close a, and in some cases, it exceeds this value. Weighted CP coverage is noticeably higher than unweighted CP, cor- roborating the results in (Barber et al., 2023). The cover- age result with weighted CP is encouraging because it illus- trates that the miscoverage gap is small all three datasets.
3DPW
ЗDHP
H3.6M
Weighted CP
86.2 2.1%
87.3 + 2.2%
89.0 + 1.5%
Theorem 2 Bound
2 83.9%
2 84.9%
Z 85.8%
Theorem 3 Bound
2 84.0%
> 85.3%
86.8%
Regular CP
81.0 3.4%
83.2 +: 2.8%
85.2 4 2.3%
Table 3: Empirical coverage with weighted vs. regular calibra- tion using the learned deep uncertainty function on three different datasets. Bounds are obtained as descrihed in Appendix B.
5.6. Implementation and Training Details
The reconstruction model takes as input video sequences of length 16, following (Shen et al., 2023). We use an Adam optimizer with a weight decay of 0.1 and momentum of 0.9. The adversarial loss weight is 0.6 and is optimized ev- ery 100 iterations. Our model trained using NVIDIA V100 GPU, where training consumes amortized GPU memory of 20GB, and CPU memory of 160 GB. We train the model for 100 epochs with an initial learning rate of 5e-5 with a cosine scheduler. The ensemble augmentation step produces 20 samples for the same input datapoint.
Input
Single Prediction
MC Prediction Set
Figure 6: In the wild video SMPL predictions with both single hypothesis and multiple hypotheses using MC Dropout.
dataset test on in-the-wild videos. We run both regular CUPS and CUPS with MC Dropout for multiple hypothe- ses results are shown in Figure 6, where the input videos are collected from TikTok. For 3D visualization, please refer to this anonymized website to interact with CUPS predictions in 3D.
6. Conclusion
5.5. Empirical Coverage
We presented CUPS, approach for learning sequence- to-sequence 3D human shapes and poses from RGB videos with uncertainty quantification. Our method uses deep uncertainty function that trained end-to-end with the 3D pose-shape estimator. The deep uncertainty function com- putes a conformity score, enabling the calibration of con- formal predictor to assess the quality of output predictions at inference time. We present two practical bounds for the miscoverage gap in CP, providing theoretical backing for the uncertainty quantification of our model. Our re- sults demonstrate that CUPS achieves state-of-the-an per- formance across various metrics and datasets, while inher- iting the probabilistic guarantees of conformal prediction.
Here we test the empirical coverage of the deep uncertainty function using the three testing datasets in Table 1. Mathe- matically, we calculate the following value:
C
Σ
1 (Y SyoreCCol cCo())
(13)
Test
and compare against the desired coverage value 1-a. We use = 1 for CP, calibrating with the 90th quantile. Fur- thermore, we compare the performance with weighted cal- ibration using weights defined Definition against un- weighted calibration as done in regular CP. From Table 3, we see the empirical coverage for weighted CP is around
8

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
Acknowledgements
Vision and Puttern Recognition (CVPR), pp. 1964-1973, 2021. 3, 6, 17
This work was partially funded by Amazon Robotics under the "Safe Autonomy Leveraging Intelligent Optimization" program, by the ARL DCIST program, and by the ONR
Ci, H., Wang, C., Ma, X., and Wang, Y. Optimizing net- work structure for 3d human pose estimation. In Intl. Conf. on Computer Vision (ICCn), pp. 2262-2271, 2019.
RAPID program.
2
Impact Statement
Dwivedi, S. K., Schmid, C.. Yi, H., Black, M. J., and Tzionas, D. Poco: 3d pose and shape estimation with confidence. In 2024 International Conference 3D Vi- sion (3DV), pp. 85-95. IEEE, 2024. 3, 6,7
This paper presents a deep-learning system on improving human mesh recovery by utilizing deep uncertainty func- tion and theoretical guarantees. can help quantity the un- certainty of deep learning models when adopted in practice. In addition, framework is fully modular, and it aims to advance research of architecture innovation and promote applications of UQ.
Gal, and Ghahramani, Z. Dropout as a bayesian approx- imation: Representing model uncertainty in deep learn- ing. In Inil. Conf. on Machine Learning (ICML), pp.
1050-1059. PMLR, 2016. 6,7
References
Goodfellow, L., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63(1 13:11391144, 2020.
Andriluka, M., Iqbal, U., Insafutdinov, E., Pishchulin, L.,
Milan, A., Gall, J., and Schiele, B. Posetrack: A benchmark for human pose estimation and tracking. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 5167-5176, 2018. 17
Harrison, M. T. Conservative hypothesis tests and confi- dence intervals using importance sampling. Biometrika, 99(1):57-69, 2012. 12
Angelopoulos, A., Candes, E., and Tibshirani, R. J. Con- formal pid control for time series prediction. Conf. Neurul Information Processing Systems (NeurIPS), 36, 2024. 3
Ho, J., Saharia, C., Chan, W., Fleet, D. Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. J. of Machine Learning Research, 23 (47):1-33, 2022.
Angelopoulos, A. N. and Bates, S. A gentle introduction to conformal prediction and distribution-Fae free uncertainty quantification. arXiv preprint arXiv:2107.075/1 2021. 2,3,4 4
Holmquist, K. and Wandt, B. Diffpose: Multi-hypothesis human pose estimation using diffusion models. In Intl. Conf. on Computer Vision (ICCV), pp. 15977-15987, 2023. 3
Angelopoulos, A. N., Kohli, P., Bates, S., Jordan, M., Malik, J., Alshaabi, T., Upadhyayula, S., and Romano, Y. Image-to-iman regression with distribution-fiee free un- certainty quantification and applications in imaging. In Intl. Conf. on Machine Learning (ICML), pp. 717-730. PMLR, 2022. 3
Ionescu, C., Papava, D., Olaru, V., and Sminchisescu, C. Human3. 6m: Large scale datasets and predictive meth- ods for 3d human sensing in natural environments. IEEE Trans. Pattern Anal. Machine Intell,, 36(7):1325-1339) 2013. 17
Barber, R. F., Candes, F. J., Ramdas, A., and Tibshirani, R. J. Conformal prediction beyond exchangeability. The Annals of Statisties, 51(2):816 845, 2023. 2, 3, 5, 8,
Kanazawa, A., Black, M. J., Jacobs, D. W., and Malik, J. End-to-end recovery of human shape and pose. In IEEE Conf. on Computer Vision and Pattern Recogni- tion (CVPR), pp. 7122-7131, 2018. 1, 3, 4, 6,7
12, 13, 14
Biggs, B., Novotny, D., Ehrhardt, S., Joo, H., Graham, B., and Vedaldi, A. 3d multi-bodies: Fitting sets of plausible 3d human models to ambiguous image data. Conf. Neural Information Processing Systems (NeurIPS), 33: 20496-20507, 2020. 3,
Kanazawa, A., Zhang, J. Y., Felsen, P., and Malik, J. Learn- ing 3d human dynamics from video. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 5614-5623, 2019. 6, 7, 17
Kocabas, M., Athanasiou, N., and Black, M. J. Vibe: Video inference for human body pose and shape estimation. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 5253-5263, 5263, 2020. 2, 3, 5, 5,7,
Choi, H., Moon, G., Chang, J. Y., and Lee, K. M. Beyond static features for temporally consistent 3d human pose and shape from a video. In IEEE Conf. on Computer

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
Kolotouros, N., Pavlakos, G., Black, M. J., and Daniilidis, K. Learning reconstruct 3d human pose and shape via model-fitting in the loop. In Intl. Conf. on Computer Vision (ICCV), pp. 2252 2221, 2019. 7
Sharma, S., Varigonda, P. T., Bindal, P., Sharma, A., and Jain, A. Monocular 3d human pose estimation by gen- eration and ordinal ranking. In Intl. Conf. on Computer Vision (ICCV), Pp. 2325-2334, 2019. 3
Li, C. and Lee, G. H. Generating multiple hypotheses for 3d human pose estimation with mixture density network. In IEEE Conf. on Computer Vision and Pattern Recogni- tion (CVPR), pp. 9887-9895, 2019.
Shen, X., Yang, Z., Wang, X., Ma, J., Zhou, C., and Yang. Global-to-local modeling for video-based 3d human pose and shape estimation. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 8887-8896, 2023. 3, 4, 6, 7, 8. 17
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., and Black, M. J. Smpl: A skimned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. pp. 851-866. 2023. 2
Stutz, D., Cemgil, A. T., Doucet, A., et al. Learning optimal conformal classifiers. Intl. Conf. on Learning Represen- tations (ICLR), 2021. 4, 7, 18
Sun, J., Jiang, Y., Qiu, J., Nobel, P., Kochenderfer, M. J., and Schwager, M. Conformal prediction for uncertainty- aware planning with diffusion dynamics model. Conf. on Neural Information Processing Systems (NeurlPS), 36, 2024. 3
Luo, Z., Golestanch, S. A., and Kitani, K. M. 3d human motion estimation via motion compression and refine- ment. In Asian Conf. Computer Viston (ACCV), 2020. 7
Ma, X., Su, J., Wang, C., Ci, H., and Wang, Y. Context modeling in 3d human pose estimation: A unified per- spective. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 6238 0248-662,2200 2021. 2
Sun, X., Xiao, B., Wei, F., Liang, S., and Wei, Y. Integral human pose regression. In European Conf. on Computer Vision (ECCV), pp. 529-545, 2018. 2
Von Marcard, T., Henschel, R., Black, M. J., Rosenhahn, B., and Pons-Moll, G. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference on computer vi- sion (ECCV), pp. 601-617, 2018. 6, 7, 17
Ma, X., Rahmani, H., Fan, Z., Yang. B., Chen, J., and Liu, J. Remote: Reinforced motion transformation network for semi-supervised 2d pose estimation in videos. Nat. Conf. on Artificial Intelligence (AAAI), volume 36, pp. 1952, 2022. 2
Wehrbein, T., Rudolph, M., Rosenhahn, B., and Wandt, B. Probabilistic monocular 3d human pose estimation with normalizing flows In Int. Conf. on Computer Vision (ICCV), pp. 1 11199-11208, 2021.
Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., and Theobalt, C. Monocular 3d human pose estimation in the wild using improved enn supervision. In 2017 international conference on 3D vision (3DV), pp. 506-516. IEEE, 2017. 7, 17
Wei, W.-L., Lin, J.-C., Liu, T.-L., and Liao, H.-Y. M. Cap- turing humans in motion: Cemporal-attentivs 3d human pose and shape estimation from monocular video. In IEEE Conf. on Computer Vision and Pattern Recogni- tion (CVPR), pp. 13211-13220, 2022. 3,7
Pavlakos, G., Zhou, X., Derpanis, K. G., and Daniilidis, K. Coarse-to-fine volumetric prediction for moagle-image 3d human pose. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 7025-7034, 2017. 2
Xu, and Takano, W. Graph stacked hourglass net- works for 3d human pose estimation. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 16105-16114 14, 2021. 2
Rempe, D., Birdal, T., Hertzmann, A., Yang, J., Sridhar, S., and Guibas, L. J. Humor: 3d human motion model for robust pose estimation. In IEEE Conf. on Computer Vi-
sion and Pattern Recognition (CVPR), pp. 11488-11499, 2021. 3
Yang, H. and Pavone, M. Object pose estimation with sta- tistical guarantees: Conformal keypoint detection and geometric uncertainty propagation. In IEEE Conf. on Computer Vision and Patlern Recognition (CVPR), pp. 8947-8958, 2023.
Shafer, G. and Vovk, V. A tutorial on conformal prediction. J. of Machine Learning Research, 9(3), 2008 2, 3,4
Shan, W., Liu, Z., Zhang, X., Wang, Z., Han, K., Wang. S., Ma, S., and Gao, W. Diffuusionbaase based 3d human pose estimation with multi- hypti-hyppotesis aggregation. In IEEE Conf. on Computer Vision and Pattern Recogni- tion (CVPR), pp. 14761-14771, 2023.
Zhan, Y., Li, F., Weng, R., and Choi, W. Ray3d: ray-based 3d human pose estimation for monocular absolute 3d lo- calization. In IEEE Conf. on Computer Vision and Pat- tern Recognition ( (CVPR), pp. 13116-13125, 2022.
10

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
Zhang, H. and Carlone, L. CHAMP: Conformalized 3D human multi-hypothesis pose estimators. arXiv preprint:
2407.06141, 2024. 2, 4,
Zhang, J., Tu, Z., Yang. J., Chen, Y., and Yuan, Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video. In IEEE Conf. Computer Vi- sion and Pattern Recognition (CVPR), pp. 13232-13242, 2022. 2
Zhang, S., Bhatnagar, B. L., Xu, Y., Winkler, A., Kadle- cek, P., Tang, S., and Bogo, F. Rohm: Robust human motion reconstruction via diffusion. In IEEE Conf. Computer Vision and Pattern Recognition (CVPR), PP. 14606-14617, 2024. 3
Zhang, W., Zhu, M., and Derpanis, K. G. From actemes action: supervvie representation for detailed action understanding. In Insl. Conf. on Computer Vision (ICCV), pp. 2248-2255, 2013. 17
Zhang, Y., Wang, H., Kephart, O., and Ji, Q. Body knowledge and uncertainty modeling for monocular 3d human body reconstruction. In JEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 9020-9032,
2023. 3

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
Supplementary Material
A. Exchangeable Distributions
First, we define exchangeability in probabilistic distribution:
Definition 4 (Exchangeability in Probabilistic Distribution). A sequence of random variables X2.X....., Xn is said to have an exchangeable distribution the joint distribution of X1, X2,X...... Xn Formally, for any permutation π. of (1,2.....7)
,X is invariant under any permutation of indices.
P(X1 = 21, X2 = 32 ... Xn = =xn)= =P(X=(1) 21, Xa(2) -2,..... Xa(n) Inn)
for all possible values 2 ,2,..., In X1, X2....,, Xn-
B. Proof of Coverage Guarantee without Exchangeability
We first recall the following definition of Deep Uncertainty Conformal Set:
Definition 2 (DUCS). The deep uncertainty conformal prediction set is the set of input-output pairs X, Y such that the deep uncertainty value S,(X,Y Y is below the calibrated threshold value T
Ca(X) (Y: Str
(6)
We further define tuple Z, (X,, Y)) and
Z (Z),... Zn+1),
well
Z Zz, 2,-1, Z n+1. Zn Ze),
which represents Z sequence after swapping the test point with the i-th calibration point.
Lemma (Weight sum upper bound (Harrison, 2012, Lemma 3)). For all a, w1. wn E oo). and all t1, ,t+1 E [-o0, ool, we have:
n+1
n+1
Σ
wgl
Σ
w (t 2 tx) a sa
Proof. We follow the sketch proof in (Harrison, 2012) and provide detailed proof for interested readers. The tuples
(li, w:) can be permuted without affecting the value of the LHS, we can assume that ti 2 > bn- This implies that
Entt we (1: 2 /x) is increasing k. There exists k* defined as follows:
bst sup &
we 11, 2 ()) S
1=1
Thus, we have:
w
w;1(t 2 (x) <a
w; 2
(x-)
S
(14)
Next, recall the nonexchangeable conformal prediction coverage guarantee:
Theorem (Nonexchangeablle Coverage (Barber et al., 2023, Thm. 2)). Under possibly non-exchangeable dataset distri-
bution, the conformal prediction set defined in Definition 2 satisfies the following coverage guarantee:
P(Y E C 21 21-00
Σ
w. Drv (1) So(2))
(9)
where W. is the normalized weight obtained via Definition 3, Drv (: II) represents the total variation distance, So(Z) = [So Z and similarly for So 2
12

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
Proof. We follow the sketch proof in (Barber et al., 2023) and provide detailed proof for interested readers. We look the complement of the above probability P (Yn+) E Co (X (++))). For simplicity, first define S So(Xi, Y.). By definition of DUCS, we have:
n & Co(Xn+1) + Su+1 > Q1-a
Σ
WZ ós.
(15)
+ 1 Σ Wi òs, Wn+1 8400
Then define an unusual ser function 14:
}
u(S) i [n [1
Q a
Σ ie òs.
(16)
which represents the indices where the deep nonconformity score values are too large. Then know that noncoverage of Y+11 implics the unusualness of point k:
n+1 $ Co ( - - E((S(((())
(17)
Thus, we have:
n+1
P(k E U(Sa(Z
P (k - (x==)iE((((((())
P (((ie(()((Z'))
E DrV (So(Z) (z)))
(18)
E
wi
Drv (So(Z) (z
H 22))
< a +
w Div (So(Z) || So(Z))
[by Lemma 4]
By complement, P (Y,,11 E (x(xn
2 a Σ
w Drv (Sa(Z) Sz(Z)))
C. Upper Bounds of Coverage Gap
We provide two possible bounds for the coverage gap. The first bound comes from an example usage in Barber al. (2023) and the second bound is derived by assuming the outputs follow beta distribution.
C.1. Bounded Periodic Changes
We are interested in finding an upper bound of the coverage gap in Theorem Specifically, we are trying to bound Drv (2x(2) /) || So (Z))). One interesting case analyze for video data is that we might have periodic large changes in the distribution rather than gradual drift (i.e. there might be changepoint in the calibration set).
Assumption 1 (k-step changepoints in dataset). Suppose that the most recent changepoint occurred k steps ago, so that Drv (So Za+1) (Z)) + for > n k. However, we might have an arbitrarily large difference from the new test point before the changepoint step: Drv (S Zn41) (z)) + for i & n k.
Now, we try to bound the coverage gap from Theorem 1. We first design the weights amenable to our analysis. Recall the definition of weights:
13

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
Definition 3 (Feature Distance Weight), For our SMPL conformal calibration, the weight is defined based on the feature distance between the predicted SMPP.feature and the ground-truth SMPL.feature:
11 110p Prod
W/ exp
(8)
T
where T is the temperature hyperparameter, of MLP( yt (x.) is the predicted embedding in Definition I and of GT
0.,
00 ,3(5)) is the ground-truth embedding.
Again, recall the miscoverage gap under bounded period changes:
Theorem 2 (Miscoverage under Periodic Change (Barber et al., 2023, 4.4)). Using wi in Definition 3, define the auxiliary weight w
w = -==11=((xc))
(10)
where is a decay hyperparameter and (w:) maps WN its ranked position E [n]] among all weights. Then the normalized mi
weights are W Σ, wi Assuming that the most recent changepoint the video dataset occurred k time steps ago such
that DTv Z || nn) for > n k and could be arbitrarily large otherwise- we have the following bound:
a w Drv (So())) (z ( pp
(11)
Proof. We follow the sketch proof in (Barber et al., 2023) and provide detailed proof for interested readers. From Assump- tion suppose the variation before the changepoint could be arbitrarily large:
n
D.D (So(Z) || Sz
-
Wi
カー #1--
1
i= patl- (())
(19)
p n
D"
p
Intuitively, this tells us that the coverage gap will be small as long as k is large namely, as long as we have enough data after the changepoint.
Remark (Measuring P and k). As explained in Barber et al. (2023), is a decay parameter less than and the above miscoverage gap is small as long as k is sufficiently large. To measure k. for each of the three test datasets in Table 3, we measure the average number of video sequences between the two closest sequence datapoints belonging to different subjectssacttiitties.
C.2. Bounded Covariates
Next, we are trying to bound Drv (2x || So())) under distributional modeling. To find this bound, we make use of the Hellinger distance. The Hellinger distance (2(P, Q) between two probability measure and Q on a measure space X' with respect to an auxiliary measure (e.g. joint) defined as:
1 H²)PQQ)= = / (V((a)-V(x1))
V A(dx),
where P(dx) ((x) and Q(dx) ((2x))(())))))))))(()))))))))( Succinctly, we denote:
1 H(P
-
V2
14

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
turns that we can make of the Hellinger distance bound the total variation distance.
Drv (P Q) (P,().Q)
(20)
Proof. This is fairly well-known results in statistics and we provide a detailed proof for the sake of completeness.
Div (P Q)
Σ pi ail?)
Σ (vvei voD(VVi V
4
Σ(ve: (WVei-va)') [iivR vaa))
$
(21)
$ #P Q) (2+2 VVivv
< (P HP, Q)) S (PH)
Hence, Drv (P Q) S (2H(P.Q)
We can now focus on providing an upper-bound Hellinger distance instead. Since our conformity score is outputted by sigmoid function via Monte Carlo Dropout during test time, one reasonable assumption that the conformity score outputs follow Beta distribution. Assumption (Conformity scores follow Beta distribution).
Sz B(a,, n -1), So ~ B(a2, n -22)
where aj and 02 are defined by the permuted dataset. Without loss of generality, we then assume:
a1 S 02 S 01
k
That is, assume the difference in the Beta parameters is bounded.
The assumption makes intuitive sense in that it can be thought of as the proportion of calibration data points that are close point. Furthermore, the expected proportion change between the original and permuted dataset essentially assuming that after swapping one pair of data
(conform to) the is bounded by k. This assumption makes sense in that points, the change in proportion of data that conform to the test data is bounded.
With the Beta distributions defined, recall the bound to prove:
Theorem 3 (Miscoverage under Beta Distribution). Assume the deep uncertainty values of the calibration set of size follow Beta distributions: So(Z) ~ 3(a1, 01), So 3(az, n. - -a2). If we assume that the difference between purameters on and 02 is bounded by k, we get the following bound without any assumplion on the weights:
2 2
Σ Dwv (So(Z) || So(Z)) S 2-
(12)
Proof. First, we can express the Hellinger distance between two Beta-distributed measures in closed form:
B ( 2 n 21ta2
H²(S((2) -
(22)
B(an B1) 02
where B(m, n) is the beta function defined as:
[(m)[(n)
B(m,n) n)
T(m + +n)
(23)
(m 1)!(n 1)!
m2n m
(m + 1)!
15

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
We define short-hand notation a 22 and then by the definition of beta function, the numerator of Eq. 22 becomes:
2
+
01 + 02
(a - 1) - 1 a)!
B
(24)
2
(n- 1)!
Similarly, in the denominator:
B(ai, n -1)) = (a) 1)1(n -1 a1)!
(a2 1)!(n 02)!
Baa2, u2)
(25)
(n - 1)!
(n 1)!
Combining the expressions above and plugging them into Eq. 22, we have:
B 01 a1
(((5s()))5((2'') VB(01, 01 (B2,, 02)
10-1)(n-1-a))
(n-
(26)
[) 1-0
(ag 1)(n-1-00)
(n-1)) 1]]
(z-1)]
(a - 1).(n a)!
V(a 1)!(22 - 1)lnn - a1 (I a2)
With Assumption 2 in place, we are able to bound the fraction in Eq. 26 as follows:
(a 1))(n 1 a)! V(a 1))(02 1)1(n -1))(nn(n az)!
(()) on
(aa))[(a)T(( [n) 02)
I(a + (1 -
01 5) k)
[by bounded difference between aj and 02]
[(a)
+ (m)
[by definition of T function]
(a1 + k) (n a1) _ (n 01 1:)
01 (n 01 k)
[rearranging the terms]
IV
(a1 + k) ( 01)
Q1n of ank
[expanding the terms]
01n of 01 kn
kn
(27)
01n
of
aik hi + kn
kn
[denominator extremum at 01
n-k 2
( k
+ kn
k))2
IV
n k)
+ 4kmn
π k
n + h:
(a- 1)!(n - a)!
2k
Hence, (a1 1)l(a2 1)1(n 11-ai)n(n
02)
n2. + 1
Putting all together, we know that
2k
- (2) 2
1:
16

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
and equivalently,
2k k:
(2) S
Hence, we have the final upper bound on the coverage gap via Lemma 20:
2k
Drv (So(Z) Sz)) & 1/2
(28)
n + k
Now, we analyze the behavior of this bound. We rewrite the upper bound as follows:
(1
2k
v2. V 11-(6
+ k
We first define p k Then we have:
2k
n1 - k + K
1 --1 P
+ k
1
By Taylor series, we have the following series expansion:
In p
p2 pd
p² 2
p²
In - - In(1 + p)
=
p
+ + t...
1+p
(29)
2
=
3
p
+
Then we have the following behavior:
2k
p
k
p
exp
In
+ k
+ p
2
+ P
k
2
)...)
exp
2p
3 p
2
(30)
k
exp
kp
kp
2
exp
Thus, we have:
Drv (So(2) So( < 12 2 (1
2k: R
12
0
-exp exp
(31)
This behavior indicates that the bounds gets weaker exponentially with larger k.
Remark (Measuring k). The parameter in the bound can be measured empirically for each video calibration dataset. To measure k. for each of the three test datasets in Table 3, we measure the average changes of labels of subjects/activities
after swapping the i-th data point with the last one. This value is usually low (in most cases 2).
D. Training and Testing Datasets Details
We use the standard 3D human shape-pose datasets: 3DPW (Von Marcard et al., 2018). Human3 6M (lonescu et al., 2013), MPII-3DHP (Mehta et al., 2017), Penn Action (Zhang et al., 2013), PoscTrack (Andriluka et al., 2018), and Insta Insta (Kanazawa et al., 2019) where the preprocessed data is provided by (Shen et al., 2023), (Choi et al., 2021), and (Kocabas et al., 2020), and evaluated on 3DPW, Human3.6. M,,, MPII-3DHP. Note that our training dataset about 2.5% smaller than previous works because we hold out small portion (~ 1500 datapoints) for calibration.
17

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
E. Details of Choice of Conformity Score Function
Note that all three mentioned conformity score functions were trained end-to-end with training time ensemble augmen- tation setting since learning the score function after the human reconstructor is trained does not improve the performance. For the score function augmented with inefficiency loss, we are essentially controlling the size of the conformal predic- tion set during training (Stutz et al., 2021). For the classifier-style conformity classify if the mean 3D keypoints L. loss is within 40mm from the groundtruth using BCE loss and we use the logit as the conformity score. It is worth noting that all three variants result in better performance, demonstrating the importance of training time ensemble augmentation. Note that Ineff. needs more proposals during training than others and converges more slowly.
function, the training objective is to
F. Test-Time Multi-Hypothesis Aggregation
In the above experiments, compare hypohhssss outputs across methods for faireess, many baselines are single- output by design. MC Dropout is optional and provides two key advantages: (1) it naturally yields multiple hypotheses, and (2) using conformity scores, we can aggregate them via a weighted average that suppresses low-quality predictions based on the DUF score value.
To clarify its benefit, we provide comparison below showing how multi-hypothhsis (H=20, cutting off samples below calibrated threshold) aggregation improves performance over the single-sample case.
Table 4: Improvement on 3DPW
MPJPE
MPVPE
Accel
PA-MPJPE
Improvement
-0.9
-1.2
-1.1
-1.1
Table 5: Improvement on MPI-INF-3DHP
MPJPE
Accel
PA-MPJPE
Improvement
-23
-0.1
-1.6
Table 6: Improvement Humans.6.. 6M
MPJPE
Accel
Improvement
-2.2
-0.2
As we can see, with multi-hypothesis aggregation, the results get further improved. The GPU usage under 12 GB.
G. Performance under Occlusion
Regarding occlusion/rruncation tests, have completed larger-scale quantitative experiments. We rerun Table experi- ments (using the same trained model), but with all input image sequences' bottom 25% truncated.
Table 7: Performance comparison with and without truncation
3DPW
MPI-INF-3DHP
Human3.6M,
Method
PA-MPJPE
MPJPE
MPVPE
Accel
PA-MPJPE
MPJPE
Accel
PA-MPIPE
MPJPE
W/ Truncation
48.7
76.2
91.7
6.9
61.3
92.8
7.2
44.0
63.8
W/ Truncation
50.7
79.7
94.8
7.0
62.1
93.2
7.8
46.1
64.9
18

---

CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty
Despite being 25% truncated, the performance loss is not much, and in many cases, still better than the untruncated baselines in Table indicating CUPS's robustness to occlusion.
H. Limitations
While CUPS performs well on various benchmarks, we acknowledge that it does have some limitations. First, many samples need be proposed during training to improve the learned nonconformity score, which consumes a lot more GPU memory (30% more going from 10 proposals to 20) and slows down the training process. Second, the method does utilize human joint-level information, which could potentially improve the performance.
19


Reviews:
[{"review_id": "AFz4c89bjV", "reviewer": "Reviewer_Xcn3", "summary": "This paper introduces a novel method for human pose and shape estimation, utilizing the SMPL representation, from video sequences. The proposed approach incorporates conformalized deep uncertainty modeling, which allows for the generation of multiple samples, in contrast to the single-output methods commonly found in the literature. The uncertainty is theoretically calibrated, providing a safety guarantee for robotics and other downstream applications. The experiments are comprehensive and demonstrate state-of-the-art performance compared to existing baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Xcn3 for their thoughtful feedback and insightful questions. We particularly appreciate their recognition of the **theoretical soundness** of our work, **the need for uncertainty prediction in human pose estimation in the community**, and the applicability to **safety-critical areas**. Below, we respond to their comments in detail.\n\n> On the lack of diversity in Figure 6:\n\nWe appreciate the reviewer’s observation. CUPS was trained on multiple human pose-shape datasets, many of which, such as Human3.6M, were collected in controlled indoor environments. This may limit the diversity of foot and lower-body movements represented during training. The bottom example in Figure 6 depicts an outlier instance with highly dynamic motion that is not well-represented in the training set. Despite this, CUPS performs reasonably well, **tracking both pose and shape in such out-of-distribution (OOD) scenarios**.\n\nWe note that Figure 6 presents only static snapshots. We encourage the reviewer to explore [our website](https://sites.google.com/view/champpp), where full prediction videos are available. These showcase significantly **more diversity in predicted meshes**—including many cases where foot motion is well captured.\n\nTo improve diversity further, two directions are promising: (1) Incorporating more in-the-wild video data during training to reduce reliance on constrained datasets like Human3.6M, and (2) Introducing entropy regularization into the output ensemble [1], encouraging greater variability in the predictions.\n\n> On our choice of the conformal prediction (CP) framework:\n\nThis is an excellent question. We selected CP due to its **flexibility and generalizability**. CP is a distribution-free framework for uncertainty quantification that can be applied to any machine learning model (Angelopoulos & Bates, 2021; Shafer & Vovk, 2008). Crucially, even without strict exchangeability (*as is the case with human video data*), CP enables us to estimate a theoretical lower bound on performance by leveraging the properties of the calibration dataset.\n\nThis has two key advantages:\n\n- Theoretical guarantees: CP allows us to offer probabilistic performance guarantees via conformity scores—effectively certifying the reliability of predictions.\n\n- Modularity: CUPS requires only minimal changes to the backbone architecture. Because CP is model-agnostic, one could replace the GLoT-based backbone with other architectures in the future to improve accuracy, while still benefiting from CUPS’s theoretical guarantees.\n\nAlternative approaches, such as Dwivedi et al. (2024), introduce learned occlusion confidences but lack the theoretical coverage guarantees that CP offers. Moreover, Dwivedi et al. (2024)’s method is only applied to pose, while uncertainty quantification for shape estimation remains unexplored. CUPS provides a unified method for uncertainty quantification for both pose and shape since CP is used in SMPL space. Other prior work relies on injecting robustness via additional constraints, which often demands *substantial modifications* to the prediction pipeline. In contrast, CUPS offers **robust and theoretically grounded predictions with minimal architectural overhead**.\n\n\n\n\n\n[1] Chen, Rongyu, Linlin Yang, and Angela Yao. \"Mhentropy: Entropy meets multiple hypotheses for pose and shape recovery.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "claims_and_evidence": "The authors claim to propose a new method that provides both human pose and shape, along with the conformalized uncertainty scores. These claims are supported by detailed methods and extensive experiments demonstrating the performance.", "methods_and_evaluation_criteria": "The high-level approach involves a transformer-based architecture that predicts the parameters of the SMPL representation, utilizing a global regressor and a local corrector that focuses on different parts of the video sequences.\n\nThe uncertainty is predicted by training a neural network that takes both the input X and the output Y, providing a probability score between 0 and 1. This design is similar to a discriminator, with the authors employing a similar loss function to supervise the network.\n\nThe authors also address the important issue of \"calibration\" to ensure that the output uncertainty score is well-calibrated within a probabilistic framework. Detailed and reasonable proofs are provided both in the main text and the supplementary material.", "theoretical_claims": "The architecture of the network follows standard deep learning practices, such as Transformers and MLPs. The main theoretical contribution lies in the calibration of uncertainty, which is sound and well-justified.", "experimental_designs_or_analyses": "The experiments are extensive, and the results are impressive. Under uncertainty modeling, the method achieves state-of-the-art performance across nearly all metrics when compared to the baselines.", "relation_to_broader_scientific_literature": "A human pose and shape estimator with uncertainty scores can be applied in safety-critical areas, such as robotics, enabling a wide range of applications.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Uncertainty modeling is a highly valuable yet underexplored area in the literature. Typically, we focus on achieving high scores while overlooking the inherent ambiguity within neural networks. This method offers an additional measure to obtain robust outputs and certification with uncertainty, which is an important contribution to the community.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. The multiple samples seem to lack diversity. For instance, in Figure 6, there is no sample covering the foot. What is the main reason for this, and how can this issue be addressed to improve the method further?\n\n2. The motivation for using conformal prediction to model uncertainty is not entirely clear. This method requires exchangeable input data. While the related work discusses alternative methods that use, for example, explicit confidence values, the authors do not provide a summary of their limitations or how this paper's method stands out. What led the authors to choose this particular uncertainty modeling approach over others?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kC2IUy0kOf", "reviewer": "Reviewer_Xj5H", "summary": "This paper presents CUPS, an approach to infer 3D human shapes and poses from videos. At the core is a deep uncertainty function that is trained with 3D pose estimation, and it computes a conformity score to optimize the pose prediction in inference. Experimental results on different datasets and metrics demonstrate that the proposed method outperforms existing baseline methods on human pose estimation tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer Xj5H for their thoughtful feedback and insightful questions. We especially appreciate their recognition of the **strength of our experiments**, the **theoretical contributions** of our work, and the **effectiveness** of the approach. Below, we address each of the reviewer’s comments in detail.\n\n> On comparison with CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators\n\nThank you for highlighting this connection. We are indeed aware of CHAMP and have cited it in Section 2. CHAMP served as a key inspiration for our work, as it is among the few papers that explore conformal prediction for human pose estimation. However, CUPS addresses two major limitations noted in CHAMP:\n\n- Pose only: CHAMP focuses solely on pose estimation, whereas CUPS extends conformal prediction to pose-shape models, enabling richer and more expressive representations of human motion.\n\n- Theoretical soundness: CHAMP’s application of CP to **non-exchangeable** datasets like human motion videos leads to mostly empirical coverage guarantees since the **CP assumptions are violated**. On the other hand, CUPS builds on recent advances in **CP beyond exchangeability**, providing *a rigorous theoretical framework* that accounts for the structure and characteristics of video-based datasets.\n\n> On the global and local transformer components:\n\nWe apologize for any confusion. As cited in the paper, CUPS builds upon the GLoT backbone (Shen et al., 2023), which introduced the global and local transformer modules. While these are not novel contributions of CUPS (and this should be correctly represented in the paper), we summarize their roles here for clarity:\n\n- The global transformer captures *long-range temporal dependencies* to ensure consistency in human motion across frames.\n\n- The local transformer focuses on *fine-grained temporal dynamics*, refining predictions by modeling short-term variations around mid-frames.\n\nThe combination of these modules produces a decoupled global-local representation. For implementation details, please refer to GLoT’s [official codebase link](https://github.com/sxl142/GLoT/blob/main/lib/models/GLoT.py#L53).\n\n> On the deep uncertainty function (DUF) for pose and shape:\n\nCUPS applies conformal prediction to the output SMPL parameters, thereby **quantifying uncertainty over both pose and shape**. The learned conformity score function *implicitly* decouples the two: when either pose or shape is off, the conformity score may be low; when only one is off, the score may still be high depending on the calibration distribution.\n\nWhile it is possible to explicitly decouple pose and shape by using two separate conformity scores (and thus two CP procedures), doing so would complicate the theoretical analysis. Modeling the *interdependencies* between pose and shape would be necessary to maintain valid performance bounds.\n\n> On comparison with VIBE’s discriminator:\n\nThank you for raising this point. While both VIBE’s motion discriminator and CUPS’s DUF are trained adversarially, they serve distinct roles:\n\n- VIBE’s discriminator relies on **motion priors** from AMASS, introducing supervision from external datasets.\n\n- CUPS’s DUF, in contrast, is **self-supervised**. It is trained using an ensemble of predictions generated via Monte Carlo dropout, requiring no additional data.\n\nThis self-supervised setup makes CUPS **more modular and efficient**, introducing minimal changes to the backbone architecture. Moreover, because CP is model-agnostic, CUPS can be easily adapted to other backbones beyond GLoT while retaining its theoretical guarantees.", "claims_and_evidence": "This paper conducts experiments on different datasets including 3DPW, MPI-INF-3DHP, and Human3.6M, and the experimental results illustrate the effectiveness of the the approach.", "methods_and_evaluation_criteria": "The method was evaluated on different metrics including PA-MPJPE, MPJPE, and MPVPE. The results show that CUPS outperforms existing methods.", "theoretical_claims": "I have checked the Definition and Theorem 1-12, which are technically solid.", "experimental_designs_or_analyses": "The experimental analysis is sound.\n\nThis paper conducts experiments on different datasets including in-the-wild videos, and also provides experimental analysis (e.g., Empirical Coverage) to verify the proposed paper.", "relation_to_broader_scientific_literature": "This paper discussed the relations to existing methods in Related Work.", "essential_references_not_discussed": "The references are good.", "other_strengths_and_weaknesses": "Strength:\nThis paper is well-written and easy to follow.\nThe paper solves an interesting problem of 3D human pose and shape prediction from videos.\n\n\nWeakness:\nSome components in the paper are not thoroughly verified. \nFor example, how do the global and local transformer improve the results?\nHow to define the global and local features, and how are they decoupled?", "other_comments_or_suggestions": "Cite and discuss CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators.", "questions_for_authors": "How does the deep uncertainty function decouple pose and shape for motion correction? For example, we can adjust the pose or shape to make the prediction aligned with the ground truth in training.\n\nWhat’s the advantage of the deep uncertainty function over the pose discriminator used in VIBE?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wwf9rY2Qu0", "reviewer": "Reviewer_aBRZ", "summary": "This paper introduces CUPS, a method that integrates conformal prediction with deep uncertainty learning for 3D human pose-shape estimation from monocular videos. The key innovation lies in training an end-to-end deep uncertainty function alongside the reconstruction model, which serves as a conformity score for constructing prediction sets with statistical guarantees. By addressing non-exchangeability in video data through weighted conformal calibration, CUPS achieves state-of-the-art performance on benchmarks, while providing theoretical bounds on coverage gaps. The method is validated through extensive experiments, ablation studies, and in-the-wild tests.\n\n## update after rebuttal\n\nAs the author addresses most of my and others' concerns, esp. for the last clarification on the Multi-Hypo, alignment and additional quantitative truncation test, I decided to keep my recommendation of Weak Accept. It would be interesting to also test the method on a more powerful backbone.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer aBRZ for their thoughtful feedback and insightful questions. We especially appreciate their recognition of the **strength of our experiments** and **practical and theoretical contributions,** offering **novel synergies to advance safety critical vision systems**. Below, we address their comments in detail.\n\n> On training augmentation and multi-hypothesis aggregation:\n\nThe reviewer is correct: we compare single-hypothesis outputs across methods for **fairness**, as many baselines are single-output by design. MC Dropout is optional and provides two key advantages: (1) it naturally yields multiple hypotheses, and (2) using conformity scores, we can aggregate them via a weighted average that suppresses low-quality predictions: $ \\bar{x} = \\sum_i w_i \\cdot x_i, \\text{ where } w_i \\sim \\text{conformity score}(x_i)$\n\nTo clarify its benefit, we provide a comparison below showing how multi-hypothesis (H=20, cutting off samples below calibrated threshold) aggregation improves performance over the single-sample case.\n\n|  | **Improvement on 3DPW** |        |        |        | **Improvement on MPI-INF-3DHP** |        |        |      **Improvement on Human3.6M** |        |        |\n|-----------|--------------------------|--------|--------|--------|-----------------------------|--------|------------|----------------------------|--------|--------|\n|           | PA-MPJPE               | MPJPE | MPVPE | Accel | PA-MPJPE                | MPJPE| Accel | PA-MPJPE    | MPJPE | Accel |\n|           | -0.9  | -1.2 | -1.1 | -0.2 | -1.1   | -2.3 | -0.1 |-1.6 |  -2.2  | -0.2 |\n\nAs we can see, with multi-hypothesis aggregation, the results get further improved. The GPU usage is under 12 GB.\n\n> On the score function and hyperparameters:\n\nWhile the score function resembles a discriminator, it is fundamentally different from a GAN setup. Its role is not to classify real/fake samples but to **rank predictions** based on conformity. We use a weaker adversarial loss, and backpropagate the score loss only once every 100 SMPL updates to ensure training stability—more frequent updates (e.g., every 10–50 steps) lead to instability, as we will discuss in the final version.\n\nMC Dropout also induces **distributional oscillations** during training, which can prevent full convergence of the score function. This is acceptable: conformal prediction requires only that the conformity score be consistent across calibration samples, not fully converged.\n\n> On the choice of the beta distribution:\n\nWe chose the beta distribution based on empirical evidence:\n\n- Calibration scores fitted well via MLE. Please checkout this [new link](https://sites.google.com/view/cups-occlusion-supp/home) for **plots of fitted vs. theoretical Beta alignment**.\n- The Kolmogorov–Smirnov test failed to reject the beta hypothesis.\n\nThis choice allows for analytical tractability in deriving theoretical bounds while remaining grounded in the observed data distribution.\n\n> On hard OOD examples and occlusions:\n\nSome in-the-wild videos we show are fast-paced and more challenging than the training data. While a few examples in the paper may not illustrate this well, we encourage the reviewer to visit [our website](https://sites.google.com/view/champpp) for full videos. These show **more diversity in foot/hand motion and depth variation**. We also add **two new visualizations** highlighting CUPS’s robustness under heavy occlusions in a separate [anonymous website](https://sites.google.com/view/cups-occlusion-supp/home).\n\nCUPS does occasionally inherit failure cases from its GLoT backbone, but it requires only minimal architectural changes. Importantly, since CP is **model-agnostic**, stronger backbones can be **easily swapped** in for future improvements while preserving theoretical guarantees.\n\n> On missing related work:\n\nWe thank the reviewer for flagging the two missing works:\n\nMHEntropy introduces entropy-based methods for hand pose-shape recovery, especially effective under occlusion. As discussed above, while CUPS does not explicitly model occlusions (we assume the underlying backbone is somewhat robust), we could further improve CUPS by incentivizing diverse outputs during training via entropy regularization. This could potentially yield more robust and diverse outputs.\n\nPlausible Uncertainties focuses on 2D pose regression. While useful, our choice of Conformal Prediction offers broader applicability, minimal architectural modification, and theoretical guarantees, making it better suited for general 3D human pose-shape tasks.\n\n> On set prediction’s computation cost:\n\nMulti-hypo prediction in CUPS is **lightweight**. We are able to achieve, on average, 20ms/segment on video data on a single V100 GPU. This is because MC Dropout itself is not memory-expensive. Other approaches, such as diffusion-based probabilistic backbones, might consume more memory, but the bottleneck does not come from CP or calibration.", "claims_and_evidence": "The claims are well-supported:\n- Good performance is evidenced by quantitative comparisons. The effectiveness of uncertainty-aware training is demonstrated via ablation studies (e.g., training-time ensemble augmentation)\n- Some mathematical proofs are provided to support the correctness of the theorems. Theoretical coverage guarantees are derived for non-exchangeable data, with empirical validation. Minor improvements could include clarifying the practical implications of the coverage bounds.", "methods_and_evaluation_criteria": "- The methodology is sound: the global-local transformer architecture aligns with recent advances, and the integration of adversarial training for uncertainty scoring seems interesting.\n- Evaluation metrics (MPJPE, PA-MPJPE, Accel) and datasets are standard for 3D pose estimation. The inclusion of in-the-wild tests strengthens practical relevance.\n\nMay I ask:\n- **(training & inference)** Is it true that augmenting multiple training samples $H$ during the training but only doing **SINGLE** inference during test on standard benchmarks (e.g. Tab. 1)? MCDropout multi-hypo generation and set selection according to the DUF is an optional function? Or inference also samples several and aggregates them with the DUF.\n- **(Degenerated discriminator)** GAN studies find the goal of discriminator is to help the learning of the generator and it will eventually degrade. However, it seems discriminator in the paper works well. Could the author elaborate and provide some insights?", "theoretical_claims": "To my perspective, the proofs for Theorems 1–3 are logically structured. The adaptation of Barber et al.’s framework to handle non-exchangeable data is appropriate.\n\n- **($\\beta$ distribution)** Theorem 3 assumes that the conformity scores follow a Beta distribution. How to validate this assumption in practice?", "experimental_designs_or_analyses": "Experiments are comprehensive, covering some datasets and baselines and ablation studies.\n\nMay I ask:\n- **(Comparisons with multi-hypo methods)** The authors seem not to clarify why they discuss but do not compare with multi-hypo aggregation methods?\n- **(Figs. 1 & 6 multi hypos)** seem not so meaningful as like hands are not aligned with the clear image? I would expect to see they span along the depth from the side view.\n- **(GAN DUF)** Introducing adversarial loss is indeed meaningful, but could the instability of adversarial training potentially led to difficulties in model convergence? Additionally, to what extent does adversarial loss impact the results? I am not talking about $\\lambda$ in Fig. 5 but GAN training setting and hyperparams. Corresponding ablation experiments can be conducted to explore these questions.\n- **(CP under occlusion)** Could analysis be done 3DPW-occlusion (like PARE), 3DPW-truncation (like NIKI)? What will the result change to?", "relation_to_broader_scientific_literature": "This work bridges three key research threads: conformal prediction, 3D human pose-shape estimation, and deep uncertainty quantification, offering novel synergies to advance safety critical vision systems.", "essential_references_not_discussed": "To my knowledge, the paper cites relevant essential works. A few literature could also be discussed, including probabilistic multi-hypo methods (MHEntropy, ICCV'23) and uncertainty HPE (though 2D, PlausibleUncertainties DER, ICCV'23).", "other_strengths_and_weaknesses": "**[Strengths]**\n- Reasonable integration of conformal uncertainty learning and multi-hypo HMR.  \n- Superior empirical results across datasets.\n- also provides some practical contributions (e.g., MC dropout seems to well suit BERT-like masking for DUCS construction).\n\n**[Weaknesses]**\n- Limited discussion on **computational overhead** as multi-hypo DUCS prediction.\n- **(Corner hard cases)** The model's ability to quantify uncertainty under extreme scenarios like severe occlusions or rapid motion is crucial. While tested on in-the-wild videos, the paper does not explicitly evaluate performance under these situations.", "other_comments_or_suggestions": "Please see other sections.", "questions_for_authors": "Please see the above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA."}}, {"review_id": "7S3KVbLL7o", "reviewer": "Reviewer_wgxo", "summary": "The paper introduces CUPS, a video-based HMR approach with uncertainty quantification. Specifically, the method uses GLoT as a base model to extract global and local features from videos. An adversarial loss is defined on the output meshes during training, similar to VIBE. The discriminator output (from a sigmoid layer) estimates the uncertainty of the predictions (whether they belong to the real dataset or not). Next, the paper applies the statistical tools developed by Barber et al. to calibrate the uncertainties. The paper presents two theoretical bounds and demonstrates state-of-the-art performance on standard benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer wgxo for their thoughtful feedback and insightful questions. We especially appreciate their recognition of our **creativity** and **theoretical soundness**. Below, we address their comments in detail.\n\n> On calibration’s computational cost:\n\nCalibration in CUPS is **lightweight**. We hold out a small subset (<1000 samples) from the training data as a calibration set. In practice, we only use 500 calibration samples (i.e., about 8 forward-pass batches), and once the conformity threshold is computed, it remains fixed. No additional inference-time cost is incurred.\n\n> On hyperparameters and distance function:\n\nThank you for pointing this out. We **will include a full list of hyperparameters** in the appendix. The missing one now is the **temperature**, set to 20. While both temperature and the ablated parameter $\\lambda$ affect score loss strength, $\\lambda$ has a greater impact.\n\nRegarding the distance function, since we operate in high-dimensional feature space, rotation-based metrics are inapplicable. An alternative we explored was using Cosine similarity, which also works well.\n\n> On alternative uncertainty methods and adapting other backbones:\n\nWe appreciate the reference to RLE [1]. RLE formulates pose estimation as distribution matching, embedding uncertainty into the architecture via flow matching. In contrast, CUPS uses MC Dropout to emulate a probabilistic output space but remains modular—*any probabilistic/generative backbone like RLE could be used within the Conformal Prediction (CP) framework*.\n\nCP’s model-agnostic nature allows CUPS to **adapt to new backbones with minimal changes while maintaining theoretical coverage** guarantees. For example, across all metrics, CUPS outperforms Dwivedi et al. (2024), which uses learned occlusion confidences for pose uncertainty measurement but lacks shape uncertainty modeling and theoretical coverage guarantees. Unlike methods that inject robustness via heavy architectural modifications and constraints, CUPS provides **unified, reliable uncertainty quantification for both pose and shape** with minimal overhead from conformal prediction. \n\nThus, when it comes to selecting other models for our pipeline, we would *still use conformal prediction* but maybe incorporate probabilistic (generative) models, which give multi-hypothesis outputs efficiently. \n\n> On comparison with CHAMP:\n\nThank you for highlighting this. We cite CHAMP in Section 2, as it inspired CUPS. However, CUPS addresses key limitations noted in CHAMP’s paper:\n\n- Pose-only: CHAMP focuses solely on pose estimation, whereas CUPS extends conformal prediction to pose-shape models, enabling richer and more expressive representations of human motion.\n\n- Theoretical rigor: CHAMP’s application of CP to **non-exchangeable** datasets like human motion videos leads to mostly empirical coverage guarantees since the CP assumptions are violated. CUPS builds on recent advances in CP **beyond exchangeability**, providing a **rigorous theoretical framework** that accounts for the structure and characteristics of video-based datasets.\n\n> On shape parameter consistency (Line 160):\n\nWe apologize for any confusion we might have caused. The reviewer is right in pointing out that the shape parameters should remain consistent across frames, but for the sake of **mathematical formulation**, especially with the use of a sequential transformer model, the pose and shape parameters are outputted as a **sequence**. While we do not enforce each frame’s shape to be exactly the same, empirical outputs show consistent shapes across frames, and it might be interesting to **explore losses that regularize shape consistency**.\n\n> Misc.\n\nWe thank the reviewer for pointing out the redundancy in some introduction of the methods. We will make our writing more concise.\n\nThe main uncertainty-based baselines we compare with are Dwivedi et al. (2024) and 3DMB (Biggs et al., 2020), which we outperformed by a noticeable margin. Other uncertainty-based baselines, which we are exploring right now, include swapping out GLoT with different probabilistic HMR models.\n\nAdding uncertainty itself **does not necessarily improve the results**. Our main contribution is a framework that quantifies the uncertainty with theoretical rigor while backpropagating the uncertainty into the pose learning process with delicate designs that improve the final outputs. Most UQ methods such as CP do not \"close the loop\"; they assume a well-trained model, and **very few prior works have explored incorporating uncertainty into the learning process.** As Reviewer aBRZ said, CUPS \"offers novel synergies to advance **safety critical** vision systems.\"", "claims_and_evidence": "The claims are generally supported, including achieving state-of-the-art and improved generalizability. The proofs also follow prior references.", "methods_and_evaluation_criteria": "The paper uses a recent paper (GLoT) as a base model, which is appropriate. Additionally, it tests the model on popular benchmarks like 3DPW and Human3.6m, according to prior research. The evaluation metrics also follow the standards in this field.", "theoretical_claims": "All proofs are according to prior research from Barber et al. 2023.", "experimental_designs_or_analyses": "All experiments were reviewed, and they are sound.", "relation_to_broader_scientific_literature": "Uncertainty quantification and conformal prediction are valuable tools for machine learning tasks, especially for HMR, where a large portion of the SMPL parameter space does not represent a plausible pose. As a result, obtaining the uncertainty and calibrating it can significantly improve the estimation performance. Such approaches are often found in medical papers, where uncertainty is paramount. However, more recent papers have adapted uncertainty estimation into HMR, showing significant performance gains. Therefore, given the context of recent research, this paper is timely and addresses an interesting topic.", "essential_references_not_discussed": "All references in the manuscript are adequate. However, I am interested in the authors' opinions on other uncertainty prediction approaches, like RLE [1] or similar works, and how they could be adapted into their framework.\n\n[1] Li, Jiefeng, et al. \"Human pose regression with residual log-likelihood estimation.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.", "other_strengths_and_weaknesses": "### Strengths\n1. The paper provides a creative combination of existing ideas, resulting in improvements.\n2. The paper is well-written.\n\n### Weaknesses\n1. The paper does not fully explore the limitations of calibration methods, which often involve memory and computation costs. More info on this (more than supplementary) would be appreciated.\n2. Some parts are unnecessarily explained, especially where it is not the paper's main contribution. Given the recency of the tools, it could be acceptable.\n3. No comparisons are provided with other uncertainty modeling papers.\n4. The results of the method are not very surprising, as adding any uncertainty-aware modules may result in better performance.\n5. The contributions are somewhat limited, especially given similar works like [1]. The paper's main focus, uncertainty modeling, and calibration is not experimented upon, and there are minimal comparisons with other approaches.\n\nReferences:\n[1] Zhang, Harry, and Luca Carlone. \"CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators.\" arXiv preprint arXiv:2407.06141 (2024).", "other_comments_or_suggestions": "I suggest including the values of hyperparameters, conformal dataset size, and architectural details in the manuscript or supplementary materials. Please see my other comments for more suggestions/questions.", "questions_for_authors": "1. Could you include the hyperparameter's chosen values in the manuscript or supplementary materials?\n2. How impactful is the choice of feature distance function? How important is the choice of the temperature? And did you consider using rotational distance instead of L2 distance of rotations? \n3. Why is the beta defined with different values across frames? Specifically, in the problem formulation in line 160 (left), you define a separate beta for each frame. If it represents the shape parameters, should it not be the same for all frames?\n4. I understand if it is not possible to incorporate other uncertainty estimation approaches and provide their results, but could you elaborate and discuss how they could be implemented into your pipeline? and if it would be an appropriate choice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel method for learning 3D human shapes and poses from RGB videos with built-in uncertainty quantification using conformal prediction. By generating and scoring multiple hypotheses during training, the method integrates a deep uncertainty function trained jointly with the pose estimator. This enables conformal prediction with theoretical coverage bounds, leading to strong performance with probabilistic guarantees.

The paper was reviewed by four experts, and all reviewers unanimously support acceptance of the paper. While there were some concerns initially around the limitations of calibration methods, need for comparisons with other uncertainty modeling work, as well as some questions on the stability of the proposed method, the rebuttal helped answer these questions, and reviewers (Rev#wgxo) appreciated the thoughtful responses and the value of this method for video HMR.

Given the consensus, this paper is recommended for acceptance. As noted by reviewers, the authors are recommended to incorporate the rebuttal responses in the final version.
Decision: Accept (poster)