Multi-Hypothesis 3D human pose estimation metrics favor miscalibrated distributions
Authors: Paweł A. Pierzchlewicz
R. James Cotton
Mohammad Bashiri
Fabian H. Sinz
Venue: Submitted to ICLR 2023 (2023)
Keywords: Pose estimation
calibration
metrics
graph neural networks

Abstract: Due to depth ambiguities and occlusions, lifting 2D poses to 3D is a highly ill-posed problem. Well-calibrated distributions of possible poses can make these ambiguities explicit and preserve the resulting uncertainty for downstream tasks. This study shows that previous attempts, which account for these ambiguities via multiple hypotheses generation, produce miscalibrated distributions. We identify that miscalibration can be attributed to the use of sample-based metrics such as $\operatorname{minMPJPE}$. In a series of simulations, we show that minimizing $\operatorname{minMPJPE}$, as commonly done, should converge to the correct mean prediction. However, it fails to correctly capture the uncertainty, thus resulting in a miscalibrated distribution. To mitigate this problem, we propose an accurate and well-calibrated model called Conditional Graph Normalizing Flow (cGNFs). Our model is structured such that a single cGNF can estimate both conditional and marginal densities within the same model - effectively solving a zero-shot density estimation problem. We evaluate cGNF on the Human 3.6M dataset and show that cGNF provides a well-calibrated distribution estimate while being close to state-of-the-art in terms of overall $\operatorname{minMPJPE}$. Furthermore, cGNF outperforms previous methods on occluded joints while it remains well-calibrated.

Under review as a conference paper at ICLR 2023
MULTI-HYPOTHESIS 3D HUMAN POSE ESTIMATION METRICS FAVOR MISCALIBRATED DISTRIBUTIONS DISTRBBTTONSSSSSS
Anonymous authors Paper under double-blind review
ABSTRACT
Due to depth ambiguities and occlusions, lifting 2D poses to 3D is a highly ill- posed problem. Well-calibrated distributions of possible poses can make these ambiguities explicit and preserve the resulting uncertainty for downstream tasks. This study shows that previous attempts, which account for these ambiguities via multiple hypotheses generation, produce miscalibrated distributions. We identify that miscalibration can be attributed to the use of sample-based metrics such as minMPJPE. In a series of simulations, we show that minimizing minMP. JPE. as commonly done, should converge to the correct mean prediction. However, it fails to correctly capture the uncertainty, thus resulting in a miscalibrated dis- tribution. To mitigate this problem, we propose accurate and well-calibrated model called Conditional Graph Normalizing Flow (cGNFs). Our model struc- tured such that single cGNF can estimate both conditional and marginal den- sities within the same model - effectively solving a zero-shot density estimation problem. We evaluate cGNF on the Human 3.6M dataset and show that cGNF provides a well-calibrated distribution estimate while being close to state-of-the- art in terms of overall MPP. Furthermore, cGNF outperforms previous methods on occluded joints while remaining well-calibraedd
NEW
INTRODUCTION
The task of estimating the 3D human pose from 2D images is classical problem in computer vision and has received significant attention over the years (Agarwal & Triggs) 2004 Mori & Malik,
2006 Bo et al. 2008). With the advent of deep learning, various approaches have been applied to this problem with many of them achieving impressive results (Martinez et al., 2017 Pavlakos
al. 2016 2018 Zhao al. 2019: Zou & Tang 202T). However, the task of 3D pose estimation from 2D images is highly ill posed: A single 2D joint can often be associated with multiple 3D) positions, and due to occlusions, many joints can be entirely missing from the image. While many previous studies still estimate one single solution for each image (Martinez et al. 2017 Pavlakos et al. al. 2017 Sun et al. 2017] Zhao et al. 2019 Zhang et al. (021), some attempts have been made to generate multiple hypotheses to account for these ambiguities (Li & Lee, 2019: Sharma al. 2019 Biggs et al., 2020 Oikarinen et al. 2020 Li & Lee 2020 Kolotouros et al. 2021 Wehrbein et al. 2021). Many of these approaches rely estimating the conditional distribution of 3D poses given the 2D observation implicitly through sample-based methods. Since direct likelihood estimation in sample-based methods is usually not feasible, different sample-based evaluation metrics have become popular. As a result, the field's focus has been on the quality of individual samples with respect to the ground truth and not the quality of the probability distribution of 3D poses itself.
FIX
In this study, we show that common sample-based metrics in lifting, such as mean per joint position error, encourage overconfident distributions rather than correct estimates of the true distribution. As result, they do not guarantee that the estimated density of 3D poses is faithful representation of the underlying data distribution and its ambiguities. As a consequence, their predicted uncertainty cannot be trusted in downstream decisions, which would be one of the key benefits of probabilistic model (Fig.
NEW
In series of experiments, we show that probabilistic lifting model trained with likelihood provides a higher-quality estimated distribution. First, we evaluate the distributions learned by minimizing
"ode and pretrained model weights available at httpt::t////... thub. com/xxxx

---

Under review as a conference paper at ICLR 2023
Wehrbein et
Ours
Figure 1: Examples showcasing the consequences of an overconfident distributions vs.our well- calibrated distribution. Ground truth marked with colored poses. Uses artificial 2D keypoint failures.
minMPJPE instead of negative log- likelihood (NLL) observing that, although minMP.JPE opti- mal distributions have a good mean they are not well-calibrated. Next, we use the SimpleBaseline (Martinez et al., 2017) lifting model with simple Gaussian noise model on Human3. 6M to demon- strate that a model optimized for NLT is well-calibrated but underperforms on minMP The same model optimized for minMPJPE performs well in that metric but turns out to be miscali- brated. To balance this trade-off, we propose interpretable evaluation strategy that allows com- paring sample-base methods, while retaining calibration. Finally, we introduce a novel method to learn the distribution of 3D poses conditioned the available 2D keypoint positions. To that end, we propose a Conditional Graph Normalizing Flow (cGNF). Unlike previous methods, cGNF does not require training separate model for the prior and posterior. Thus, model does not require an adversarial loss term, as opposed to Wehrbein et al. 2021). By evaluating the cGNF's performance on the Human 3.6M dataset (Ionescu et al., 2014). we show that, in contrast previous methods, our model is well-calibrated while being close to state-of tfrt--rr- in terms of overall minMPJPPE, and that significantly outperforms prior work on occluded joints.
2 RELATED WORK
Lifting Models Estimating the human 3D pose from 2D image is an active research area (Pavlakos et al. 2016 Martinez et al.) 2017: Zhao et al. 2019 Wu et al. 2022). An effective ap- proach is to decouple 2D keypoint detection from 3D pose estimation Martinez et al. 2017). First, the 2D keypoints are estimated from the image using a 2D keypoint detector, then a lifting model uses just these keypoints obtain a 3D pose estimate. Since the task of estimating a 3D pose from 2D data is a highly ill-posed problem, approaches have been proposed to estimate multiple hypothe- (Li & Lee [2019] Sharma al. 2019 Oikarinen al.) 2020 Kolotouros et al. 2021 Li al.] 2021 Wehrbein et al. 2021). However, these approaches i) do not explicitly account for occluded or missing keypoints and ii) do not consider the calibration of the estimated densities. Wehrbein et al.
(2021) incorporate a Normalizing Flow (Tabak, (200) architecture to model the well-defined 3D to 2D projection and exploit the invertible nature Normalizing Flows to obtain 2D to 3D estimates. Albeit structured as Normalizing Flow it is not trained as probabilistic model. Instead, the au- thors optimize the model by minimizing a set of cost functions. All some form depend on the distance of hypotheses to the ground truth. In addition, they utilize adversarial loss to improve the quality of the hypotheses. The proposed model achieves high performance on popular met- rics in multi-hypothesis pose estimation, which are all sample-based distance measures rather than distribution-base metrics. [Sharma et al.] (2019) introduces conditional variational autoencoder architecture with an ordinal ranking disambiguate depth. Similarly Wehrbein al.] (2021). the authors additionally optimize the poses on sample-based reconstruction metrics and report per- formance on sample-based metries only. Oikarinen et al. (2020) utilize a graph-based approach to construct a mixture density network of Gaussian distributions, Kolotouros et al.] (202I) use a Salume-preserving normalizing flow model based on the GLOW architecture (Kingma & Dhariwal 2018).
NEW
Sample-Based Metrics in Pose Estimation The most widely used metric in pose estimation is the mean per joint position error (MPJPE) (Wang et al. 2021). h. is defined as the mean Euclidean distance between the K ground truth joint positions X E R and the predicted joint positions X E RRx3 Multi-hypothesis pose estimation considers N hypotheses of positions X E Nx
2

---

Under review as a conference paper at ICLR 2023
and adapts the error to consider the hypothesis closest to the ground truth (Jahangiri & Yuille 2017
minMPJJEE(X. X) min
Σ X
X
K
In this work, refer this minimum version of the MPJPE as minMPPPE. Procrustes-Aligned MPJPE (PA-MPJPE) is a variation on MPJPE which first aligns the test pose to the ground truth pose. The percentage of correct keypoints (PCK) (Toshev Szegedy) 2013 Tompson al.) 2014 Mehta et al. 2016) is another widely accepted metric in pose estimation which measures the percentage of keypoints in circle of 150mm around the ground truth in terms of minMPJPE. Conrect pose score (CPS) proposed by Wandt et al. (2021) considers pose to be correct if all the keypoints are within a radius E [0 mm, 300 mm the ground-truth terms of minMPJPEE CPS is defined as the area under the curve of percentage correct poses and r.
NEW
Calibration an important property of a probabilistic model measuring model's ability to cor- rectly reflect the uncertainty in the data. Thus, the confidence of an event assigned by a well- calibrated model should equal to the true probability of the event (Brier 1950). Guo et al. (2017) show that calibration of densities is especially important in the field of deep learning. where dif- ferent architecture choices have been shown to lead to miscalibrated. Nacini et al. (2015) propose to measure the expected calibration error (ECE) metric which approximates the expectation of the absolute difference between the predicted probability and the true probability.
FIX
N
ECE
Σ
P - Pxl
(1)
N
1=1
The lower the ECE the better the calibration of the distribution. A model which predicts the same probability for all samples has an ECE of 0.5, whereas perfectly calibrated model has ECE 0. DeGroot & Fienberg (183) and Niculescu-Mizi & Caruana (2005) provide a visual representation of calibration using reliability diagrams. They display the calibration curve, which is function of confidence against the true probability. If the calibration curve is an identity function then the model perfectly calibrated.
OBSERVING MISCALIBRATION
In this section, we demonstrate that the current We consider two of the latest methods: SShamma et al. (2019) and WWerbein al. (202T). We compute the ECE for the two models and visualize their reliability diagrams (Fig. 2
lifting models are not well-calibrated.
3.1 QUANTILE CALIBRATION FOR POSE ESTIMATION
Quantile calibration (Song al. 2019) defines a per- fectly calibrated distribution as one for which ground- truth values X" fall within the q-th quantile 9% the time. However, for high dimensions estimating whether a point is contained within a given quantile is non-trivial We, therefore, propose to simplify the problem by pro- jecting to the univariate space of squared errors e from the median X of N hypotheses X conditioned on 2D poses C with K keypoints. We then compute ECE in the space of over the set of quantiles Q [0, 1] (Algorithm []. As a measurement of central tendency we choose the median statistic, which is more robust to outlier samples. How- ever, in practice, the choice of median vs. mean results in minor differences in the calibration outcomes (sec. A.3
Algorithm 1 Quantile calibration for pose estimation
for each X", and C.,, do
NEW
draw N hypotheses X C.n.
Xm, median( {,m,)))
En,m,k 1x X,n,kl2
Pm t CDF(E, (E),,,k)
e'n,k | X Xm,
end for
M ΣM 1om(en w(g) +- median(wx (g))
Wie + 1
FIX
ECE = [θ] Egeo ((()) - ql
3.2 SAMPLE- BASED METRICS PROMOTE MISCALIBRATTON
In this section, we show that sample- based metrics are a major component that contributes to mis- calibration. In principle, minMPJPF, could be good surrogate metric for NLL. However, as it
3

---

Under review as a conference paper at ICLR 2023
a
b
Sharma et al. (2019)
Wehrbein et al. (2021)
minMPIPE optimal 0
100
Overco meenconfdddett Undercentidett
mimWp3
Frequency ((o)
ECE 0.36
ECE 0.18
32-
0.6
Perfectly calibratte
Misral. Gracle
0.4
40
20-
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1
22
106
500
Micral, Oracle
Quantile
Samples
C
minMPpE
minMP3PE
ECE constraint
40
7
High
20
PLL
0.4
NIL
mininum
3
Law
-1
11 2
MinMP.FENN NIL
MiMPP_PE mirimum
Figure 2: a) Calibration curves of previous lifting models with the corresponding expected calibra- tion error (FCF) scores. b) Standard deviation a of a Gaussian distribution optimized to minimize minMPJPF for different numbers of samples and dimensions. The true is 0.5 (black line), un- derconfident a 0.5 (blue), overconfident a < 0.5 (pink). The human pose equivalent distribution (black point, 45 dimensions, 200 samples) compared to oracle distribution (with true μ and in terms of minMP.JPE and NLL. e) Gaussian noise model schematic to the left. The SimpleBaseling model weights are not trained. Right bar plots compare the performance minMPJPE and ECE when optimizing for minMPJJP and NLL. d) Loss landscapes of minMPJJE and ECE for a 1D Gaussian distribution with parameters and μ.. The gold star represents the ground truth values of a² 4 and j = 0. To the right is schematic of the ECE constrained optimization.
became a common metric for selecting models it might become subject to Goodhart's 's Law (Good- hart 1975) "When measure becomes a target, ceases be a good measure" (Strathern) 1997). In the case of minimizing the mean MPJPE over hypotheses, the posterior distribution collapses onto the mean (sec. A.T. Similarly, simulations indicate that minMPJPPE converges the correct mean, but it encourages miscalibration (Fig. 2p.d and sec. A.2).
We illustrate this with small toy example. Consider M samples X* RMXD from D- dimensional Isotropic Normal distribution with mean fr" E RDD and variance x2 RDD and an ap- proximate isotropic Normal posterior distribution (X) with mean μ E E. and variance o² E EDD We assume the ground truth mean to be known μ imize minMPJPE with N hypotheses. We optimize o² for different numbers dimensions D and hypotheses N. If the distribution converges to variance lower than the true variance *2 call such a distribution overconfident. However, if the converged variance is larger than the true variance then the distribution is considered to be underconfident. Intuitively, for small sampling budget drawing samples at the mean constitutes the least risk of generating a bad sample. With an increase in the number of hypotheses, increasing variance should gradually become beneficial, as the samples cover more of the volume. For sufficiently large number of hypotheses, we can expect the variance to increase beyond the true variance, as the low-probability samples can have sufficient representation. Increasing dimensions should have an inverse effect since the volume to be covered increases with each dimension. We observe these effects the toy example (Fig. (p). When we consider the case which corresponds to the 3D pose estimation problem (D 45 and N 200, black point in Fig. 2p), we expect an overconfident distribution based on our toy example. This is also what observe for the current oftte-of-the-arr lifting models (Fig, 2h) Furthermore, show that the minMPJPP optimal distribution outperforms the ground truth distribution in terms of JPPMPJPF, but not in terms of negative log-likelihood (Fig. 2p). Together, the results imply
and only optimize the variance o² to min-
NEW

---

Under review as a conference paper at ICLR 2023
that minimizing minMPJJEE. directly by model selection, is expected to result in miscalibrated distributions and thus minMP.IPE by itself is not sufficient to identify the best model.
3.3 UNCONDITIONAL GAUSSIAN NOISE BASELINE ON HUMAN 3.6M
To verify the conclusions from the toy model in section 3.2 we test the prediction with a simplified
model on the Human3.66 6M dataset (Catalin lonescu 2011: lonescu et al.) 2014) (see section (S) for
more details about the dataset). We train an additive Gaussian noise model on top of the SimpleBase- line (Martinez et al. 2017) a well-established single-hypothesis model. We generate N hypotheses X E X3 of poses with K keypoints for M observations C E [R [RMM x2 according to:
X n,m Simple (Cm) o²2,
where SimpleeBaselnne Cm.) estimates the mean of the noise and o is the standard deviation param- eter scaling the standard normal samples
(2z 0, I) (Fig. 2 It is important to note that we do not condition a on the 2D observation C. i.e. the same noise model is used for every input. We test two optimization setups: 1) minimizing minMPJPE and 2) maximizing likelihood. Based on the predictions from the toy problem (sec. 3. we expect the minMPJPE. model to be overconfident and outperform the NLL model on the minMPPJPF but the NLL model to be better calibrated. This is exactly what we observe (Fig.Zp) Furthermore, each of these models achieve minMPJPE performance in a range similar to state-of-the-ar multi-hypothesis methods and even outperform some established single-hypothesis methods (Table T.
NEW
3.4 EVALUATING SAMPLE- BASED METHODS
Given that minMPJPF. is not sufficient to fully evaluate multi-hypothesis methods, we propose an evaluation strategy that remains interpretable and promotes calibrated distributions. Consider the landscapes of minMP and ECE with respect to the mean and variance of an approximate distribution (Fig. 2 Simulations indicate that optimizing MinMPJPE identifies the correct mean (sec. A but not the correct α.. ECE, however, is minimized by manifold of μu, values and converges to a good standard deviation for each mean, but does not guarantee an accurate model. We thus hypothesize that likelihood-optimal distribution can be approximated when minMPPPP minimized on the FCFEooptimal manifold. Thus, minMPJPE can become a measure of accuracy if it is constrained by ECE, but it should not be considered as accuracy if calibration is not matched.
NEW
NEW
NEW
4 CONDITIONAL GRAPH NORMALIZING FLOW
Given the observations made in sec. 3 we conclude that MP objective functions are not
NEW
sufficient to obtain a well-calibrated distribution. The objective function should instead be based on likelihood, which in this case is maximized if and only if the estimated distribution recovers the
ground truth distribution i.e. if the distribution is well-calibrated (Hastie et al, 2009). Therefore, in
this section, we propose method that can be optimized purely based on likelihood. Moreover, utilize the natural graph structure of the human pose providing zero-shot generalization capabilities to occluded and unobserved body parts. We propose learn the conditional distribution p(ar e) of the 3D pose x given the 2D pose using conditional graph normalizing flows (cGNF).
We define target graph x [H E²) of 3D poses and a context graph c (Ho E²) of 2D detec- tions. H" E D. and Ex are the edges between the nodes of the target graph and H E m and Es are the edges between the nodes of the context graph. In the case that an observation is not present, the corresponding node is removed from c. The model is built of L transformation blocks, each of which consists of a per-node feature split step. graph merging step, an actnorm (Kingmal & Dhariwal 2018) and two graph neural network layers (Gori et al. (205) (Fig. 3). These elements construct an affine coupling layer (Dinh et al., 2016), which is then followed by permutation layer. The transformation blocks are only applied to the target graph, while the context graph is passed through unchanged.
Per-Node Feature Split Step splits the target node features H into two parts, H. H and H across the feature dimension. We incorporate a saave-one-or strategy for splitting the features. The ith feature dimension is propagated directly to the affine coupling layer and the remaining
FIX
5

---

Under review as a conference paper at ICLR 2023
Inference
Original space
Latent space
Generation
X
~ N
Xilet X3)
per-node feature split
affine coupling
a(xg, c)
t(x3, c)
target graph
context graph
graph merge
actnorm
GNN
GNN
2: lafent representation
H feature mabri
H
e
A: adjacency matrix
Figure 3: schematic of the cGNF. Target variables æ: are represented by graph with the feature matrix H and the adjacency matrix AA. The context variables represented by context graph with the feature matrix He and adjacency matrix A². In the inference path the target graph æ is transformed into latent space which follows standard normal distribution. The transformation achieved through transformation blocks.
dimensions are passed to the graph neural network layers. the next block, the next ith dimension used.
Graph Merging When utilizing conditional normalizing flows Winkler et al. 2019 on graph- structured data, key challenge is incorporating the context graph in the transformation. We propose to merge the context graph e with the target graph æ into heterogeneous graph æ c. The context graph forms directed edges from nodes R (Fig.
nodes in æ as defined by R the relations matrix. indicates that node in the context graph forms an edge with node j in the target graph
Graph Neural Network Layers We define the graph neural network layers relational graph convolutions (R-GCNs) (Schlichtkrull et al.] (2018). In the message passing step, the message re- ceived by node from the neighboring nodes defined as
(v)
Σ
Be-rx hb(") clu,))
Σ
Σ
vir (n(")) cla.v)
NEA+ (x)
PER wEN"(u)
where v, P., + P and dooz P. + P with Dn the number of latent dimensions. dle-zz should flexible enough to allow the network to learn to distinguish between missing obser- vations and zero observations i.c. veoz (0, (u.r)) o O. The Update step is defined by the mapping Pr + P. which maps the latent space to the output dimension of size D.. We implement the Update step single fully connected linear layer.
Affine Coupling Layer Similarly to Liu et al. (2019) the output of the GNN layers models the scale sa c) and translation t(#2, e) functions. The scale and translation functions then applied to the unchanged split æ1 to produce the transformed graph z/.
œi @ exp (s(x2, + t(#2.
x2
The 22 is copied to 2/ unchanged. The o and x/ are then concatenated to form the transformed graph z which is passed the next transformation block.

---

Under review as a conference paper at ICLR 2023
Estimating Conditional and Marginal Densities The cGNF architecture allows for estimating both the conditional and marginal densities within a single model. The conditional density p( estimated by merging the target graph z with the context graph c. Consequently, the output density becomes constrained by the context. By removing nodes from c, the associated conditioning variables removed from p(=: c), functionally conditioning only on subset of the possible nodes in c. Finally, if the context graph is empty, the model provides a marginal density p(æ).
NEW
Loss The standard optimization procedure for normalizing flows is maximize the log probability of the observed data z obtained through the inverse path (x + 2) (Fig. B Assuming are i.i.d. the task of the flow is to model (æ c) = II N p (æ; c2) where æ; are the 3D poses and the corresponding 2D observations. We thus define the loss as the negative log probability of pairs of observations and c.
postt. = In go c)) + detV [A((Zk-1,))
where 9o NV
the source distribution. We augment the training data by randomly re- moving context variables to simulate new observations with missing keypoints in The augmented observations contain 20%, 40%, 60% or 80% of all observable keypoints. For all 3D poses, we addi- tionally compute the prior loss, which expresses the likelihood of pose given that no 2D keypoints were observed.
Sprior = - lo 2)) + Σ In ve
Our overall loss function is thus the sum of the two partial losses.
FIX
L -= Cprior Lpost
The proposed training strategy and architecture formulate pose estimation as a zero-shot density estimation problem. The cGNF model is trained subset of possible observations and is required to evaluate previously unseen conditional densities. Such zero-shot capabilities are useful in reliably estimating occluded poses. The graph structure allows the cGNF to share information between nodes and as result allows modeling distributions with sets of conditioning variables that have not been seen before. We observe that eGNF can solve these zero-shot density estimation problems comparably to specialized conditional normalizing flow problems (sec B.3
NEW
Root Node 3D poses are relative to a root node (usually the pelvis). Hence, the root node's position deterministic. We, therefore, remove the root node and corresponding edges from the target graph x and represent it as a noo node-type r, which has features H E R and a message generation function s/, which is fully connected neural network with 100 units.
Graph Symmetries The human pose graph has symmetries, e.g. the left and right limbs are mirrored. We impose hierarchical structure on the nodes of the target graph æ. A node may have a parent and a child, for example, the elbow node is the child of the shoulder node and the parent of the wrist node. Messages passed from the parent to the child are forward messages generated by diz- and messages from the child to the parent are backward messages generated by Vare--
Occlusion Representation We use 2D keypoint positions published by Wehrbein et al. 2021) estimated using the HRNet model (Sun et al. 2019) and the provided Gaussian distribution fits for evaluating occluded keypoints. a keypoint is classified as occluded (2D detection or > 5px) its corresponding node is removed from the context graph. To adjust for the differences between the pose definitions used by HRNet and H36M we employ an embedding network using the SageConv architecture (Hamilton transforms the observed 2D keypoints into a 10-dimensional embedding vector for each of the key- points. Additional implementation details of the architecture are given in the appendix (sec. B.
al. 2017) with learnable adjacency matrix. The embedding network

---

Under review as a conference paper at ICLR 2023
a
b
K
K.
C
Human 3.6M
Occluded bints
1.0
cGNF ECE 0.08
cGNF ECE 0.07
Frequency (a(g)
0.8
ProMPE 0.18
ProHPE 0.26
0.6
perfectly calibrated
0.4
0.0
0.4
1.0
Quantile
Figure 4: a) Hypotheses generated by the cGNF (gray) vs the ground truth pose (blue). Original im- age is shown to the right. b) Example of samples from the prior learned by the cGNE. e) Calibration of the conditional density. Comparison of the frequency that the distance of the ground truth from the median pose is within given quantile. Median calibration curves for model (cGNF, orange) and /Wehrbiin et al. (202T) (ProHPE, gray). Left shows calibration curves for the whole Human 3.6M test set and right for only the occluded joints.
5
LIFTING HUMAN3.6N
Data We use the Human3.66 66 Dataset (H36M) on the academic use only license (Catalin lonescu, 2011 Tonescu al. 2014) which is the largest dataset for 3D human pose estimation. It consists of tuples of 2D images, 2D poses, and 3D poses for 7 professional actors performing 15 different activ- ities captured with 4 cameras. Accurate 3D positions
obtained from 10 motion capture cameras and markers placed on the subjects. For evaluation, we additionally use the Human 3.6M Ambigu- (H36MA) dataset introduced by Wehrbein al. (202). H36MA is subset of the H36M dataset containing only ambiguous poses from subjects 9 and 11. pose is defined as ambiguous when the 2D keypoint detector highly uncertain about at least one of the keypoints.
Evaluation We evaluate the model on every 64th frame of subjects and 11 and the H36MA subset. We compare model's performance prior work on minMPJPE and ECE using 200 samples (Table Ty. As expected from the observations made in section our method underperforms on minMPJJPE but significantly outperforms on ECE (Fig. 4b). We further compare our method to
NEW
Kolotouros et al. (2021) which utilizes similar likelihood- based loss and normalizing flow archi- tecture, but does not account for occlusions and does not utilize graph inductive biases (Table As predict in section 3 we find that Kolotouros et al. (2021) is well-calibrated. We show that cGNF outperforms Kolotouros et al. (221)) even though fewer samples are used and remains comparably well-calibrated. Samples from the posterior and prior shown in figure 4 and b. Additional amples are included in the appendix (posterior samples Fig. S3 prior samples Fig. SS4. We further compare model performance in scenarios where other models exhibit overconfidence (Fig. and (56) and explore failure cases (Fig. S7
NEW
Performance individual occluded joints The poses contained in H36MA not only oc- cluded but also generally more difficult than the average pose in H36M. Therefore, we propose to evaluate the performance on solely the occluded joints instead of the whole poses. We report these errors in table [ (Occluded, where we show that our method outperforms the competing methods by significant margin on both minMPJPE and ECE. Thus, this shows that our model is able to learn posterior distribution that is more calibrated than previous methods and is able to outperform prior methods on mPEMPJPE for the occluded joints.
8

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of the cGNF model state-of-the-art methods for multi-hypotthsis pose estimation using expected calibration error (ECE) and minimum mean per joint position error (minMP.JPE) between the ground truth 3D pose and N hypotheses. Best model row is printed in bold font. Reporting the mean across the outcomes of 3 different seeds and the standard deviation (SD). For ECE the SD smaller than 0.001 in all cases. Thus, we do not report the SD value for ECE in this table. For all the metrics lower is better. We underlined the results that we did not compute instead used the originally reported value. With we mark results which used ground truth 2D keypoints and not estimated 2D keypointss and these not included in the comparison.
Method
H36M (mm)
H36MA (mm)
ECE
Occluded (mm)
ECE
Params
Martinez 2017
62.9
4,288,587
Zhao et 2019
60.8
434,703
(minMPJPP))
54.8 0.002
(84%)
200
4,288,572
Gaussian (NLL)
60.1 = 0.002
0.07 (14%)
200
4,288,572
Kolotouros
2021
(GT)
37.1
0.07 (14%)
200
25,475,744
Lee 2019
52.7
4,498,682
Sharma ot al.
2019
46.7
78.3
0.36 (72%)
200
9, 9,100,080
Oikarinen ct m
2020
46.2
0,16 (32%)
200
440,357
Wehrbein
2021
44,3
71.0
(36%)
51.1 = 0.13
0,26 (52%)
200
1,157,176
cGNF
57.5 = 0.06
87.3 +: 0.13
0.08 (16%)
47.0 =: 0.18
0.07 (14%)
200
852,546
cGNF Example
53.0 0.06
79.3 0.05
0.08 (16%)
41.8 0.04
0.03 (6%)
200
852,5466
eGNF slarge Cxample
48.5
72.6
0.23 (46%)
0.07 (14%)
200
8,318,741
Improving minMPJPE performance and the effect on calibration We can incorpo- rate a couple of additional steps to improve the minMPJPE performance. We introduce an additional loss term
Table 2: Comparison of methods on the Procrustes-Aligned minMPPPE metric.
Method
PA-MPJPE
Kolotouros et al. 2021
42.4 mm
4095
Lsampte MPJPE (x= (O,).
cGNF w Lsample
40.7 mm
200
that encourages the model to predict the ground truth pose. The sample-based loss term is added to the vanilla loss (equation 2 with scaling coefficient Asample 0.1. Analogously Kolotouros et al. (2021) sample pose from the mode of the source distribution and minimize the minMPJPP between the sampled pose and the ground truth pose. This additional loss term shown to improve the minMP performance. At the original model capacity, the minMP and calibration performance show improvement. However, while the model performance on minMP.PPF increases further with model capacity, cali- bration decreases significantly. We compare the performances in table Additional model capacity evaluations are made sec. B.4
NEW
FIX
CONCLUSION
In this study, we explored the problem of miscalibration in multi-hypothesis 3D pose estimation. Obtaining calibrated density estimates is important for safety-critical applications, such as health- care or autonomous driving. Here we provide evidence that a focus on sample-based metrics for multi-hypothesis 3D pose estimation (e.g. minMPJPE) can lead miscalibrated distributions. We propose a flexible model which can be trained to minimize the negative log-likelihood loss and show that, unlike previous methods, model can learn well-calibrated posterior distribution and
NEW
outperforms comparably calibrated methods on min JPF... However, in particularly ambiguous situations, i.c. for the occluded joints, we show that our model outperforms the Mtate-ffthe-ant on MinMPJPE while maintaining a well-calibrated distribution. We believe that our findings will be useful for future work in identifying and mitigating miscalibration in multi-hp helti-hhyothesis pose estima- tion and will lead to more robust and safer applications of multi- hypothesis pose estimation.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our code for reproducing the results is open-sourced at httts::/ ctmptuu.ooom//oooooooooooooo com. XXXX also include code to reproduce the results we obtained for other works. Experimental logs and downloadable pretrained models are fully available at lhttpss//wandb: ai/xxxX
We NEW
REFERENCES
Abien Fred Agarap. Deep learning using rectified
linear units (relu). arXiv preprint
arXiv:1803.08375. 2018.
A Agarwal and B Triggs. 3D human pose from silhouettes by relevance vector regression. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pp. II-II, June 2004.
Benjamin Biggs, David Novotny, Sebastien Ehrhardt, Hanbyul Joo, Ben Graham, and Andrea Vedaldi. 3d multi-bodies: Fitting sets of plausible 3d human models to ambiguous image data. Adv. Neural Inf. Process. Syst. 33:20496 20507, 2020.
Liefeng Bo, Cristian Sminchisescu, Atul Kanaujia, and Dimitris Metaxas. Fast algorithms for large scale conditional 3D prediction. In 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, June 2008.
Glenn W Brier. Verification of forecasts expressed in terms of probability. Mon. Weather Rev., 78 (1):1-3, January 1950.
Cristian Sminchisescu Catalin lonescu, Fuxin Li. Latent structured models for human pose estima- tion. In International Conference on Computer Vision, 2011.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society. Series (The Statistician), 32(1/)):12-22, 1983.
Laurent Dinh, Jascha Sock-Dicks and Samy Bengio. Density estimation using real nvp, 2016.
URL http::tt:://// wttp:///avvvvvviiii org/ abs/1bl5. 08803
C A E Goodhart. Problems of monetary management: The UK experience. In C A E Goodhart (ed.), Monetary Theory and Practice: The UK Experience, pp. 91-121. Macmillan Education UK, London, 1975.
M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In Pro- ceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp- 729-734 vol. 2, 2005. doi: 10.1 N,.109//CNN
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. June 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovee. Inductive representation learn- ing large graphs. tems. 30, 2017.
Advances in Neural Information Processing Syy-
URL https://papers.rrt 7ebeea Awstrttttbsttttttt...... html
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference and prediction. Springer, edition, 2009. URL http::/www--st wtw.//www.ttt..t...
rraanorddedu/-tibs.. EtteElllettttteerreeeeeea
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions Pattern Analysis and Machine Intelligence, 2014.
Ehsan Jahangiri and Alan L Yuille. Generating multiple diverse hypotheses for human 3D pose consistent with 2D joint detections. February 2017.
Diederik Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions. Jul 2018. URL //ct::///ae /tte://axxiii wttp://wwww...//////////.. 03039
10

---

Under review as a conference paper at ICLR 2023
Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic mod- cling for human mesh recovery. In ICCV, 2021.
Chen Li and Gim Hee Lee. Generating multiple hypotheses for 3d human pose estimation with mixture density network. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Chen Li and Gim Hee Lee. Weakly supervised generative network for multiple 3d human pose hypotheses. BMVC, 2020.
Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, and Luc Van Gool. Mhformer: Multi-hypothesis transformer for 3d human pose estimation. arXiv /cs.CV], Nov 2021. doi: 10.48550/ARXIV. 2111.12707. URL http::::/aax...oo/raaba/////////
Jenny Liu, Aviral Kumar, Jimmy Jamie Kiros, and Kevin Swersky. Graph normalizing flows. Ad- vances in Neural Information Processing Systems, 32, 2019. URL httpttttt/pppperrree Abstract.
CC/
html
Julieta Martinez, Rayat Hossain, Javier Romero, and James J. Little. A simple yet effective baseline for 3d human pose estimation. In ICCV, 2017.
Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3D human pose estimation in the wild using improved CNN supervision. November 2016.
Greg Mori and Jitendra Malik. Recovering 3D human body configurations using shape contexts. IEEE Trans. Puttern Anal. Mach. Intell., 28(7):1052-1062 July 2006.
Mahdi Pakdaman Nacini, Gregory F Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Nitth AAAI Conference on Artificial Intelligence, AAAI' 15, pp.
AAAI Press, January 2015.
Alexandru Niculescu- Mizil and Rich Caruana. Predicting good probabilities with supervised learn- ing. In Proceedings of the 22nd international conference on Machine learning, ICML '05, pp. 625-632, New York, NY, USA, August 2005. Association for Computing Machinery.
Tuomas P. Oikarinen, Daniel C. Hannah, and Sohrob Kazerounian. Graphmdn: Leveraging graph structure and deep learning to solve inverse problems. arXiv Lcs, Oct 2020. doi: 10.48550/ ARXIV.2010.13668. URL http//tttp://xxx..ooggg/g abs 2010 13668
Georgios Pavlakos. Xiaowei Zhou, Konstantinos Derpanis, and Kostas Daniilidis, Coarse-to-Fine volumetric prediction for Single-Image 3D human pose. November 2016.
Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Danilidis. Harvesting multiple views for marker-less 3d human pose annotations. In CVPR, 2017.
Georgios pavlakos, Luyang Zhu, Xiaowei Zhou, and Kostas Daniilidis. Learning to estimate 3D human pose and shape from single color image. May 2018
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. The Semantic Web, pp. 593-607, 2018. doi: (00007978--331999441111 com/ htmm//chater/rin 100
URL https: wwr.inkk.pringer...
1007/978 wttpchteee...1///77111.11
319 93417 -004111733886
Saurabh Sharma, Pavan Teja Varigonda, Prashast Bindal, Abhishek Sharma, and Arjun Jain. Monoc- ular 3d human pose estimation by generation and ordinal ranking. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. Distribution calibration for regression. arXiv [stat.MLJ, May 2019.
Marilyn Strathern. 'improving ratings': audit the british university system. Eur. Rev. 5(3):305- 321, July 1997.
11

---

Under review as a conference paper at ICLR 2023
Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolutionn representation learning for human pose estimation. In CVPR, 2019.
Xiao Sun, Jiaxiang Shang, Shuang Liang, and Yichen Wei. Compositional human pose regression. In The IEEE International Conference on Computer Vision (ICCV), volume 2, 2017.
G Tabak. A family of non-parametric density estimation algorithms. Commun, Pure Appl. Math., 000:0001-0020, 2000.
Jonathan Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training of convolutional network and graphical model for human pose estimation. June 2014.
Alexander Toshev and Christian Szegedy. DeepPose: Human pose estimation via deep neural net- works. December 2013.
Bastian Wandt, Marco Rudolph. Petrissa Zell. Helge Rhodin, and Bodo Rosenhahn. Canonpose: Self-supervised monocular 3d human pose estimation in the wild. In Computer Vision and Pattern Recognition (CVPR), June 2021.
Jinbao Wang, Shujie Tan, Xiantong Zhen, Shuo Xu, Feng Zheng, Zhenyu He, and Ling Shao. Deep 3D human pose estimation: A review. Comput. Vis. Image Underst., 210: 103225, September 2021.
Tom Wehrbein, Marco Rudolph, Bodo Rosenhahn, and Bastian Wandt. Probabilistic monocular 3d human pose estimation with normalizing flows. In International Conference on Computer Vision (ICCV), October 2021.
Christina Winkler, Daniel Worrall, Emiel Hoogeboom, and Max Welling, Learning likelihoods with conditional normalizing flows. Nov 2019. URL http:://txxxi.///////// wwwpaaiwww.....abaoaao abs 1t0.a//...00000000.0202
Yiqi Wu, Shichao Ma, Dejun Zhang. Weilun Huang, and Yilin Chen. An improved mixture density network for 3D human pose estimation with ordinal ranking. Sensors, 22(13), July 2022.
Changgong Zhang, Fangneng Zhan, and Yuan Chang. Deep monocular 3d human pose estimation via cascaded dimension-lifting ArXiv, abs/2104.03520, 2021.
Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N. Metaxas. Semantic graph volutional networks for 3d human pose regression. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3425-3435 2019.
Zhiming Zou and Wei Tang. Modulated graph convolutional network for 3D human pose estimation. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) IEEE, October 2021.
A METRICS
A.1 MEAN PER JOINT POSITION ERROR
popular optimization metric is the MPJPE. While this metric especially popular in single-pose estimation methods, it has also been used in various forms in multi-hypothesis methods. Optimizing this metric causes the distribution dimensional distribution, the generalization to the multi-dimension samples x ~ p(z|c) from a data distribution given a particular context c, such as keypoints from a image, consider approximate distribution ((/c) supposed to reflect the uncertainty about w/c.
poses to overconfident. We show this for a simple one-
case is straightforward. Given
12

---

Under review as a conference paper at ICLR 2023
This below objective is equivalent to the mean position enror for single joint. Note that and # are conditionally independent given c, i... xL6fc. The objective can then be expanded as follows:
L = (( [(x - = Ee Ru [(x Me + Me =
E, Varlæ 22z 1/ ne)( Me]] + Egjc [[ mo)
indep. of
= const. - 2Ec
Erje [(x - Hc)l Ezjc 1( - [c)] + Eaje [ - Hc)P]
=0
const. + E. [Es]c [(t- 20
The expectation in the final line is non-negative and can be minimized by ((t(c) = 8(1 Me), ic. setting å = He and shrinking the variance to zero. This means that g would be extremely overconfident.
A.2 minMPJPE CONVERGES TO THE CORRECT MEAN
Consider ID samples x" from a data distribution p(z) and an approximate Gaussian distribution q(x)) with parameters y and We sample N hypotheses from g(xr) and minimize the minMPJPR objective:
minMPJPE = Eq(x)) Ep(()) minn(x* - - 02.)²]]
Consider 2) as the sample which minimizes the expression for the j-th data sample v..
minMPJPE -Eg+)) [Epp)) [(x" - 1 ==
Thus the derivative can be computed be
minMPJPE = -2)g(=) (Ep()) (x" 1 o23 =0
= Ep(x) [x"] - By(z) [zj]
Simulations indicate that (x()) [+]] can be approximated by sigmoid function
(=)=) [2]]
S (Ep(a) [x=] 14) N)a
where C(a, .) scalar scaling value dependent a and the number of hypotheses. Thus the root of the derivative can be computed to be:
/ = Ep(a)) [x*]
A.3 IMPACT OF CENTER TENDENCY MEASURE ON EXPECTED CALIBRATION ERROR
The choice of center tendency measure should be considered when computing the expected cali- bration error. Therefore on a subset the models presented in table we compare the effect of choosing different reference points. 1) The median of the samples 2) the mean of the samples and 3) the mode of the samples. We showcase the results in table 3 We observe that the use of median in contrast to mean has little no effect on the computation ECE. Using the mode as a reference point results in generally smaller values of ECE. Finally, it can be observed that regardless of the reference point type our eGNF model remains better calibrated than the other methods.
NEW
B CONDITIONAL GRAPH NORMALIZING FLOW
B.
ARCHITECTURE DETAILS
The cGNF model consists of 10 flow layers. Each flow layer fx consists of two GNN layers each performing message-passing step each defined in eq. equation 4. In the first GNN layer each
13

---

GNN layer, the message generation functions e are single layer fully connected neural networks with 100 units and ReI.U activation, The Update is neural network layer with 3 output units. In Under review as a conference paper at ICLR 2023
Supplementary Table 3: Comparison of different reference points definitions the resulting ECE score. In bold we mark the method that under the particular reference point has the lowest ECE.
Method
Median
Mean
Mode
Sharma al. 2019)
0.36
0.36
0.14
Wehrbein et al. (202)
0.18
0.18
0.08
cGNF (ours)
0.08
0.09
0.04
message generation function oن is single layer fully-connected neural network with 100 umits and ReLU activation (Agarap 2018). All the messages to a node are summed together resulting in the output of the Message in eq. equation 4 Then the Update step takes the message output as its input to a single-layer fully connected neural network with 100 units and linear activation. The context is transformed via the to 100 dimensions and passed to the next GNN layer. In the next
the next flow layer of the original context graph is used and not the transformed context.
B.2 TRAINING DETAILS
We train the model subjects 5, 6, 7, and 8 on every 4th frame. We reduce the learning rate plateau with an initial learning rate 0.001 and patience of 10 steps reducing the learning rate by factor of 10. Training is stopped after the 3rd decrease the learning rate or 200 epochs. The model was trained on a single Nvidia Tesla V100 GPU, for about 6 days.
FIX
B.3 SERO-SHOT DENSITY ESTIMATION
We evaluate cGNF's zero-shot capability to estimate a previously unseen conditional density. We simulated 50 different triple pendulums with initial velocities sampled from normal distribution V (0, 10) for 25 timesteps each. Each pendulum was constructed from nodes connected in chain. The zeroth node was fixed and the remaining 21, 32 and I3 were freely moving. The nodes were observed with C; = I; + with ~ (0, 5 10-2). On this dataset, we trained models. CNF trained to estimate the density when all positions are observed p(æ: c e3) II. A CNF trained on a density where only one node is observed p(: e1) III. A cGNF trained on the densities where at most 2 nodes are observed i.. the cGNF never sees examples of p(x C1, C2, C3)
To test zero-shot capabilities we compare the performances of these 3 models on p(z C1, C22 C3, Model (CNF) is used reference for estimating this distribution when p(x C1, C2 03) is dis- tribution. Model II (CNF) is used to reference a model which cannot zero-shot estimate densities as is out of distribution. Model III (cGNF) shows that our model can zero-shot estimate previously unseen conditional density (Fig. ST)..
B.4 CONSEQUENCES OF MODEL SCALE
We explore the effect increasing the number of parameters of the model. We train sizes of models: 1) small with 852 546 parameters, 2) large with 3 301 546 parameters, and 3) xlarge with 8 318 741 parameters. The individual architectures were found by architecture search. We observe that the size increases the performance of cGNF applied to the lifting task improves decreasing the gap to the state-of-the-art methods. The performance further improves outperforming the state- of-the-art method occluded joints. However, the improvement in performance comes cost of calibration (Fig. S2
14

---

Under review as a conference paper at ICLR 2023
CNF in distribution
CNF out of distribution
cGNF zero-shot
2
2
X1
1-
1-
X2
X3
0
0-
-1-
-1-
-2-
-3
-3
-3
-1.0
-0.5
0.5
-1.0
-0.5
0.0
1.0
-1.0
-0.5
0.5
1.0
Supplementary Figure S1: Zero-shot capabilities of the cGNF model. Density estimates for 3 mod- els: 1) CNF trained on p(z|c1 C22 c3) 2) CNF trained on p( 3) cGNF which has never seen p(z|c), C2 0z). The black points represent the true positions of the triple pendulum, and the orange, magenta and cyan represent samples for each node T1, T2, Fg respectively.
b
Model Scale ECE
Model Scale minMPJPE
0.4
90
H36M
H36MA
80
Occl.
Best MPIPE (mm)
0.3
Wehrbein al.
70
Sharma et al.
ECE
0.2
60
Li et al.
50
0.1
40
0.0
30
0.75M
3M # Params
10M
Supplementary Figure S2: minMPJPE and ECE across different model sizes. a) shows the effect of model scaling on calibration. b) shows the effect of model scale on accuracy. Performance on both metrics compared to prior methods at their respective model sizes.
15

---

Under review as a conference paper at ICLR 2023
Supplementary Figure S3: Examples of samples from the posterior distribution learned by the cGNF (gray) vs the ground truth pose (blue). Pink points show the 50 sampled hypotheses for the right wrist positions. These examples are non-cherry picked and generated for subjects 59 and S11 from the test dataset.
16

---

Under review as a conference paper at ICLR 2023
Supplementary Figure S4: Examples of non-cherry picked samples from the prior learned by the cGNF. Each are generated by randomly sampling a latent z A( I) and inverting to the pose space x fz, ()) without any context c. For each pose images are shown for two rotations.
17

---

Under review as a conference paper at ICLR 2023
Unoccluded
Occluded
Foot
Hip
Wrist
Shoulder
Knee
Spine
Elbow
Supplementary Figure S5: Examples of the effect of artificially occluding joint. Free joints like the foot or wrist show a high increase in variance, while internal joints like the shoulder and hip show smaller changes in variance. Gray poses mark samples from the cGNF and blue poses represent the ground truth 3D poses. Yellow points mark the sampled positions of the occluded joint.
18

---

Under review as a conference paper at ICLR 2023
Wehrbein et al.
Ours
Supplementary Figure S6: Examples showcasing the consequences of overconfident distributions. Cases were artificially generated by shifting the 2d input and increasing the detection variance such that the joint classified as occluded. Comparison between Wehrbein et al. failure cases where overconfident and wrong distributions were predicted and ours which produces well-calibrated distribution. Gray poses mark samples from the eGNF and blue and pink poses represent the ground truth 3D poses. Yellow points mark the sampled positions of the joint of interest.
19

---

Under review as a conference paper at ICLR 2023
2D Ground Truth
2D Predicted
3D Predicted
Supplementary Figure S7: Examples showing failure cases. For each of these poses, the minMPJPE exceeds 100 mm. In our experience, the most common cause of failure over- confident and incorrect 2D detections. Gray poses mark samples from the cGNF and blue poses represent the ground truth 3D poses.
20


Reviews:
[{"review_id": "gqTpYQI1POo", "reviewer": "Reviewer_Sgm5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a method to lift 2D human body keypoints to 3D by estimating the distribution of 3D joint locations. Special care is taken to ensure that, apart from the correct estimation of the mean, the estimated distributions are also well calibrated: It is argued and experimentally shown with toy examples that sample-based metrics for multi-hypothesis generation lead to overconfident distribution with variance lower than the true one. To mitigate this, a normalizing-flow based approach is proposed that takes advantage of the natural graph structure of the human kinematics. The resulting approach regresses the mode of the estimated distribution with small accuracy loss compared to current SoTA methods, while being better calibrated.", "summary_of_the_review": "After the issues raised by the other reviewers and the response of the authors, I think this work is not yet ready for acceptance in ICLR. Although there is a motivation for the problem, it is not clear whether the proposed approach tackles it explicitly.", "strength_and_weaknesses": "This work is generally well presented, and the results are convincing.\n\nOne issue regarding the motivation of the work is the actual need for a well-calibrated method, especially if there is a trade-off with respect to accuracy: One may favor over-confident and more accurate models over (slightly) less accurate ones that are well-calibrated. It is perhaps useful to \"know when you don't know\", but arguably it is more useful to have more accurate estimations on average.\n\nAnother issue that may not be as important is the use of isotropic distributions: for the problem at hand, arguably the uncertainty of depth is possibly higher than that for the other two dimensions. This in turn may skew the estimation of the variance when assuming isotropic behavior.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X_ic2yrZNp", "reviewer": "Reviewer_F59S", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper reveals an interesting finding that minMPJPE metric would result in miscalibrated distribution in a 2D-to-3D lifting problem, which is particularly unsuitable for using MPJPE to evaluate those distribution-based methods that generate multiple hypotheses to solve the ambiguity. It provides a toy example that shows optimizing minMPJPE would have a larger ECE than NLL and lead to overconfident distribution, while a good NLL may not guarantee excellent MPJPE, but helps to calibrate the distribution. To this end, they propose cGNFs and train it by likelihood maximization to learn the conditional 3D human pose distribution. Together with optimizing minMPJPE, their method would achieve comparable performance to SOTA. From experimental results, their method has better ECE performance on H36M, and an obvious boost in MPJPE can be seen on occluded samples.  ", "summary_of_the_review": "This paper starts with a nice motivation, but the writing and presentation are suboptimal. This paper claims the necessity of using ECE to evaluate the regression model, but I think Sec.3 is not solid enough to let me believe the way they formulate ECE is correct, which further affects the credibility of comparison results in Sec.5. I would consider changing my rating If the above comments are addressed.", "strength_and_weaknesses": "Strength\n* This paper starts with good motivation and provides a toy example to provide a possible explanation about the miscalibration caused by minMPJPE.\n* The performance of the proposed method is impressing on occluded samples by both MPJPE and ECE metrics.\n\nWeaknesses\n* The introduction in Sec.3 is hard to follow. I think this section is significant for helping construct a basic understanding of how ECE is used in 3D pose regression. But I fail to read it when I came to Sec.3.1. The authors should explain why using median is reasonable. Why not use other statistics, such as mean? If I understand correctly, this cumulative distribution function should give the probability of being less than a certain distance. However, I think the probability of the distance between the ground truth pose and the median is what we need. In Figure 1, why the model is underconfident if the optimal variance is larger than 0.5?   \n\n* The comparison with Biggs et al. (2020), Kolotouros et al. (2021), Oikarinen et al., (2020) should be added. Biggs et al. (2020) and Kolotouros et al. (2021) share a similar idea with this paper that simultaneously optimizes likelihood and joint distance. Oikarinen et al., (2020) use 2D pose as input and train with GNN. I would suggest the authors add more discussions with these papers to highlight the differences and explain the advantages of cGNFs. These paper also publicize their code with accessible pretrained models, it would be better if the authors provide experimental comparisons.\n\n* Although the good performance could be seen on ECE metric, the loss on MPJPE metric is also noticeable. Through the measurement of the article design, it seems that this method has made a great contribution to improving the credibility of the model. However, the results also show that certain gains on ECE would result in a loss on MPJPE. It’s hard to say whether such consequences are acceptable to us, especially when the loss is not subtle (7.7mm increase on MPJPE comparing cGNF w L_sample and Wehrbein et al. (2021)).\n\n[1] Biggs, Benjamin, et al. 3D Multi-bodies: Fitting Sets of Plausible 3D Human Models to Ambiguous Image Data, 2020\n\n[2] Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic modeling\nfor human mesh recovery, 2021\n\n[3] Tuomas P. Oikarinen, Daniel C. Hannah, and Sohrob Kazerounian. Graphmdn: Leveraging graph\nstructure and deep learning to solve inverse problems, 2020", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HjUKfVtI2gt", "reviewer": "Reviewer_AasC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work focuses on two main aspects of lifting 2D pose into 3D: 1) proving that the popular MPJPE metric is causing models to be overconfident and not accounting for uncertainty, and proposing a new metric in pose estimation that also measures calibration, ECE; 2) proposing a new method, cGNF conditioned Graph Normalizing Flows, that performs comparably in the MPJPE spectrum while outperforming other methods in ECE.", "summary_of_the_review": "I find the analysis presented in the paper novel and a good contribution to the community, while the proposed model is less so. There are a few remaining questions that I would like to discuss the authors and am open to more clarifications. ", "strength_and_weaknesses": "## Strength\n\n- Analysis of calibration and the metric minMPJPE: this work conducts a thorough analysis of the issues with the popular minMPJPE metric. Namely, probabilistic methods trained to minimize MPJPE tend to collapse to the mean and not account for the uncertainty (overconfident). By approximating the uncertainty of the pose by using model prediction error and uncertainty of the samples by using deviation from the median, the analysis shows that directly utilizing MPJPE as a loss/metric will cause the methods to produce a distribution that has high calibration error.\n    - The authors conducted a series of well-thought-out toy examples to showcase this phenomenon.\n- The proposed cGNF model achieves comparable results against SOTA method in terms of MPJPE while also producing well-calibrated pose distribution that captures the uncertainty of the input 2D pose.\n- This work conducts a comprehensive experimental analysis of state-of-the-art methods and showcased the strength of the proposed method. The paper is also clear and well-written.\n\n## Weakness\n\n- While the proposed shortcoming of MPJPE and ECE metric makes intuitive sense, I find the proposed method is quite disconnected from the main motivation. It is hard for me to find how the design choices made for cGNF relate to a better measure of the underlying distribution. The proposed “training using subset of observation” is close to a masking strategy and the training loss is a fairly standard NLL loss for normalizing flow in pose estimation [2]. As a result, I could not make a connection between the objective of obtaining a better-clibrated model and the actual proposed method (is the innovation in the architecture? I could not make a connection there either). While I find the analysis interesting and well-designed and quantifies a known pose estimation issue well (that SOTA methods often do not measure uncertainty well), the method does not seem to draw insight from it.\n    - The claimed to estimate both conditional (which I see) and marginal (which I do not see) using the cGNF model needs to be further explained.\n    - I also do not see how this is a “zero-shot density estimation problem” and how randomly using a subset to train can lead to this. (It could be me not understanding it and if the authors could further elaborate on this I could consider raising the score.).\n- While the paper focuses on quantifying uncertainty and occlusion in pose estimation, few examples and results were actually shown showcasing the strength of the model. It would significantly strengthen the claim if **extensive** visual examples could be shown the benefit of the model (e.g. uncertain 2D keypoints actually correspond to the more spread-out hypothesis, and, equally important, that 2D keypoints with little ambiguity leads to a model that is closer to the mean).\n\n## Question to authors\n\n- During the analysis of the miscalibration behavior in using minMPJPE, the models’ samples’ deviation from the median is used to construct the error distribution. The difference between the ground truth and the median is then used to approximate the ground truth error.\n    - I am not sure how the second part approximates the actual uncertainty in the ground truth samples. Each ground truth sample $m$ has a unique uncertainty associated with it. For instance, occluded poses lead to more significant errors. In equation (3) the summation term lumps all of them together and forms a distribution. This amounts to measuring the uncertainty at a per-joint level ($\\epsilon_{m, k}$) and not per sample level. On the other hand, the uncertainty of the model is done at a per-sample model. I understand that it would be difficult to measure the uncertainty at a per-sample model for ground truth data, but the current model seems questionable.\n\n[1] Wehrbein, T., Rudolph, M., Rosenhahn, B., & Wandt, B. (2021). Probabilistic Monocular 3D Human Pose Estimation with Normalizing Flows. *2021 IEEE/CVF International Conference on Computer Vision (ICCV)*, 11179-11188.\n\n[2] Kolotouros, N., Pavlakos, G., Jayaraman, D., & Daniilidis, K. (2021). Probabilistic Modeling for Human Mesh Recovery. *2021 IEEE/CVF International Conference on Computer Vision (ICCV)*\n, 11585-11594.\n\nSmall error: right above page 3 equation (3), should it be $\\epsilon_{m, k}^*$?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2l5nu3MqtM7", "reviewer": "Reviewer_8uoB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper raises an interesting problem, which is the miscalibrated distributions caused by using $minMPJPE$ to choose the best estimates in multi-hypothesis 3D pose estimation. To resolve the miscalibration, the paper uses the existing ECE metric proposed by Naeini et al., and proposes cGNF to learn the conditional distribution $p(x|c)$.", "summary_of_the_review": "This paper is novel and interesting, but is not ready for publication at ICLR at this stage due to limited analysis and experiments.", "strength_and_weaknesses": "Pros\n\n- The topic and the perspective of this paper are interesting. \n- The analysis and the plots demonstrate the miscalibration issue to some extent.\n\nCons\n\n- Except for the $minMPJPE$ metric, how about other factors, such as the model itself? For example, the distribution of the multiple hypotheses is depending on the design of the normalizing flow model, therefore, the miscalibration of the variance may also originate from the capacity of the model itself. \n- The logic is not so clear. I'm agreed that one of the miscalibration reasons is using $minMPJPE$. But how could cGNF solve this and does it have a theoretical guarantee? And there are already some works using normalizing flow as mentioned in the paper, what's the difference between cGNF and [1][2]?\n- Experiments are too few. The goal of multi-hypothesis is to solve the ambiguity in monocular 3D pose estimation by approaching the real distribution, which is expected to improve the ability to generalize to a new scene. However, all the experiments are conducted on H36M, which is a quite restricted and constrained scenario. Therefore, could the method also solve the miscalibration issue on the unseen 3DHP dataset (which is a common practice to evaluate the generalization ability of the model in this field)?\n- The authors also mention that the uncertainty could not be corrected captured due to the miscalibration, but no experiments about the uncertainty are presented using the cGNF. Could cGNF provide correct uncertainty? Quantitative or quality results may be needed. \n- Writing is not so clear and consistent, some notations are not clearly explained, and some even have typos. E.g. the $H_c$ in Fig. 2 caption, is actually $\\bf{H}^{c}$ in the main text of Sec. 4, right? If wrong, please correct me.\n\n[1] Nikos Kolotouros, Georgios Pavlakos, Dinesh Jayaraman, and Kostas Daniilidis. Probabilistic modeling for human mesh recovery. In ICCV, 2021\n[2] Tom Wehrbein, Marco Rudolph, Bodo Rosenhahn, and Bastian Wandt. Probabilistic monocular 3d\nhuman pose estimation with normalizing flows. In International Conference on Computer Vision\n(ICCV), October 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject