ManiPose: Manifold-Constrained Multi-Hypothesis 3D Human Pose Estimation
Authors: Cédric Rommel
Victor Letzelter
Nermin Samet
Renaud Marlet
Matthieu Cord
Patrick Perez
Eduardo Valle
Venue: NeurIPS 2024 poster (2024)
Keywords: human pose estimation
depth ambiguity
multiple choice learning
Primary area: machine_vision
TL;DR: We prove previous 2D-to-3D human pose lifting methods suffer from topology inconsistencies invisible to standard evaluation metrics, and propose both new metrics and a new method circumventing these issues via constrained multiple hypotheses.

Abstract: We propose ManiPose, a manifold-constrained multi-hypothesis model for human-pose 2D-to-3D lifting. We provide theoretical and empirical evidence that, due to the depth ambiguity inherent to monocular 3D human pose estimation, traditional regression models suffer from pose-topology consistency issues, which standard evaluation metrics (MPJPE, P-MPJPE and PCK) fail to assess.  ManiPose addresses depth ambiguity by proposing multiple candidate 3D poses for each 2D input, each with its estimated plausibility. Unlike previous multi-hypothesis approaches, ManiPose forgoes generative models, greatly facilitating its training and usage. By constraining the outputs to lie on the human pose manifold, ManiPose guarantees the consistency of all hypothetical poses, in contrast to previous works. We showcase the performance of ManiPose on real-world datasets, where it outperforms state-of-the-art models in pose consistency by a large margin while being very competitive on the MPJPE metric.

ManiPose: Manifold-Constrained Multi-Hypothesis 3D Human Pose Estimation
Cédric Rommel'
Victor Letzelter!
Nermin Samet'
Renaud Marlet!:5
Matthieu Cord
Patrick Pérez!
Eduardo Valle 1.4
Valeo.ai, Paris, France Sorbonne Université, Paris, France
LTCI, Télécom Paris, Institut Polytechnique de Paris, France
4 Recod.ai Lab, School of Electrical and Computing Engineering, University of Campinas, Brazil LIGM, Ecole des Ponts, Uniy Gustave Eiffel, CNRS, Marne-la- Vallee, France
Abstract
We propose ManiPose, cuunffold-consrrainee multi hypothesis model for human- pose 2D-to-3D lifting. We provide theoretical and empirical evidence that, due to the depth ambiguity inherent to monocular 3D human pose estimation, traditional regression models suffer from pose-topology consistency issues, which standard evaluation metrics (MPJPE, P-MPJPE and PCK) fail to assess. ManiPose addresses depth ambiguity by proposing multiple candidate 3D poses for each 2D input, each with its estimated plausibility. Unlike previous multi-hypothesis approaches, ManiPose forgoes generative models, greatly facilitating its training and usage. By constraining the outputs to lie the human pose manifold, ManiPose guarantees the consistency of all hypothetical poses, in contrast to previous works. We show- case the performance of ManiPose on real-world datasets, where it outperforms state-of-the-as models in pose consistency by a large margin while being very competitive on the MPJPE metric.
Introduction
We propose ManiPose, novel approach for human-pose 2D-to-3D lifting. ManiPose directly addresses the depth ambiguity inherent to monocular 3D human pose estimation by being both multi- hypothesis and regression-barte based methods. Unlike previous multi-hypothesis approaches, ManiPose forgoes the use of costly generative models, while still estimating the plausibility of each hypothesis.
thus avoiding pose consistency issues, which plague traditional
Monocular 3D human pose estimation (HPE) is a challenging learning problem that aims to predict 3D human poses given image video from single camera. Often, the problem is split into two successive steps: first 2D human pose estimation, then 2D-to-3D lifting. Such separation is favorable because 2D-HPE much more mature, leading better overall results. Due to depth ambiguity and occlusions, 2D-to-3D lifting is intrinsically ill-posed: multiple 3D poses correspond to the same projection observed in 2D. Despite that, the field has experienced fast developments, with substantial improvements in terms of mean-pers jor-jottr--o prediction error (MPJPE) and derived metrics (e.g., P-MPJPE, PCK) [52, 53, 42, 47].
per-jiiitt-rediiction
However, recent studies [49, 12, 40] noted that poses predicted by state-of-the-art models fail to respect basic invariances of human morphology, such bilateral sagittal symmetry, constant length across time of rigid body segments connecting the joints. Not only do we address those concerns with ManiPose (see Fig. 1), but also provide theoretical elements clarifying the cause of those issues. We show in particular that pose consistency and traditional performance metrics (such as MPJPE)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Constraints
unsonstmined
Wehrbein et
bad
Figure 1: Optimizing both 3D position and pose consistency requires combin- ing constraints and multiple hypothe- ses. Results from Tables 2 and 4. Pre- vious unconstrained methods provide in- consistent poses (top). Regularization (MR) and disentanglement constraints improve consistency, but degrade joint position error (bottom-right). Ours is the only method that achieves both good joint error and consistency, thanks to combination of disentanglement and a few hypotheses (see circles sizes).
Hypatheses
12
3-5
good Inconsistency MPSCE (mm)
10-20
200
MsSTE
Shamaa et al.
DSDP
MHFamar
VideoPose3O
PoseFormet
MISTEE MR
mutt-hyyothesis
manifold constraints
Anstomy33
ManiPase (ours)
ManiPose wio MH
40
42
44
good Joint
(mm) bad
cannot be optimized simultancously by a standard regression model, because MPJPE ignores the topology the space of human poses, and traditional regression models imply unimodality, thus overlooking the inherently ambiguous nature of 3D-HPE.
Our contributions include:
ManiPose, novel, multi-hypothesis manifold-constrainee model for human-poss 2D-to- 3D lifting, which is able to estimate the plausibility of each hypothesis without resorting to costly generative models.
Theoretical insights that elucidate why traditional regression models associated with standard metrics such MPJPE fail enforce pose consistency.
Extensive empirical results, including comparison to strong baselines, evaluation on two challenging datasets (Human 3.6M and MPI-INT-3DHP), and ablations. ManiPose outper- forms state-of-the-art methods by a substantial margin in terms of pose consistency, while still beating them in the MPJPE metric. The ablations confirm the importance of both multiple hypotheses and of constraining the poses to their manifold.
The PyTorch [37] implementation of ManiPose and code used for all our experiments can be found at https://go //gi thub. com/ cedri wwwomee/mannnppooooeeeeeee
2 Related work
Regression-based 2D-to-3D pose lifttin. While 2D-to-3D human pose lifting was initially restricted to static frames [31, 3], the field embraced recurrent [13], convolutional [38] and graph neural networks [2, 55, 14, 51] to handle motion. Spatial-temporal transformers appear more recently [42, 53], including MixSTE [52], arguably becoming the state of the art. We adopt them in work. few previous works constrain predicted poses to respect human symmetries [50, 4], an idea we advance with novel constraint implementation, in multi-hypothesis setting.
SMPL-based methods. While 3D human pose lifting's objective is to predict 3D joint positions based on 2D keypoints, the neighboring field of human pose and shape reconstruction (HPSR) aims at estimating whole 3D body meshes from images. HPSR is hence more challenging than 3D-HPE, which explains why models are often larger, frame-based and more reliant on optimization-be based post-pros post-processing [16, 39, 46, 9]. Nonetheless, our work shares some ideas from this field. Indeed, modern HPSR methods often predict joint angles (and body shape parameters), which are fed to the pre-trained parametric model SMPL [29] to produce human body meshes, thus ensuring that limbs' sizes remain constant along a movement. Note, however, that these are also single-hypothesis regression methods and hence share the same caveats most 3D-HPE approaches.

---

Multi-hypothesis 3D-HPE. The intrinsic hapthaabbigutty of 3D-HPE led the community to inves- tigate multi-hyppp hypothsssss approaches, including Mixture Density Networks [25, 36, 1], variational autoencoders [44], normalizing flows [18, 49] and diffusion models [12, 6, 10]. Contrary to ours, those methods rely on generative model to sample 3D pose hypotheses conditioned on the input. A notable exception is MHFormer [27]. which, like ManiPose, is deterministic, but treats the hypotheses as intermediate representations to be aggregated at the final network layers, thus concluding with one-to-one 2D-to-3D mapping. We strive to avoid such injectivity and to preserve the multiple hypotheses, for reasons we will justify both empirically and theoretically the next sessions. Moreover, none of the previous multi-hyp sultiihhypothesis approaches constrain hypotheses to lie on the human pose manifold, thus failing to guarantee good pose consistency.
Multiple choice learning (MCL) [11] is simple approach for estimating multimodal distributions, suited for ambiguous tasks, using the winner-takes-al all loss. Adapted for deep learning by Lee et al. [20, 21]. it produces diverse predictors, each specialized in particular subset of the data distribution. MCL has proved its effectiveness in several computer vision tasks [41, 19, 33, 8, 30, 45], and first applied to 2D-HPE in [41]. Our work is the first employ MCL. for the 3D-HPE task, by leveraging recent innovations Letzelter et al. [22].
3 ManiPose
Hyp.
Legend:
Head
2D keypoints
Learne
Rotations Module
:
Pose
Not leerned
Segments lengih
Hyp.
Head K
Segments
Scores
Module
Figure 2: Overview of ManiPose. The rotations module predicts K possible sequences of segment rotations with their corresponding likelihoods (scores), while the segments module estimates the shared segment lengths. Hence, predicted poses are constrained to manifold defined by the estimated lengths, guaranteeing their consistency.
Following the previous state of the art, split 3D-HPE into two steps, first estimating J human 2D keypoints in the pixel space from a sequence of video frames [Xi...., .....,,, xy] E ==2xxxT and then lifting them 3D joint positions (p.....,pyyl E 33x We focus the second step (i.e., lifting) in the rest of the paper, assuming the availability of 2D keypoints x;. Our method aims to both ensure pose consistency and resolve depth ambiguity,
will discuss in the next section.
3.1 Constraining predictions to the pose manifold
Rationale. Human morphology prevents the joints from arbitrarily occupying the whole space. Instead, the poses within movement are restricted to manifold, reflecting the human skeleton's 's rigidity. If we knew the length of each segment connecting pairs of joints for a given subject, we could guarantee that the predicted poses lie on the correct pose manifold by only predicting the body part's rotations with respect to a reference skeleton. Since we not have access to ground-truch segment lengths in real use cases, we propose to predict them, thus disentangling the estimation of the reference lengths (fixed across time) from the estimation of the joint rotations (variable across time).
Disentangled representations. We constrain model predictions to lie on an estimated manifold by predicting parametrized disentangled transformations of a reference pose u E (RR3),, for which all segments have unit length. Namely, we propose to split the network into two parts (cf. Fig. 2):
3

---

1. Segments module, which predicts segment lengths E R /-1, ..saared by the T frames (time steps) of the imput sequence;
2. Rotations module, which predicts the rotation r = [1,,0, TT,J-1] E (Rd)JxT of each joint relative to their parent joint at each time step.
Rotations representation. We represent rotations using 6D continuous embeddings (i.e., d = 6). Compared quaternions axis-angles, those representations continuous and, hence, better learned by neural networks, as demonstrated by their proposers [54].
Pose decoding. To deliver pose predictions in (R3)JxT, the intermediate representations (s,r) must be decoded. We achieve that in three steps (cf. Fig. 3):
1. We scale the unit segments of the reference pose E (R))) using s, forming scaled reference pose u': u'y = "-6) + sy (u, u+(i) for < JJJJJ 1, where maps the index of joint to its parent's, s, if any-
2. For each time step 1 S t S T and joint 0 S < J. we convert the predicted rotation representations Tr,j into rotation matrices Rr. SO(3) (Algorithm 1).
3. We apply those rotation matrices Re, at each time step to the scaled reference pose u' using forward kinematics (Algorithm 2).
3.2 Multiple choice learning
ManiPose architecture. As explained in the introduction, the inherent depth ambiguity of pose lifting requires multiple hypotheses to conciliate pose consistency and MPJPE performance. To address this, we adopt the multiple choice learning (MCL) [21] framework, more precisely leveraging the resilient MCL approach as proposed by Letzelter er al. [22]. This methodology allows the estimation of conditional distributions for regression tasks, enabling our model to predict multiple plausible 3D poses for each 2D imput. Specifically, instead single rotation r, E (RR") per time step, ManiPose's rotations module predicts an intermediate representation é₁ E (R=") that feeds K linear heads (with weights W and WH),, each predicting its own rotation hypothesis rf E (IR") with corresponding likelihood 7f' E [o, 1). That is, for all < / < T, r W and 7. = a [7] TR of intermediate values
where the softmax function is applied to the vector 7, [5] - K W.
All rotation hypotheses are decoded together with the shared segment-length predictions s, result- ing in K hypothetical pose sequences B* (p?))= with corresponding likelihood sequences 7tt = (Tt) T called scores hereafter (Fig. 2).
Loss function. As in [22], ManiPose is trained with composite loss
L L L
(1)
The first term, Lwms is the ainner-takes-all loss [21]
Lw Lwu(((x),,) T
Σ min (x), kE|LKK]
where (x),,x,, p.) A J E |Ip. pt,, (x) and p (x) denotes the pose prediction time t using the 4h head. The second term, Lycore is the scoring loss
T
Seco (f(x), T
(3)
where H(---) ) is the cross-entropy, P: = (ft)) ( and
[8(pr, Po)la _ k arg min e (pf p.)
(4)
KE[LAA]

---

is the indicator function of the winner pose hypothesis, which is the closest to the ground truth. Eq. (3) the average entropy between target and predicted scores (x(x) E [o. 1] each time t.
Those losses are complementary. The winner-takes-all loss updates only the best predicted hypothesis, specializing each head on part of the data distribution [21]. The scoring loss allows the model to learn how likely each head is to winning, thus avoiding overconfidence of non- non-winner heads (cf. [19, 45]).
Conditional distribution estimation. As detailed in [22]. the model may be interpreted probabilisti- cally as multimodal conditional density estimator. More precisely, it models the distribution P ((p|x) of 3D poses conditioned on 2D poses as mixture of Dirac distributions:
K
P(pxx) 2 Σ (x))
(5)
Hence, the predicted conditional distribution has, at each predicted hypothesis p*, a peak whose likelihood is given by the predicted score y*. As described in Section 4, interpreting hypotheses and scores probabilistically enables us to handle depth ambiguity.
Formal analysis
ManiPose, as outlined in Section 3, is crafted to address the flaws inherent in unconstrained, single-hypothesis lifting-based 3D-HPE methods (see Fig. 1). This section illustrates that without ManiPose's critical components (multiple hypotheses and manifold constraint), it is im- possible to simultaneously minimize joint error and en- sure pose consistency (Section 4.1). To illustrate this, toy example within simplified 1D-to-2D framework is provided in Section 4.2.
Unit ref. pose
Soaled ref, pose u
8
Segmants
Scaling
Prodicted movement
th
Rotations Conversion
Forward Kinemalics
4.1 Single-hypothesis position-error minimization leads to inconsistent skeleton lengths
Figure 3: Pose decoder overview.
We formally highlight the limitations of unconstrained single-hypothesis 3D-HPE, justifyin our ap- proach, which combines consistency constraints and multiple hypotheses to resolve depth ambiguity.
Let = "ip',,...,''"... p'' E / be a human pose, defined by the Cartesian 3D coordinates of each of the J joints of predefined skeleton. Then, sequence of T poses of the same subject at increasing time steps t1 tr E IR forms a movement m = =Poo.... Pr] J3 Assuming bone length is fixed during movement (which is empirically verifiable human pose datasets), then the poses Pe of m must all lie on the same smooth manifold.
Proposition 4.1 (Human pose manifold). Assuming a rigid skeleton, all poses of a movement m [pe] lie on manifold M of dimension 2(J - -1):
VI E (1, (I,...,T), T} Pt M.
Proof sketch. (Detailed in Appendix B). Skeleton rigidity implies that, if i is joint connected to the root, then it lies on a 2D sphere 5 (0, ss,o) centered at the origin with fixed radius $ Another joint j linked to i has a position expressible by its spherical coordinates relative to í with fixed radius 8j.1- That implies an homeomorphism between the position pt.j of joint j and the direct product of spheres centered at the origin 52 (0, s2,0) x S2 (0, s),). By induction, one can show that p. lies on a subspace of (R3) which is homeomorphic to a product of spheres centered at the origin.
Proposition 4.1 implies that all poses predicted for video sequence should ideally lie on the same manifold M as the ground truth data, which is homeomorphic to the direct product of 2D unit spheres (S2)J-1 (cf. Appendix B). Crucially, we can further show that minimizing joint position error using a single-hypothesis model necessarily leads predicted poses lying outside the true manifold:
Proposition 4.2 (Inconsistency of MSE minimizer). With rigid skeleton and mild assumptions on the training distribution, predicted 3D poses minimizing the traditional mean squared error (MSE) loss lie outside the pose manifold M.
5

---

Proof sketch. (See Appendix B). Consider a skeleton with J joints, with (x, p), as pairs of cone sponding 2D inputs and 3D poses. Let the function L = (B)) compute the lengths of the segments in pose, which shall remain constant. On a dataset {(x,, },) drawn from the joint distribution of 2D and 3D poses, let the expected MSE of a traditional predictive model f be 1xx. [|]p f(x) [8]
Let the ideal model f be the one minimizing that expected MSE, which is the conditional expec- tation (x)
EBlp x]. Jensen inequality and the rigidity assumption imply that, for any joint j. ej (f* (x)) < s² where 5j is the true length of the segment associated with joint j. This shows that the poses predicted by f* violate the original segment length constraints, and thus, the original rigidity assumption.
Proposition 4.2 has the following implications:
1. Traditional unconstrained single-hypothese approaches are bound to predict inconsistent movements, where bone lengths may vary.
2. With a single hypothesis, models constrained to the manifold will always lose to uncon- strained models in terms of MPJPE performance (formalized Corollary B.
3. The only way of reaching both optimal MPJPE and consistency is through multiple hypothe- ses (formalized in Corollary B.3).
Therefore, the MPJPE metric (and its traditional extensions) is insufficient assess 3D-HPE, as it completely ignores pose consistency. Furthermore, we are able to prove in Appendix B.2 that multiple hypotheses (constrained or not) can always reach better joint position errors than models.
4.2 Insights to the formal argument on a simplified setting
We illustrate the argument of Section 4.1 with a simplified ID-to-2D setup. We further gener- alize this intuitive illustration to the 2D-to-3D setting in Appendix C of the supplementary.
Table 1: 1D-to-2D performance. Fig. 4-D setting, results averaged over five random seeds.
MPJPE
Distance circle
As in human pose lifting, we take root joint Jo as reference, fixed at (0, O). For a joint Jr, the problem amounts to predicting the 2D position (Ir, y), given its ID projection u = 2i assuming a constant distance s = 1 between them. This simplification ignores the camera perspective and considers the joints to be connected by a rigid segment as in the case of human poses.
Unconst. MLP
0.753
0.008
0.42 1 0.01
Constrained MLP
0.777
0.027
0.00 1 0.00
ManiPose
0.752 0.012
0.00 + 0.00
We train three different models with comparable architectures on two datasets {(x,, (Iz y.)) > sampled from the angular distributions represented in blue on Fig. 4. The models correspond to:
1. A 2-layer MLP (x) trained minimize the mean squared error between true (a, y) and predicted joint positions (ă, 0):
2. A constrained MLP of the same size (x), predicting the angle i instead of the joint position;
3. ManiPose: our constrained
model capable of predicting 2 possible angles (0*) )== with their corresponding likelihoods.
Fig. 4 shows that the traditional unconstrained tigll-hhypothesss approach (x) leads good results in an easy unimodal scenario (C), but fails when facing a more challenging bimodal distribution (D), leading to predictions outside the circle manifold, as depth ambiguity makes the lifting problem ill-posed. The single-hypothesn
constrained model (x) delivers predictions on the circle, at the cost of worse MPJPE performance than the unconstrained MLP. Such performance decrease is due the Euclidean topology of the MPJPE metric having its minimum outside the manifold (Fig. 4-B).
Crucially, this implies that the unconstrained predictions, with varying "bone lengths" (the circle radius). It also shows that models constrained to the manifold (circle) will always be outcompeted by unconstrained models on MPJPE performance.
models bound to make inconsistent
Predicting multiple hypotheses constrained to the circle, with their respective likelihoods (* in Fig. 4-B) allows escaping this dilemma, which is exactly what ManiPose does (A in Fig. 4-D). The

---

11
pixkk 0.87
XX)) 0.33
Inputs
Constr. MLP
Constr: MH min.
Outputs
Unconstr.
hyps.
GT probability
MSE minimizer
ManiPose scores
Figure 4: (A) ID-to-2D articulated pose lifting problem. (B) True MSE minimizers under a multi- modal distribution. One-to-one mappings cannot both reach optimal performance and stay the pose manifold (dashed circle). (C) Without depth ambiguity, unconstrained models are effective. (D) Ambiguity from multimodal distributions challenges both constrained and unconstrained models. Multi-hypothesis approaches can deliver an acceptable solution to the problem.
predicted hypotheses are all on the circle, contrary to the unconstrained MLP, and spread between the two distribution modes, unlike the constrained single-hypothesis method.
Moreover, the predicted scores (length of green lines) match the 2 and 1 3 ground-truti likelihoods of the two modes. Those advantages translate into perfect pose consistency and into comparable MPJPE performance with respect to the unconstrained MLP (Table 1).
3
5 Experiments
5.1 Experimental setup
Datasets. We evaluate our model on two 3D-HPE datasets. Human 3.6M [15] contains 3.6 million images of 7 actors performing 15 different indoor actions. It is the most widely used dataset for 3D-HPE. Following previous works [52, 27, 53,38), 38], we train on subjects S1, S5, S6, S7, S8, and test on subjects S9 and S11, adopting 17-joint skeleton (ef. Fig. 5). We employ pre- pre-trained CPN [5] to compute the input 2D keypoints, as in [38, 52]. MPI-INF-3DHP [32] also adopts a 17-joint skeleton, but, with fewer samples and containing both indoor and outdoor scenes, it is more challenging than Human 3.6M. We used ground-truti 2D keypoints for this dataset, as usually done [53, 4, 52].
Traditional evaluation metrics. The mean per-joint position error (MPJPE) is the usual performance metric for the datasets above, under different protocols, both reported in mm. In protocol #1, the root joint position is set as a reference, and the predicted root position is translated to 0. In protocol #2 (P-MPJPE), predictions are additionally Procrustes-corrected. For MPI-INF-3DHP. additional thresholded metrics derived from MPJPE are often reported, such as AUC (Area Under Curve) and PCK (Percentage Correct Keypoints) with a threshold at 150 mm, as explained in [32].
Pose consistency metries. MPJPE being insufficient assess pose consistency (Section 4), we further assess to which extent predicted skeletons are rigid by measuring the average standard deviations of segment lengths across time in predicted action sequences:
J-1
MPSCE A
(L. - 83,(1)2
(7)
with 8t.j. |lÉr,j - Pr,lll and 8j. =
St.j.s where 7 was defined Section 3.1. We call this metric, reported in mm, the Mean Per Segment Consistency Error (MPSCE).
E
7

---

Following [12, 40], we also assess the bilateral symmetry of predicted skeletons through the Mean Per Segment Symmetry Error (MPSSE), mm:
MPSSE
Σ t=1 Jun
Σ (s...r)
St.d', st.d with -()
(8)
T [hen
where Jee denotes the set of indices of left-side joints and Ç maps left-side joint indices to their right-side counterparts.
Multi-hypothesis evaluation protocol. One must decide how to use multiple hypotheses to compute the metrics. The dominant approach [24, 25, 36, 44, 49, 12] is the oracle evaluation, í.e., using the predicted hypothesis closer to the ground truth (iee, Eq. (2) for MPJPE). That makes sense for multi-hypothesis methods, as the oracle metric measures the distance between the target and the discrete set of predicted hypotheses. It aligns with the idea of many possible outputs for given input.
Hypotheses can also be aggregated into final pose, e.g., through unweighted weighted averaging (using predicted scores). The latter has the disadvantage of falling back to a one-to-one mapping scheme, which precisely what we want avoid in multi-hypother setting.
We report both oracle and aggregated metrics in our experiments, favoring oracle results.
Implementation details. ManiPose, as presented in Section 3, is compatible with any backbone. Here, we chose to build on the MixSTE [52] network for both the rotations and the segment modules (the latter in reduced scale). Details about our architecture and training appear in Appendix D.
5.2 Comparison with the state of the art
Table 2:
Pose consistency evaluation of state-of-the-art methods on Human3.6M. MPJPE perfor- mance and pose consistency not correlated; only ManiPose excels in both. T: sequence length. K: number hypotheses. Orac.: Metric computed using oracle hypothesis. Grey lines: Methods where the Oracle MPJPE is computed with non-comparable number of hypotheses with respect the other baselines. Bold: best; Underlined: second best. *; Method with unavailable code MPSSE values reported in [12]. t: Results with comparable number of hypotheses. f: Results computed with official checkpoint and code.
T
K
Orac.
MPJPE 1
MPSSE L
MPSCE 1
Single-hypottheii methods:
ST-GCN [2]
7
48.8
8.9
10.8
VidcoPose3D [38]
243
46.8
6.5
7.8
PoseFormer [53]
81
44.3
4.3
7.2
Anatomy3D [4]
243
44.1
1.4
2.0
MixSTE [52]
243
40.9
8.8
9,9
Multihyyootheiis methods:
Wehrhein al. [49]
200
44.3
12.2
14.8
DiffPose (Holmquist et at.) [12]*
200
43.3
14.9
GFPose [6]
200
35.6
13.1
16.5
D3DP (P-Best) [43]
243
20
39.5
6.9
9.0
GFPose [6]]
10
45.1
13.1
16.5
Sharma el al. [44]
10
46.8
13.0
9,9
DiffPosc (Gong et al.) [10]T
243
39.3
5.2
6.1
MHForrer [27]
351
43.0
5.7
8.0
ManiPose (Ours)
243
42.1
0.4
0.8
ManiPose (Ours)
243
39.1
0.3
0.5
Human 3.6M. Comparisons with state-of-the-arr single- and multi-hypothesis methods presented in Table 2 and illustrated in Fig. 1. ManiPose outperforms previous methods in terms of Oracle MPJPE comparable scenarios, while reaching nearly perfect consistency. Moreover, note that MPJPE and consistency metrics are not positively correlated for tanglehhypothestt methods. As predicted in Section 4.1, our empirical results show that MPJPE improvements achieved by MixSTE come at the cost of poorer consistency compared to previous models. In contrast, the only single- hypothesis constrained model, Anatomy3D [4], achieves good consistency the expense of inferior MPJPE. Those results empirically validate the theoretical predictions of Sections 4.1 and B, further

---

confirming what we have shown, intuitively, in the simplified ID-to-2D setting (Section 4.2), Note that while ManiPose is deterministic, previous multi-hy hupthishypoheeiis methods are generative, except for MHFormer. Table 2 shows that they require up two orders of magnitude more hypotheses than ManiPose to reach competitive performance (see, e.g., the performance GFPose). This property expected. Indeed, optimization based on Mainer-Takes-AAe theoretically leads to an optimal coverage of the modes of the conditional distribution with fixed number samples [23], in contrast to generative-based approaches. This is reflected in the oracle metric, which approximates the so-called quantization (or Distortion) error, as defined in (27), when the number of data points is large. More detailed MPJPE results per action appear in Tables 8 and 9 in the supplemental. We also complement our analysis on the diversity of ManiPose in Fig. of the appendix.
Fig. 6 showcases qualitative results, where multiple hypotheses help in depth-ambight situations.
Table 3: Comparison with the tate-off MPI-INF-3DHP using ground-truth 2D poses. T: sequence length.
T
PCK
AUC
MPJPE
MPSSE
MPSCE |
VideoPose3D [38]
81
85.5
51.5
84,8
10.4
27.5
PoseFormer [53]
86.6
56.4
77.1
10.8
14.2
MixSTE [52]
94.4
66.5
54.9
17.3
21.6
P-STMO [42]
81
97.9
75.8
32.2
8.5
11.3
ManiPose (Ours) Aggr.
27
98.0
75.3
37.7
0.6
1.3
ManiPose (Ours) Orac.
27
98.4
77.0
34.6
0.6
1.3
MPI-INF-3DHP Similar results obtained for this dataset (ef. Table 3). Not only does ManiPose reach consistency errors close to 0, but also best PCK and AUC performance. As for MPJPE, only [42] achieves slightly better performance, at the cost of large pose consistency errors.
5.3
Ablation study
Table 4: Ablation study: Single hypothesis cannot optimize both MPJPE and consistency. ManiPose uses the same backbone as MixSTE. MR: with manifold regularization. MC: manifold- constrained. Bold: best. Underlined: second best.
MR
MC
K
Params.
MPJPE +
MPSSE
MPSCE
ManiPose (Ours)
X
5
34.44 M
39.1
0,3
0.5
w/o MH
34.42
44.6
MC, w/o MR (MiXSTE)
33.78
42.3
5.7
7.3
33.78
40.9
8.8
9.9
MuSSTE
ManiPase
Ground Truth
13
11.6
14
30
Segment [mm]
15
IPSSSE per
25
20
20
5
10
27
910
11
11:12
12:13
8:14
14-15
15-16
0-4
45
81
12
6
Segment
Segment
Coordinate
Figure 5: MPSCE, MPSSE and MPJPE per segment/coordinate (lower is better). ManiPose mostly helps deal with the depth ambiguity (z coordinate). Ground-truth poses represented but not visible because they have perfect consistency.
Impact of components. We evaluate the impact of removing each component of ManiPose on the Human 3.6M performance (Table 4). The components tested are the multiple hypotheses (MH) and the manifold constraint (MC). We also compare MC to more standard manifold regularization (MR), i.e., adding Eq. (7) to the loss. Note that without all these components, we fall back to MixSTE [52], and that the performances reported in Table 4 also appear in Fig.
We see that MR helps to improve pose consistency, but not as much as MC. However, without multiple hypotheses, MC consistency improvements come at the cost of degraded MPJPE performance, as

---

foreseen by our formal analysis (Section 4). Only the combination of both MC and MH allows us to optimize both consistency and MPJPE.
Fine error analysis. We can see in Fig. 5 that, compared to MixSTE, ManiPose reaches substantially superior MPSSE and MPSCE, consistency across all skeleton segments. Furthermore, note that larger MixSTE errors occur for segments KNEE-FOOT and BLBOW-WRIST which are the most prone to depth ambiguity. That agrees with coordinate- wise errors depicted in Fig. 5, showing that ManiPose improvements mostly translate into reduction of MixSTE depth errors, which are twice as large for other coordinates. Further ablations, including the effect of the number of hypotheses K, the score loss weight 3 and the rotations representation choice appear in the supplemental.
Input
MiXSTE
ManiPose (ours)
Input
MiXSTE
ManiPose (ours)
Single hypothesis
Hypathesis t:
Hypothesis p²
Hypothesis 3: p/())
Score: xpx)
Figure 6: Qualitative comparison between ManiPose and state-of-the-an regression method, MixSTE, Two pairs of predicted hypotheses by ManiPose illustrated in green-pink (left) and green-purple (right), where opacity is used to represent the predicted scores, Multiple hypotheses and constraints help to deal with depth ambiguities and avoids predicting shorter limbs (red circles).
6 Conclusion
We presented a new manifold-constraing multi-hypothesis human pose lifting method (ManiPose) and demonstrated its empirical superiority to the existing state-of-the-ard Further, we provided theoretical evidence supporting the tenets of our method, by showing the inherent limitation of unconstrained single-hypothesis approaches 3D-HPE. We established that unconstrained single-hypothesi methods cannot deliver consistent poses and that constraining or regularizing single-hypothesis models leads to worse position errors. We also showed that traditional MPJPE-like metrics are insufficient to assess consistency.
challenging datasets.
Limitations. To guarantee its consistency, ManiPose relies the forward kinematics algorithm, which is inherently sequential across joints. Removing that dependence an interesting avenue for accelerating the method. On another note, while ManiPose ensures the rigidity of the predicted poses, imposing constraints within human body articulation limits presents another area for enhancement.
Acknowledgments and Disclosure of Funding
This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD01 made by GENCI. It was also partly funded by the French Association for Technological Research (ANRT CIFRE contract 2022- 1854). We are grateful the reviewers for their insightful comments.
10

---

References
[1] Bishop, C.M.: Mixture density networks. Working paper, Aston University (1994) 3
[2] Cai, Y., Ge, L., Liu, J., Cai, J., Cham, T.J,, Yuan, J., Thalmann, N.M.: Exploiting spatial- temporal relationships for 3d pose estimation via graph convolutional networks. In: Proceedings of the IEEE/CVF international conference on computer vision. 2272-2281 (2019) 2, 8, 23,
24
[3] Chen, C.H., Ramanan, D.: 3d human pose estimation= pose estimation+ matching. In: Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 7035-7043 (2017) 2
[4] Chen, T., Fang. C., Shen, X., Zhu, Y. Chen, Z., Luo, J.: Anatomy- aware 3d human pose estimation with bone-basee pose decomposition. IEEE Transactions Circuits and Systems for Video Technology 32(1), 198-209 (2021) 2, 7, 8, 23, 24
[5] Chen, Y., Wang, Z., Peng. Y., Zhang. Z., Yu, G., Sun, J.: Cascaded pyramid network for multi-person pose estimation, In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7103-7112 (2018)
[6] Ci, H., Wu, M., Zhu, W. Ma, X., Dong, H., Zhong, F., Wang, Y: Gfpose: Learning 3d human pose prior with gradient fields. In: Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4800-4810 (2023) 3,
[7] Du, Q., Faber, V., Gunzburger, M.: Centroidal voronoi tessellations: Applications and algo- rithms. SIAM review 41(4), 637-676 (1999) 19
[8] Firman, M., Campbell, N.D., Agapito, L., Brostow, G.J.: Diversenet: When one right answer not enough. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5598-5607 (2018) 3
[9] Goel, S., Pavlakos, G., Rajasegaran, J., Kanazawa, A., Malik, J:: Humans in 4d: Reconstruct- ing and tracking humans with transformers. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 14783-14794 (2023)
[10] Gong, Foo, L.G., Fan, Z.,, Ke, Q., Rahmani, H., Liu, J.: Diffpose: Toward reliable 3d pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13041-1305I (2023) 3, 8, 24, 26
[11] Guzman-Rivera, A., Batra, D., Kohli, Multiple choice learning: Learning to produce multiple structured outputs. Advances in neural information processing systems 25 (2012) 3
[12] Holmquist, K., Wandt, B.: Diffpose: Multi- hultiihypothesis human pose estimation using diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15977-15987 (2023) 1, 3, 8, 23
[13] Hossain, M.R.I., Littl, J.J.: Exploiting temporal information for 3d human pose estimation. In: Proceedings of the European conference on computer vision (ECCV), pp. 68-84 (2018) 2, 22
[14] Hu, W. Zhang, C., Zhan, F., Zhang, L., Wong, T.T. Conditional directed graph convolution for 3d human pose estimation. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 602-611 (2021)* 2
[15] Ionescu, C.,, Papava, D., Olaru, V., Sminchisescu, C.: Human3.6M 66M Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. IEEE Transactions on Pattern Analysis and Machine Intelligence 36(7), 1325-1333 1339 (Jul 2014) 7, 15
[16] Kanazawa, A., Black, M.J., Jacobs, D. W. Malik, J.: End-to-end recovery of human shape and pose. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp.
7122-7131 (2018) 2
[17] Kingma, D.P., Ba, J.: Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 20, 22

---

[18] Kolotouros, N., Pavlakos, G., Jayaraman, D., Daniilidis, K.; Probabilistic modeling for human mesh recovery. In: Proceedings of the IEEE/CVE international conference on computer vision. pp. 1 11165-11614 1614 (2021)3 3
[19] Lee, K., Hwang, C., Park, K., Shin, Confident multiple choice learning. In: International Conference on Machine Learning. pp- 2014-2023, PMLR (2017) 3, 5
[20] Lee, S., Purushwalkam, S., Cogswell, M., Crandall, D., Batra, D.: Why m heads are better than one: Training diverse ensemble of deep networks. arXiv preprint arXiv: 1111016114 (2015) 3
[21] Lee, S., Purushwalkam Shiva Prakash, S., Cogswell, M., Ranjan, V., Crandall, D., Batra, D.: Stochastic multiple choice learning for training diverse deep ensembles. Advances in Neural Information Processing Systems 29 (2016) 4,5
[22] Letzelter, V., Fontaine, M., Chen, M., Pérez, P., Essid, S., Richard, G.: Resilient multiple choice learning: A learned scoring scheme with application to audio scene analysis. Advances in neural information processing systems 36 (2024) 3, 4,
[23] Letzelter, V., Perera, D., Rommel, C., Fontaine, M., Essid, S., Richard, G., Pérez, P.: Winner- takes-all learners are geometry-awant conditional density estimators. In: Proceedings of the 41 st International Conference on Machine Learning (2024) 9, 19
[24] Li, C., Lee, G.H.: Generating multiple hypotheses for 3d human pose estimation with mixture density network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9887-9895 (2019)* 8, 23
[25] Li, C., Lee, G.H.: Weakly supervised generative network for multiple 3d human pose hypotheses. In: British Machine Vision Conference (BMVC) (2020) 3, 8, 23
[26] Li, J., Xu, C., Chen, Z., Bian, S., Yang, L., Lu, C.: Hybrik: A hybrid analytical -eeural inverse kinematics solution for 3d human pose and shape estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3383-3393 (2021) 22
[27] Li, W., Liu, H., Tang, H., Wang. P., Van Gool, L.: Mhformer: Multi-hypothesis transformer for 3d human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13147-13156 (2022) 3, 8, 23
[28] Liu, R., Shen, J., Wang, H., Chen, C., Cheung, S.c., Asari, V.: Attention mechanism exploits temporal contexts: Real-time 3d human pose reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5064-5073 (2020) 23, 24
[29] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: A skinned multi- person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34(6), 248:1-244 166::16 (Oct 2015)< 2
[30] Makansi, O., Ilg, E., Cicek, O., Brox, T.: Overcoming limitations of mixture density networks: sampling and fitting framework for multimodal future prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7144-7153 (2019)
[31] Martinez, J., Hossain, R., Romero, J., Little, JJJ.: A simple yet effective baseline for 3d human pose estimation. In: Proceedings of the IEEE international conference on computer vision. pp. 2640 (2017)* 2
[32] Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O.,, Xu, W., Theobalt, C.: Monocular 3d human pose estimation the wild using improved cnn supervision. In: 2017 international conference on 3D vision (3DV). pp. 506-516. IEEE (2017)*
[33] Mun, J., Lee, K., Shin, J., Han, B.: Learning to specialize with knowledge distillation for visual question answering. Advances in neural information processing systems 31 (2018) 3
[34] Murray, R.M., Li, Z., Sastry, S.S.: mathematical introduction to robotic manipulation. CRC press (2017) 17, 22
12

---

[35] Naeem, M.F., Oh, S.J., Uh, Y., Choi, Y., Yoo, J.:
Reliable fidelity and diversity metrics for generative models, In: International Conference on Machine Learning, pp. 7176-7185. PMLR (2020) 24, 26
[36] Oikarinen, Hannah, D., Kazerounian, S.: Graphmdn: Leveraging graph structure and deep learning to solve inverse problems. In: 2021 International Joint Conference on Neural Networks (JJCNN). pp. 1-9. IEEE (2021) 3, 8, 23
[37] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury. J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019) 2
[38] Pavilo, D., Feichtenhofer, C., Grangier, D., Auli, M.: 3d human pose estimation in video with temporal convolutions and semi-supervised training. In: Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7753-7762 (2019) 2, 7, 9, 22,
23, 24
[39] Rempe, D., Birdal, T., Hertzmann, A., Yang. J., Sridhar, s., Guibas, L.J.: Humor: 3d human motion model for robust pose estimation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11488-11499 (2021)
[40] Rommel, C., Valle, E., Chen, M., Khalfaoui, S., Marlet, R., Cord, M., Pérez, P.: DiffHPE: Robust, Coherent 3D Human Pose Lifting with Diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3220-3229 (2023) 1,
[41] Rupprecht, C., Laina, I., DiPietro, R., Baust, M., Tombari, F., Navab, N., Hager, G.D.: Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In: Proceedings of the IEEE international conference on computer vision. pp. 3591-3600 (2017) 3, 19
[42] Shan, W., Liu, Z., Zhang, X., Wang, S., Ma, S., Gao, W.: P-STMO: Pre- Tre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation (Jul 2022) 1, 2,
[43] Shan, W., Liu, Z., Zhang, X., Wang, Z., Han, K., Wang, S., Ma, S., Gao, W.: Diffusion- based 3d human pose estimation with multi-hypothesis aggregation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14761-14771 (2023) 8, 23
[44] Sharma, S., Varigonda, P.T., Bindal, P., Sharma, A., Jain, A.: Monocular 3d human pose estimation by generation and ordinal ranking, In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2325-2334 (2019) 3, 8,
[45] Tian, K., Xu, Y., Zhou, S., Guan, J.: Versatile multiple choice learning and its application to vision computing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 0349-6357 (2019) 5,
[46] Tiwari, G., Antić, D., Lenssen, J.E., Sarafianos, N., Tung, T., Pons-Moll, G.: Pose-ndf: Modeling human pose manifolds with neural distance fields. In: European Conference Computer Vision. pp- 572-589. Springer (2022)*
[47] Wang, J., Yan, S., Xiong, Y., Lin, D.: Motion guided 3d pose estimation from videos. In: European Conference on Computer Vision. pp. 764-780. Springer (2020) 23, 24
[48] Waskom, M.L.: seaborn: statistical data visualization. Journal of Open Source Software 6(60), 3021 (2021). https://doi.org/10.. 105/100s.00002.. https://doi.orgg. org/10....0. 21105/ joss. 03021 16
[49] Wehrbein, T., Rudolph, M., Rosenhahn, B., Wandt, B.: Probabilistic monocular 3d human pose estimation with normalizing flows. In: Proceedings of the IEEF/CVF international conference on computer vision. pp. 11 1119991199 1208 (2021) 1, 23
[50] Xu, J., Yu, Z., Ni, B., Yang. J., Yang. X., Zhang, W.: Deep kinematics analysis for monocular 3d human pose estimation. In: Proceedings of the IEEE/CVF Conference on computer vision and Pattern recognition. pp. 899-908 (2020)
13

---

[51] Xu, T., Takano, W.: Graph stacked hourglass networks for 3d human pose estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16105-16114 (2021) 2, 23
[52] Zhang, J., Tu, Z., Yang, J.. Chen, Y., Yuan, J.: Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13232-13242 (2022) 2, 7, 8, 9, 21, 22, 23, 24
[53] Zheng, C., Zhu, S., Mendieta, M., Yang, T.,, Chen, C., Ding, Z.: 3D Human Pose Estima- tion with Spatial and Temporal Transformers. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 11636-1 1645. IEEE, Montreal, QC, Canada (Oct 2021). http::/doi.oor/110 100./ 109/1CCV88922.20222
145 1, 7, 8, 9, 9, 23, 24
[54] Zhou, Y., Barnes, C., Lu, J., Yang. J., Li. H.: On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9745-5755 (2019) 4, 22, 24
[55] Zou, Z., Tang, W.: Modulated graph convolutional network for 3d human pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1477-114887
(2021) 2, 23, 24
14

---

A supplemental material
This supplemental material is organized as follows:
Appendix A contains empirical verification of our assumptions,
Appendix B presents the proofs of theoretical results, together with few corollaries, Appendix C provides further implementation details concerning the ID-to-2D experiment, as well as an extension to the 2D-to-3D setting,
Appendix D contains implementation and training details concerning ManiPose,, well as compared baselines,
Appendix presents further results of the Human 3.6M experiment, and finally, Appendix explains the provided experiment code.
A Assumption verifications
Let us first define a few elements that we will need needed for our derivations.
Definition A.1 (Human skeleton). We define human skeleton as undirected connected graph G=(V, with J = [V] nodes, called joints, associated with different human body articulation points. We assume a predefined order of joints and denote A [Ai]losaj<] adjacency matrix of G, defining joints connections.
(),1) the
Definition A.2 (Human pose and movement). Let G be a skeleton of J joints. We attach to each joint i a position p in IR3 and call the vector p = .ps....,,f. pf-1] E ( human pose. Furthermore, given series of increasing time steps t t2 sequence of poses of the same subject at those instants m = .p.......PP.
< tr E we define human movement m as
P E
(RR3)XT.
We base the theoretical results of Section 4.1 the following assumptions. The first states the reference frame traditionally used for assessing 3D-HPE models:
Assumption A.3 (Reference root joint). For any skeleton G and movement of length T, the joint of index O, called the root joint, is at the origin P [0, 0, 0j at all times f1 S t < tr. That is equivalent to measuring positions p in reference frame attached to the root joint.
The second assumption concerns the rigidity of human body parts:
Assumption A.4 (Rigid segments). We assume that the Euclidean distance between adjacent joints constant within a movement m: for any pair of instants and and for any joints i, j such that Ag 1, assume that
$t,6,1 = 3.. 8i.
where St,i,j IIf,, - 12 O.
Finally, we assume that the conditional distribution of poses does not collapse to a single point, i.e. that we have a one-to-many problem:
Assumption A.5 (Non-degeneratt conditional distribution). Given joint distribution P(x c) of 3D poses g ( and corresponding 2D inputs x E (R2)², we assume that the conditional distribution (pp x is non-degenerate, i.e., it is not single Dirac distribution.
Note that can be true even when P(xC, C) is unimodal (e.g., Fig. 4).
We verified on Human 3.6M [15] ground tround-truth data that assumptions A.4 and A.5 hold for actual poses in both training and test splits.
Segments rigidity. As shown on Figs. 5 and 9, ground-truth 3D poses have perfect MPSSE (8) and MPSCE (7) metrics, meaning that ground-truth skeletons are perfectly symmetric, with rigid segments. Assumption A.4 is thus verified in actual training and test data.
Non-degenerate distributions. As shown on Fig. 7, the conditional distribution ground-truth poses given 2D keypoints position is clearly multimodal, and, thus, non-degenerate (not reduced to single Dirae distribution). That validates assumption A.5 and explains why multi-hypothesis techniques are necessary.
15

---

0.4
0.2
0.0
-0.4
-0.5
0.0
0.5
1.0
-0.5
0.0
0.5
1.0
(a) S9, Walking
0.50
0.00
-0.25
0.50
-0755
-0.5
-0.5
0.0
0.5
(h) S1, Greeting
0.50
0.25
0.00
-0.25
-0.50
-0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
v
(c) S11, Directions
0.0
-0.5
0.5
0.0
0.5
(d) S1, SittingDown
Figure Estimated joint distributions of ground-truth 2D inputs (u, v pixel coordinates) together with 3D z-coordinates (depth) for different subjects and actions. The depth density conditional on inputs clearly multimodal. Vertical red lines are examples of depth-ambiguous inputs. Distributions are estimated with kernel density estimator from the Seaborn plotting library [48].
16

---

B Proofs and additional corollaries
B.1 Properties of manifold constraint and multi-hypotheses models
This section contains the proofs of the theoretical results presented in Section 4.1, together with few corollaries.
PROOF. [Proposition I] Let i be a joint connected to the root po (i.e., A. = 1). From assumptions A.3 and A.4, we know that at any instant t. P lies on the sphere (2, s1,0) centered at 0 with radius $4,0 independent of time. Therefore, its position can be fully parameterized in spherical coordinates by two angles i 21.)). Let j be joint connected to i. Like before, assumption A.4 implies that at any instant t. PL, lies the moving sphere S (2(p,,, 85,2) centered P with radius $3,8 independent of time. Thus, we can fully describe P with the position of its center, P and the spherical coordinates (0..).O..)) of joint j relative to the center of the sphere, i.e, joint i. That means that there is a bijection between the possible positions attainable by p at any instant and the direct product of spheres 5² $1,D) & s5.)). That bijection is an homeomorphism since is a composition of homcomorphisms: we can compute pi from (0r,t, St.s, 02,, Qt.,) following the forward kinematics algorithm [34] (cf. algo. 2), i.e., using a composition of rotations and translations.
Now let us assume for some arbitrary joint that PÊ, lies at all times on a space M22 homeomorphic G to a product of spheres of dimension 2d. That means that Pi,k can be fully parametrized using 2d spherical angles (01, .1...., Od, Φa). Let be a joint connected to k (typically one further step away from the root joint Po and not already represented in M22). As before, at any instant t. PL needs to lie on the sphere centered on G of constant radius sp,t- Thus. we can fully describe Pt, using the 2(d + 1) -ttple of angles obtained by concatenating its spherical coordinates relative to joint k, together with the 2d-tuple describing PÊ, i.e. the center of the sphere. So P lies on a space M2( homeomorphic to a product of spheres of dimension 2(d + 1).
We can conclude by induction that at any instant t, = Ipf...... Pf.j] lies on the same subspace of (R which is homeomorphic to product of spheres centered at the origin:
80,
(10)
Kj/A,==
Finally, the previous space is trivially homeomorphic to (8")J-1 through the scaling (1/8t.)scs/A,.. (5 is a manifold of dimension 2(J 1) as the direct product of J manifolds of dimension 2.
PROOF. [Proposition 4.2] Let G be a skeleton with J joints, x E (R²)) 2D pose, p (IR3) its corresponding 3D pose, and P p) a joint distribution of poses in 2D and 3D. We define E (Lg) as the function allowing us to compute the length of the segments of a pose p:
E p + [|p; - Pro)l22 0 < J
(11)
where T : (l...... J - 1) + t0, - 1} maps joint indices to the index of their parent joint:
7i) = j < i, s.t. Aij 1.
(12)
From assumption A.4, we know that for any pose p from the training distribution,
Vj, (p) = (j.(())
(13)
some i.i.d. evaluation data, the MSE of a model f is defined as: Given D = P ~ P(x
MSE(F:N) N llp f(x))11
(14)
and converges to
MSE Fx.p
[[pp- [(x)
(15)
s sj.a) is homeomorphic to S (pC.i
17

---

as the dataset size N goes to infinity. We then define the oracle MSE minimizer as
arg min MSE" (f).
(16)
f
The quantity in (15) is known statistics as the expected L2-risk and is a well-known fact that its minimizer is the conditional expectation:
(*(x) = Elp|x = x]-
(17)
Thus, since lj are strictly convex and P(p[x) is non-degenerate according to assumption A.5, can conclude from Jensen's i strict inequality that for all j.
(f (x)) (E[p)x x)) ElG x] = S))))
(18)
where the last equality arises from the fact that (p)) is not random according to (13). Thus, given that l > 0 and $j.(() > O, we can say that (x)) < $).(()) for all joints j. We conclude that the model f minimizing MSE" predicts poses that violate assumption A.4 and are inconsistent.
As an immediate corollary of proposition 4.2, we may state the following result, which was empirically illustrated in many parts of our paper:
Corollary B.1. Given a fixed training distribution P(x, p) respecting assumptions A.3-A.5, for all 3D-HPE model predicting consistent poses, i.e., that respect assumption A.4, there is an inconsistent model f with lower mean-squared error.
PROOF. Let f' E arg min MSE" (f). According to proposition 4.2, f' is inconsistent. Suppose that the consistent model f is such that
(19) Since MSE* reaches its minimum at f', we have MSE = MS) Thus. f E arg min MSE" (f), which means that f is also inconsistent according to proposition 4.2. That is impossible given that we assumed f to be consistent. We conclude that Eq. (19) is wrong and that MSE* (f) > MSE*(f)).
(f) S MSE*
(20)
Note that propositions 4.2 and B. assume the of the MSE loss, which is the most widely used loss in 3D-HPF. We can however extend them to the case where MPJPR serves as optimization criteria under an additional technical assumption:
Corollary B.2. The predicted poses minimizing the mean-per preen-per-oii--ppsiioneeee
loss are inconsis- tent if the training poses distribution P(x, p) verifies Asm. A.3-A.5 and if the joint-wise residuals' norm standard deviation is small compared to the joint-wise loss:
VxD [ll., - [x)]1]
OSi<J,
20
(21)
Bu.p [llp, (x)]l2
PROOF. From proposition 4.2 we know that the poses predicted by the minimizer f of
MSE" (f) = Ex.p [llp - [x)
(22)
are inconsistent. Let / be the component of corresponding to the 7th joint. We define the jth maan-pprr-jontt-ppitoooerrrrr component as:
MPJPE A - [x(x)]
(23)
Under the small variance assumption, we have:
Vxp [p; - (x)
(24)
Em.p [lp; - fz (x)
Ex.p |lp - f(x) Ex.p [lle, - (x))2]
(25)
Er.p [llp, (x)
MSE; (f) MP) 2= 0,
(26)
(2)
so both criteria, MSE and MPJPF, are asymptotically equivalent and have the same minimizer f*, which is inconsistent according to proposition 4.2.
18

---

Corollary B.3. Under Asm. A.4-A.5 and under (21), the only way to get both optimal MPJPE and consistency is to multiple hypotheses.
models (constrained or not) deliver either suboptimal MPJPE or inconsistent pose predictions. Hence, by negation, we get our result.
PROOF. Corollary B.1 and Proposition 4.2 imply that
In the next section, we further show that multi-hypotheses models, constrained or not, can theoretically show better L2-risk (or quantization) performance compared with single-hypotheses models.
B.2 Multiple hypotheses (constrained or not) can improve L2-risk over single-hypothesis models
Let X' 2 denote the space of input 2D poses and P 3x the space of 3D poses. Also, let R(f) = Expplpp fx) be the L2-risk of some pose estimator f under some underlying continuous joint distribution of 2D-3D pose pairs P(x, p), with density p (when it exists).
Before stating the proposition, we need define an adapted notion of risk for multi-hyyothesis models under the oracle aggregation scheme:
Definition B.4 Wiinerr-takes-al risk, [41]). As in [41] (section 3.2) and in [23] (section 2.2), we define the L2-risk for K-head models fWIA (WWA...... (WIA) as:
E.E k= V+ (x plıp((x p) dpdx,
(SWAA) A
(27)
where (g)'s denotes the htth cell of the Voronoi tesselation of the output space P defined by generators (g) "g..........................
ggk) gx)e E pk:
v (g) A (pEP
p12 < Ig - vr 4 k}}
(28)
The risk above translates the notion of oracle pose, since it partitions the space of ground-truth poses P into regions where some hypothesis the closest, and uses only that hypothesis to compute the risk in that region. Note that RTA (f) R(()) for any function f, since a single-cell tessellation of P is P itself.
In the following, we assume that is expressive enough, so that, minimizing the risk (27) comes down minimizing
E/ V (fwa(()))
Σ
II (x) p)
for each X.
Proposition B.5 (Optimality of manifold constrained multi-hypothesis models), A K K-hypotheses model fiTa = (f ......... f minimizing (27) has always a risk lower or equal to a single- hypothesis model fase minimizing R:
R (füm)) S (.ix(ffss)) R(fūse)
(29)
PROOF. Following [23] (Section 2.2), we decouple the cell generators from the risk arguments in (27):
K Σ
K(g, x) A
||* -1pppppp))) dp,
(30)
(t(g)
k=1
for any generators (gt..... ...... )*)
PK and arguments (2t,..., E PK, Note that
(f) = fx K(f((), (xx)) dx.
According to Proposition 3.1 of [7] (or Proposition 2.1 [23]), fwr minimizes R then (fiTa (x)) has to minimize K. for all x X:
((fiTa(x), (x)a I K(g,2),
Vg,z E pk x pk
(31)
19

---

Let's choose g such that g" -WIA pk. (x) and 2 such that 2t = (x) for all 1 k K. Then
S Σ
R
[I pllp(pl)))p(x) R(SMSE). (32)
(((m((()))
where the last equality comes from the fact that y (fWTA(x)) defines partition of P.
C Further details of ID-to-2D case study
C.1 Implementation details
divided into 000 training examples, 000 validation examples and 100 test examples. Since the 2D position of J1 is fully determined by the angle between the segment J1) and the r-axis, the dataset is generated by first sampling from a von Mises mixture distribution, then converting it into Cartesian coordinates (Iri, y:) form the outputs, and finally projecting them into the x-axis to obtain the inputs.
Datasets. We created dataset of input-output pairs {(r,, (2t, (e),, 1
Distribution scenarios. We considered three different distribution scenarios with different levels of difficulty:
Easy scenario: a unimodal distribution centered at 9 = 2m, where the axis of maximum 2D variance is approximately parallel to the z-axis (Fig. 4-A).
2. Difficult unimodal scenario: a unimodal distribution centered at 0 = 0, where the axis of maximum 2D variance perpendicular to the r-axis (Fig. 4-B).
3. Difficult multimodal scenario: a bimodal distribution, with modes at 01 3 and 0g and mixture weights wi = 3/ and W2 = 3. i.e., where the projection of modes onto the z-axis are close to each other (Fig. 4-C).
All von Mises components in all scenarios had concentrations equal to 20.
Architectures and training. All three models were based on a multi-layer perceptron (MLP) with 2 hidden layers of neurons each, using tanh activation.
The constrained and unconstrained MLPs were trained using the mean-squared loss \ ΣN ((ē; az,32 + (Ú. )). ManiPose was trained with the loss Eq. (1), and had trained all models with batches of 100 examples for maximum of 50 epochs. We used the Adam optimizer [17], with default hyperparameters and no weight decay. Learning rates were searched for each model and distribution independently over a small grid: [10-3 10 10-- 100 (ef. selected values in Table 5). They were scheduled during training using a plateau strategy of factor 0.5, patience
heads. We
of 10 epochs and threshold of 10
Table 5: Selected learning rates for 1D-to-2D synthetic experiment.
Distribution
A
B
C
Unconstr. MLP
10-3
10-3 3
10 10-2 2
Constrained MLP
10-2
10-4
10-2
ManiPose
10-2
10
10 10-2
C.2 Extension to 2D-to-3D setup with more joints
We further extend the two-joint ID-to-2D lifting experiment of Section 4.2 to 2D-to-3D with three joints, aiming providing scenario that is closer to real-world 3D-HPE, but that can still be fully dissected and visualized.
As in Section 4.2, we suppose that joint Jo is at the origin at all times, that Ji is connected to Jo through a rigid segment of length 81 and that Ja connected Jı through second rigid segment of length so. We further assume that both J1 and J2 are allowed to rotate around two axes
20

---

orthogonal to each other. Thus, J1 is constrained to lie on circle S' (0, su), while Jz lies on torus homeomorphic S (0,00))) so) @ S' s1). Without loss of generality, we set the radii so 2 and and assume them to be known.
predict the 3D pose (21, = (I1, y1, 21, 22, 22, 22) E RR, given its 2D projection 1K1, K2) (41, 21, 22) E R. We create a dataset comprising 20000 training, 2000 validation, and 2000 test examples, sampled using an arbitrary von Mises mixture of poloidal and toroidal angles (0, 6) in T. We set the modes of such mixture at [(-T,, 0), (0. //44, (2. -T -/4), (2 x /2)], with concentrations of [2, 3, 10] and weights [0.3, 0.4, 0.2, 0.1]. Similarly to Fig. 4-C, that creates difficult multimodal distribution, depicted in Fig.
Given that setup,
interested in learning
10
(a)
(b)
Figure 8: Visualisation of the von Mises mixture distribution on the torus T. The different colors (blue, green, red, purple) represent the modes of the sampled points. We are only representing joint J2 here for clarity.
We train and evaluate the same baselines in Section 4.2 in that new scenario, using similar setup (ef. Appendix C. Architectures and training). Note that for those experiments, we used an initial learning rate of 10-3 for each bascline, and batch size of 1000 examples. The corresponding Mcan Per Segment Consistency Error (MPSCE) and Mean Per Joint Position Error (MPJPE) results are reported in Table 6.
Table 6: Mean per joint prediction error (MPJPE) and mean per segment consistency error (MPSCE) in 2D-to-3D scenario. Results averaged over five random seeds. ManiPose reaches perfect MPSCE consistency without degrading MPJPE performance.
MPJPE +
MPSCE 4
Unconst, MI.P
1.152 + 0.021
0.269
0.018
Constrained ML.P
1.166 + 0.028
0.000 0.000
ManiPose
1.149 1 0.036
0.000 0.000
We that the same observations in Section 4.2 also apply here: although the unconstrained MLP yields competitive MPJPE results, its predictions are not consistently aligned with the manifold, as indicated by its poor MPSCE performance. Again, we show here that ManiPose offers an effective balance between maintaining manifold consistency and achieving high josit-position-error performance.
D Further ManiPose implementation details
D.1 Architectural details
Our architecture is backbone-agnosia as shown Fig. 2. Thus, in order to have fair comparison, we decided to implement using the most powerful architecture available, i.e., MixSTE [52].
In practice, the rotations module follows the MiXSTE architecture with d₁ 8 spatio-tempora trans- former blocks of dimension d,, = 512 and time receptive field of T 243 frames for Human 3.6M experiments and T 43 frames for MPI-INF-3DHP experiments. Contrary to MixSTE, that network
21

---

outputs rotation embeddings of dimension 6 for each joint and frame, instead of Cartesian coordinates of dimension 3.
Concerning the segment module, it was also implemented with a smaller MixSTE backhone of depth d: 2 and dimension d 128.
The ablation study presented in Table shows that the increase in the number of parameters between MixSTE and ManiPose is negligible.
D.2 Pose decoding details
The pose decoding block from Fig. 2 described in Section 3.1 and is based on Algorithms and 2. The whole procedure illustrated on Fig. 3.
Table Joint wise weights used the Winner-takes-all loss Eq. (2) (as in [52]).
Joint
2
12
14
15
16
Weight
2,5
2.5
2.5
2.5
1.5
15
4
1.5
4
4
Algorithm 6D rotation representation conversion [54]
Require: Predicted 6D rotation representation E R.
x' +- [ro, T1, 72
2: y' t T4.
3: x < x//[|<='11a.
4: z' + .AA'
5: z + z 2//11='122,
6: y + 2 a
7: return I [x|y||]] E p3x3
Algorithm 2 Forward Kinematics [34, 26]
Require: Scaled reference pose u' E (R&)), predicted rotation matrices Re,, 0 S J.
1: Rg_0- t- Ri,0
2: Pr, u/
3: for = I.....J-- do
4: Ry, Ri.j R(,
Compose relative rotations
5: Pt.j + (u' un( + Pt.r())
6: end for
7: return P. [pc,loss<s
D.3 Training details
Training tactics. In order to have a fair comparison with MixSTE [52], we trained ManiPose using the same training tactics, such pose flip augmentation both at training and test time. Moreover, the training loss (1) was complemented with two additional terms described in [52]]
1. TCloss term, initially introduced in [13];
2. a velocity loss term, introduced [38].
We also weighted the Winner-takes-all MPJPE loss (2) as in [52] (ef. weights in Table 7). The score loss weight, B, was to 0.1 according to our hyperparameter study (Appendix E), while TCloss and velocity loss terms had respective weights of 0.5 and 2 (values from [52]).
Training settings. We trained our model for maximum of 200 epochs with the Adam optimizer [17]. using default hyperparameters, weight decay of 10-6 and an initial learning rate x 10-5 The latter was reduced with a plateau scheduler of factor 0.5, patience of 11 epochs and threshold of 0. mm. Batches contained sequences of T 243 frames each for the Human 3.6M training, and 30 sequences of T' 43 frames for MPI-INF-3DHIP
22

---

Compute resources. Trainings were carried out on single NVIDIA RTX 2000 GPU with around 11GB of memory. The training of the large model with 243 frames on Human 3.6M dataset took around 26 hours.
Dataset licences. Human 3.6M is a dataset released under a research-only custom license, and available upon request this URL: httt://vision. imar ro hhman3.. 6m/ description. php. MPI-INF- is released under non commmrerraal custom license and can be found at: https: //vcai mpi- iifp--if..ie mpg. de/3dhpp-dttad
D.4
Baselines evaluation.
All Human 3.6M evaluations MPSSE and MPSCE listed in Tables and 4 were performed using the official checkpoints of these methods and their corresponding official evaluation scripts. Concerning UPI-INF-3DHI evaluations from Table checkpoints were not available (except for P-STMO). Thus, baseline models were retrained from scratch using the official MPI-INF-3DHP training scripts provided by the authors of each work, using hyperparameters reported in their corresponding papers. We checked that we were able to reproduce the reported MPJPE results.
E Further results on the Human 3.6M dataset
Table 8: Quantitative comparison with the state-of-the-art methods on Human3.6M under Protocol #1 (MPJPE in mm), using detected 2D poses. T: sequence length. K: number of hypotheses. Orac.: Metric computed using oracle hypothesis. Bold: best; Underlined: second best.
Orac.
Smoke
WIITT. Avg.
Stepledyenbeel
GraphSHiSSt
49.9
47.5
50.9
54.9
48.5
46.3
51.4
43.5
44.1
51.9
MOCI [55]
49.2
49.4
46,0
69.7
40.3
49.4
SCG
7
47.4
48.8
50.5
59.0
43.9
29.7
29.4
48.6
243
45.2
46.7
43.3
45.6
48.1
S5.1
44.6
44.3
$7.3
65.8
47.1
44.3
40.0
32.8
33.0
46.8
UGCN [-7]
96
41.3
43.9
44.0
42.2
48.0
$7.1
42.2
43.2
57.3
61.3
47.0
43.5
47.0
32.6
31.5
45.6
Liu of [25]
243
41.8
44.8
41.1
44.9
47.4
$4.1
43.4
42.2
56.2
63.6
45.3
43.5
45.3
31.3
32.2
45.1
PooeFomeer [53]
$1
41.5
44.3
39.8
42.5
46.5
$1.6
42.1
41.0
53.5
50.7
45.5
43.3
46.1
31.8
32.2
44.3
Amsiomy 3D [4]
24%
41.4
43.
40.
42.9
46.6
$1.9
41.5
43.3 5
$3.9
45.4
41.7
46.0
11.5
10.7
44.1
MisSTE [52]
243
37.6
40.9
57.3
30.7
42.3
49.9
40.1
30.8
51.7
$5.0
43.1
30.8
41.0
27.9
27.0
40.9
Muli Expotheri aettoedi:
ei [S]]
73.6
75.1
60.0
40.8
54.8
Olkaiinn [36]
200
40.2
Shorma
[10]
DiniPowe
MitFomer [27]
351
40.9
51.2
30.6
DSDP
245
37.3
ManiPose (Oun)
243
99.6
458
41.9
97.1
42.7
52.3
47.5
98.5
43.7
53.3
43.6
40.9
48.2
27.0
10.0
47.1
ManiPowe (Ouns)
243
36.0
41.5
38.9
34.5
39.6
48.5
42.7
37.4
39.8
30.9
40.2
30.7
45.3
15.9
18.6
39.1
Protocol #1 and #2 detailed results. A detailed quantitative comparison in terms of MPJPE per action on Human3.0m 6M dataset between ManiPose and state-of ofethe-rrt methods is shown in Table 8. We see that ManiPose reaches the best MPJPE performance on average and on most actions. Table 9 contains poses). We observe the
similar analysis in terms of P-MPJPE (i.e., MPJPE with procrust-alignn same patterns as in Table 8, namely that ManiPose reaches the second-best P-MPJPE performance on average and for most actions. We confirm here again that the substantial improvements in pose consistency brought by ManiPose are not obtained at the expense of traditional metrics derived from MPJPE.
Errors per joint. On the top of Fig. 9 we see that most of MixSTE errors come from feet, elbows and wrist joints, which are most prone to depth ambiguity. ManiPose helps reduce the position errors for most of those ambiguous joints, probably byproduct of its major consistency improvements shown in Fig. 5.
Impact of hypepparameters. ManiPose introduces two additional hyperparameters when compared to MixSTE: the number K of hypotheses and the score loss weight (cf. Eq. (1)). We further assess the impact of their respective values on MPJPF. For computational cost reasons, we used smaller version of our model for this study, with transformer blocks of dimension d,, = 64 and time receptive field of T 27 frames. Fig. 10 (left) shows that more hypotheses help, but that the performance
23

---

Table 9: Quantitative comparison with the state-of-the-art methods Human3.6M under Proto- col #2 (P-MPJJPE in mm), using detected 2D poses. Bold: best; Underlined: second best. ManiPose results using the oracle evaluation. Actions: Directions, Discussion, Eating, Greeting, Talking on the Phone, Taking photo, Posing, Makes purchases, Sitting on chair, Activities while seated, Smoking, Waiting, Walking dog, Walking, Walking together.
Dir.
Doc
Eat
Greet
Phone
Photo
Pose
Purch.
Sit
SItD.
Smokk
Wait
WalDD
Walk
WalkT.
Avg.
MOCN[SS] [S5]
15.7
38.6
36.3
40.5
59.2
44.5
37.0
35.4
46.4
51.2
40.5
35.6
41.7
30.7
33.9
39.1
ST [2]
1
35.7
37.8
36.9
40.7
39.6
45.2
37.4
34,5
46.9
50.1
40.5
36.1
41.0
29.5
33.2
39.0
Pacilo et [38]
243
34.2
36.8
33.9
37.5
37.1
43.2
34.4
23.5
45.3
52.7
37.7
34.1
38.0
25.8
22.7
36.3
Zheng of [53]
34.1
36.1
34.4
37.2
56.4
42.2
34.4
33.6
45.0
52.5
37.4
33.8
37.8
25.6
27.3
36.5
[25]
243
35.2
33.3
25.8
35.9
33.2
32.7
44.6
50.9
37.0
32.4
37.0
25.2
27.2
35.6
UGCN [4]
243
35.4
36.0
25.5
1531EE122
30.3
31.8
31.1
42.5
30.8
22.1
ManiPowe (Oum)
243
31.9
35.7
30,8
33.5
34.0
33.0
11.4
41.1
45.9
36.0
32.3
35.4
24.2
23.8
36.1
improvements saturate around hypotheses. Concerning 3. Fig. 10 (right) shows that lower values help to improve the MPJPE performance.
Impact of the rotations representations used.
The disentanglement between segments' length and orientation not novel, and was proposed in previous works restricted the single-hypothesis case, such Anatomy3D [4]. While ManiPose represents segments' orientations as full 3D rotations relatively to parent segments in the kinematics tree, Anatomy3D simply predicts segments' absolute directions, i.e., normalized vectors in the 3D space. This solution has the advantage of not over- parametrizing the segments orientations (which invariant to rotations around the segment axis) and being lower dimensional (3 vs 6). One might hence wonder whether Anatomy3D's parametrization is not preferable. As shown in Table 10, Anatomy3D's implementation led to poorer results when compared to rotations parametrization multi-hypottesis setting. This motivated us to use full 3D rotations' representations proposed in [54] in our experiments, despite their caveats. Note that [54] also shows good empirical results in the related problem of inverse kinematics of human 3D poses.
Table 10: Rotations representation ablation: learning 3D directions instead of full rotations yields poorer results. Dim.: Dimension of rotations or directions representations. K: Number of hypotheses, Scores regularization. Bold: best. Underlined: second best.
Learn
Dim.
MPIPE.
MPSSE.
MPSCE.
ManiPose (Ours)
Rotations
0.1
39.1
Anatomy3D-ll [4]
Directions
39.6
Directions
0.5
41.8
3,9
6.9
Directions
3
0.5
43,2
4.4
7.5
Diversity of predicted poses. As explained in Section 5.2, ManiPose's state-of-the-art oracle MPJPE results show that excels in terms of diversity when the latter is assessed using the quantization етгоr. There are many other ways of assessing distribution diversity. In an attempt to quantify the diversity of pose distributions learned by ManiPose by other means, have computed the coverage (as defined in [35]) of generated poses relatively to the ground-truth test set of Human 3.6M. For computation cost reasons grows quadratically with sample size), we limited our analysis actions from subject $11. We compare ManiPose to DiffPose [10], using 5 hypotheses for both, and observe similar diversity on average (ef. Fig. 11).
F Code
We provide the code to reproduce all experiments under https://githhe cedri wwwcccooommmaaeeaee mmw.:rrmmmmmmmiis......
24

---

MixSTE
ManiPose
70
60
Joint-wise MPJPE [mm]
50
40
30
20
1
2
3
5
6
8
10
11
12
13
14
15
16
Joint
10
9
16
11
13
7
12
2
6
Figure 9: Detailed results on H3.6M. Top: Mean position errors per joint. Bottom: Human 3.6M
skeleton.
Aggregated MPJPE
Oracle MPJPE
80
[mm]
60
55
60
2
3
5
0.1
0.5
1.0
# Hypotheses
Score loss weight B
Figure 10: Impact of the number K of hypotheses (left) and score loss weight 3 (right) on ManiPose aggregated and oracle performance. Results are obtained on H3.6M with a smaller network (d,, = 64) and shorter sequence (T 27). Left plot obtained with / = 0.1 and right plot with K 5.
25

---

Madel ManiPose
Purchases
Waittng
Action
Phoning
SittingDown
Average
0.000
0.005
0.010
0.015
0.020
0.025
0.030
0.035
Coverage
Figure 11: ManiPose achieves similar diversity to DoffPose [10]. Diversity is assessed through the coverage [35] over test data from subject 11 from Human 3.6M. 5 hypotheses were predicted/sampled for each frame by both models.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims are enumerated the bottom of the introduction and backed by our experiments from Section 5 and our theoretical results from Section 4.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in separate section in the last page of the
manuscript.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All propositions start by stating their assumptions (some might be detailed in the appendix, such for Proposition 4.2). All results proved formally in the supplemen- tary material, with a proof sketch provided within the main article.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The experimental setting of all experiment is described in details, either in the main article (Sections 4.2 and 5) or in the supplementary material (Appendices C.I and D).
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code for the toy experiments made completely available to reviewers in the supplementary material, together with instructions to reproduce the paper figures and tables, well environment configurations. Concerning Human3.61 M.6 and MPI-INF-3DHP experiments, their code has been open-sourced to the community under publication (cf. Appendix F).
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The experimental setting of all experiment is described in details, either in the main article (Sections 4.2 and 5) or in the supplementary material (Appendices C. and D).
Justification:
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
27

---

Justification: In toy experiments, we do report error bars corresponding to the standard- deviation across several 5 runs (cf. Tables and 6). For real-world datasets, however, error bars are not reported because it would be too computationally expensive. This is customary in computer vision and practiced by all competing baselines.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are described in Appendix D.3.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https http:///...ppbboo/ccaicecostadd.
Answer: [Yes]
Justification: Our experiments comply with NeurIPS Code of Ethics, For instance, we only work with well-known publicly available datasets.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is foundational work 3D human pose lifting, and is hence not tied to any particular application. We do not believe that the enhancements it proposes could be used to make existing 3D human pose technology more dangerous any way.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The main assets used in this work are the human pose datasets Human 3.6M and MPI-INF-3DHP, whose papers are duely cited. We also provide their licenses, availability conditions and download URL in the supplementary material (Appendix D.3).
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The only asset introduced in this paper is the experimental code, which is well documented in the its own README.mo file provided alongside its (open) license.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
28

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
29


Reviews:
[{"review_id": "GMaY77IJXe", "reviewer": "Reviewer_tj82", "summary": "This paper propose ManiPose, a manifold-constrained multi-hypothesis model for 3D human pose lifting. The authors provide empirical and experimental evidence to show that joint position regression leads to inconsistent skeleton lengths. And they propose to predict globally consistent pose scale and individual joint rotations per frame (rather than joint positions) to constrain the predictions to the pose manifold. Empirical results demonstrates that the proposed ManiPose framework improves the pose consistency.", "questions": "Please review the Weaknesses Section. If the author can address or respond to the above issues well in the rebuttal stage, I will consider increasing my score.", "limitations": "As the authors discussed in the Limitations Section, they used forward kinematics to obtain joint positions, which can lead to error accumulation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper provides valuable theoretical analysis to support their arguments and provides intuitive toy examples to illustrate the ambiguity in pose lifting.\n* The paper conducts extensive experiments on H36M and MPI-INF-3DHP datasets.", "weaknesses": "* The paper uses a multi-head design to predict multiple hypotheses. This design loses the flexibility of sampling different numbers of hypotheses and limits the maximum number of hypotheses to a small number. This often results in limited hypothesis diversity. In the experimental section, the authors do not provide numerical of visual measurements of hypothesis diversity.\n* According to the comparison in Table 4, the manifold constraint proposed in this paper sacrifices MPJPE to improve pose consistency, serving as a trade-off approach between accuracy and consistency. Although the consistency is improved, it lags behind the traditional position regression or manifold regularization in accuracy, and does not bring essential improvement (improve both in accuracy and consistency) compared with these two methods.\n* Missing comparison with two recent multi-hypothesis methods. [1] GFPose: Learning 3D Human Pose Prior with Gradient Fields. [2] DiffPose: Toward More Reliable 3D Pose Estimation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and answer here their concerns in the same order.\n1. **Diversity and fixed number of hypotheses concern:**\n\n- _SOTA oracle MPJPE is evidence of good diversity:_ It is true that ManiPose produces a fixed number of poses per forward pass. While methods based on generative models (GFPose, DiffPose, etc.) typically control the number of samples drawn at test-time, we argue that in predictive tasks, selecting the right number of samples is important to capture the diversity of the plausible targets, i.e., the modes of the conditional distribution. It was indeed proven in prior works that the winner-takes-all (WTA) training scheme, on which ManiPose is based, can achieve an optimal quantization of the target distribution [A] given sufficient data. This diversity in the predicted poses can be measured through Oracle MPJPE performance, for which ManiPose achieves state-of-the-art results.\n\n- _Few hypotheses with scores are more informative:_ Moreover, note that generative methods assign equal weight/likelihood to all sampled poses for a given input. Since they require to sample a large number of poses (~200) to achieve competitive results, it becomes difficult to practically use such an uniform and high-cardinality output in a real scenario. ManiPose, on the other hand, provides more information to the user by predicting just a few relevant consistent poses with their corresponding scores/likelihoods, which is easier to process.\n\n- _Similar coverage of 3D poses distribution as DiffPose:_ In an attempt to quantify the diversity of poses predicted by ManiPose by other means than oracle MPJPE, we have computed the coverage (c.f. [B]) of generated poses over the ground-truth test distribution. For computation cost reasons (it grows quadratically with sample size), we limited our analysis to 5 actions from subject S11 of Human 3.6M. We compare ManiPose to _[2] DiffPose: Toward More Reliable 3D Pose Estimation_, using 5 hypotheses for both, and observe similar diversity on average (c.f. Figure 1 in the pdf attached to the main response).\n\n2. **Concern with Table 4 and trade-off between consistency and accuracy:**\n\n- _With a single hypothesis, one has to choose between minimizing MPJPE or pose consistency:_ We agree with the reviewer reading of Table 4 and would like to highlight that these are precisely the main messages of our work: 1) in a single-hypothesis setting, constraints will necessarily trade MPJPE for consistency (2nd row) and 2) multiple hypotheses (1st row) are needed to conciliate both MPJPE and consistency metrics.\n\n- _Table 4 is hence empirical evidence supporting our proposition 4.2 and corollary B.1.:_ This result is precisely what makes our approach principled and novel, since there are previous works proposing to constrain predicted poses to estimated manifolds (e.g. Anatomy3D) or proposing to use multiple unconstrained hypotheses (e.g. Diffpose, GFPose, …), but never both together. Please let us know if we misunderstood the reviewers’ point.\n\n3. **Concern with missing comparison with two recent multi-hypothesis methods:**\n\n- _We will add GFPose and DiffPose to our experimental results._ We thank the reviewer for pointing us to these excellent recent works. We have used their official code and checkpoints to generate 3D poses over the test split of Human3.6M and computed their pose consistency metrics.\n\n- _GFPose delivers worse pose consistency and joint position error in a comparable setting:_ Indeed, we measured an MPSCE of 16.5 mm and an MPSSE of 13.1 mm, considerably worse than ManiPose (MPSCE=0.5 and MPSSE=0.3). In terms of MPJPE, the paper reports an impressive 35.6 mm, but this score makes use of 200 sampled hypotheses, which is 2 orders of magnitude higher than us. In a fairer setting, when using a similar number of hypotheses for both methods, GFPose then stands at 45.1 mm of MPJPE with 10 hypotheses, which is considerably worse than ManiPose (39.1 mm with 5 hypotheses).\n\n- _DiffPose delivers worse pose consistency and competitive joint position error when using their checkpoint and code:_ We computed an MPSCE = 6.1 mm and MPSSE = 5.2 mm for them, which is again inferior to ManiPose. In terms of MPJPE, the authors also report an impressive number of 36.9 mm in their paper for their video model. However, using their official code and checkpoint, and setting the number of hypotheses to 5 as mentioned in the paper, we could only obtain 39.3 mm, which is competitive with ManiPose. Note that there are two other issues on their github page (which we can’t link here) mentioning precisely that the code and checkpoint do not allow to reproduce their reported results.\n\n[A] Letzelter, V., Perera, D., Rommel, C., Fontaine, M., Essid, S., Richard, G., & Perez, P. Winner-takes-all learners are geometry-aware conditional density estimators. In Forty-first International Conference on Machine Learning.\n\n[B] Naeem, M. F., Oh, S. J., Uh, Y., Choi, Y., & Yoo, J. (2020, November). Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning (pp. 7176-7185). PMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BTVnbkAVRh", "reviewer": "Reviewer_B5U1", "summary": "This paper presents a new method to estimate 3D human pose from 2D observations (lifting). To ensure the body symmetry and temporal consistency, the authors disentangle human skeleton to two parts: temporally consistency bone scales and temporally variable bone rotations. The authors use fancy formulas to prove that, minimizing MSE loss could not gurantee manifold consistency. The quantitative and qualitative results on Human3.6m and MPI-INF-3DHP datasets show the superiority of the proposed method.", "questions": "1. The citation style is weird. They are not NeurIPS style, please correct them.", "limitations": "The authors addressed limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The evalution results in this paper is quite impressive, especially the newly proposed consistency metric. Figure 1 clearly shows the superiority of the proposed method. \n\n2. The authors try to prove the theoretical optimal of the proposed method, which is worth encouraging.", "weaknesses": "I am not an expert in manifold theory, therefore my questions only relate to human pose estimation. \n\n1. How to constrain the rotation space during training? \n\n2. The pose lifting method is quite similar to Anatomy3D (bone length + rotations). Can I view this paper an multi-hypothesis extension of Anatomy3D? Why? \n\n3. Previous paper \"POSE-NDF: MODELING HUMAN POSE MANIFOLDS WITH NEURAL DISTANCE FIELDS\" is similar to this paper in concepts. SMPL naturally guarantees bone length symmetry, and the learnable parameters (rotations and shape parameters) are similar to this paper in its functionality. It would be better to cite it. \n\n4. Suppose that, there is a virtual dataset, all 2D human joints are rendered (projected) from strictly symmetric 3D joints, then, could learning the lifting function on this virtual dataset using MSE loss guarantee the results all lie on manifold? \n\n5. (An optional question) The ground truth 3D joints of Human3.6M datasets come from the marker tracking on body surface, which naturally could not guarantee skeleton length consistency. Why learning symmetric bones yields better results (both Anatomy3D and the proposed methods)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and provide hereafter our response to their concern, in the same order.\n\n1. **How to constrain the rotation space during training?**\n\n- _Our method can be adapted to incorporate angle constraints._ This is possible for example if one chooses to use rotation representations where angles appear explicitly (Axis-angle, Euler angles, …). This is not straightforward when using the 6D representations that we chose in this work though, where angles are implicit. We chose these representations because there is a bijection between them and rotations matrices, which presents optimization advantages and allows us to avoid training instabilities (cf. [43] On the continuity of rotation representations in neural networks. CVPR 2019.)\n- _A possible solution:_ If we chose to predict directions instead of rotations (cf. answer to reviewer fpqy Q2), then angles could be easily made explicit in our representation (using spherical coordinates for instance) and we could constrain them to stay within a certain interval by using simple sigmoid activations at the end of our rotations network. This idea could be a nice future extension of our work.\n\n2. **Is ManiPose a multi-hypothesis extension of Anatomy3D?**\n\n- _Yes, in a sense:_ It is true that, similar to Anatomy3D, ManiPose disentangles limbs length and orientation in order to constrain predicted poses to lie in an estimated manifold. In this sense, we can indeed see it as an extension to the multi-hypothesis setting.\n\n- _But with new theoretical results and a very different message:_ Please note however that, unlike Anatomy3D, we provide theoretical proofs and empirical evidence that both constraints and multiple hypotheses are needed if one wants to optimize joint position error and pose consistency together. This is quite a different message than the one in Anatomy3D paper.\n\n- _And a different way of constraining poses:_ Also note that we use a different representation of limbs orientations than Anatomy3D to constrain our predicted poses (cf. answer to reviewer fpqy Q2). New ablation results in Table 1 of the pdf attached to our main response show that our representation yields better results in the multi-hypothesis setting.\n\n3. **Concern with SMPL-methods and Pose-NDF citation:**\n\n- _We will eagerly cite Pose-NDF._ We knew and hold this work in high regard. We agree with the reviewer that it is related to our work, as it implicitly learns a plausible sub-manifold of $SO(3)^J$ for poses.\n- _Please note however that Pose-NDF is part of the “body pose and shape estimation” field, which is related but different from 3D human pose lifting._ The objective of most SMPL-based methods is to predict whole 3D body meshes from images instead of 3D joint positions based on 2D keypoints. The task is more challenging, which means that algorithms are heavier and more reliant on optimization-based post-processing. For instance, Pose-NDF estimates 3D meshes in a single image by initializing pose angles and using gradient descent to project them into the learned pose manifold, which is different from doing a simple forward pass with ManiPose.\n\n4. **Question regarding synthetic data and MSE loss:**\n\n- We can guarantee that if multiple hypotheses _and_ constraints are not used together, then the predictions of the lifting function learned using just MSE loss on the synthetic data proposed by the reviewer _will not_ lie on the manifold (unless the distribution of 3D poses conditioned on 2D poses are Dirac measures, i.e., if there is always a single possible 3D pose projecting into 2D space).\n- This is proved in Proposition 4.2 and shown (with simplified 2-joint and 3-joint articulated objects) in experiments of sections 4.2 and C.2.\n\n5. **Optional question regarding MoCap:**\n\n- The Human3.6M dataset, as many other datasets, does not contain raw MoCap measurements, but rather post-processed estimations of 3D joint positions obtained through heavy optimization. One of the constraints that is enforced during such post-processing steps is precisely that limb lengths do not vary over time for a given subject. We verify this for ground-truth poses of Human3.6M dataset in figure 5 of our paper (cf. legend).\n\n6. **Citation style**\n- We apologize for the citation style and will correct it in our revision to comply with the NeurIPS format.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BGrsQODrp8", "reviewer": "Reviewer_5BLY", "summary": "This paper proposes a MCL-based framework for multi-hypothesis 3D human pose estimation. This framework predicts skeletal parameters so that the predicted 3D poses in a sequence are constrained to one smooth manifold. To prove the superiority of such a framework, the paper presents detailed theoretical analysis on the drawback of unconstrained single-hypothesis HPE and why MPJPE alone is not enough for pose evaluation. The experiments show the proposed framework is capable of keeping the consistency of predicted poses and achieving state-of-the-art MPJPE in the meantime.", "questions": "What is the quality of the score for each hypothesis? If the multiple hypotheses are fused to one (e.g. by taking the one with the largest confidence or taking the weighted average), then how will the MPJPE, MPSCE, and MPSSE change?", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* Simple and reasonable manifold representation. The proposed framework keeps the predicted human pose on the target manifold by representing the human pose with bone lengths and orientations, and the 3D pose is a direct inference from forward kinematics. The manifold is represented by the kinematics itself.\n  \n* Inspiring theoretical analysis on basic problems in 3D HPE. The paper arrives at some theoretical conclusions (line178-183), along with detailed proofs. They can provide some refreshing ideas on the innate drawbacks of traditional loss functions and MPJPE metrics.\n  \n* Good performance under both MPJPE and consistency measures, as validated in Table 2 and 3.", "weaknesses": "* Theoretical analysis on the advantage of multi-hypothesis methods over single-hypothesis ones could be added. Specifically, why a **constrained multi-hypothesis** method performs better than an **unconstrained single-hypothesis** method in MPJPE? Though this is already validated by the experiments, I personally believe it would make the paper more solid if the authors could make this analysis.\n\nMinor problem:\n* In Fig.4 (C) and (D), it is not quite clear how the estimations (crosses and triangles) correspond with the inputs (black dots). There might be some unexpected shifts, as the projections of the predicitons do not strictly align with the inputs (like in B).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We answer their remarks below, following the same order.\n\n1. **New theoretical analysis on the advantage of multi-hypothesis methods over single-hypothesis:**\n\nWe agree with the reviewer and provide the proposed theoretical result hereafter.\n\n- Let $\\mathcal{X}=\\mathbb{R}^{2 \\times J}$ denote the space of input 2D poses and $\\mathcal{P}=\\mathbb{R}^{3 \\times J}$ the space of 3D poses. Following Rupprecht et al. 2017 [31] and Letzelter et al. 2024 [A], we define the “oracle risk” for a K-hypothesis model $f_{\\text{WTA}} = (f_{\\text{WTA}}^1, \\dots, f_{\\text{WTA}}^K)$ as:  \n$$  \\mathcal{R}^K (f_{\\text{WTA}}) \\triangleq \\int_\\mathcal{X}\n\\sum_{k=1}^{K} \\int_{\\mathcal{V}^k(f_\\text{WTA}(\\mathrm{x}))} \\\\|f_{\\text{WTA}}^k(\\mathrm{x}) - \\mathrm{p} \\\\|^2_2 \\rho(\\mathrm{x}, \\mathrm{p}) \\mathrm{d} \\mathrm{p} \\mathrm{d} \\mathrm{x},  $$\nwhere $\\mathcal{V}^k(g)$ denotes the $k^{th}$ cell of the Voronoi tesselation of the output space $\\mathcal{P}$ defined by generators $g=(g^1, \\dots, g^K) \\in \\mathcal{P}^K$:\n$$\\mathcal{V}^k(g) \\triangleq \\Big\\\\{ \\mathrm{p} \\in \\mathcal{P} \\;\\Big|\\; \\\\| g^k - \\mathrm{p} \\\\|^2_2 < \\\\| g^r - \\mathrm{p} \\\\|^2_2, \\forall r \\neq k \\Big\\\\}.$$\nThe risk above translates the notion of oracle pose, since it partitions the space of ground-truth poses $\\mathcal{P}$ into regions where some hypothesis is the closest, and uses only that hypothesis to compute the risk in that region.\n\n_Now we can state our new proposition:_\n\n- A $K$-hypothesis model $f_{\\text{WTA}}^*=(f_{\\text{WTA}}^{1*}, \\dots, f_{\\text{WTA}}^{K*} )$ minimizing $\\mathcal{R}^K$ has always a risk lower or equal to a single-hypothesis model $f_\\text{MSE}^*$ minimizing $\\mathcal{R}^1$:\n$$\\mathcal{R}^K (f_{\\text{WTA}}^*) \\leq \\mathcal{R}^1(f_\\text{MSE}^*) = \\min_f \\mathbb{E}_{\\mathrm{x}, \\mathrm{p}}[\\\\| \\mathrm{p} - f(\\mathrm{x})\\\\|^2_2]  = \\mathbb{E}\\_{\\mathrm{x}, \\mathrm{p}} [ \\mathrm{p}  \\| \\mathrm{x} ].$$\n\n_The proof relies on the following steps:_\n\n- First we assume that $f_{\\text{WTA}}$ is expressive enough, so that, minimizing the risk $\\mathcal{R}^K$ comes down to minimizing\n$$\\mathcal{R}\\_{\\mathrm{x}}^K (f\\_{\\text{WTA}}) \\triangleq \\sum_{k=1}^{K} \\int_{\\mathcal{V}^k(f_\\text{WTA}(\\mathrm{x}))} \\\\|f_{\\text{WTA}}^k(\\mathrm{x}) - \\mathrm{p} \\\\|^2_2 \\rho(\\mathrm{p} | \\mathrm{x}) \\mathrm{d} \\mathrm{p} ,$$\nfor each $\\mathrm{x} \\in \\mathcal{X}$.\n\n- Following [A] (Section 2.2), we decouple the cell generators from the risk arguments:\n$$\\mathcal{K}(g, z) \\triangleq \\sum_{k=1}^{K} \\int_{\\mathcal{V}^k(g)} \\\\|z^k - \\mathrm{p} \\\\|^2_2 \\rho(\\mathrm{p} | \\mathrm{x}) \\mathrm{d} \\mathrm{p},$$\nfor any generators $g=(g^1, \\dots, g^K) \\in \\mathcal{P}^K$ and arguments $z=(z^1, \\dots, z^K) \\in \\mathcal{P}^K$.\nNote that $\\mathcal{R}_{\\mathrm{x}}^K(f) = \\mathcal{K}(f(\\mathrm{x}), f(\\mathrm{x}))$.\n\n- Next, according to Proposition 3.1 of [B], if $f_{\\text{WTA}}^*$ minimizes the input-dependent risk $\\mathcal{R}\\_{\\mathrm{x}}^K (f\\_{\\text{WTA}})$, then $(f_{\\text{WTA}}^*(\\mathrm{x}), f_{\\text{WTA}}^*(\\mathrm{x}))$ has to minimize $\\mathcal{K}$:\n$$\\mathcal{K}(f_{\\text{WTA}}^*(\\mathrm{x}), f_{\\text{WTA}}^*(\\mathrm{x})) \\leq \\mathcal{K}(g, z), \\qquad \\forall g, z \\in \\mathcal{P}^K \\times \\mathcal{P}^K.$$\n\n- Finally, let's choose $g$ such that $g^k=f_{\\text{WTA}}^{k*}(\\mathrm{x})$ and $z$ such that $z^k = f_\\text{MSE}^*(\\mathrm{x})$ for all $1 \\leq k \\leq K$. Then\n$$\\mathcal{R}\\_\\\\mathrm{x}^K (f\\_{\\text{WTA}}^{k*}) \\leq\n\\sum_{k=1}^{K} \\int_{\\mathcal{V}^k(f_\\text{WTA}^*(\\mathrm{x}))} \\\\|f_\\text{MSE}^*(\\mathrm{x}) - \\mathrm{p} \\\\|^2_2 \\rho(\\mathrm{p} | \\mathrm{x}) \\mathrm{d} \\mathrm{p} = \\mathcal{R}^1_\\mathrm{x}(f_\\text{MSE}^*) ,$$\nwhere the last equality comes from the fact that $\\mathcal{V}^k(f_\\text{WTA}^*(\\mathrm{x}))$ defines a partition of $\\mathcal{P}$.\n\n2. **Fig.4 misalignments:**\n\nIndeed, it seems that some misalignment was introduced during editing and should be corrected in our revision.\n\n3. **Metrics with hypotheses fused to one:**\n\n- _MPJPE degrades:_ When we compute the weighted average (cf. penultimate row in tables 3 and 8 in the paper), we see that MPJPE performance is hurt (42.1 mm in H3.6M instead of 39.1). This might indeed indicate that scores could be better estimated, but it could also just be a consequence of the new proposition proved above in point 1.\n\n- _Expected from our new theoretical result:_ Indeed, according to [17, Equation 8], the weighted average is an estimate of the conditional expectation $\\mathbb{E}[\\mathrm{p} \\| \\mathrm{x}]$, i.e., the best single-hypothesis model $f\\_\\text{MSE}^*$. Hence, according to the proposition proven above, it should underperform the multi-hypothesis model at the limit.\n\n- _MPSSE and MPSCE also degrade as expected:_ Concerning pose consistency, we computed an MPSSE of 0.4 mm and an MPSCE of 0.8 mm, which are again worse than for the oracle pose. This is of course expected, since Proposition 4.2 in the paper proves that single-hypothesis models (which the aggregation approximates) are bound to lie outside the pose manifold.\n\n[A] Letzelter, V., Perera, D., Rommel, C., Fontaine, M., Essid, S., Richard, G., & Perez, P. Winner-takes-all learners are geometry-aware conditional density estimators. In Forty-first International Conference on Machine Learning.\n\n[B] Du, Q., Faber, V., & Gunzburger, M. (1999). Centroidal Voronoi tessellations: Applications and algorithms. SIAM review, 41(4), 637-676.\n\n[17] Letzelter, V., Fontaine, M., Chen, M., Pérez, P., Essid, S., & Richard, G. (2023). Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysis. Advances in neural information processing systems, 36.\n\n[31] Rupprecht, Christian, et al. \"Learning in an uncertain world: Representing ambiguity through multiple hypotheses.\" Proceedings of the IEEE international conference on computer vision. 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5uQkjgVeRT", "reviewer": "Reviewer_fpqy", "summary": "This paper presents a method to estimate 3D human keypoints from a sequence of monocular 2D keypoints observations. It builds upon an existing sequence-to-sequence architecture (MixSTE), with a different output parameterization exploiting a kinematic skeletton prior, and different training losses. Lengths of the skeletton bones are predicted for the whole sequence to ensure consistency across frames (and maybe also left/right symmetry of the skeletton), and five 3D pose hypotheses with associated scores are predicted for each frame, parameterized as a list of 3D relative orientation for each bone with respect to its parent in the kinematic tree.\n\nThe authors develop theoretical arguments regarding the benefits of enforcing such structural priors in the predictions, and illustrate with a toy example the interest of having multiple predictions in case of ambiguous multimodal output. They validate their approach on Human3.6M and MPI-INF-3DHP datasets.", "questions": "See the weaknesses section for a list of suggestions.", "limitations": "N/A", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "The motivation for exploiting bone lengths constraints is well expressed, with a clear and detailed discussion provided in Section 4. The discussion of experimental and ablation results is insightful and shows – in a setting dependent on an oracle – benefits of the proposed approach.", "weaknesses": "The idea of enforcing body priors (constant bone length here) is not novel and has actually been heavily exploited in a whole line of work relying on more advanced parametric models such as SMPL [100]. This line of work would deserve being considered in the paper, as it encompass approaches suitable for 2D-to-3D sequence lifting such as e.g. [101].\n\nThe authors present a pose space consisting in 3D coordinates of joints linked by some rigid segments. Based on this definition, a natural pose parameterization would consist in the 3D direction of each segment, yet the authors chose to overparameterize poses by using relative 3D bone orientation instead. I understand that such choice can have practical benefits in term of biomechanical constraints and additional supervision signal when ground truth data is available, but such choice should be properly motivated, discussed and ablated in the paper.\n\n\nThe authors describe two ways of aggregating results L247 but do not state which one they use for MPI-INF-3DHP, and they only report oracle results on Human3.6M and for the ablations.\n\nIn my understanding, pose hypotheses are selected independently for each frame and there are no temporal terms in the training objectives or aggregation method. Since the proposed approach deals with temporal sequences, it would be worth evaluating the temporal consistency of the predictions, through qualitative video examples and quantitatively e.g. using joint acceleration metrics. Having multiple hypotheses for each frame brings combinatorial questions worth discussing in my opinion.\n\nReferences:\n- [100] Loper at al., “SMPL: A Skinned Multi-Person Linear Model”, at SIGGRAPH Asia 2015.\n- [101] Baradel et al., “PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling”, in TPAMI 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We answer their remarks below, following the same order.\n\n1. **Citation of SMPL-based methods:**\n\n- _We will properly cite these important works._ We agree with the reviewer that SMPL-based methods share the same constant-bone-length assumption that we present, and that it represents an extensive line of work which is worth mentioning. We thank the reviewer for pointing this out to us.\n\n- We would like to highlight however a few differences between these works and ours:\n\n    - _HMR is a field solving a more complex task:_ “Human body pose and shape estimation” (a.k.a. “human body reconstruction”, “human mesh reconstruction”, “HMR”, …) is a separate field, different from our “3D human pose lifting”: the objective of the former is to predict whole 3D body meshes from images, as opposed to 3D joint positions from 2D keypoints for the latter. This means that these works tackle a different task, which is more challenging.\n    - _HMR is often frame-based:_ Because their task is more challenging, SMPL methods are more computation-heavy and often restricted to single-frame predictions, which are then smoothed across frames using optimization-based post-processing. A good example of this methodology can be seen in [A].\n    - _SMLP methods deliver worse MPJPE:_ Of course, predicting the human mesh across time includes the skeleton pose, which means that some of these methods do evaluate MPJPE. Note however that, while [A] is SOTA in Human3.6M data last time we checked, their performance (MPJPE=44.8 mm) lags behind 3D human pose lifting methods (c.f. Table 2 and 8 in our paper).\n    - _Our novelty lies on multiple hypotheses with constraints:_ Furthermore, please note that our work proposes not just to restrict predicted poses to have constant bone length, but proves that this is not sufficient to optimize both pose fitting (i.e., joint position error) and consistency (i.e., avoiding bone stretching). We prove that multiple hypotheses are needed to accommodate both objectives and propose a practical implementation, show-cased in our experimental results.\n\n2. **Concerns regarding rotations parametrization:**\n\n- _Learning 3D directions instead of full rotations yielded poorer results:_ We understand the reviewer’s concern. It is true that, when compared to full 3D rotation matrices, predicting simple direction vectors (as done in Anatomy3D [4] for example) is a simpler way of parametrizing poses where joints are connected by segments. As advised by the reviewer, we hence performed a new ablation study, where we compare our parametrization to the one used in Anatomy3D, i.e., where the rotations module predicts normalized vectors representing the direction of each segment. As shown in Table 1 of the pdf attached to our global answer, the latter leads to poorer oracle MPJPE of 39.6 mm when compared to our rotations parametrization (39.1 mm) in a multi-hypothesis setting.\n\n- _Our choice was guided by relevant previous works:_ To clarify our choice, we opted for our rotations representation building on results from [43], where the benefits of 6D representations of SO(3) matrices is studied. Please note that [43] includes good results in an inverse kinematics problem with \"stick\" human poses similar to ours.\n\n3. **Concern regarding results with aggregated poses:**\n\n- _Values for MPI-INF-3DHP are reported in Table 4:_ The penultimate row corresponds to a weighted averaged pose, while the last row corresponds to the oracle pose.\n\n- _Values for Human3.6M are reported in Table 8 in the appendix:_ Metrics of the aggregated pose can be found at the penultimate row of Table 8. We understand that this can be missed since it is in the appendix.\n\n- _We will clarify this in our revision._ We apologize if the results with aggregated poses are not stated with enough clarity in our manuscript.\n\n- _MPSCE performance is reported above:_ As explained to reviewer 5BLY in Q3, both MPJPE and MPSCE performance are hurt when using the averaged pose. While the latter is expected according to Proposition 4.2 in the paper, the MPJPE degradation is explained by a new theoretical result proved in the answer to reviewer 5BLY.\n\n4. **Concern regarding temporal consistency of predictions:**\n- _The training objective includes a temporal term in the Human 3.6M and MPI-INF-3DHP experiments._ We apologize if this was not very clear in our manuscript. It is mentioned in Appendix D.3 lines 602-609, where we explain that TCloss and velocity loss terms were added to our original objective in order to have a fair comparison with existing art. We chose not to add this in the main manuscript because of space limitations and because they are relatively standard practice. We agree that it may lead to confusion and will hence add a longer note on this in our revision to the main paper.\n\n- _We provide a gif demonstrating temporal consistency:_ Although we understand the reviewers concern, we found no important issues regarding this point. We believe this to be explained by the aforementioned loss terms and the fact that the model predicts sequences instead of single frames. The reviewer can find a gif in the pdf attached to the main response (NeurIPS rules forbid us from providing a link to a video). Note that Adobe Reader is needed to visualize it.\n\n[A] Goel, Shubham, et al. \"Humans in 4D: Reconstructing and tracking humans with transformers.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[4] Chen, Tianlang, et al. \"Anatomy-aware 3d human pose estimation with bone-based pose decomposition.\" IEEE Transactions on Circuits and Systems for Video Technology 32.1 (2021): 198-209.\n\n[43] Zhou, Yi, et al. \"On the continuity of rotation representations in neural networks.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their work. We provide answers to all their concerns individually, referring sometimes to the pdf attached to this general answer.

We would like to highlight that our rebuttal includes:
- a new theoretical result, together with its proof sketch,
- a new ablation study related to our rotations representation,
- the evaluation of two new baseline methods,
- and new evaluations of our method in terms of diversity and consistency of aggregated poses.

Meta-review: This paper presents a manifold-constrained multi-hypothesis model for 3D human pose lifting.  To ensure the body symmetry and temporal consistency, the method disentangles human skeleton to two parts: temporally consistency bone scales and temporally variable bone rotations. The method then predicts skeletal parameters so that the predicted 3D poses in a sequence are constrained to one smooth manifold. The paper provides theoretical arguments regarding the benefits of enforcing such structural priors in the prediction. The proposed method is validated on Human3.6M and MPI-INF-3DHP datasets. Although theoretical analysis to support the arguments in the paper is appreciated, the reviewers raised concerns regarding diversity and fixed number of hypotheses, unclear advantage of multi-hypothesis methods, undetailed explanation about the method, trade-off between accuracy and consistency, missing comparison, and difference from Anatomy3D and Pose-NDF.  The authors addressed in their rebuttal most of the concerns by providing new arguments, new experiments and new theoretical analysis. During post-rebuttal discussion, the authors further clarified the contributions of the paper, and convinced three of the four reviewers.  The reviewer tj82 suggested to specify the definition of diversity, which AC thinks should be fixed.  The review fpqy, on the other hand, was not satisfied the feedbacks from the authors.  He/She keeps concern on the positioning of the paper, meaning that theoretical contribution or 3D pose lifting application is unclear.  It was also pointed out that the new theoretical results provided in the rebuttal are obvious.  AC does not agree with the reviewer fpqy in the sense that even obvious theoretical results are sometimes valuable.  This is because results in good harmony with our intuition bring to us the strong ground of the method rather than heuristically/empirically developed methods. AC also thinks the balance between theoretical analysis and the proposed lifting method is good and does not reduce the strengths of the paper. This paper should be accepted, accordingly.
Decision: Accept (poster)