Title: ManiPose: Manifold-Constrained Multi-Hypothesis 3D Human Pose Estimation
Venue: NeurIPS 2024
Accepted: yes
Keywords: human pose estimation

depth ambiguity

multiple choice learning
Abstract: We propose ManiPose, a manifold-constrained multi-hypothesis model for human-pose 2D-to-3D lifting. We provide theoretical and empirical evidence that, due to the depth ambiguity inherent to monocular 3D human pose estimation, traditional regression models suffer from pose-topology consistency issues, which standard evaluation metrics (MPJPE, P-MPJPE and PCK) fail to assess.  ManiPose addresses depth ambiguity by proposing multiple candidate 3D poses for each 2D input, each with its estimated plausibility. Unlike previous multi-hypothesis approaches, ManiPose forgoes generative models, greatly facilitating its training and usage. By constraining the outputs to lie on the human pose manifold, ManiPose guarantees the consistency of all hypothetical poses, in contrast to previous works. We showcase the performance of ManiPose on real-world datasets, where it outperforms state-of-the-art models in pose consistency by a large margin while being very competitive on the MPJPE metric.

ManiPose: Manifold-Constrained Multi-Hypothesis 3D Human Pose Estimation

Abstract

We propose ManiPose , a manifold-constrained multi-hypothesis model for humanpose 2D-to-3D lifting. We provide theoretical and empirical evidence that, due to the depth ambiguity inherent to monocular 3D human pose estimation, traditional regression models suffer from pose-topology consistency issues, which standard evaluation metrics (MPJPE, P-MPJPE and PCK) fail to assess. ManiPose addresses depth ambiguity by proposing multiple candidate 3D poses for each 2D input, each with its estimated plausibility. Unlike previous multi-hypothesis approaches, ManiPose forgoes generative models, greatly facilitating its training and usage. By constraining the outputs to lie on the human pose manifold, ManiPose guarantees the consistency of all hypothetical poses, in contrast to previous works. We showcase the performance of ManiPose on real-world datasets, where it outperforms state-of-the-art models in pose consistency by a large margin while being very competitive on the MPJPE metric.

1 Introduction

We propose ManiPose , a novel approach for human-pose 2D-to-3D lifting. ManiPose directly addresses the depth ambiguity inherent to monocular 3D human pose estimation by being both multihypothesis and manifold-constrained, thus avoiding pose consistency issues, which plague traditional regression-based methods. Unlike previous multi-hypothesis approaches, ManiPose forgoes the use of costly generative models, while still estimating the plausibility of each hypothesis.

Monocular 3D human pose estimation (HPE) is a challenging learning problem that aims to predict 3D human poses given an image or a video from a single camera. Often, the problem is split into two successive steps: first 2D human pose estimation, then 2D-to-3D lifting. Such separation is favorable because 2D-HPE is much more mature, leading to better overall results. Due to depth ambiguity and occlusions, 2D-to-3D lifting is intrinsically ill-posed: multiple 3D poses correspond to the same projection observed in 2D. Despite that, the field has experienced fast developments, with substantial improvements in terms of mean-per-joint-prediction error (MPJPE) and derived metrics ( e.g. , P-MPJPE, PCK) [52, 53, 42, 47].

However, recent studies [49, 12, 40] noted that poses predicted by state-of-the-art models fail to respect basic invariances of human morphology, such as bilateral sagittal symmetry, or constant length across time of rigid body segments connecting the joints. Not only do we address those concerns with ManiPose (see Fig. 1), but we also provide theoretical elements clarifying the cause of those issues. We show in particular that pose consistency and traditional performance metrics (such as MPJPE)

+ manifold constraints

+ multi-hypothesis

cannot be optimized simultaneously by a standard regression model, because MPJPE ignores the topology of the space of human poses, and traditional regression models imply unimodality, thus overlooking the inherently ambiguous nature of 3D-HPE.

Our contributions include:

ManiPose, a novel, multi-hypothesis, manifold-constrained model for human-pose 2D-to3D lifting, which is able to estimate the plausibility of each hypothesis without resorting to costly generative models.

Theoretical insights that elucidate why traditional regression models associated with standard metrics such as MPJPE fail to enforce pose consistency.

Extensive empirical results, including comparison to strong baselines, evaluation on two challenging datasets (Human 3.6M and MPI-INF-3DHP), and ablations. ManiPose outperforms state-of-the-art methods by a substantial margin in terms of pose consistency, while still beating them in the MPJPE metric. The ablations confirm the importance of both multiple hypotheses and of constraining the poses to their manifold.

The PyTorch [37] implementation of ManiPose and code used for all our experiments can be found at https://github.com/cedricrommel/manipose .

2 Related work

Regression-based 2D-to-3D pose lifting. While 2D-to-3D human pose lifting was initially restricted to static frames [31, 3], the field embraced recurrent [13], convolutional [38] and graph neural networks [2, 55, 14, 51] to handle motion. Spatial-temporal transformers appear more recently [42, 53], including MixSTE [52], arguably becoming the state of the art. We adopt them in our work. A few previous works constrain predicted poses to respect human symmetries [50, 4], an idea we advance with a novel constraint implementation, in a multi-hypothesis setting.

SMPL-based methods. While 3D human pose lifting's objective is to predict 3D joint positions based on 2D keypoints, the neighboring field of human pose and shape reconstruction (HPSR) aims at estimating whole 3D body meshes from images. HPSR is hence more challenging than 3D-HPE, which explains why models are often larger, frame-based and more reliant on optimization-based post-processing [16, 39, 46, 9]. Nonetheless, our work shares some ideas from this field. Indeed, modern HPSR methods often predict joint angles (and body shape parameters), which are fed to the pre-trained parametric model SMPL [29] to produce human body meshes, thus ensuring that limbs' sizes remain constant along a movement. Note, however, that these are also single-hypothesis regression methods and hence share the same caveats as most 3D-HPE approaches.

Multi-hypothesis 3D-HPE. The intrinsic depth-ambiguity of 3D-HPE led the community to investigate multi-hypothesis approaches, including Mixture Density Networks [25, 36, 1], variational autoencoders [44], normalizing flows [18, 49] and diffusion models [12, 6, 10]. Contrary to ours, those methods rely on a generative model to sample 3D pose hypotheses conditioned on the 2D input. A notable exception is MHFormer [27], which, like ManiPose, is deterministic, but treats the hypotheses as intermediate representations to be aggregated at the final network layers, thus concluding with a one-to-one 2D-to-3D mapping. We strive to avoid such injectivity and to preserve the multiple hypotheses, for reasons we will justify both empirically and theoretically in the next sessions. Moreover, none of the previous multi-hypothesis approaches constrain hypotheses to lie on the human pose manifold, thus failing to guarantee good pose consistency.

Multiple choice learning (MCL) [11] is a simple approach for estimating multimodal distributions, suited for ambiguous tasks, using the winner-takes-all loss. Adapted for deep learning by Lee et al. [20, 21], it produces diverse predictors, each specialized in a particular subset of the data distribution. MCL has proved its effectiveness in several computer vision tasks [41, 19, 33, 8, 30, 45], and was first applied to 2D-HPE in [41]. Our work is the first to employ MCL for the 3D-HPE task, by leveraging recent innovations of Letzelter et al. [22].

3 ManiPose

Rotations

Module

Segments

Module

Pose

Decoder

Head 1

Head

K

...

...

Hyp. 1

Hyp.

K

Learned

Not learned

Rotations

Scores

Segments

length

2D keypoints

Legend:

K

Yt

K

K

Yt

S

Following the previous state of the art, we split 3D-HPE into two steps, first estimating J human 2D keypoints in the pixel space from a sequence of T video frames [x 1 , . . . , x T ] ∈ R 2 × J × T , and then lifting them to 3D joint positions [ˆ p 1 , . . . , ˆ p T ] ∈ R 3 × J × T . We focus on the second step ( i.e. , lifting) in the rest of the paper, assuming the availability of 2D keypoints x i . Our method aims to both ensure pose consistency and resolve depth ambiguity, as we will discuss in the next section.

3.1 Constraining predictions to the pose manifold

Rationale. Human morphology prevents the joints from arbitrarily occupying the whole space. Instead, the poses within a movement are restricted to a manifold, reflecting the human skeleton's rigidity. If we knew the length of each segment connecting pairs of joints for a given subject, we could guarantee that the predicted poses lie on the correct pose manifold by only predicting the body part's rotations with respect to a reference skeleton. Since we do not have access to ground-truth segment lengths in real use cases, we propose to predict them, thus disentangling the estimation of the reference lengths (fixed across time) from the estimation of the joint rotations (variable across time).

Disentangled representations. We constrain model predictions to lie on an estimated manifold by predicting parametrized disentangled transformations of a reference pose u ∈ ( R 3 ) J , for which all segments have unit length. Namely, we propose to split the network into two parts ( cf. Fig. 2):

Segments module , which predicts segment lengths s ∈ R J -1 , shared by the T frames (time steps) of the input sequence;

Rotations module , which predicts the rotation r = [ r 1 , 0 , . . . , r T,J -1 ] ∈ ( R d ) J × T of each joint relative to their parent joint at each time step.

Rotations representation. We represent rotations using 6D continuous embeddings ( i.e. , d = 6 ). Compared to quaternions or axis-angles, those representations are continuous and, hence, better learned by neural networks, as demonstrated by their proposers [54].

Pose decoding. To deliver pose predictions in ( R 3 ) J × T , the intermediate representations ( s, r ) must be decoded. We achieve that in three steps ( cf. Fig. 3):

We scale the unit segments of the reference pose u ∈ ( R 3 ) J using s , forming a scaled reference pose u ′ : u ′ j = u ′ τ ( j ) + s j (u j -u τ ( j ) ) for 0 < j ≤ J -1 , where τ maps the index of a joint to its parent's, if any.

For each time step 1 ≤ t ≤ T and joint 0 ≤ j < J , we convert the predicted rotation representations r t,j into rotation matrices R t,j ∈ SO(3) (Algorithm 1).

We apply those rotation matrices R t,j at each time step t to the scaled reference pose u ′ using forward kinematics (Algorithm 2).

3.2 Multiple choice learning

ManiPose architecture. As explained in the introduction, the inherent depth ambiguity of pose lifting requires multiple hypotheses to conciliate pose consistency and MPJPE performance. To address this, we adopt the multiple choice learning (MCL) [21] framework, more precisely leveraging the resilient MCL approach as proposed by Letzelter et al. [22]. This methodology allows the estimation of conditional distributions for regression tasks, enabling our model to predict multiple plausible 3D poses for each 2D input. Specifically, instead of a single rotation r t ∈ ( R d ) J per time step, ManiPose's rotations module predicts an intermediate representation e t ∈ ( R d ′ ) J that feeds K linear heads (with weights W k r and W k γ ), each predicting its own rotation hypothesis r k t ∈ ( R d ) J with a corresponding likelihood γ k t ∈ [0 , 1] . That is, for all 1 ≤ t ≤ T , r k t = W k r e t and γ k t = σ [˜ γ t ] k , where the softmax function σ is applied to the vector ˜ γ t = [˜ γ 1 t , . . . , ˜ γ t K ] ∈ R K of intermediate values ˜ γ k t = W k γ e t .

All rotation hypotheses are decoded together with the shared segment-length predictions s , resulting in K hypothetical pose sequences ˆ p k = (ˆ p k t ) T t =1 , with corresponding likelihood sequences γ k = ( γ k t ) T t =1 , called scores hereafter (Fig. 2).

Loss function. As in [22], ManiPose is trained with a composite loss

The first term, L wta, is the winner-takes-all loss [21]

where ℓ (ˆ p k t (x) , p t ) ≜ 1 J ∑ J -1 j =0 ∥ p t,j -ˆ p k t,j (x) ∥ 2 , and ˆ p k t (x) denotes the pose prediction at time t using the k th head. The second term, L score, is the scoring loss

where H ( · , · ) is the cross-entropy, ˆ p t = (ˆ p k t ) K k =1 , and

is the indicator function of the winner pose hypothesis, which is the closest to the ground truth. Eq. (3) is the average cross-entropy between target and predicted scores γ t (x) ∈ [0 , 1] K at each time t .

Those losses are complementary. The winner-takes-all loss updates only the best predicted hypothesis, specializing each head on part of the data distribution [21]. The scoring loss allows the model to learn how likely each head is to winning, thus avoiding overconfidence of non-winner heads ( cf. [19, 45]).

Conditional distribution estimation. As detailed in [22], the model may be interpreted probabilistically as a multimodal conditional density estimator. More precisely, it models the distribution P(p | x) of 3D poses conditioned on 2D poses as a mixture of Dirac distributions:

Hence, the predicted conditional distribution has, at each predicted hypothesis ˆ p k , a peak whose likelihood is given by the predicted score γ k . As described in Section 4, interpreting hypotheses and scores probabilistically enables us to handle depth ambiguity.

4 Formal analysis

ManiPose, as outlined in Section 3, is crafted to address the flaws inherent in unconstrained, single-hypothesis lifting-based 3D-HPE methods (see Fig. 1). This section illustrates that without ManiPose's critical components (multiple hypotheses and manifold constraint), it is impossible to simultaneously minimize joint error and ensure pose consistency (Section 4.1). To illustrate this, a toy example within a simplified 1D-to-2D framework is provided in Section 4.2.

4.1 Single-hypothesis position-error minimization leads to inconsistent skeleton lengths

Rotations

Conversion

Forward

Kinematics

Segments

Scaling

Predicted

movement

Scaled ref. pose

Unit ref. pose

We formally highlight the limitations of unconstrained single-hypothesis 3D-HPE, justifying our approach, which combines consistency constraints and multiple hypotheses to resolve depth ambiguity.

Let p = [p 1 , . . . , p J ] ∈ R 3 × J be a human pose, defined by the Cartesian 3D coordinates of each of the J joints of a predefined skeleton. Then, a sequence of T poses of the same subject at increasing time steps t 1 . . . t T ∈ R forms a movement m = [p 0 , . . . , p T ] ∈ R 3 × J × T . Assuming bone length is fixed during a movement (which is empirically verifiable in human pose datasets), then the poses p t of m must all lie on the same smooth manifold.

Proposition 4.1 (Human pose manifold) . Assuming a rigid skeleton, all poses of a movement m = [p t ] T t =1 lie on a manifold M of dimension 2( J -1) :

Proof sketch. (Detailed in Appendix B). Skeleton rigidity implies that, if i is a joint connected to the root, then it lies on a 2D sphere S 2 (0 , s i, 0 ) centered at the origin with fixed radius s i, 0 . Another joint j linked to i has a position expressible by its spherical coordinates relative to i with fixed radius s j,i . That implies an homeomorphism between the position p t,j of joint j and the direct product of spheres centered at the origin S 2 (0 , s i, 0 ) × S 2 (0 , s j,i ) . By induction, one can show that p t lies on a subspace of ( R 3 ) J , which is homeomorphic to a product of spheres centered at the origin. ■

Proposition 4.1 implies that all poses predicted for a video sequence should ideally lie on the same manifold M as the ground-truth data, which is homeomorphic to the direct product of 2D unit spheres ( S 2 ) J -1 ( cf. Appendix B). Crucially, we can further show that minimizing joint position error using a single-hypothesis model necessarily leads to predicted poses lying outside the true manifold:

Proposition 4.2 (Inconsistency of MSE minimizer) . With a rigid skeleton and mild assumptions on the training distribution, predicted 3D poses minimizing the traditional mean squared error (MSE) loss lie outside the pose manifold M .

Proof sketch. (See Appendix B). Consider a skeleton with J joints, with (x , p ), as pairs of corresponding 2D inputs and 3D poses. Let the function ℓ = ( ℓ j ) J -1 j =1 compute the lengths of the segments in a pose, which shall remain constant. On a dataset { (x i , p i ) } N i =1 drawn from the joint distribution of 2D and 3D poses, let the expected MSE of a traditional predictive model f be E x , p [ ∥ p -f (x) ∥ 2 2 ] . Let the ideal model f ∗ be the one minimizing that expected MSE, which is the conditional expectation f ∗ (x) = E [p | x] . Jensen inequality and the rigidity assumption imply that, for any joint j , ℓ 2 j ( f ∗ (x)) < s 2 j where s j is the true length of the segment associated with joint j . This shows that the poses predicted by f ⋆ violate the original segment length constraints, and thus, the original rigidity assumption. ■

Proposition 4.2 has the following implications:

Traditional unconstrained single-hypothesis approaches are bound to predict inconsistent movements, where bone lengths may vary.

With a single hypothesis, models constrained to the manifold will always lose to unconstrained models in terms of MPJPE performance (formalized in Corollary B.1).

The only way of reaching both optimal MPJPE and consistency is through multiple hypotheses (formalized in Corollary B.3).

Therefore, the MPJPE metric (and its traditional extensions) is insufficient to assess 3D-HPE, as it completely ignores pose consistency. Furthermore, we are able to prove in Appendix B.2 that multiple hypotheses (constrained or not) can always reach better joint position errors than single-hypothesis models.

4.2 Insights to the formal argument on a simplified setting

We illustrate the argument of Section 4.1 with a simplified 1D-to-2D setup. We further generalize this intuitive illustration to the 2D-to-3D setting in Appendix C of the supplementary.

As in human pose lifting, we take a root joint J 0 as reference, fixed at (0 , 0) . For a joint J 1 , the problem amounts to predicting the 2D

position ( x, y ) , given its 1D projection u = x , assuming a constant distance s = 1 between them. This simplification ignores the camera perspective and considers the joints to be connected by a rigid segment as in the case of human poses.

We train three different models with comparable architectures on two datasets { ( x i , ( x i , y i )) } N i =1 sampled from the angular distributions represented in blue on Fig. 4. The models correspond to:

A 2-layer MLP ( ) trained to minimize the mean squared error between true ( x, y ) and predicted joint positions (ˆ x, ˆ y ) ;

A constrained MLP of the same size ( ), predicting the angle ˆ θ instead of the joint position;

ManiPose: our constrained multi-hypothesis model capable of predicting K = 2 possible angles ( ˆ θ k ) K k =1 with their corresponding likelihoods.

Fig. 4 shows that the traditional unconstrained single-hypothesis approach ( ) leads to good results in an easy unimodal scenario (C), but fails when facing a more challenging bimodal distribution (D), leading to predictions outside the circle manifold, as depth ambiguity makes the lifting problem ill-posed. The single-hypothesis constrained model ( ) delivers predictions on the circle, at the cost of worse MPJPE performance than the unconstrained MLP. Such performance decrease is due to the Euclidean topology of the MPJPE metric having its minimum ( ) outside the manifold (Fig. 4-B).

Crucially, this implies that the unconstrained single-hypothesis models are bound to make inconsistent predictions, with varying 'bone lengths' (the circle radius). It also shows that models constrained to the manifold (circle) will always be outcompeted by unconstrained models on MPJPE performance.

Predicting multiple hypotheses constrained to the circle, with their respective likelihoods ( ⋆ in Fig. 4-B) allows escaping this dilemma, which is exactly what ManiPose does ( in Fig. 4-D). The

x

x

y

J

0

J

1

s

A

p

(

y

|

x

) = 0.67

p

(

y

|

x

) = 0.33

B

C

D

Inputs

Outputs

GT probability

Constr

.

MLP

Unconstr

.

MLP

MSE minimizer

Constr

.

MH min

.

ManiPose

-

hyps

.

ManiPose

-

scores

predicted hypotheses are all on the circle, contrary to the unconstrained MLP, and spread between the two distribution modes, unlike the constrained single-hypothesis method.

Moreover, the predicted scores (length of green lines) match the 2 3 and 1 3 ground-truth likelihoods of the two modes. Those advantages translate into perfect pose consistency and into comparable MPJPE performance with respect to the unconstrained MLP (Table 1).

5 Experiments

5.1 Experimental setup

Datasets. We evaluate our model on two 3D-HPE datasets. Human3.6M [15] contains 3.6 million images of 7 actors performing 15 different indoor actions. It is the most widely used dataset for 3D-HPE. Following previous works [52, 27, 53, 38], we train on subjects S1, S5, S6, S7, S8, and test on subjects S9 and S11, adopting a 17-joint skeleton ( cf. Fig. 5). We employ a pre-trained CPN [5] to compute the input 2D keypoints, as in [38, 52]. MPI-INF-3DHP [32] also adopts a 17-joint skeleton, but, with fewer samples and containing both indoor and outdoor scenes, it is more challenging than Human3.6M. We used ground-truth 2D keypoints for this dataset, as usually done [53, 4, 52].

Traditional evaluation metrics. The mean per-joint position error (MPJPE) is the usual performance metric for the datasets above, under different protocols, both reported in mm. In protocol #1, the root joint position is set as a reference, and the predicted root position is translated to 0. In protocol #2 (P-MPJPE), predictions are additionally Procrustes-corrected. For MPI-INF-3DHP, additional thresholded metrics derived from MPJPE are often reported, such as AUC (Area Under Curve) and PCK (Percentage of Correct Keypoints) with a threshold at 150 mm, as explained in [32].

Pose consistency metrics. MPJPE being insufficient to assess pose consistency (Section 4), we further assess to which extent predicted skeletons are rigid by measuring the average standard deviations of segment lengths across time in predicted action sequences:

with s t,j,i = ∥ ˆ p t,j -ˆ p t,i ∥ 2 and ¯ s j,i = 1 T ∑ T t =1 s t,j,i , where τ was defined in Section 3.1. We call this metric, reported in mm, the Mean Per Segment Consistency Error (MPSCE).

Following [12, 40], we also assess the bilateral symmetry of predicted skeletons through the Mean Per Segment Symmetry Error (MPSSE), in mm:

where J left denotes the set of indices of left-side joints and ζ maps left-side joint indices to their right-side counterparts.

Multi-hypothesis evaluation protocol. One must decide how to use multiple hypotheses to compute the metrics. The dominant approach [24, 25, 36, 44, 49, 12] is the oracle evaluation, i.e. , using the predicted hypothesis closer to the ground truth ( i.e. , Eq. (2) for MPJPE). That makes sense for multi-hypothesis methods, as the oracle metric measures the distance between the target and the discrete set of predicted hypotheses. It aligns with the idea of many possible outputs for a given input.

Hypotheses can also be aggregated into a final pose, e.g. , through unweighted or weighted averaging (using predicted scores). The latter has the disadvantage of falling back to a one-to-one mapping scheme, which is precisely what we want to avoid in a multi-hypothesis setting.

We report both oracle and aggregated metrics in our experiments, favoring oracle results.

Implementation details. ManiPose, as presented in Section 3, is compatible with any backbone. Here, we chose to build on the MixSTE [52] network for both the rotations and the segment modules (the latter in a reduced scale). Details about our architecture and training appear in Appendix D.

5.2 Comparison with the state of the art

Table 2: Pose consistency evaluation of state-of-the-art methods on Human3.6M. MPJPE performance and pose consistency are not correlated; only ManiPose excels in both. T : sequence length. K : number of hypotheses. Orac.: Metric computed using oracle hypothesis. Grey lines: Methods where the Oracle MPJPE is computed with non-comparable number of hypotheses with respect to the other baselines. Bold : best; Underlined: second best. *: Method with unavailable code ; MPSSE values reported in [12]. † : Results with comparable number of hypotheses. ‡ : Results computed with official checkpoint and code.

Human3.6M. Comparisons with state-of-the-art single- and multi-hypothesis methods are presented in Table 2 and illustrated in Fig. 1. ManiPose outperforms previous methods in terms of Oracle MPJPE in comparable scenarios, while reaching nearly perfect consistency. Moreover, note that MPJPE and consistency metrics are not positively correlated for single-hypothesis methods. As predicted in Section 4.1, our empirical results show that MPJPE improvements achieved by MixSTE come at the cost of poorer consistency compared to previous models. In contrast, the only singlehypothesis constrained model, Anatomy3D [4], achieves good consistency at the expense of inferior MPJPE. Those results empirically validate the theoretical predictions of Sections 4.1 and B, further

confirming what we have shown, intuitively, in the simplified 1D-to-2D setting (Section 4.2). Note that while ManiPose is deterministic, previous multi-hypothesis methods are generative, except for MHFormer. Table 2 shows that they require up to two orders of magnitude more hypotheses than ManiPose to reach competitive performance (see, e.g. , the performance of GFPose). This property is expected. Indeed, optimization based on Winner-Takes-All theoretically leads to an optimal coverage of the modes of the conditional distribution with a fixed number of samples [23], in contrast to generative-based approaches. This is reflected in the oracle metric, which approximates the so-called quantization (or Distortion) error, as defined in (27), when the number of data points is large. More detailed MPJPE results per action appear in Tables 8 and 9 in the supplemental. We also complement our analysis on the diversity of ManiPose in Fig. 11 of the appendix.

Fig. 6 showcases qualitative results, where multiple hypotheses help in depth-ambiguous situations.

MPI-INF-3DHP. Similar results were obtained for this dataset ( cf. Table 3). Not only does ManiPose reach consistency errors close to 0 , but also best PCK and AUC performance. As for MPJPE, only [42] achieves slightly better performance, at the cost of large pose consistency errors.

5.3 Ablation study

Impact of components. We evaluate the impact of removing each component of ManiPose on the Human3.6M performance (Table 4). The components tested are the multiple hypotheses (MH) and the manifold constraint (MC). We also compare MC to a more standard manifold regularization (MR), i.e. , adding Eq. (7) to the loss. Note that without all these components, we fall back to MixSTE [52], and that the performances reported in Table 4 also appear in Fig. 1.

We see that MR helps to improve pose consistency, but not as much as MC. However, without multiple hypotheses, MC consistency improvements come at the cost of degraded MPJPE performance, as

foreseen by our formal analysis (Section 4). Only the combination of both MC and MH allows us to optimize both consistency and MPJPE.

Fine error analysis. We can see in Fig. 5 that, compared to MixSTE, ManiPose reaches substantially superior MPSSE and MPSCE, consistency across all skeleton segments. Furthermore, note that larger MixSTE errors occur for segments KNEE-FOOT and ELBOW-WRIST, which are the most prone to depth ambiguity. That agrees with coordinate-wise errors depicted in Fig. 5, showing that ManiPose improvements mostly translate into a reduction of MixSTE depth errors, which are twice as large as for other coordinates. Further ablations, including the effect of the number of hypotheses K , the score loss weight β and the rotations representation choice appear in the supplemental.

Ground-truth

Hypothesis 1: p 1 (x)

Single hypothesis

Hypothesis 2: p 2 (x)

1

0

Score :

γ

k (x)

Hypothesis 3: p

3

(x)

6 Conclusion

We presented a new manifold-constrained multi-hypothesis human pose lifting method (ManiPose) and demonstrated its empirical superiority to the existing state-of-the-art on two challenging datasets. Further, we provided theoretical evidence supporting the tenets of our method, by showing the inherent limitation of unconstrained single-hypothesis approaches to 3D-HPE. We established that unconstrained single-hypothesis methods cannot deliver consistent poses and that constraining or regularizing single-hypothesis models leads to worse position errors. We also showed that traditional MPJPE-like metrics are insufficient to assess consistency.

Limitations. To guarantee its consistency, ManiPose relies on the forward kinematics algorithm, which is inherently sequential across joints. Removing that dependence is an interesting avenue for accelerating the method. On another note, while ManiPose ensures the rigidity of the predicted poses, imposing constraints within human body articulation limits presents another area for enhancement.

Acknowledgments and Disclosure of Funding

This work was granted access to the HPC resources of IDRIS under the allocation 2023-AD011014073 made by GENCI. It was also partly funded by the French Association for Technological Research (ANRT CIFRE contract 2022-1854). We are grateful to the reviewers for their insightful comments.

References

Bishop, C.M.: Mixture density networks. Working paper, Aston University (1994) 3

Cai, Y., Ge, L., Liu, J., Cai, J., Cham, T.J., Yuan, J., Thalmann, N.M.: Exploiting spatialtemporal relationships for 3d pose estimation via graph convolutional networks. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2272-2281 (2019) 2, 8, 23, 24

Chen, C.H., Ramanan, D.: 3d human pose estimation= 2d pose estimation+ matching. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7035-7043 (2017) 2

Chen, T., Fang, C., Shen, X., Zhu, Y., Chen, Z., Luo, J.: Anatomy-aware 3d human pose estimation with bone-based pose decomposition. IEEE Transactions on Circuits and Systems for Video Technology 32 (1), 198-209 (2021) 2, 7, 8, 23, 24

Chen, Y., Wang, Z., Peng, Y., Zhang, Z., Yu, G., Sun, J.: Cascaded pyramid network for multi-person pose estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7103-7112 (2018) 7

Ci, H., Wu, M., Zhu, W., Ma, X., Dong, H., Zhong, F., Wang, Y.: Gfpose: Learning 3d human pose prior with gradient fields. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4800-4810 (2023) 3, 8

Du, Q., Faber, V., Gunzburger, M.: Centroidal voronoi tessellations: Applications and algorithms. SIAM review 41 (4), 637-676 (1999) 19

Firman, M., Campbell, N.D., Agapito, L., Brostow, G.J.: Diversenet: When one right answer is not enough. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 5598-5607 (2018) 3

Goel, S., Pavlakos, G., Rajasegaran, J., Kanazawa, A., Malik, J.: Humans in 4d: Reconstructing and tracking humans with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14783-14794 (2023) 2

Gong, J., Foo, L.G., Fan, Z., Ke, Q., Rahmani, H., Liu, J.: Diffpose: Toward more reliable 3d pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13041-13051 (2023) 3, 8, 24, 26

Guzman-Rivera, A., Batra, D., Kohli, P.: Multiple choice learning: Learning to produce multiple structured outputs. Advances in neural information processing systems 25 (2012) 3

Holmquist, K., Wandt, B.: Diffpose: Multi-hypothesis human pose estimation using diffusion models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15977-15987 (2023) 1, 3, 8, 23

Hossain, M.R.I., Little, J.J.: Exploiting temporal information for 3d human pose estimation. In: Proceedings of the European conference on computer vision (ECCV). pp. 68-84 (2018) 2, 22

Hu, W., Zhang, C., Zhan, F., Zhang, L., Wong, T.T.: Conditional directed graph convolution for 3d human pose estimation. In: Proceedings of the 29th ACM International Conference on Multimedia. pp. 602-611 (2021) 2

Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. IEEE Transactions on Pattern Analysis and Machine Intelligence 36 (7), 1325-1339 (Jul 2014) 7, 15

Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7122-7131 (2018) 2

Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 20, 22

Kolotouros, N., Pavlakos, G., Jayaraman, D., Daniilidis, K.: Probabilistic modeling for human mesh recovery. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11605-11614 (2021) 3

Lee, K., Hwang, C., Park, K., Shin, J.: Confident multiple choice learning. In: International Conference on Machine Learning. pp. 2014-2023. PMLR (2017) 3, 5

Lee, S., Purushwalkam, S., Cogswell, M., Crandall, D., Batra, D.: Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314 (2015) 3

Lee, S., Purushwalkam Shiva Prakash, S., Cogswell, M., Ranjan, V., Crandall, D., Batra, D.: Stochastic multiple choice learning for training diverse deep ensembles. Advances in Neural Information Processing Systems 29 (2016) 3, 4, 5

Letzelter, V ., Fontaine, M., Chen, M., Pérez, P., Essid, S., Richard, G.: Resilient multiple choice learning: A learned scoring scheme with application to audio scene analysis. Advances in neural information processing systems 36 (2024) 3, 4, 5

Letzelter, V., Perera, D., Rommel, C., Fontaine, M., Essid, S., Richard, G., Pérez, P.: Winnertakes-all learners are geometry-aware conditional density estimators. In: Proceedings of the 41 st International Conference on Machine Learning (2024) 9, 19

Li, C., Lee, G.H.: Generating multiple hypotheses for 3d human pose estimation with mixture density network. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9887-9895 (2019) 8, 23

Li, C., Lee, G.H.: Weakly supervised generative network for multiple 3d human pose hypotheses. In: British Machine Vision Conference (BMVC) (2020) 3, 8, 23

Li, J., Xu, C., Chen, Z., Bian, S., Yang, L., Lu, C.: Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3383-3393 (2021) 22

Li, W., Liu, H., Tang, H., Wang, P., Van Gool, L.: Mhformer: Multi-hypothesis transformer for 3d human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13147-13156 (2022) 3, 7, 8, 23

Liu, R., Shen, J., Wang, H., Chen, C., Cheung, S.c., Asari, V.: Attention mechanism exploits temporal contexts: Real-time 3d human pose reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5064-5073 (2020) 23, 24

Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: A skinned multiperson linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34 (6), 248:1-248:16 (Oct 2015) 2

Makansi, O., Ilg, E., Cicek, O., Brox, T.: Overcoming limitations of mixture density networks: A sampling and fitting framework for multimodal future prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7144-7153 (2019) 3

Martinez, J., Hossain, R., Romero, J., Little, J.J.: A simple yet effective baseline for 3d human pose estimation. In: Proceedings of the IEEE international conference on computer vision. pp. 2640-2649 (2017) 2

Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., Theobalt, C.: Monocular 3d human pose estimation in the wild using improved cnn supervision. In: 2017 international conference on 3D vision (3DV). pp. 506-516. IEEE (2017) 7

Mun, J., Lee, K., Shin, J., Han, B.: Learning to specialize with knowledge distillation for visual question answering. Advances in neural information processing systems 31 (2018) 3

Murray, R.M., Li, Z., Sastry, S.S.: A mathematical introduction to robotic manipulation. CRC press (2017) 17, 22

Naeem, M.F., Oh, S.J., Uh, Y., Choi, Y., Yoo, J.: Reliable fidelity and diversity metrics for generative models. In: International Conference on Machine Learning. pp. 7176-7185. PMLR (2020) 24, 26

Oikarinen, T., Hannah, D., Kazerounian, S.: Graphmdn: Leveraging graph structure and deep learning to solve inverse problems. In: 2021 International Joint Conference on Neural Networks (IJCNN). pp. 1-9. IEEE (2021) 3, 8, 23

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019) 2

Pavllo, D., Feichtenhofer, C., Grangier, D., Auli, M.: 3d human pose estimation in video with temporal convolutions and semi-supervised training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7753-7762 (2019) 2, 7, 8, 9, 22, 23, 24

Rempe, D., Birdal, T., Hertzmann, A., Yang, J., Sridhar, S., Guibas, L.J.: Humor: 3d human motion model for robust pose estimation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11488-11499 (2021) 2

Rommel, C., Valle, E., Chen, M., Khalfaoui, S., Marlet, R., Cord, M., Pérez, P.: DiffHPE: Robust, Coherent 3D Human Pose Lifting with Diffusion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3220-3229 (2023) 1, 8

Rupprecht, C., Laina, I., DiPietro, R., Baust, M., Tombari, F., Navab, N., Hager, G.D.: Learning in an uncertain world: Representing ambiguity through multiple hypotheses. In: Proceedings of the IEEE international conference on computer vision. pp. 3591-3600 (2017) 3, 19

Shan, W., Liu, Z., Zhang, X., Wang, S., Ma, S., Gao, W.: P-STMO: Pre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation (Jul 2022) 1, 2, 9

Shan, W., Liu, Z., Zhang, X., Wang, Z., Han, K., Wang, S., Ma, S., Gao, W.: Diffusion-based 3d human pose estimation with multi-hypothesis aggregation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14761-14771 (2023) 8, 23

Sharma, S., Varigonda, P.T., Bindal, P., Sharma, A., Jain, A.: Monocular 3d human pose estimation by generation and ordinal ranking. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2325-2334 (2019) 3, 8, 23

Tian, K., Xu, Y., Zhou, S., Guan, J.: Versatile multiple choice learning and its application to vision computing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6349-6357 (2019) 3, 5

Tiwari, G., Anti´ c, D., Lenssen, J.E., Sarafianos, N., Tung, T., Pons-Moll, G.: Pose-ndf: Modeling human pose manifolds with neural distance fields. In: European Conference on Computer Vision. pp. 572-589. Springer (2022) 2

Wang, J., Yan, S., Xiong, Y., Lin, D.: Motion guided 3d pose estimation from videos. In: European Conference on Computer Vision. pp. 764-780. Springer (2020) 1, 23, 24

Waskom, M.L.: seaborn: statistical data visualization. Journal of Open Source Software 6 (60), 3021 (2021). https://doi.org/10.21105/joss.03021, https://doi.org/10.21105/ joss.03021 16

Wehrbein, T., Rudolph, M., Rosenhahn, B., Wandt, B.: Probabilistic monocular 3d human pose estimation with normalizing flows. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11199-11208 (2021) 1, 3, 8, 23

Xu, J., Yu, Z., Ni, B., Yang, J., Yang, X., Zhang, W.: Deep kinematics analysis for monocular 3d human pose estimation. In: Proceedings of the IEEE/CVF Conference on computer vision and Pattern recognition. pp. 899-908 (2020) 2

Xu, T., Takano, W.: Graph stacked hourglass networks for 3d human pose estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16105-16114 (2021) 2, 23

Zhang, J., Tu, Z., Yang, J., Chen, Y ., Yuan, J.: Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13232-13242 (2022) 1, 2, 7, 8, 9, 21, 22, 23, 24

Zheng, C., Zhu, S., Mendieta, M., Yang, T., Chen, C., Ding, Z.: 3D Human Pose Estimation with Spatial and Temporal Transformers. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 11636-11645. IEEE, Montreal, QC, Canada (Oct 2021). https://doi.org/10.1109/ICCV48922.2021.01145 1, 2, 7, 8, 9, 23, 24

Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation representations in neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5745-5753 (2019) 4, 22, 24

Zou, Z., Tang, W.: Modulated graph convolutional network for 3d human pose estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11477-11487 (2021) 2, 23, 24

Appendix / supplemental material

This supplemental material is organized as follows:

Appendix A contains empirical verification of our assumptions,

Appendix B presents the proofs of our theoretical results, together with a few corollaries,

Appendix C provides further implementation details concerning the 1D-to-2D experiment, as well as an extension to the 2D-to-3D setting,

Appendix D contains implementation and training details concerning ManiPose, as well as compared baselines,

Appendix E presents further results of the Human 3.6M experiment,

and finally, Appendix F explains the provided experiment code.

A Assumption verifications

Let us first define a few elements that we will need needed for our derivations.

Definition A.1 (Human skeleton) . We define a human skeleton as an undirected connected graph G = ( V, E ) with J = | V | nodes, called joints , associated with different human body articulation points. We assume a predefined order of joints and denote A = [ A ij ] 0 ≤ i,j<J ∈ { 0 , 1 } J × J the adjacency matrix of G , defining joints connections.

Definition A.2 (Human pose and movement) . Let G be a skeleton of J joints. We attach to each joint i a position p G i in R 3 and call the vector p G = [p G 0 , . . . , p G J -1 ] ∈ ( R 3 ) J a human pose . Furthermore, given a series of increasing time steps t 1 < t 2 < · · · < t T ∈ R , we define a human movement m as a sequence of poses of the same subject at those instants m = [p G t 1 , . . . , p G t T ] ∈ ( R 3 ) J × T .

We base the theoretical results of Section 4.1 on the following assumptions. The first states the reference frame traditionally used for assessing 3D-HPE models:

Assumption A.3 (Reference root joint) . For any skeleton G and movement m of length T , the joint of index 0 , called the root joint , is at the origin p G t, 0 = [0 , 0 , 0] at all times t 1 ≤ t ≤ t T . That is equivalent to measuring positions p G t in a reference frame attached to the root joint.

The second assumption concerns the rigidity of human body parts:

Assumption A.4 (Rigid segments) . We assume that the Euclidean distance between adjacent joints is constant within a movement m: for any pair of instants t and t ′ and for any joints i, j such that A ij = 1 , we assume that

Finally, we assume that the conditional distribution of poses does not collapse to a single point, i.e. , that we have a one-to-many problem:

Assumption A.5 (Non-degenerate conditional distribution) . Given a joint distribution P(x G , p G ) of 3D poses p G ∈ ( R 3 ) J and corresponding 2D inputs x G ∈ ( R 2 ) J , we assume that the conditional distribution P(p G | x G ) is non-degenerate, i.e. , it is not a single Dirac distribution.

Note that can be true even when P(x G , p G ) is unimodal ( e.g. , Fig. 4).

We verified on Human 3.6M [15] ground-truth data that assumptions A.4 and A.5 hold for actual poses in both training and test splits.

Segments rigidity. As shown on Figs. 5 and 9, ground-truth 3D poses have perfect MPSSE (8) and MPSCE (7) metrics, meaning that ground-truth skeletons are perfectly symmetric, with rigid segments. Assumption A.4 is thus verified in actual training and test data.

Non-degenerate distributions. As shown on Fig. 7, the conditional distribution of ground-truth 3D poses given 2D keypoints position is clearly multimodal, and, thus, non-degenerate (not reduced to a single Dirac distribution). That validates assumption A.5 and explains why multi-hypothesis techniques are necessary.

0.5

0.0

0.5

1.0

u

0.4

0.2

0.0

0.2

0.4

z

0.5

0.0

0.5

1.0

v

(a) S9, Walking

0.5

0.0

0.5

u

0.75

0.50

0.25

0.00

0.25

0.50

z

0.5

0.0

0.5

v

(b) S1, Greeting

0.5

0.0

0.5

u

0.75

0.50

0.25

0.00

0.25

0.50

z

0.5

0.0

0.5

v

(c) S11, Directions

0.0

0.5

u

0.5

0.0

0.5

z

0.0

0.5

v

(d) S1, SittingDown

B Proofs and additional corollaries

B.1 Properties of manifold constraint and multi-hypotheses models

This section contains the proofs of the theoretical results presented in Section 4.1, together with a few corollaries.

PROOF. [Proposition 4.1] Let i be a joint connected to the root p 0 ( i.e. , A i 0 = 1 ). From assumptions A.3 and A.4, we know that at any instant t , p G t,i lies on the sphere S 2 (0 , s i, 0 ) centered at 0 with radius s i, 0 independent of time. Therefore, its position can be fully parameterized in spherical coordinates by two angles ( θ t,i , ϕ t,i ) . Let j be a joint connected to i . Like before, assumption A.4 implies that at any instant t , p G t,j lies on the moving sphere S 2 ( p G t,i , s j,i ) centered at p G t,i with radius s j,i independent of time. Thus, we can fully describe p G t,j with the position of its center, p G t,i and the spherical coordinates ( θ t,j , ϕ t,j ) of joint j relative to the center of the sphere, i.e. , joint i . That means that there is a bijection between the possible positions attainable by p G t,j at any instant and the direct product of spheres S 2 (0 , s i, 0 ) ⊗ S 2 (0 , s j,i ) . 1 That bijection is an homeomorphism since it is a composition of homeomorphisms: we can compute p G t,j from ( θ t,i , ϕ t,i , θ t,j , ϕ t,j ) following the forward kinematics algorithm [34] ( cf. algo. 2), i.e. , using a composition of rotations and translations.

Now let us assume for some arbitrary joint k that p G t,k lies at all times on a space M 2 d homeomorphic to a product of spheres of dimension 2 d . That means that p G t,k can be fully parametrized using 2 d spherical angles ( θ 1 , ϕ 1 , . . . , θ d , ϕ d ) . Let l be a joint connected to k (typically one further step away from the root joint p 0 and not already represented in M 2 d ). As before, at any instant t , p G t,l needs to lie on the sphere centered on p G t,k of constant radius s k,l . Thus, we can fully describe p G t,l using the 2( d +1) -tuple of angles obtained by concatenating its spherical coordinates relative to joint k , together with the 2 d -tuple describing p G t,k , i.e. the center of the sphere. So p G t,l lies on a space M 2( d +1) homeomorphic to a product of spheres of dimension 2( d +1) .

We can conclude by induction that at any instant t , p t = [p G t, 1 , . . . , p G t,J ] lies on the same subspace of ( R 3 ) J , which is homeomorphic to a product of spheres centered at the origin:

Finally, the previous space is trivially homeomorphic to ( S 2 ) J -1 through the scaling (1 /s i,j ) i<j/A ij =1 . ( S 2 ) J -1 is a manifold of dimension 2( J -1) as the direct product of J -1 manifolds of dimension 2 . ■

PROOF. [Proposition 4.2] Let G be a skeleton with J joints, x ∈ ( R 2 ) J a 2D pose, p ∈ ( R 3 ) J its corresponding 3D pose, and P(x , p) a joint distribution of poses in 2D and 3D. We define ℓ = ( ℓ j ) J -1 j =1 as the function allowing us to compute the length of the segments of a pose p :

where τ : { 1 , . . . , J -1 } → { 0 , . . . , J -1 } maps joint indices to the index of their parent joint:

From assumption A.4, we know that for any pose p from the training distribution,

Given D = { (x i , p i ) } N i =1 ∼ P(x , p) , some i.i.d. evaluation data, the MSE of a model f is defined as:

and converges to

as the dataset size N goes to infinity. We then define the oracle MSE minimizer as

The quantity in (15) is known in statistics as the expected L 2 -risk and it is a well-known fact that its minimizer is the conditional expectation:

Thus, since ℓ 2 j are strictly convex and P(p | x) is non-degenerate according to assumption A.5, we can conclude from Jensen's strict inequality that for all j ,

where the last equality arises from the fact that ℓ 2 j (p) is not random according to (13). Thus, given that ℓ j > 0 and s j,τ ( j ) > 0 , we can say that ℓ j ( f ∗ (x)) < s j,τ ( j ) for all joints j . We conclude that the model f ∗ minimizing MSE ∗ predicts poses that violate assumption A.4 and are inconsistent. ■

As an immediate corollary of proposition 4.2, we may state the following result, which was empirically illustrated in many parts of our paper:

Corollary B.1. Given a fixed training distribution P(x , p) respecting assumptions A.3-A.5, for all 3D-HPE model f predicting consistent poses, i.e. , that respect assumption A.4, there is an inconsistent model f ′ with lower mean-squared error.

PROOF. Let f ′ ∈ arg min ˜ f MSE ∗ ( ˜ f ) . According to proposition 4.2, f ′ is inconsistent. Suppose that the consistent model f is such that

∗ ′ ∗ ∗ ′

Since MSE reaches its minimum at f , we have MSE ( f ) = MSE ( f ) . Thus, f ∈ arg min ˜ f MSE ∗ ( ˜ f ) , which means that f is also inconsistent according to proposition 4.2. That is impossible given that we assumed f to be consistent. We conclude that Eq. (19) is wrong and that

■

Note that propositions 4.2 and B.1 assume the use of the MSE loss, which is the most widely used loss in 3D-HPE. We can however extend them to the case where MPJPE serves as optimization criteria under an additional technical assumption:

Corollary B.2. The predicted poses minimizing the mean-per-joint-position-error loss are inconsistent if the training poses distribution P(x , p) verifies Asm. A.3-A.5 and if the joint-wise residuals' norm standard deviation is small compared to the joint-wise loss:

PROOF. From proposition 4.2 we know that the poses predicted by the minimizer f ∗ of

are inconsistent. Let f j be the component of f corresponding to the j th joint. We define the j th mean-per-joint-position-error component as:

Under the small variance assumption, we have:

so both criteria, MSE and MPJPE, are asymptotically equivalent and have the same minimizer f ∗ , which is inconsistent according to proposition 4.2. ■

Corollary B.3. Under Asm. A.4-A.5 and under (21) , the only way to get both optimal MPJPE and consistency is to use multiple hypotheses.

PROOF. Corollary B.1 and Proposition 4.2 imply that single-hypothesis models (constrained or not) deliver either suboptimal MPJPE or inconsistent pose predictions. Hence, by negation, we get our result.

In the next section, we further show that multi-hypotheses models, constrained or not, can theoretically show a better L 2 -risk (or quantization ) performance compared with single-hypotheses models.

B.2 Multiple hypotheses (constrained or not) can improve L2-risk over single-hypothesis models

Let X = R 2 × J denote the space of input 2D poses and P = R 3 × J the space of 3D poses. Also, let R ( f ) = E x , p [ ∥ p -f (x) ∥ 2 2 ] be the L 2 -risk of some pose estimator f under some underlying continuous joint distribution of 2D-3D pose pairs P(x , p) , with density ρ (when it exists).

Before stating the proposition, we need to define an adapted notion of risk for multi-hypothesis models under the oracle aggregation scheme:

Definition B.4 (Winner-takes-all risk, [41]) . As in [41] (section 3.2) and in [23] (section 2.2), we define the L 2 -risk for K -head models f WTA = ( f 1 WTA , . . . , f K WTA ) as:

where V k ( g ) 's denotes the k th cell of the Voronoi tesselation of the output space P defined by generators g = ( g 1 , . . . , g K ) ∈ P K :

̸

The risk above translates the notion of oracle pose, since it partitions the space of ground-truth poses P into regions where some hypothesis is the closest, and uses only that hypothesis to compute the risk in that region. Note that R 1 WTA ( f ) = R ( f ) for any function f , since a single-cell tessellation of P is P itself.

In the following, we assume that f is expressive enough, so that, minimizing the risk (27) comes down to minimizing

for each x ∈ X .

Proposition B.5 (Optimality of manifold constrained multi-hypothesis models) . A K -hypotheses model f ∗ WTA = ( f 1 , ∗ WTA , . . . , f K, ∗ WTA ) minimizing (27) has always a risk lower or equal to a singlehypothesis model f ∗ MSE minimizing R :

PROOF. Following [23] (Section 2.2), we decouple the cell generators from the risk arguments in (27):

for any generators g = ( g 1 , . . . , g K ) ∈ P K and arguments z = ( z 1 , . . . , z K ) ∈ P K . Note that R K WTA ( f ) = ∫ X K ( f (x) , f (x)) ρ (x) dx .

According to Proposition 3.1 of [7] (or Proposition 2.1 in [23]), if f ∗ WTA minimizes R K WTA , then ( f ∗ WTA (x) , f ∗ WTA (x)) has to minimize K for all x ∈ X :

Let's choose g such that g k = f k, ∗ WTA ( x ) and z such that z k = f ∗ MSE ( x ) for all 1 ≤ k ≤ K . Then

where the last equality comes from the fact that V k ( f ∗ WTA (x)) defines a partition of P .

C Further details of 1D-to-2D case study

C.1 Implementation details

Datasets. We created a dataset of input-output pairs { ( x i , ( x i , y i )) } N i =1 , divided into 1 000 training examples, 1 000 validation examples and 1 000 test examples. Since the 2D position of J 1 is fully determined by the angle θ between the segment ( J 0 , J 1 ) and the x -axis, the dataset is generated by first sampling θ from a von Mises mixture distribution, then converting it into Cartesian coordinates ( x i , y i ) to form the outputs, and finally projecting them into the x -axis to obtain the inputs.

Distribution scenarios. We considered three different distribution scenarios with different levels of difficulty:

Easy scenario : a unimodal distribution centered at θ = 2 π 5 , where the axis of maximum 2D variance is approximately parallel to the x -axis (Fig. 4-A).

Difficult unimodal scenario : a unimodal distribution centered at θ = 0 , where the axis of maximum 2D variance is perpendicular to the x -axis (Fig. 4-B).

Difficult multimodal scenario : a bimodal distribution, with modes at θ 1 = π 3 and θ 2 = -π 3 and mixture weights w 1 = 2 3 and w 2 = 1 3 , i.e. , where the projection of modes onto the x -axis are close to each other (Fig. 4-C).

All von Mises components in all scenarios had concentrations equal to 20 .

Architectures and training. All three models were based on a multi-layer perceptron (MLP) with 2 hidden layers of 32 neurons each, using tanh activation.

The constrained and unconstrained MLPs were trained using the mean-squared loss 1 N ∑ N i =1 ((ˆ x i -x i ) 2 + (ˆ y i -y i ) 2 ) . ManiPose was trained with the loss in Eq. (1), and had K = 2 heads. We trained all models with batches of 100 examples for a maximum of 50 epochs. We used the Adam optimizer [17], with default hyperparameters and no weight decay. Learning rates were searched for each model and distribution independently over a small grid: [10 -5 , 10 -4 , 10 -3 , 10 -2 ] ( cf. selected values in Table 5). They were scheduled during training using a plateau strategy of factor 0 . 5 , patience of 10 epochs and threshold of 10 -4 .

C.2 Extension to 2D-to-3D setup with more joints

We further extend the two-joint 1D-to-2D lifting experiment of Section 4.2 to 2D-to-3D with three joints, aiming at providing a scenario that is closer to real-world 3D-HPE, but that can still be fully dissected and visualized.

As in Section 4.2, we suppose that joint J 0 is at the origin at all times, that J 1 is connected to J 0 through a rigid segment of length s 1 and that J 2 is connected to J 1 through a second rigid segment of length s 1 < s 0 . We further assume that both J 1 and J 2 are allowed to rotate around two axes

orthogonal to each other. Thus, J 1 is constrained to lie on a circle S 1 (0 , s 0 ) , while J 2 lies on a torus T homeomorphic to S 1 (0 , s 0 ) ⊗ S 1 (0 , s 1 ) . Without loss of generality, we set the radii s 0 = 2 and s 1 = 1 and assume them to be known.

Given that setup, we are interested in learning to predict the 3D pose ( J 1 , J 2 ) = ( x 1 , y 1 , z 1 , x 2 , y 2 , z 2 ) ∈ R 6 , given its 2D projection ( K 1 , K 2 ) = ( x 1 , z 1 , x 2 , z 2 ) ∈ R 4 . We create a dataset comprising 20000 training, 2000 validation, and 2000 test examples, sampled using an arbitrary von Mises mixture of poloidal and toroidal angles ( θ, ϕ ) in T . We set the modes of such a mixture at [( -π, 0) , (0 , π/ 4) , ( 1 2 , -π/ 4) , (2 π/ 3 , π/ 2)] , with concentrations of [2 , 4 , 3 , 10] and weights [0 . 3 , 0 . 4 , 0 . 2 , 0 . 1] . Similarly to Fig. 4-C, that creates a difficult multimodal distribution, depicted in Fig. 8.

Distribution of Points on the T orus

X axis

3

2

1

0

1

2

3

Y axis

3

2

1

0

1

2

3

Z axis

1.0

0.5

0.0

0.5

1.0

(a)

X axis

3

2

1

0

1

2

3

Y axis

3

2

1

0

1

2

3

Z axis

1.0

0.5

0.0

0.5

1.0

(b)

We train and evaluate the same baselines as in Section 4.2 in that new scenario, using a similar setup ( cf. Appendix C.1, Architectures and training). Note that for those experiments, we used an initial learning rate of 10 -3 for each baseline, and a batch size of 1000 examples. The corresponding Mean Per Segment Consistency Error (MPSCE) and Mean Per Joint Position Error (MPJPE) results are reported in Table 6.

We see that the same observations as in Section 4.2 also apply here: although the unconstrained MLP yields competitive MPJPE results, its predictions are not consistently aligned with the manifold, as indicated by its poor MPSCE performance. Again, we show here that ManiPose offers an effective balance between maintaining manifold consistency and achieving high joint-position-error performance.

D Further ManiPose implementation details

D.1 Architectural details

Our architecture is backbone-agnostic, as shown on Fig. 2. Thus, in order to have a fair comparison, we decided to implement it using the most powerful architecture available, i.e. , MixSTE [52].

In practice, the rotations module follows the MixSTE architecture with d l = 8 spatio-temporal transformer blocks of dimension d m = 512 and time receptive field of T = 243 frames for Human 3.6M experiments and T = 43 frames for MPI-INF-3DHP experiments. Contrary to MixSTE, that network

outputs rotation embeddings of dimension 6 for each joint and frame, instead of Cartesian coordinates of dimension 3 .

Concerning the segment module, it was also implemented with a smaller MixSTE backbone of depth d l = 2 and dimension d m = 128 .

The ablation study presented in Table 4 shows that the increase in the number of parameters between MixSTE and ManiPose is negligible.

D.2 Pose decoding details

The pose decoding block from Fig. 2 is described in Section 3.1 and is based on Algorithms 1 and 2. The whole procedure is illustrated on Fig. 3.

Algorithm 1 6D rotation representation conversion [54]

Require: Predicted 6D rotation representation r ∈ R 6 . 1: x ′ ← [ r 0 , r 1 , r 2 ] , 2: y ′ ← [ r 3 , r 4 , r 5 ] , 3: x ← x ′ / ∥ x ′ ∥ 2 , 4: z ′ ← x ∧ y ′ , 5: z ← z ′ / ∥ z ′ ∥ 2 , 6: y ← z ∧ x, 7: return R = [ x | y | z ] ∈ R 3 × 3 .

Algorithm 2 Forward Kinematics [34, 26]

Require: Scaled reference pose u ′ ∈ ( R 3 ) J , predicted rotation matrices R t,j , 0 ≤ j < J .

▷ Compose relative rotations

1: R ′ t, 0 ← R t, 0 , 2: p t, 0 ← u ′ 0 , 3: for j = 1 , . . . , J -1 do 4: R ′ t,j ← R t,j R ′ t,τ ( j ) , 5: p t,j ← R ′ t,j (u ′ j -u ′ τ ( j ) ) + p t,τ ( j ) , 6: end for

7: return p t = [p t,j ] 0 ≤ j<J

D.3 Training details

Training tactics. In order to have a fair comparison with MixSTE [52], we trained ManiPose using the same training tactics, such as pose flip augmentation both at training and test time. Moreover, the training loss (1) was complemented with two additional terms described in [52]:

a TCloss term, initially introduced in [13]; 2. a velocity loss term, introduced in [38].

We also weighted the Winner-takes-all MPJPE loss (2) as in [52] ( cf. weights in Table 7). The score loss weight, β , was set to 0 . 1 according to our hyperparameter study (Appendix E), while TCloss and velocity loss terms had respective weights of 0 . 5 and 2 (values from [52]).

Training settings. We trained our model for a maximum of 200 epochs with the Adam optimizer [17], using default hyperparameters, a weight decay of 10 -6 and an initial learning rate of 4 × 10 -5 . The latter was reduced with a plateau scheduler of factor 0 . 5 , patience of 11 epochs and threshold of 0 . 1 mm. Batches contained 3 sequences of T = 243 frames each for the Human 3.6M training, and 30 sequences of T = 43 frames for MPI-INF-3DHP.

Compute resources. Trainings were carried out on a single NVIDIA RTX 2000 GPU with around 11GB of memory. The training of the large model with 243 frames on Human 3.6M dataset took around 26 hours.

Dataset licences. Human3.6M is a dataset released under a research-only custom license, and is available upon request at this URL: http://vision.imar.ro/human3.6m/description.php . MPI-INF-3DHP is released under non-commercial custom license and can be found at: https: //vcai.mpi-inf.mpg.de/3dhp-dataset/ .

D.4 Baselines evaluation.

All Human 3.6M evaluations of MPSSE and MPSCE listed in Tables 2 and 4 were performed using the official checkpoints of these methods and their corresponding official evaluation scripts. Concerning MPI-INF-3DHP evaluations from Table 3, checkpoints were not available (except for P-STMO). Thus, baseline models were retrained from scratch using the official MPI-INF-3DHP training scripts provided by the authors of each work, using hyperparameters reported in their corresponding papers. We checked that we were able to reproduce the reported MPJPE results.

E Further results on the Human 3.6M dataset

Protocol #1 and #2 detailed results. Adetailed quantitative comparison in terms of MPJPE per action on Human3.6M dataset between ManiPose and state-of-the-art methods is shown in Table 8. We see that ManiPose reaches the best MPJPE performance on average and on most actions. Table 9 contains a similar analysis in terms of P-MPJPE ( i.e. , MPJPE with procrust-aligned poses). We observe the same patterns as in Table 8, namely that ManiPose reaches the second-best P-MPJPE performance on average and for most actions. We confirm here again that the substantial improvements in pose consistency brought by ManiPose are not obtained at the expense of traditional metrics derived from MPJPE.

Errors per joint. On the top of Fig. 9 we see that most of MixSTE errors come from feet, elbows and wrist joints, which are most prone to depth ambiguity. ManiPose helps to reduce the position errors for most of those ambiguous joints, probably as a byproduct of its major consistency improvements shown in Fig. 5.

Impact of hyperparameters. ManiPose introduces two additional hyperparameters when compared to MixSTE: the number K of hypotheses and the score loss weight β ( cf. Eq. (1)). We further assess the impact of their respective values on MPJPE. For computational cost reasons, we used a smaller version of our model for this study, with transformer blocks of dimension d m = 64 and time receptive field of T = 27 frames. Fig. 10 (left) shows that more hypotheses help, but that the performance

improvements saturate around 5 hypotheses. Concerning β , Fig. 10 (right) shows that lower values help to improve the MPJPE performance.

Impact of the rotations representations used.

The disentanglement between segments' length and orientation is not novel, and was proposed in previous works restricted to the single-hypothesis case, such as Anatomy3D [4]. While ManiPose represents segments' orientations as full 3D rotations relatively to parent segments in the kinematics tree, Anatomy3D simply predicts segments' absolute directions, i.e. , normalized vectors in the 3D space. This solution has the advantage of not over-parametrizing the segments orientations (which are invariant to rotations around the segment axis) and being lower dimensional (3 vs 6). One might hence wonder whether Anatomy3D's parametrization is not preferable. As shown in Table 10, Anatomy3D's implementation led to poorer results when compared to our rotations parametrization in a multi-hypothesis setting. This motivated us to use full 3D rotations' representations proposed in [54] in our experiments, despite their caveats. Note that [54] also shows good empirical results in the related problem of inverse kinematics of human 3D poses.

Diversity of predicted poses. As explained in Section 5.2, ManiPose's state-of-the-art oracle MPJPE results show that it excels in terms of diversity when the latter is assessed using the quantization error. There are many other ways of assessing distribution diversity. In an attempt to quantify the diversity of pose distributions learned by ManiPose by other means, we have computed the coverage (as defined in [35]) of generated poses relatively to the ground-truth test set of Human 3.6M. For computation cost reasons (it grows quadratically with sample size), we limited our analysis to 5 actions from subject S11. We compare ManiPose to DiffPose [10], using 5 hypotheses for both, and observe similar diversity on average ( cf. Fig. 11).

F Code

We provide the code to reproduce all our experiments under https://github.com/ cedricrommel/manipose .

2

3

4

5

K

55

60

[mm]

0.1

0.5

1.0

Score loss weight

60

80

Aggregated MPJPE

Oracle MPJPE

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

Coverage

Discussion

Purchases

Waiting

Phoning

SittingDown

Average

Action

Model

DiffPose

ManiPose

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our claims are enumerated at the bottom of the introduction and backed by our experiments from Section 5 and our theoretical results from Section 4.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in a separate section in the last page of the manuscript.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All propositions start by stating their assumptions (some might be detailed in the appendix, such as for Proposition 4.2). All results are proved formally in the supplementary material, with a proof sketch provided within the main article.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental setting of all experiment is described in details, either in the main article (Sections 4.2 and 5) or in the supplementary material (Appendices C.1 and D).

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code for the toy experiments was made completely available to reviewers in the supplementary material, together with instructions to reproduce the paper figures and tables, as well as environment configurations. Concerning Human3.6M and MPI-INF-3DHP experiments, their code has been open-sourced to the community under publication ( cf. Appendix F).

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details ( e.g. , data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setting of all experiment is described in details, either in the main article (Sections 4.2 and 5) or in the supplementary material (Appendices C.1 and D).

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In toy experiments, we do report error bars corresponding to the standarddeviation across several 5 runs ( cf. Tables 1 and 6). For real-world datasets, however, error bars are not reported because it would be too computationally expensive. This is customary in computer vision and practiced by all competing baselines.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute resources are described in Appendix D.3.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our experiments comply with NeurIPS Code of Ethics. For instance, we only work with well-known publicly available datasets.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a foundational work on 3D human pose lifting, and is hence not tied to any particular application. We do not believe that the enhancements it proposes could be used to make existing 3D human pose technology more dangerous in any way.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse ( e.g. , pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets ( e.g. , code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The main assets used in this work are the human pose datasets Human 3.6M and MPI-INF-3DHP, whose papers are duely cited. We also provide their licenses, availability conditions and download URL in the supplementary material (Appendix D.3).

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The only asset introduced in this paper is the experimental code, which is well documented in the its own README.md file a provided alongside its (open) license.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method to estimate 3D human keypoints from a sequence of monocular 2D keypoints observations. It builds upon an existing sequence-to-sequence architecture (MixSTE), with a different output parameterization exploiting a kinematic skeletton prior, and different training losses. Lengths of the skeletton bones are predicted for the whole sequence to ensure consistency across frames (and maybe also left/right symmetry of the skeletton), and five 3D pose hypotheses with associated scores are predicted for each frame, parameterized as a list of 3D relative orientation for each bone with respect to its parent in the kinematic tree.\n\nThe authors develop theoretical arguments regarding the benefits of enforcing such structural priors in the predictions, and illustrate with a toy example the interest of having multiple predictions in case of ambiguous multimodal output. They validate their approach on Human3.6M and MPI-INF-3DHP datasets.\n\nSTRENGTHS:\nThe motivation for exploiting bone lengths constraints is well expressed, with a clear and detailed discussion provided in Section 4. The discussion of experimental and ablation results is insightful and shows – in a setting dependent on an oracle – benefits of the proposed approach.\n\nWEAKNESSES:\nThe idea of enforcing body priors (constant bone length here) is not novel and has actually been heavily exploited in a whole line of work relying on more advanced parametric models such as SMPL [100]. This line of work would deserve being considered in the paper, as it encompass approaches suitable for 2D-to-3D sequence lifting such as e.g. [101].\n\nThe authors present a pose space consisting in 3D coordinates of joints linked by some rigid segments. Based on this definition, a natural pose parameterization would consist in the 3D direction of each segment, yet the authors chose to overparameterize poses by using relative 3D bone orientation instead. I understand that such choice can have practical benefits in term of biomechanical constraints and additional supervision signal when ground truth data is available, but such choice should be properly motivated, discussed and ablated in the paper.\n\n\nThe authors describe two ways of aggregating results L247 but do not state which one they use for MPI-INF-3DHP, and they only report oracle results on Human3.6M and for the ablations.\n\nIn my understanding, pose hypotheses are selected independently for each frame and there are no temporal terms in the training objectives or aggregation method. Since the proposed approach deals with temporal sequences, it would be worth evaluating the temporal consistency of the predictions, through qualitative video examples and quantitatively e.g. using joint acceleration metrics. Having multiple hypotheses for each frame brings combinatorial questions worth discussing in my opinion.\n\nReferences:\n- [100] Loper at al., “SMPL: A Skinned Multi-Person Linear Model”, at SIGGRAPH Asia 2015.\n- [101] Baradel et al., “PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling”, in TPAMI 2022.\n\nQUESTIONS:\nSee the weaknesses section for a list of suggestions.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a MCL-based framework for multi-hypothesis 3D human pose estimation. This framework predicts skeletal parameters so that the predicted 3D poses in a sequence are constrained to one smooth manifold. To prove the superiority of such a framework, the paper presents detailed theoretical analysis on the drawback of unconstrained single-hypothesis HPE and why MPJPE alone is not enough for pose evaluation. The experiments show the proposed framework is capable of keeping the consistency of predicted poses and achieving state-of-the-art MPJPE in the meantime.\n\nSTRENGTHS:\n* Simple and reasonable manifold representation. The proposed framework keeps the predicted human pose on the target manifold by representing the human pose with bone lengths and orientations, and the 3D pose is a direct inference from forward kinematics. The manifold is represented by the kinematics itself.\n  \n* Inspiring theoretical analysis on basic problems in 3D HPE. The paper arrives at some theoretical conclusions (line178-183), along with detailed proofs. They can provide some refreshing ideas on the innate drawbacks of traditional loss functions and MPJPE metrics.\n  \n* Good performance under both MPJPE and consistency measures, as validated in Table 2 and 3.\n\nWEAKNESSES:\n* Theoretical analysis on the advantage of multi-hypothesis methods over single-hypothesis ones could be added. Specifically, why a **constrained multi-hypothesis** method performs better than an **unconstrained single-hypothesis** method in MPJPE? Though this is already validated by the experiments, I personally believe it would make the paper more solid if the authors could make this analysis.\n\nMinor problem:\n* In Fig.4 (C) and (D), it is not quite clear how the estimations (crosses and triangles) correspond with the inputs (black dots). There might be some unexpected shifts, as the projections of the predicitons do not strictly align with the inputs (like in B).\n\nQUESTIONS:\nWhat is the quality of the score for each hypothesis? If the multiple hypotheses are fused to one (e.g. by taking the one with the largest confidence or taking the weighted average), then how will the MPJPE, MPSCE, and MPSSE change?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new method to estimate 3D human pose from 2D observations (lifting). To ensure the body symmetry and temporal consistency, the authors disentangle human skeleton to two parts: temporally consistency bone scales and temporally variable bone rotations. The authors use fancy formulas to prove that, minimizing MSE loss could not gurantee manifold consistency. The quantitative and qualitative results on Human3.6m and MPI-INF-3DHP datasets show the superiority of the proposed method.\n\nSTRENGTHS:\n1. The evalution results in this paper is quite impressive, especially the newly proposed consistency metric. Figure 1 clearly shows the superiority of the proposed method. \n\n2. The authors try to prove the theoretical optimal of the proposed method, which is worth encouraging.\n\nWEAKNESSES:\nI am not an expert in manifold theory, therefore my questions only relate to human pose estimation. \n\n1. How to constrain the rotation space during training? \n\n2. The pose lifting method is quite similar to Anatomy3D (bone length + rotations). Can I view this paper an multi-hypothesis extension of Anatomy3D? Why? \n\n3. Previous paper \"POSE-NDF: MODELING HUMAN POSE MANIFOLDS WITH NEURAL DISTANCE FIELDS\" is similar to this paper in concepts. SMPL naturally guarantees bone length symmetry, and the learnable parameters (rotations and shape parameters) are similar to this paper in its functionality. It would be better to cite it. \n\n4. Suppose that, there is a virtual dataset, all 2D human joints are rendered (projected) from strictly symmetric 3D joints, then, could learning the lifting function on this virtual dataset using MSE loss guarantee the results all lie on manifold? \n\n5. (An optional question) The ground truth 3D joints of Human3.6M datasets come from the marker tracking on body surface, which naturally could not guarantee skeleton length consistency. Why learning symmetric bones yields better results (both Anatomy3D and the proposed methods)?\n\nQUESTIONS:\n1. The citation style is weird. They are not NeurIPS style, please correct them.\n\nLIMITATIONS:\nThe authors addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose ManiPose, a manifold-constrained multi-hypothesis model for 3D human pose lifting. The authors provide empirical and experimental evidence to show that joint position regression leads to inconsistent skeleton lengths. And they propose to predict globally consistent pose scale and individual joint rotations per frame (rather than joint positions) to constrain the predictions to the pose manifold. Empirical results demonstrates that the proposed ManiPose framework improves the pose consistency.\n\nSTRENGTHS:\n* The paper provides valuable theoretical analysis to support their arguments and provides intuitive toy examples to illustrate the ambiguity in pose lifting.\n* The paper conducts extensive experiments on H36M and MPI-INF-3DHP datasets.\n\nWEAKNESSES:\n* The paper uses a multi-head design to predict multiple hypotheses. This design loses the flexibility of sampling different numbers of hypotheses and limits the maximum number of hypotheses to a small number. This often results in limited hypothesis diversity. In the experimental section, the authors do not provide numerical of visual measurements of hypothesis diversity.\n* According to the comparison in Table 4, the manifold constraint proposed in this paper sacrifices MPJPE to improve pose consistency, serving as a trade-off approach between accuracy and consistency. Although the consistency is improved, it lags behind the traditional position regression or manifold regularization in accuracy, and does not bring essential improvement (improve both in accuracy and consistency) compared with these two methods.\n* Missing comparison with two recent multi-hypothesis methods. [1] GFPose: Learning 3D Human Pose Prior with Gradient Fields. [2] DiffPose: Toward More Reliable 3D Pose Estimation.\n\nQUESTIONS:\nPlease review the Weaknesses Section. If the author can address or respond to the above issues well in the rebuttal stage, I will consider increasing my score.\n\nLIMITATIONS:\nAs the authors discussed in the Limitations Section, they used forward kinematics to obtain joint positions, which can lead to error accumulation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}