ManiPose: Manifold-Constrained Multi-Hypothesis 3D Human Pose Estimation

Authors: cédric  rommel
victor  letzelter
nermin  samet
renaud  marlet
matthieu  cord
patrick  pérez
eduardo  valle
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:52:01

Abstract: We propose ManiPose, a manifold-constrained multi-hypothesis model for human-pose 2D-to-3D lifting. We provide theoretical and empirical evidence that, due to the depth ambiguity inherent to monocular 3D human pose estimation, traditional regression models suffer from pose-topology consistency issues, which standard evaluation metrics (MPJPE, P-MPJPE and PCK) fail to assess.  ManiPose addresses depth ambiguity by proposing multiple candidate 3D poses for each 2D input, each with its estimated plausibility. Unlike previous multi-hypothesis approaches, ManiPose forgoes generative models, greatly facilitating its training and usage. By constraining the outputs to lie on the human pose manifold, ManiPose guarantees the consistency of all hypothetical poses, in contrast to previous works. We showcase the performance of ManiPose on real-world datasets, where it outperforms state-of-the-art models in pose consistency by a large margin while being very competitive on the MPJPE metric.

Main paper: 
 introduction: 
 We propose ManiPose, a novel approach for human-pose 2D-to-3D lifting. ManiPose directly addresses the depth ambiguity inherent to monocular 3D human pose estimation by being both multihypothesis and manifold-constrained, thus avoiding pose consistency issues, which plague traditional regression-based methods. Unlike previous multi-hypothesis approaches, ManiPose forgoes the use of costly generative models, while still estimating the plausibility of each hypothesis. Monocular 3D human pose estimation (HPE) is a challenging learning problem that aims to predict 3D human poses given an image or a video from a single camera. Often, the problem is split into two successive steps: first 2D human pose estimation, then 2D-to-3D lifting. Such separation is favorable because 2D-HPE is much more mature, leading to better overall results. Due to depth ambiguity and occlusions, 2D-to-3D lifting is intrinsically ill-posed: multiple 3D poses correspond to the same projection observed in 2D. Despite that, the field has experienced fast developments, with substantial improvements in terms of mean-per-joint-prediction error (MPJPE) and derived metrics (e.g., P-MPJPE, PCK) [52, 53, 42, 47] .However, recent studies [49, 12, 40] noted that poses predicted by state-of-the-art models fail to respect basic invariances of human morphology, such as bilateral sagittal symmetry, or constant length across time of rigid body segments connecting the joints. Not only do we address those concerns with ManiPose (see Fig. 1 ), but we also provide theoretical elements clarifying the cause of those issues. We show in particular that pose consistency and traditional performance metrics (such as MPJPE) Figure 1 : Optimizing both 3D position and pose consistency requires combining constraints and multiple hypotheses. Results from Tables 2 and 4 . Previous unconstrained methods provide inconsistent poses (top) . Regularization (MR) and disentanglement constraints improve consistency, but degrade joint position error (bottom-right). Ours is the only method that achieves both good joint error and consistency, thanks to a combination of disentanglement and a few hypotheses (see circles sizes). cannot be optimized simultaneously by a standard regression model, because MPJPE ignores the topology of the space of human poses, and traditional regression models imply unimodality, thus overlooking the inherently ambiguous nature of 3D-HPE.Our contributions include:• ManiPose, a novel, multi-hypothesis, manifold-constrained model for human-pose 2D-to-3D lifting, which is able to estimate the plausibility of each hypothesis without resorting to costly generative models.• Theoretical insights that elucidate why traditional regression models associated with standard metrics such as MPJPE fail to enforce pose consistency.• Extensive empirical results, including comparison to strong baselines, evaluation on two challenging datasets (Human 3.6M and MPI-INF-3DHP), and ablations. ManiPose outperforms state-of-the-art methods by a substantial margin in terms of pose consistency, while still beating them in the MPJPE metric. The ablations confirm the importance of both multiple hypotheses and of constraining the poses to their manifold.The PyTorch [37] implementation of ManiPose and code used for all our experiments can be found at https://github.com/cedricrommel/manipose. 
related work: 
 Regression-based 2D-to-3D pose lifting. While 2D-to-3D human pose lifting was initially restricted to static frames [31, 3] , the field embraced recurrent [13] , convolutional [38] and graph neural networks [2, 55, 14, 51] to handle motion. Spatial-temporal transformers appear more recently [42, 53] , including MixSTE [52] , arguably becoming the state of the art. We adopt them in our work.A few previous works constrain predicted poses to respect human symmetries [50, 4] , an idea we advance with a novel constraint implementation, in a multi-hypothesis setting.SMPL-based methods. While 3D human pose lifting's objective is to predict 3D joint positions based on 2D keypoints, the neighboring field of human pose and shape reconstruction (HPSR) aims at estimating whole 3D body meshes from images. HPSR is hence more challenging than 3D-HPE, which explains why models are often larger, frame-based and more reliant on optimization-based post-processing [16, 39, 46, 9] . Nonetheless, our work shares some ideas from this field. Indeed, modern HPSR methods often predict joint angles (and body shape parameters), which are fed to the pre-trained parametric model SMPL [29] to produce human body meshes, thus ensuring that limbs' sizes remain constant along a movement. Note, however, that these are also single-hypothesis regression methods and hence share the same caveats as most 3D-HPE approaches.Multi-hypothesis 3D-HPE. The intrinsic depth-ambiguity of 3D-HPE led the community to investigate multi-hypothesis approaches, including Mixture Density Networks [25, 36, 1] , variational autoencoders [44] , normalizing flows [18, 49] and diffusion models [12, 6, 10] . Contrary to ours, those methods rely on a generative model to sample 3D pose hypotheses conditioned on the 2D input. A notable exception is MHFormer [27] , which, like ManiPose, is deterministic, but treats the hypotheses as intermediate representations to be aggregated at the final network layers, thus concluding with a one-to-one 2D-to-3D mapping. We strive to avoid such injectivity and to preserve the multiple hypotheses, for reasons we will justify both empirically and theoretically in the next sessions. Moreover, none of the previous multi-hypothesis approaches constrain hypotheses to lie on the human pose manifold, thus failing to guarantee good pose consistency.Multiple choice learning (MCL) [11] is a simple approach for estimating multimodal distributions, suited for ambiguous tasks, using the winner-takes-all loss. Adapted for deep learning by Lee et al. [20, 21] , it produces diverse predictors, each specialized in a particular subset of the data distribution. MCL has proved its effectiveness in several computer vision tasks [41, 19, 33, 8, 30, 45] , and was first applied to 2D-HPE in [41] . Our work is the first to employ MCL for the 3D-HPE task, by leveraging recent innovations of Letzelter et al. [22] . Legend:Figure 2 : Overview of ManiPose. The rotations module predicts K possible sequences of segment rotations with their corresponding likelihoods (scores), while the segments module estimates the shared segment lengths. Hence, predicted poses are constrained to a manifold defined by the estimated lengths, guaranteeing their consistency.Following the previous state of the art, we split 3D-HPE into two steps, first estimating J human 2D keypoints in the pixel space from a sequence of T video frames [x 1 , . . . , x T ] ∈ R 2×J×T , and then lifting them to 3D joint positions [p 1 , . . . , pT ] ∈ R 3×J×T . We focus on the second step (i.e., lifting) in the rest of the paper, assuming the availability of 2D keypoints x i . Our method aims to both ensure pose consistency and resolve depth ambiguity, as we will discuss in the next section. 
constraining predictions to the pose manifold: 
 Rationale. Human morphology prevents the joints from arbitrarily occupying the whole space. Instead, the poses within a movement are restricted to a manifold, reflecting the human skeleton's rigidity. If we knew the length of each segment connecting pairs of joints for a given subject, we could guarantee that the predicted poses lie on the correct pose manifold by only predicting the body part's rotations with respect to a reference skeleton. Since we do not have access to ground-truth segment lengths in real use cases, we propose to predict them, thus disentangling the estimation of the reference lengths (fixed across time) from the estimation of the joint rotations (variable across time).Disentangled representations. We constrain model predictions to lie on an estimated manifold by predicting parametrized disentangled transformations of a reference pose u ∈ (R 3 ) J , for which all segments have unit length. Namely, we propose to split the network into two parts (cf. Fig. 2 ):1. Segments module, which predicts segment lengths s ∈ R J-1 , shared by the T frames (time steps) of the input sequence; 2. Rotations module, which predicts the rotation r = [r 1,0 , . . . , r T,J-1 ] ∈ (R d ) J×T of each joint relative to their parent joint at each time step.Rotations representation. We represent rotations using 6D continuous embeddings (i.e., d = 6).Compared to quaternions or axis-angles, those representations are continuous and, hence, better learned by neural networks, as demonstrated by their proposers [54] .Pose decoding. To deliver pose predictions in (R 3 ) J×T , the intermediate representations (s, r) must be decoded. We achieve that in three steps (cf. Fig. 3 ):1. We scale the unit segments of the reference pose u ∈ (R 3 ) J using s, forming a scaled reference pose u ′ : u ′ j = u ′ τ (j) + s j (u j -u τ (j) ) for 0 < j ≤ J -1, where τ maps the index of a joint to its parent's, if any.2. For each time step 1 ≤ t ≤ T and joint 0 ≤ j < J, we convert the predicted rotation representations r t,j into rotation matrices R t,j ∈ SO(3) (Algorithm 1).3. We apply those rotation matrices R t,j at each time step t to the scaled reference pose u ′ using forward kinematics (Algorithm 2).ManiPose architecture. As explained in the introduction, the inherent depth ambiguity of pose lifting requires multiple hypotheses to conciliate pose consistency and MPJPE performance. To address this, we adopt the multiple choice learning (MCL) [21] framework, more precisely leveraging the resilient MCL approach as proposed by Letzelter et al. [22] . This methodology allows the estimation of conditional distributions for regression tasks, enabling our model to predict multiple plausible 3D poses for each 2D input. Specifically, instead of a single rotation r t ∈ (R γ k = (γ k t ) T t=1, called scores hereafter (Fig. 2 ).As in [22] , ManiPose is trained with a composite lossEQUATIONThe first term, L wta , is the winner-takes-all loss [21] L wta (p(x), p) = 1EQUATIONwhere ℓ(p k t (x), p t ) ≜ 1 J J-1 j=0 ∥p t,j -pk t,j (x)∥ 2 , and pk t (x) denotes the pose prediction at time t using the k th head. The second term, L score , is the scoring lossEQUATIONwhere H(•, •) is the cross-entropy, pt = (p k t ) K k=1 , andEQUATIONis the indicator function of the winner pose hypothesis, which is the closest to the ground truth. Eq. ( 3) is the average cross-entropy between target and predicted scores γ t (x) ∈ [0, 1] K at each time t.Those losses are complementary. The winner-takes-all loss updates only the best predicted hypothesis, specializing each head on part of the data distribution [21] . The scoring loss allows the model to learn how likely each head is to winning, thus avoiding overconfidence of non-winner heads (cf. [19, 45] ).Conditional distribution estimation. As detailed in [22] , the model may be interpreted probabilistically as a multimodal conditional density estimator. More precisely, it models the distribution P(p|x) of 3D poses conditioned on 2D poses as a mixture of Dirac distributions:EQUATIONHence, the predicted conditional distribution has, at each predicted hypothesis pk , a peak whose likelihood is given by the predicted score γ k . As described in Section 4, interpreting hypotheses and scores probabilistically enables us to handle depth ambiguity. ManiPose, as outlined in Section 3, is crafted to address the flaws inherent in unconstrained, single-hypothesis lifting-based 3D-HPE methods (see Fig. 1 ). This section illustrates that without ManiPose's critical components (multiple hypotheses and manifold constraint), it is impossible to simultaneously minimize joint error and ensure pose consistency (Section 4.1). To illustrate this, a toy example within a simplified 1D-to-2D framework is provided in Section 4.2. 
single-hypothesis position-error minimization leads to inconsistent skeleton lengths: 
 We formally highlight the limitations of unconstrained single-hypothesis 3D-HPE, justifying our approach, which combines consistency constraints and multiple hypotheses to resolve depth ambiguity.Let p = [p 1 , . . . , p J ] ∈ R 3×J be a human pose, defined by the Cartesian 3D coordinates of each of the J joints of a predefined skeleton. Then, a sequence of T poses of the same subject at increasing time steps t 1 . . . t T ∈ R forms a movement m = [p 0 , . . . , p T ] ∈ R 3×J×T . Assuming bone length is fixed during a movement (which is empirically verifiable in human pose datasets), then the poses p t of m must all lie on the same smooth manifold. Proof sketch. (Detailed in Appendix B). Skeleton rigidity implies that, if i is a joint connected to the root, then it lies on a 2D sphere S 2 (0, s i,0 ) centered at the origin with fixed radius s i,0 . Another joint j linked to i has a position expressible by its spherical coordinates relative to i with fixed radius s j,i . That implies an homeomorphism between the position p t,j of joint j and the direct product of spheres centered at the origin S 2 (0, s i,0 ) × S 2 (0, s j,i ). By induction, one can show that p t lies on a subspace of (R 3 ) J , which is homeomorphic to a product of spheres centered at the origin. ■ Proposition 4.1 implies that all poses predicted for a video sequence should ideally lie on the same manifold M as the ground-truth data, which is homeomorphic to the direct product of 2D unit spheres (S 2 ) J-1 (cf. Appendix B). Crucially, we can further show that minimizing joint position error using a single-hypothesis model necessarily leads to predicted poses lying outside the true manifold: Proposition 4.2 (Inconsistency of MSE minimizer). With a rigid skeleton and mild assumptions on the training distribution, predicted 3D poses minimizing the traditional mean squared error (MSE) loss lie outside the pose manifold M.Proof sketch. (See Appendix B). Consider a skeleton with J joints, with (x, p), as pairs of corresponding 2D inputs and 3D poses. Let the function ℓ = (ℓ j ) J-1 j=1 compute the lengths of the segments in a pose, which shall remain constant. On a dataset {(x i , p i )} N i=1 drawn from the joint distribution of 2D and 3D poses, let the expected MSE of a traditional predictive model f be E x,p ∥p -f (x)∥ 2 2 . Let the ideal model f * be the one minimizing that expected MSE, which is the conditional expectationf * (x) = E[p | x].Jensen inequality and the rigidity assumption imply that, for any joint j, ℓ 2 j (f * (x)) < s 2 j where s j is the true length of the segment associated with joint j. This shows that the poses predicted by f ⋆ violate the original segment length constraints, and thus, the original rigidity assumption. ■ Proposition 4.2 has the following implications:1. Traditional unconstrained single-hypothesis approaches are bound to predict inconsistent movements, where bone lengths may vary. 2. With a single hypothesis, models constrained to the manifold will always lose to unconstrained models in terms of MPJPE performance (formalized in Corollary B.1). 3. The only way of reaching both optimal MPJPE and consistency is through multiple hypotheses (formalized in Corollary B.3).Therefore, the MPJPE metric (and its traditional extensions) is insufficient to assess 3D-HPE, as it completely ignores pose consistency. Furthermore, we are able to prove in Appendix B.2 that multiple hypotheses (constrained or not) can always reach better joint position errors than single-hypothesis models. We illustrate the argument of Section 4.1 with a simplified 1D-to-2D setup. We further generalize this intuitive illustration to the 2D-to-3D setting in Appendix C of the supplementary.As in human pose lifting, we take a root joint J 0 as reference, fixed at (0, 0). For a joint J 1 , the problem amounts to predicting the 2D position (x, y), given its 1D projection u = x, assuming a constant distance s = 1 between them. This simplification ignores the camera perspective and considers the joints to be connected by a rigid segment as in the case of human poses.We train three different models with comparable architectures on two datasets {(x i , (x i , y i ))} N i=1 sampled from the angular distributions represented in blue on Fig. 4 . The models correspond to:1. A 2-layer MLP ( ) trained to minimize the mean squared error between true (x, y) and predicted joint positions (x, ŷ);2. A constrained MLP of the same size ( ), predicting the angle θ instead of the joint position; 3. ManiPose: our constrained multi-hypothesis model capable of predicting K = 2 possible angles ( θk ) K k=1 with their corresponding likelihoods.Fig. 4 shows that the traditional unconstrained single-hypothesis approach ( ) leads to good results in an easy unimodal scenario (C), but fails when facing a more challenging bimodal distribution (D), leading to predictions outside the circle manifold, as depth ambiguity makes the lifting problem ill-posed. The single-hypothesis constrained model ( ) delivers predictions on the circle, at the cost of worse MPJPE performance than the unconstrained MLP. Such performance decrease is due to the Euclidean topology of the MPJPE metric having its minimum ( ) outside the manifold (Fig. 4-B ).Crucially, this implies that the unconstrained single-hypothesis models are bound to make inconsistent predictions, with varying "bone lengths" (the circle radius). It also shows that models constrained to the manifold (circle) will always be outcompeted by unconstrained models on MPJPE performance.Predicting multiple hypotheses constrained to the circle, with their respective likelihoods (⋆ in Fig. 4-B ) allows escaping this dilemma, which is exactly what ManiPose does ( in Fig. 4-D ). The predicted hypotheses are all on the circle, contrary to the unconstrained MLP, and spread between the two distribution modes, unlike the constrained single-hypothesis method.Moreover, the predicted scores (length of green lines) match the 2 3 and 1 3 ground-truth likelihoods of the two modes. Those advantages translate into perfect pose consistency and into comparable MPJPE performance with respect to the unconstrained MLP (Table 1 ). 
experimental setup: 
 Datasets. We evaluate our model on two 3D-HPE datasets. Human 3.6M [15] contains 3.6 million images of 7 actors performing 15 different indoor actions. It is the most widely used dataset for 3D-HPE. Following previous works [52, 27, 53, 38] , we train on subjects S1, S5, S6, S7, S8, and test on subjects S9 and S11, adopting a 17-joint skeleton (cf. Fig. 5 ). We employ a pre-trained CPN [5] to compute the input 2D keypoints, as in [38, 52] . MPI-INF-3DHP [32] also adopts a 17-joint skeleton, but, with fewer samples and containing both indoor and outdoor scenes, it is more challenging than Human 3.6M. We used ground-truth 2D keypoints for this dataset, as usually done [53, 4, 52] .Traditional evaluation metrics. The mean per-joint position error (MPJPE) is the usual performance metric for the datasets above, under different protocols, both reported in mm. In protocol #1, the root joint position is set as a reference, and the predicted root position is translated to 0. In protocol #2 (P-MPJPE), predictions are additionally Procrustes-corrected. For MPI-INF-3DHP, additional thresholded metrics derived from MPJPE are often reported, such as AUC (Area Under Curve) and PCK (Percentage of Correct Keypoints) with a threshold at 150 mm, as explained in [32] .Pose consistency metrics. MPJPE being insufficient to assess pose consistency (Section 4), we further assess to which extent predicted skeletons are rigid by measuring the average standard deviations of segment lengths across time in predicted action sequences:EQUATIONwith s t,j,i = ∥p t,j -pt,i ∥ 2 and sj,i = 1 T T t=1 s t,j,i , where τ was defined in Section 3.1. We call this metric, reported in mm, the Mean Per Segment Consistency Error (MPSCE).Following [12, 40] , we also assess the bilateral symmetry of predicted skeletons through the Mean Per Segment Symmetry Error (MPSSE), in mm:EQUATIONwhere J left denotes the set of indices of left-side joints and ζ maps left-side joint indices to their right-side counterparts.Multi-hypothesis evaluation protocol. One must decide how to use multiple hypotheses to compute the metrics. The dominant approach [24, 25, 36, 44, 49, 12] is the oracle evaluation, i.e., using the predicted hypothesis closer to the ground truth (i.e., Eq. ( 2) for MPJPE). That makes sense for multi-hypothesis methods, as the oracle metric measures the distance between the target and the discrete set of predicted hypotheses. It aligns with the idea of many possible outputs for a given input.Hypotheses can also be aggregated into a final pose, e.g., through unweighted or weighted averaging (using predicted scores). The latter has the disadvantage of falling back to a one-to-one mapping scheme, which is precisely what we want to avoid in a multi-hypothesis setting.We report both oracle and aggregated metrics in our experiments, favoring oracle results.Implementation details. ManiPose, as presented in Section 3, is compatible with any backbone.Here, we chose to build on the MixSTE [52] network for both the rotations and the segment modules (the latter in a reduced scale). Details about our architecture and training appear in Appendix D. 2 shows that they require up to two orders of magnitude more hypotheses than ManiPose to reach competitive performance (see, e.g., the performance of GFPose). This property is expected. Indeed, optimization based on Winner-Takes-All theoretically leads to an optimal coverage of the modes of the conditional distribution with a fixed number of samples [23] , in contrast to generative-based approaches. This is reflected in the oracle metric, which approximates the so-called quantization (or Distortion) error, as defined in (27) , when the number of data points is large. More detailed MPJPE results per action appear in Tables 8 and 9 in the supplemental. We also complement our analysis on the diversity of ManiPose in Fig. 11 of the appendix.Fig. 6 showcases qualitative results, where multiple hypotheses help in depth-ambiguous situations. MPI-INF-3DHP. Similar results were obtained for this dataset (cf. Table 3 ). Not only does ManiPose reach consistency errors close to 0, but also best PCK and AUC performance. As for MPJPE, only [42] achieves slightly better performance, at the cost of large pose consistency errors. Impact of components. We evaluate the impact of removing each component of ManiPose on the Human 3.6M performance (Table 4 ). The components tested are the multiple hypotheses (MH) and the manifold constraint (MC). We also compare MC to a more standard manifold regularization (MR), i.e., adding Eq. ( 7) to the loss. Note that without all these components, we fall back to MixSTE [52] , and that the performances reported in Table 4 also appear in Fig. 1 .We see that MR helps to improve pose consistency, but not as much as MC. However, without multiple hypotheses, MC consistency improvements come at the cost of degraded MPJPE performance, as foreseen by our formal analysis (Section 4). Only the combination of both MC and MH allows us to optimize both consistency and MPJPE.Fine error analysis. We can see in Fig. 5 that, compared to MixSTE, ManiPose reaches substantially superior MPSSE and MPSCE, consistency across all skeleton segments. Furthermore, note that larger MixSTE errors occur for segments KNEE-FOOT and ELBOW-WRIST, which are the most prone to depth ambiguity. That agrees with coordinate-wise errors depicted in Fig. 5 , showing that ManiPose improvements mostly translate into a reduction of MixSTE depth errors, which are twice as large as for other coordinates. Further ablations, including the effect of the number of hypotheses K, the score loss weight β and the rotations representation choice appear in the supplemental.Ground-truth  
conclusion: 
 We presented a new manifold-constrained multi-hypothesis human pose lifting method (ManiPose) and demonstrated its empirical superiority to the existing state-of-the-art on two challenging datasets. Further, we provided theoretical evidence supporting the tenets of our method, by showing the inherent limitation of unconstrained single-hypothesis approaches to 3D-HPE. We established that unconstrained single-hypothesis methods cannot deliver consistent poses and that constraining or regularizing single-hypothesis models leads to worse position errors. We also showed that traditional MPJPE-like metrics are insufficient to assess consistency.Limitations. To guarantee its consistency, ManiPose relies on the forward kinematics algorithm, which is inherently sequential across joints. Removing that dependence is an interesting avenue for accelerating the method. On another note, while ManiPose ensures the rigidity of the predicted poses, imposing constraints within human body articulation limits presents another area for enhancement. Distributions are estimated with a kernel density estimator from the Seaborn plotting library [48] . 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. 
 
 Appendix: This section contains the proofs of the theoretical results presented in Section 4.1, together with a few corollaries.PROOF. [Proposition 4.1] Let i be a joint connected to the root p 0 (i.e., A i0 = 1). From assumptions A.3 and A.4, we know that at any instant t, p G t,i lies on the sphere S 2 (0, s i,0 ) centered at 0 with radius s i,0 independent of time. Therefore, its position can be fully parameterized in spherical coordinates by two angles (θ t,i , ϕ t,i ). Let j be a joint connected to i. Like before, assumption A.4 implies that at any instant t, p G t,j lies on the moving sphere S 2 (p G t,i , s j,i ) centered at p G t,i with radius s j,i independent of time. Thus, we can fully describe p G t,j with the position of its center, p G t,i and the spherical coordinates (θ t,j , ϕ t,j ) of joint j relative to the center of the sphere, i.e., joint i. That means that there is a bijection between the possible positions attainable by p G t,j at any instant and the direct product of spheres S 2 (0, s i,0 ) ⊗ S 2 (0, s j,i ). 1 That bijection is an homeomorphism since it is a composition of homeomorphisms: we can compute p G t,j from (θ t,i , ϕ t,i , θ t,j , ϕ t,j ) following the forward kinematics algorithm [34] (cf. algo. 2), i.e., using a composition of rotations and translations. Now let us assume for some arbitrary joint k that p G t,k lies at all times on a space M 2d homeomorphic to a product of spheres of dimension 2d. That means that p G t,k can be fully parametrized using 2d spherical angles (θ 1 , ϕ 1 , . . . , θ d , ϕ d ). Let l be a joint connected to k (typically one further step away from the root joint p 0 and not already represented in M 2d ). As before, at any instant t, p G t,l needs to lie on the sphere centered on p G t,k of constant radius s k,l . Thus, we can fully describe p G t,l using the 2(d + 1)-tuple of angles obtained by concatenating its spherical coordinates relative to joint k, together with the 2d-tuple describing p G t,k , i.e.the center of the sphere. So p G t,l lies on a space M 2(d+1) homeomorphic to a product of spheres of dimension 2(d + 1).We can conclude by induction that at any instant t, p t = [p G t,1 , . . . , p G t,J ] lies on the same subspace of (R 3 ) J , which is homeomorphic to a product of spheres centered at the origin:EQUATION)Finally, the previous space is trivially homeomorphic to (S 2 ) J-1 through the scaling (1/s i,j ) i<j/Aij =1 . (S 2 ) J-1 is a manifold of dimension 2(J -1) as the direct product of J -1 manifolds of dimension 2. ■ PROOF. [Proposition 4.2] Let G be a skeleton with J joints, x ∈ (R 2 ) J a 2D pose, p ∈ (R 3 ) J its corresponding 3D pose, and P(x, p) a joint distribution of poses in 2D and 3D. We define ℓ = (ℓ j ) J-1 j=1 as the function allowing us to compute the length of the segments of a pose p:EQUATIONwhere τ : {1, . . . , J -1} → {0, . . . , J -1} maps joint indices to the index of their parent joint:EQUATIONFrom assumption A.4, we know that for any pose p from the training distribution, ∀j , ℓ j (p) = s j,τ (j) .Given D = {(x i , p i )} N i=1 ∼ P(x, p), some i.i.d. evaluation data, the MSE of a model f is defined as:EQUATIONand converges toEQUATION)1 S 2 (0, sj,i) is homeomorphic to S 2 (p G t,j ; sj,i).as the dataset size N goes infinity. We then define the oracle MSE minimizer asEQUATION)The quantity in ( 15) is known in statistics as the expected L 2 -risk and it is a well-known fact that its minimizer is the conditional expectation:EQUATION) Thus, since ℓ 2 j are strictly convex and P(p|x) is non-degenerate according to assumption A.5, we can conclude from Jensen's strict inequality that for all j, ℓ 2 j (fEQUATIONwhere the last equality arises from the fact that ℓ 2 j (p) is not random according to (13) . Thus, given that ℓ j > 0 and s j,τ (j) > 0, we can say that ℓ j (f * (x)) < s j,τ (j) for all joints j. We conclude that the model f * minimizing MSE * predicts poses that violate assumption A.4 and are inconsistent. ■ As an immediate corollary of proposition 4.2, we may state the following result, which was empirically illustrated in many parts of our paper: Corollary B.1. Given a fixed training distribution P(x, p) respecting assumptions A.3-A.5, for all 3D-HPE model f predicting consistent poses, i.e., that respect assumption A.4, there is an inconsistent model f ′ with lower mean-squared error.EQUATION) Since MSE * reaches its minimum at f ′ , we have MSE * (f ) = MSE * (f ′ ). Thus, f ∈ arg min f MSE * ( f ),which means that f is also inconsistent according to proposition 4.2. That is impossible given that we assumed f to be consistent. We conclude that Eq. ( 19) is wrong and thatEQUATION■ Note that propositions 4.2 and B.1 assume the use of the MSE loss, which is the most widely used loss in 3D-HPE. We can however extend them to the case where MPJPE serves as optimization criteria under an additional technical assumption: Corollary B.2. The predicted poses minimizing the mean-per-joint-position-error loss are inconsistent if the training poses distribution P(x, p) verifies Asm. A.3-A.5 and if the joint-wise residuals' norm standard deviation is small compared to the joint-wise loss:EQUATIONPROOF. From proposition 4.2 we know that the poses predicted by the minimizerEQUATION)are inconsistent. Let f j be the component of f corresponding to the j th joint. We define the j th mean-per-joint-position-error component as:EQUATION)Under the small variance assumption, we have:EQUATIONEQUATIONso both criteria, MSE and MPJPE, are asymptotically equivalent and have the same minimizer f * , which is inconsistent according to proposition 4.2. ■ B.3. Under Asm. A.4-A.5 and under (21) , the only way to get both optimal MPJPE and consistency is to use multiple hypotheses.PROOF. Corollary B.1 and Proposition 4.2 imply that single-hypothesis models (constrained or not) deliver either suboptimal MPJPE or inconsistent pose predictions. Hence, by negation, we get our result.In the next section, we further show that multi-hypotheses models, constrained or not, can theoretically show a better L2-risk (or quantization) performance compared with single-hypotheses models.Let X = R 2×J denote the space of input 2D poses and P = R 3×J the space of 3D poses. Also, let R(f) = E x,p [∥p -f (x)∥ 2 2] be the L2-risk of some pose estimator f under some underlying continuous joint distribution of 2D-3D pose pairs P(x, p), with density ρ (when it exists).Before stating the proposition, we need to define an adapted notion of risk for multi-hypothesis models under the oracle aggregation scheme: Definition B.4 (Winner-takes-all risk, [41] ). As in [41] (section 3.2) and in [23] (section 2.2), we define the L2-risk for K-head models f WTA = (f 1 WTA , . . . , f K WTA ) as:EQUATIONwhere V k (g)'s denotes the k th cell of the Voronoi tesselation of the output space P defined by generators g = (g 1 , . . . , g K ) ∈ P K :EQUATIONThe risk above translates the notion of oracle pose, since it partitions the space of ground-truth poses P into regions where some hypothesis is the closest, and uses only that hypothesis to compute the risk in that region. Note that R 1 WTA (f ) = R(f ) for any function f , since a single-cell tessellation of P is P itself.In the following, we assume that f is expressive enough, so that, minimizing the risk (27) comes down to minimizingEQUATIONPROOF. Following [23] (Section 2.2), we decouple the cell generators from the risk arguments in (27) :EQUATIONfor any generators g = (g 1 , . . . , g K ) ∈ P K and arguments z = (z 1 , . . . , z K ) ∈ P K . Note thatEQUATIONLet's choose g such that g = f k, * WTA (x) and z such thatEQUATIONwhere the last equality comes from the fact that V k (f * WTA (x)) defines a partition of P.to each other. Thus, J 1 is constrained to lie on a circle S 1 (0, s 0 ), while J 2 lies on a torus T homeomorphic to S 1 (0, s 0 ) ⊗ S 1 (0, s 1 ). Without loss of generality, we set the radii s 0 = 2 and s 1 = 1 and assume them to be known.Given that setup, we are interested in learning to predict the 3D pose(J 1 , J 2 ) = (x 1 , y 1 , z 1 , x 2 , y 2 , z 2 ) ∈ R 6 , given its 2D projection (K 1 , K 2 ) = (x 1 , z 1 , x 2 , z 2 ) ∈ R 4 .We create a dataset comprising 20000 training, 2000 validation, and 2000 test examples, sampled using an arbitrary von Mises mixture of poloidal and toroidal angles (θ, ϕ) in T . We set the modes of such a mixture at [(-π, 0), (0, π/4), ( 1 2 , -π/4), (2π/3, π/2)], with concentrations of [2, 4, 3, 10] and weights [0.3, 0.4, 0.2, 0.1]. Similarly to Fig. 4-C , that creates a difficult multimodal distribution, depicted in Fig. 8 . We train and evaluate the same baselines as in Section 4.2 in that new scenario, using a similar setup (cf. Appendix C. We see that the same observations as in Section 4.2 also apply here: although the unconstrained MLP yields competitive MPJPE results, its predictions are not consistently aligned with the manifold, as indicated by its poor MPSCE performance. Again, we show here that ManiPose offers an effective balance between maintaining manifold consistency and achieving high joint-position-error performance.EQUATIONOur architecture is backbone-agnostic, as shown on Fig. 2 . Thus, in order to have a fair comparison, we decided to implement it using the most powerful architecture available, i.e., MixSTE [52] .In The ablation study presented in Table 4 shows that the increase in the number of parameters between MixSTE and ManiPose is negligible.The pose decoding block from Fig. 2 is described in Section 3.1 and is based on Algorithms 1 and 2.The whole procedure is illustrated on Fig. 3 .Table 7 : Joint-wise weights used in the Winner-takes-all loss Eq. ( 2) (as in [52] ). Algorithm 1 6D rotation representation conversion [54] Require: Predicted 6D rotation representation r ∈ R 6 . 1:x ′ ← [r 0 , r 1 , r 2 ] , 2: y ′ ← [r 3 , r 4 , r 5 ] , 3: x ← x ′ /∥x ′ ∥ 2 , 4: z ′ ← x ∧ y ′ , 5: z ← z ′ /∥z ′ ∥ 2 , 6: y ← z ∧ x , 7: return R = [x|y|z] ∈ R 3×3 .Algorithm 2 Forward Kinematics [34, 26] Require: Scaled reference pose u ′ ∈ (R 3 ) J , predicted rotation matrices R t,j , 0 ≤ j < J.1: R ′ t,0 ← R t,0 , 2: p t,0 ← u ′ 0 , 3: for j = 1, . . . , J -1 do 4: R ′ t,j ← R t,j R ′ t,τ (j) , ▷ Compose relative rotations 5:p t,j ← R ′ t,j (u ′ j -u ′ τ (j) ) + p t,τ (j) , 6: end for 7: return p t = [p t,j ] 0≤j<JTraining tactics. In order to have a fair comparison with MixSTE [52] , we trained ManiPose using the same training tactics, such as pose flip augmentation both at training and test time. Moreover, the training loss (1) was complemented with two additional terms described in [52] :1. a TCloss term, initially introduced in [13]; 2. a velocity loss term, introduced in [38] .We also weighted the Winner-takes-all MPJPE loss (2) as in [52] (cf. weights in Table 7 ). The score loss weight, β, was set to 0.1 according to our hyperparameter study (Appendix E), while TCloss and velocity loss terms had respective weights of 0.5 and 2 (values from [52] ).Training settings. We trained our model for a maximum of 200 epochs with the Adam optimizer [17] , using default hyperparameters, a weight decay of 10 -6 and an initial learning rate of 4 × 10 -5 . The latter was reduced with a plateau scheduler of factor 0. Dataset licences. Human 3.6M is a dataset released under a research-only custom license, and is available upon request at this URL: http://vision.imar.ro/human3.6m/description.php. MPI-INF-3DHP is released under non-commercial custom license and can be found at: https: //vcai.mpi-inf.mpg.de/3dhp-dataset/.All Human 3.6M evaluations of MPSSE and MPSCE listed in Tables 2 and 4 were performed using the official checkpoints of these methods and their corresponding official evaluation scripts. Concerning MPI-INF-3DHP evaluations from Table 3 , checkpoints were not available (except for P-STMO). Thus, baseline models were retrained from scratch using the official MPI-INF-3DHP training scripts provided by the authors of each work, using hyperparameters reported in their corresponding papers. We checked that we were able to reproduce the reported MPJPE results. Protocol #1 and #2 detailed results. A detailed quantitative comparison in terms of MPJPE per action on Human3.6M dataset between ManiPose and state-of-the-art methods is shown in Table 8 . We see that ManiPose reaches the best MPJPE performance on average and on most actions. Table 9 contains a similar analysis in terms of P-MPJPE (i.e., MPJPE with procrust-aligned poses). We observe the same patterns as in Table 8 , namely that ManiPose reaches the second-best P-MPJPE performance on average and for most actions. We confirm here again that the substantial improvements in pose consistency brought by ManiPose are not obtained at the expense of traditional metrics derived from MPJPE.Errors per joint. On the top of Fig. 9 we see that most of MixSTE errors come from feet, elbows and wrist joints, which are most prone to depth ambiguity. ManiPose helps to reduce the position errors for most of those ambiguous joints, probably as a byproduct of its major consistency improvements shown in Fig. 5 .ManiPose introduces two additional hyperparameters when compared to MixSTE: the number K of hypotheses and the score loss weight β (cf. Eq. ( 1)). We further assess the impact of their respective values on MPJPE. For computational cost reasons, we used a smaller version of our model for this study, with transformer blocks of dimension d m = 64 and time receptive field of T = 27 frames. Fig. 10 (left) shows that more hypotheses help, but that the performance Impact of the rotations representations used.The disentanglement between segments' length and orientation is not novel, and was proposed in previous works restricted to the single-hypothesis case, such as Anatomy3D [4] . While ManiPose represents segments' orientations as full 3D rotations relatively to parent segments in the kinematics tree, Anatomy3D simply predicts segments' absolute directions, i.e., normalized vectors in the 3D space. This solution has the advantage of not over-parametrizing the segments orientations (which are invariant to rotations around the segment axis) and being lower dimensional (3 vs 6). One might hence wonder whether Anatomy3D's parametrization is not preferable. As shown in Table 10 , Anatomy3D's implementation led to poorer results when compared to our rotations parametrization in a multi-hypothesis setting. This motivated us to use full 3D rotations' representations proposed in [54] in our experiments, despite their caveats. Note that [54] also shows good empirical results in the related problem of inverse kinematics of human 3D poses. Diversity of predicted poses. As explained in Section 5.2, ManiPose's state-of-the-art oracle MPJPE results show that it excels in terms of diversity when the latter is assessed using the quantization error. There are many other ways of assessing distribution diversity. In an attempt to quantify the diversity of pose distributions learned by ManiPose by other means, we have computed the coverage (as defined in [35] ) of generated poses relatively to the ground-truth test set of Human 3.6M. For computation cost reasons (it grows quadratically with sample size), we limited our analysis to 5 actions from subject S11. We compare ManiPose to DiffPose [10] , using 5 hypotheses for both, and observe similar diversity on average (cf. Fig. 11 ).We provide the code to reproduce all our experiments under https://github.com/ cedricrommel/manipose. Figure 11 : ManiPose achieves similar diversity to DiffPose [10] . Diversity is assessed through the coverage [35] over test data from subject 11 from Human 3.6M. 5 hypotheses were predicted/sampled for each frame by both models. 


Decision: This paper presents a manifold-constrained multi-hypothesis model for 3D human pose lifting.  To ensure the body symmetry and temporal consistency, the method disentangles human skeleton to two parts: temporally consistency bone scales and temporally variable bone rotations. The method then predicts skeletal parameters so that the predicted 3D poses in a sequence are constrained to one smooth manifold. The paper provides theoretical arguments regarding the benefits of enforcing such structural priors in the prediction. The proposed method is validated on Human3.6M and MPI-INF-3DHP datasets. Although theoretical analysis to support the arguments in the paper is appreciated, the reviewers raised concerns regarding diversity and fixed number of hypotheses, unclear advantage of multi-hypothesis methods, undetailed explanation about the method, trade-off between accuracy and consistency, missing comparison, and difference from Anatomy3D and Pose-NDF.  The authors addressed in their rebuttal most of the concerns by providing new arguments, new experiments and new theoretical analysis. During post-rebuttal discussion, the authors further clarified the contributions of the paper, and convinced three of the four reviewers.  The reviewer tj82 suggested to specify the definition of diversity, which AC thinks should be fixed.  The review fpqy, on the other hand, was not satisfied the feedbacks from the authors.  He/She keeps concern on the positioning of the paper, meaning that theoretical contribution or 3D pose lifting application is unclear.  It was also pointed out that the new theoretical results provided in the rebuttal are obvious.  AC does not agree with the reviewer fpqy in the sense that even obvious theoretical results are sometimes valuable.  This is because results in good harmony with our intuition bring to us the strong ground of the method rather than heuristically/empirically developed methods. AC also thinks the balance between theoretical analysis and the proposed lifting method is good and does not reduce the strengths of the paper. This paper should be accepted, accordingly.

Reviews: review: paper_summary: paper_summary: This paper propose ManiPose, a manifold-constrained multi-hypothesis model for 3D human pose lifting. The authors provide empirical and experimental evidence to show that joint position regression leads to inconsistent skeleton lengths. And they propose to predict globally consistent pose scale and individual joint rotations per frame (rather than joint positions) to constrain the predictions to the pose manifold. Empirical results demonstrates that the proposed ManiPose framework improves the pose consistency.
strength_weakness: strength_weakness: * The paper uses a multi-head design to predict multiple hypotheses. This design loses the flexibility of sampling different numbers of hypotheses and limits the maximum number of hypotheses to a small number. This often results in limited hypothesis diversity. In the experimental section, the authors do not provide numerical of visual measurements of hypothesis diversity.
* According to the comparison in Table 4, the manifold constraint proposed in this paper sacrifices MPJPE to improve pose consistency, serving as a trade-off approach between accuracy and consistency. Although the consistency is improved, it lags behind the traditional position regression or manifold regularization in accuracy, and does not bring essential improvement (improve both in accuracy and consistency) compared with these two methods.
* Missing comparison with two recent multi-hypothesis methods. [1] GFPose: Learning 3D Human Pose Prior with Gradient Fields. [2] DiffPose: Toward More Reliable 3D Pose Estimation.
questions: questions: Please review the Weaknesses Section. If the author can address or respond to the above issues well in the rebuttal stage, I will consider increasing my score.
limitations: limitations: As the authors discussed in the Limitations Section, they used forward kinematics to obtain joint positions, which can lead to error accumulation.
paper_summary: paper_summary: This paper presents a new method to estimate 3D human pose from 2D observations (lifting). To ensure the body symmetry and temporal consistency, the authors disentangle human skeleton to two parts: temporally consistency bone scales and temporally variable bone rotations. The authors use fancy formulas to prove that, minimizing MSE loss could not gurantee manifold consistency. The quantitative and qualitative results on Human3.6m and MPI-INF-3DHP datasets show the superiority of the proposed method.
strength_weakness: strength_weakness: I am not an expert in manifold theory, therefore my questions only relate to human pose estimation. 

1. How to constrain the rotation space during training? 

2. The pose lifting method is quite similar to Anatomy3D (bone length + rotations). Can I view this paper an multi-hypothesis extension of Anatomy3D? Why? 

3. Previous paper "POSE-NDF: MODELING HUMAN POSE MANIFOLDS WITH NEURAL DISTANCE FIELDS" is similar to this paper in concepts. SMPL naturally guarantees bone length symmetry, and the learnable parameters (rotations and shape parameters) are similar to this paper in its functionality. It would be better to cite it. 

4. Suppose that, there is a virtual dataset, all 2D human joints are rendered (projected) from strictly symmetric 3D joints, then, could learning the lifting function on this virtual dataset using MSE loss guarantee the results all lie on manifold? 

5. (An optional question) The ground truth 3D joints of Human3.6M datasets come from the marker tracking on body surface, which naturally could not guarantee skeleton length consistency. Why learning symmetric bones yields better results (both Anatomy3D and the proposed methods)?
questions: questions: 1. The citation style is weird. They are not NeurIPS style, please correct them.
limitations: limitations: The authors addressed limitations.
paper_summary: paper_summary: This paper proposes a MCL-based framework for multi-hypothesis 3D human pose estimation. This framework predicts skeletal parameters so that the predicted 3D poses in a sequence are constrained to one smooth manifold. To prove the superiority of such a framework, the paper presents detailed theoretical analysis on the drawback of unconstrained single-hypothesis HPE and why MPJPE alone is not enough for pose evaluation. The experiments show the proposed framework is capable of keeping the consistency of predicted poses and achieving state-of-the-art MPJPE in the meantime.
strength_weakness: strength_weakness: * Theoretical analysis on the advantage of multi-hypothesis methods over single-hypothesis ones could be added. Specifically, why a **constrained multi-hypothesis** method performs better than an **unconstrained single-hypothesis** method in MPJPE? Though this is already validated by the experiments, I personally believe it would make the paper more solid if the authors could make this analysis.

Minor problem:
* In Fig.4 (C) and (D), it is not quite clear how the estimations (crosses and triangles) correspond with the inputs (black dots). There might be some unexpected shifts, as the projections of the predicitons do not strictly align with the inputs (like in B).
questions: questions: What is the quality of the score for each hypothesis? If the multiple hypotheses are fused to one (e.g. by taking the one with the largest confidence or taking the weighted average), then how will the MPJPE, MPSCE, and MPSSE change?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper presents a method to estimate 3D human keypoints from a sequence of monocular 2D keypoints observations. It builds upon an existing sequence-to-sequence architecture (MixSTE), with a different output parameterization exploiting a kinematic skeletton prior, and different training losses. Lengths of the skeletton bones are predicted for the whole sequence to ensure consistency across frames (and maybe also left/right symmetry of the skeletton), and five 3D pose hypotheses with associated scores are predicted for each frame, parameterized as a list of 3D relative orientation for each bone with respect to its parent in the kinematic tree.

The authors develop theoretical arguments regarding the benefits of enforcing such structural priors in the predictions, and illustrate with a toy example the interest of having multiple predictions in case of ambiguous multimodal output. They validate their approach on Human3.6M and MPI-INF-3DHP datasets.
strength_weakness: strength_weakness: The idea of enforcing body priors (constant bone length here) is not novel and has actually been heavily exploited in a whole line of work relying on more advanced parametric models such as SMPL [100]. This line of work would deserve being considered in the paper, as it encompass approaches suitable for 2D-to-3D sequence lifting such as e.g. [101].

The authors present a pose space consisting in 3D coordinates of joints linked by some rigid segments. Based on this definition, a natural pose parameterization would consist in the 3D direction of each segment, yet the authors chose to overparameterize poses by using relative 3D bone orientation instead. I understand that such choice can have practical benefits in term of biomechanical constraints and additional supervision signal when ground truth data is available, but such choice should be properly motivated, discussed and ablated in the paper.


The authors describe two ways of aggregating results L247 but do not state which one they use for MPI-INF-3DHP, and they only report oracle results on Human3.6M and for the ablations.

In my understanding, pose hypotheses are selected independently for each frame and there are no temporal terms in the training objectives or aggregation method. Since the proposed approach deals with temporal sequences, it would be worth evaluating the temporal consistency of the predictions, through qualitative video examples and quantitatively e.g. using joint acceleration metrics. Having multiple hypotheses for each frame brings combinatorial questions worth discussing in my opinion.

References:
- [100] Loper at al., “SMPL: A Skinned Multi-Person Linear Model”, at SIGGRAPH Asia 2015.
- [101] Baradel et al., “PoseBERT: A Generic Transformer Module for Temporal 3D Human Modeling”, in TPAMI 2022.
questions: questions: See the weaknesses section for a list of suggestions.
limitations: limitations: N/A

Comments: title: Answer to comments of reviewers tj82 and B5U1
comment: Thank you for your answer. I acknowledge having read the rebuttal (but I don't have Acrobat Reader unfortunately).

For reference, and although this is not directly related to your argument, the continuity theory proposed in [43] is disputed in:
Brégier, "Deep Regression on Manifolds: A 3D Rotation Case Study", in 3DV 2021.
Thanks for the authors' response. While I still have concern about the diversity of the results, I recognize that the proposed method offers better summarization ability compared to GFPose and D3DP. Therefore, I am inclined to raise my score. I recommend that the authors include additional metrics related to diversity in their paper and avoid using "Oracle MPJPE" as a proxy for measuring diversity.
Thank the authors for addressing my concerns. The theory they prove in the rebuttal is a valuable addition to the contribution of this paper. I have also read the comments from other reviewers and agree that some additional experiments could make this paper more solid. However, I shall vote for acceptance because of the theoretical contributions. If the proofs are guaranteed correct (I only checked the proof sketches, due to limited time and expertise), then the conclusions can be very valuable for the community. Thus, I will keep my rating.
We thank the reviewer for taking into account our explanations. We agree that the definition of “diversity” used is indeed important to specify, and will clarify our quantization results in the revised manuscript.
Thanks for the authors' response and Reviewer B5U1's comments. I acknowledge that ManiPose demonstrates better quantization properties compared to generative model-based methods like GFPose and D3DP. I suggest that the authors include this comparison and emphasize this point in their revision, as it highlights a unique advantage of their approach. 

However, I believe that better summarization does not necessarily imply better diversity. I recommend that the authors avoid making such disputed claims in their paper. I also welcome input from other reviewers on this matter.
We thank the reviewers for their prompt answers. Reviewer B5U1 is correct in interpreting Figure 6 and in saying that our answer is connected to [A] "Winner-takes-all learners are geometry-aware conditional density estimators". But we understand reviewer tj82's concern. The reason why we say that **oracle MPJPE** is one way of assessing diversity is because it writes as
$$\frac{1}{N} \sum_{i} \min_k \ell(f_k(x_i), \mathrm{p}\_i),$$
which **is an approximation when $N$ is large of the quantization error**, also known as distortion:
$$\int_{\mathcal{X} \times \mathcal{P}} \min_k \ell(f_k(x), \mathrm{p}) \rho(x, \mathrm{p}) \mathrm{d} x \mathrm{d} \mathrm{p},$$
where $\ell$ is the average joint-wise $L2$ distance in our case. The latter is traditionally used to measure the efficiency of an estimator in summarizing a distribution with few representatives, commonly used to study the K-means estimator for example [C].

So the fact that we achieve better oracle MPJPE than methods requiring a large number of hypotheses (GFPose, D3DP, Wehrbein et al., …) shows that **ManiPose has better quantization properties** than the latter, i.e., it is more efficient in summarizing the diversity of the conditional distribution with fewer representatives/hypotheses.

More practically:
1. the extreme case of a model with no diversity at all (e.g., predicting $K$ times the same pose) would lead to an oracle MPJPE = vanilla MPJPE of its single-hypothesis version. This is not what we obtain in our ablation study of Table 4 (1st vs last rows).
2. In the opposite extreme case, a naive way to obtain $K$ very diverse hypotheses would be to use a regular grid of the pose space $\mathcal{P}=\mathbb{R}^{3 \times J}$. The latter learns nothing and is uninformative, but could still achieve better oracle MPJPE if given an unrealistically large number of hypotheses (cf [A] equation 16).
3. As shown in [A], the winner-takes-all learning scheme, used in ManiPose, allows it to sit between these extreme cases by learning an adaptive “grid” made of a few hypotheses, capturing the geometry of the underlying conditional distribution. Results on its quantization optimality can be found in section 5.2 of [A] for example.

Of course, oracle MPJPE is not the only way of assessing diversity, which is why **we have provided in our rebuttal additional results** measuring the coverage of ManiPose, which corresponds to the ratio of ground-truth poses whose neighborhood contains at least one generated pose. It is a common metric used in the literature on generative models to analyze diversity.

[C] Pages, Gilles, and Jacques Printems. "Optimal quadratic quantization for numerics: the Gaussian case." (2003).
I think the authors' claim in the rebuttal rooted in the conclusion of "Winner-takes-all learners are geometry-aware conditional density estimators". Maybe the authors could elaborate on this point by showing how to map the conclusion of "winner-takes-all ..." paper with the claim here. From my point of view, the diversity has been demonstrated in Figure.6. Such visualization of diversity has also been used in previous papers like HMFormer. 

In fact, I think this claim is not totally unreasonable. For example, for a side view person whose left arm is totally occluded, a good predicted distribution (with proper or enough diversity) should cover the pose as close to GT as possible, which would result in a better MPJPE in winner-takes-all evaluation scheme. If it is not overfitting, then the predicted distribution must have a good diversity, at least covering the one close to GT. If the training dataset is large enough, it can not overfit for the occluded parts. Maybe the authors could demonstrate such cases by visualization, which would be much more convincing. 

I would also like to listen to the advice of other reviewers and the authors about this good question.
Thanks for the authors' response. The response addresses some of my concerns. However, I disagree with the authors' statement that "The diversity in the predicted poses can be measured through Oracle MPJPE performance." MPJPE and diversity reflect two different aspects of the generated results: the accuracy of the best-matching pose and the ability to generate other plausible poses. If we only focus on MPJPE, it aligns more closely with the goal of single-hypothesis pose estimation rather than multi-hypothesis pose estimation.
The authors rebuttal has clarified my concerns carefully. I also notice that the authors add some experiments according to the comments of other reviewers, which makes the evaluation stronger. If the area chairs could guarantee the correctness of mathematical derivation, I think it would be a good choice to accept this paper.