3D Human Pose Estimation with Muscles
Authors: Kevin Zhu
AliAsghar MohammadiNasrabadi
Alexander Wong
John McPhee
Venue: NeurIPS 2025 poster ()
Keywords: 3D human pose estimation
physics-based computer vision
Primary area: applications

Abstract: We introduce MusclePose as an end-to-end learnable physics-infused 3D human pose estimator that incorporates muscle-dynamics modeling to infer human dynamics from monocular video. Current physics pose estimators aim to predict physically plausible poses by enforcing the underlying dynamics equations that govern motion. Since this is an underconstrained problem without force-annotated data, methods often estimate kinetics with external physics optimizers that may not be compatible with existing learning frameworks, or are too slow for real-time inference. While more recent methods use a regression-based approach to overcome these issues, the estimated kinetics can be seen as auxiliary predictions, and may not be physically plausible. To this end, we build on existing regression-based approaches, and aim to improve the biofidelity of kinetic inference with a multihypothesis approach --- by inferring joint torques via Lagrange’s equations and via muscle dynamics modeling with muscle torque generators. Furthermore, MusclePose predicts detailed human anthropometrics based on values from biomechanics studies, in contrast to existing physics pose estimators that construct their human models with shape primitives. We show that MusclePose is competitive with existing 3D pose estimators in positional accuracy, while also able to infer plausible human kinetics and muscle signals consistent with values from biomechanics studies, without requiring an external physics engine.

3D Human Pose Estimation with Muscles
Kevin Zhu AliAsghar MohammadiNassabadi University of Waterloo (k79zhu, aa27moha, a28wong, mcphee) Juvateroo.a
Alexander Wong John McPhee
Abstract
We introduce MusclePose as an end-to-end learnable physics-infused 3D human pose estimator that incorporates muscle-dynamics modeling infer human dy- namics from monocular video. Current physics pose estimators aim to predict physically plausible poses by enforcing the underlying dynamics equations that govern motion. Since this is an underconstrained problem without porce-annotated data, methods often estimate kinetics with external physies optimizers that may not be compatible with cxisting learning frameworks, or are too slow for real-time inference. While more recent methods use a regression -based approach over- come these issues, the estimated kinetics can be seen as auxiliary predictions, and may not be physically plausible. To this end, we build on existing regression- based approaches, and aim to improve the biofidelity of kinetic inference with a multihypothesis approach - by inferring joint torques via Lagrange's equations and via muscle dynamics modeling with muscle torque generators. Furthermore, MusclePose predicts detailed human anthropometrics based on values from biome- chanics studies, in contrast existing physics pose estimators that construct their human models with shape primitives. We show that MusclePose is competitive with existing 3D pose estimators in positional accuracy, while also able infer plausi- ble human kinetics and muscle signals consistent with values from biomechanics studies, without requiring external physics engine.
Introduction
3D human pose estimation (HPE) is fundamental task in computer vision that involves the localiza- tion of 3D human joints from images, which allows the user to track human movement from videos, leading a plethora of potential downstream applications. However, since many pose estimators are purely data-driven, the inferred motion is modeled implicitly, which may lead to physically impossible poses and movements.
Physics-basee based human pose estimation (PHPE) methods aim to mitigate these artifacts by enforcing the underlying dynamics equations that govern the kinematic state A. {q. q. ().
M. A) q + Clq. A) = Ta +
(I)
where q are generalized coordinates that describe motion, often in terms of translational (e.g. 3D position of the root) and rotational (e.g. joint rotations) degrees of freedom (DoF). We denote "dot" (") as the time derivative and "double dot" (") the 2nd time derivative of variable. MN is the mass matrix and contains the Coriolis, centrifugal, and gravitational forces, for a human with anthropometric features A at given state K.. Here, loosely lump together human's dimensions, mass and inertia properties, and other intrinsic and mobility features using the anthropometrics term A. On the right hand side, T, describes the human joint torques generated by each DoF, and external forces, both in the generalized space. In this paper, we deal with monocular pose estimation, where our only input source is a monocular video, without force sensors. When only one unknown external force & is applied on the human,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Transtarer enender
rertred kisernatics shapes XG,
Human estimation
estimation
Ainonatis
motion Teatures
MTG Groarsi derimiss))
Lagrange's dreeese synamiz))
SMPL
Kinernatesss essamauor
FMTG
Iiu signats
joint tarquas
positiona (S.P)
Decoder
iepuruition
Figure I: Overall framework of MusclePose.
we can solve for Ty by enforcing the entries of Tq that corresponds to the root link to be zero. However, when there are more than one external force applied to the human simultaneously at different locations, which is often the case (e.g. when both feet are in contact with the ground), Eq. (I) becomes underconstrained.
Without large-scale force-annotated video datasets, many methods estimate the corresponding kinetics via optimization with an external physics engine 20 19 41). However, these physics engines are often either cinfeifeernttaalee and cannot be trained end- enddto-end are too slow for real-time inference. Furthermore, as discussed in [85). these methods are often combined with reinforcement learning to reach a desired outcome, but the effects of changing inputs the outputs are unknown. Since joint torques can be hard to agree on the biomechanics community, as they are often computed from different models, with different assumptions and post-processing, more flexible learning framework may be preferred. More recently, PHPE methods have began regressing kinetics directly with neural networks [85] 377 [63] While the regression- baseb approach improves the kinematic reliability of the predicted motion, the inferred kinetics can seen as auxiliary predictions, which may not be directly constrained and may be physically implausible. Although these kinetic predictions the main focus these pose estimators, they may still be interest for downstream applications. In sports for example, in addition to kinematics, practitioners and researchers analyzing the whole-body musculoskeletal dynamics athletes. do so, a multibody model of skeletal dynamics commonly used in combination with optimal control algorithm generate predictive simulations of athlete movements [ 28 49). However, these optimal control algorithms
often interested in
take hours days produce results.
To this end, we build on existing regression-basee based PHPE approaches, to infer human kinetics simultaneously with kinematics, without physics engine, and propose MusclePose (Fig. D to improve the plausibility of the predicted kinetics. To mitigate the underconstrained problem of regressing kinetics,
multihypothesis approach, and compute torques via Lagrange' equations, and also via muscle dynamics modeling with muscle torque generators (MTGs) 25]
To maintain fidelity when modeling human movement, classical muscle models often represent muscles as linear actuators, and capture the nonlinear dependence muscle tension on muscle length and the rate of lengthening [66 322 using various Hill-type muscle models [2 However,
incorporating detailed muscles requires solving the actuator redundancy problem [3] and computing complex and varying musculoskeletal geometries [60 12]. To overcome these drawbacks, parametric MTG models were proposed to mimic the behavior muscles crossing a given joint to directly approximate joint torque by modeling kinematic dependence on active torque generation and passive impedance (Eq. (TT). Essentially, MTGs infer net joint torques from a joint's kinematics and activation levels, which is what ultimately want,
are not interested in isolated muscle tensions or granular joint contact forces. And since MTGs consist of differentiable equations, we are able to incorporate them into learning framework, and train our pose estimator end-to-end.
Moreover, for computational efficiency, existing PHPE methods rely on human models with anthropo- metrics estimated from the predicted human dimensions, use the intrinsics properties (e.g. inertia

---

and mass properties) of primitive shapes (e.g. spheres and simple rods), as proxies. From, Eq. ( we see that, even if the kinematics state K. and external forces are accurate, but A is not, the inferred torques Ty may not correspond to the actual human performing the motion. For example, existing pose estimators may infer the center of mass (CoM) of body parts by taking the mean of the predicted surface mesh, assuming constant density [85]. However, since the composition of bones, muscles, internal organs, etc. is different, the human body's s density is not uniform [[4]. For example, the CoM of the upper torso is slightly towards the left side [16]. whereas taking the mean vertices will be
in the center. As such, we further predict detailed anthropometrics for each human, and keep them
close to values taken from biomechanics studies.
In summary, we introduce MusclePose to comprehensively predict human kinematics, kinetics, muscle signals, and detailed anthropometries from monocular video. Specifically, we want a pose estimator with (i) a flexible learning framework easily adaptable for different scenarios, (ii) a reasonable degree of biofidelity, (iii) inference speed and (iv) positional accuracy both on par with purely kinematic pose estimators. To satisfy (i) and (iii), MusclePose is regression-based
consists of customizable and swappable components, can be trained end-to-end, and does not require an external physics engine. For (ii), MusclePose is the first pose estimator to incorporate muscle dynamics modeling and predict detailed human anthropometricss We demonstrate improvements in the inferred kinetics on actions including walking from the H36M dataset [27] and baseball pitching and golf swings from PennAction [84]. Also, the use of MTGs allows to further assess human motion at a musculoskeletal level, and we show that our inferred muscle signals are comparative to those from biomechanics studies, well as to EMG data of pertinent muscle groups. Lastly, for (iv), we evaluate our method on benchmark 3D HPE datasets, H36M (27) and 3DPWoc, 77], to show that MusclePose is kinematically competitive with sfate-of-the-ard (SOTA) pose estimators.
Examples of MTG curves for hip flexion. Tpassive models the passive torque (79] double exponential function. Tu models the active-torqu-aaggula--aao as a piecewise function. ty models the active-torque-angle rela- tionship 21 33) as the non-negative portion of a polynomial.
Figure 2:
relationship [69] 67)
2
Related work
Monocular 3D human pose estimation. Early deep learning 3D HPE approaches use convolutional neural networks directly estimate human 3D keypoint positions from images, with intermediate values represented by 3D heatmaps (57). location maps (50), or 2D heatmaps with depth regression [87]. The more recent and popular approach lifts 2D keypoints to 3D, essentially forming a monocular sparse-depth estimation task. The lifting network can be fully-connected layers [47), temporal
convolution networks [10) graph convolution networks 774 I or transformers [89 39 86).
Human pose and shape estimation (HPSE) refers to predicting 3D surface mesh of humans. The popular model- HPSE approach 38
35 31 predicts input parameters of parametric human model, such as SMPL, which infers 3D mesh from rotation and shape parameters. Non-parametric directly regress 3D coordinates of mesh vertices. Other methods combine both approaches, such as 770), which predicts a volumetric representation before fitting SMPL model, or 45,, which calibrates model-based mesh predictions with 3D keypoints.
approaches 42 52 36)
Physics based pose estimation. To achieve more physically plausible human motion, PHPE methods apply dynamic constraints encourage contact and penalize motion jitter, ground penetration, and unbalanced postures. [77 63 [1] 64) model contact forces between the foot and ground, [20] 19 [2]
include contact points between the full body and ground, while 4T also models human interaction
with stick-like hand tools. Optimization-based frameworks (20) 77 64 41 simulate physically
plausible human motion from a physies engine and minimize an objective function to keep the simulated motion close to the detections obtained from a kinematic pose estimator. These frameworks
are also combined with reinforcement learning 82 59
Recently, to overcome the need of an external directly estimate human kinetics using
physics engine, regression-based frameworks 85 37
63
neural networks. As discussed in Sec. [ this is an underconstrained problem, for which we hope to mitigate, and further inject biofidelity. We incorporate MTGs to do so.
3

---

Muscle torque generators (MTGs). Due to their simplicity, MTGs have been increasingly popular in multibody dynamics simulations as they reduce computational cost while maintaining reasonable degree of biofidelity. Recently, MTGs have been incorporated to simulate human movement post hip and knee replacement surgeries [33, human interactions with exoskeletons 22 26), and manual wheelchair propulsion [5]. For more dynamic movements, such as in sports, examples of MTG-driven simulations athlete motor control include golf [49] and cycling [28].
3 MusclePose
We propose MusclePose (Fig. D as physics-based pose estimator to directly regress comprchensive human dynamics from a monocular video of length T. We use a transformer encoder to refine initial pose estimates, and produce latent motion features (f2-T) which are used as inputs for customizable modules to infer human anthropometrics A, kinematics K., external forces F, joint torques via Lagrange's equations Ty. and joint torques via MTGs TWTG- We describe our prototype in the following subsections, where we extract muscle signals a (1:T) and residual terms E, 8(1:7) 3 from ((2:2') as inputs for the modules. All variables described in this section are sequences of length T, except E, 3 and shape parameters 3, and we drop the superscript ((I:T)..
Compared to the most recent regression- -aseed PHPE method, PhysPT [85, we do not use a trans- former decoder directly regress joint torques, and instead, regress the input parameters of MTG models. Our motivation stems from the popular HPSE approach that regresses SMPL parameters instead of the human mesh directly, which not only reduces the computation complexity but also geometrically constrains the predicted human, as SMPL infers the mesh via forward kinematics. In parallel, we use MTGs to avoid estimating complex musculoskeletal geometries and granular joint contact forces, while enforcing a constraint on the inferred torques from Lagrange's 's equations.
3.1 Kinematics estimation
We follow the common approach in PHPE and clean initial kinematic estimates (0, 3, T) generated by some existing kinematic pose estimator, to obtain the refined {0. B, T, c} as our prediction. Here, T E R represents 3D pelvis translation in the world frame, and are binary contact labels. Rotation parameters 0 = {fp..... 023} represents local rotations of the 24 SMPL keypoints, relative to their parents in the SMPL kinematic tree, with On being the pelvis orientation in the world frame.
We follow prior work 3 and predict the 6D continuous rotation representation [8] for each
O2 E RR. Shape parameter 3 E IR:00 denotes the first 10 principal components of SMPL's shape space. Since our inputs lack the shape information that RGB images provide, we follow the hybrid approach in [89] regress shape residuals that are combined with initial predictions. We also use the same approach and regress anthropometric residuals & later on in Sec. 3.2 and force residuals ô in
Sec.
The parametric human model, SMPL, then uses a collection of linear functions to map these parame- ters to a triangulated mesh V of 6890 vertices that represents the surface of the human body, and 24 SMPL keypoint positions P:
(V. P) SMPL.(0, + T
We define the kinematic loss Lusn with weights Agiin as
Lhin Asan [Lp L.
Lo
Lg Lnorm Le]
where the first five losses are from 88 which penalize joint position, linear velocity, SMPL parameter prediction Ll errors, and minimize the L2 norms of the SMPL parameters; and L. is the binary contact loss from [83
To facilitate multibody dynamics modeling in the following sections, we convert the predicted coordinates to generalized coordinates, q [Xo. qu, q1, qN IR No.F, where X, E R is the global root translation, and each qu describes the joint's rotational DoFs. Specifically, each z E Q: are ZXY euler angles converted from the predicted Oz, to match the International Society of Biomechanics (ISB) format, where joint's local z-direction corresponds to flexion/extension, for abcuuction/addduction and y for internal/external rotation. We denote the predicted kinematics as K. {q.6 q, 4) with the velocity and acceleration terms estimated via finite differences.

---

3.2 Human estimation
Human model. We assume a rigid multibody dynamics model of a human with N2 = 18 segments and NDoF = 47 total degrees of freedom (DoF). The 3D positions of the 18 joints, each corresponding to a segment, are the 24 SMPL. keypoint positions minus the 5 end-effectors and the spine3 SMPL. keypoint. The wrists, elbows, scapulas, each contain 2 rotational DoFs, knees each with rotational DoF, root with rotational and translational DoFs, and 3 rotational DoFs for each of the remaining joints, for a total of 47 DoFs. We selected this configuration as it aligns best with biomechanics studies with anthropometric measurements that we use in the remaining sections.
Anthropometrics prediction. To predict the human's 's anthropometries = Un{ma, Lo CoM& J. specifically the mass my, inertia tensor at zero rotation lo, and CoM of all segments, we scale literature values A from [16] based on the predicted human shapes 33 and add the predicted offsets E,
A 1-3A++2
(4)
The scaling term sy is computed from the predicted B. with details in the supplementary material.
3.3 Kinetics estimation
Ground reaction forces and moments (GRFM) prediction. Let = [F,, T be the GRFM applied on the CoM of each segment k. We infer J Σn Fi from our previous predictions and our regressed force residuals 6,
F GREM mX, AA
Since we trained our model on the AMASS dataset 44 and feet-ground contact labels from RoHM [83], assume feet-ground contact only for simplicity, with many PHPE methods 33 20 g2 Omitting subscript k; let F = [Fx, Fy, F2]T be the force in world cartesian coordinates where Y the vertical direction, and let coordinates where is along the length of the foot (i.e. M R Rankie2 z x F where Ra is the ankle's world orientation). From the regressed residuals ō(Y,1) C and the kinematics of each foot KJ foot we estimate the vertical force applied on the foot scaled by bodyweight FV = /V, and the CoP along the foot scaled by foot length 2 Zzz / foot.
[Za> Zy. ] be the center of pressure (CoP) in the foot's local
(FY
2, Foot 5(1))
where linear coefficients n were fitted on the forceplate data in (72.. The remaining ò terms are scaling factors between -1 and to ensure the values in the other directions are physically possible F2 S 87 F? and æ is within the foot's dimensions)
(i.e. F
+
Fx = 5x
Fz 82 52 62F9 F3
By = -1844l
2g = 8, (2w
(8)
where luc. In are the foot's width and height, respectively. Additional details of our GRFM model can be found in the supplementary material.
Inverse dynamics via Lagrange's. From here, we can analytically compute the mass matrix M, Coriolis term C, external forces in the generalized space 3. and infer joint torques in the generalized space T from the equations of motion:
Ty Lagrange's Lag Mtj C-
We include the calculations of these terms in the supplementary material. We define residual force loss frres to minimize the resulting forces and torques at the root, which correspond to the first 6 entries of T4
Lres
Tqp.o]
(10)
5

---

Inverse dynamics via MTGs. Simultaneously, we use parametric MTG models [ST] to infer joint torques TMTG from the predicted kinematics K and muscle activations a. Specifically, this kinematic dependence is separated into active torque generation Tactioe and passive impedance "passioe- For each joint rotational DoF q E [ with angular veloeity 4, let muscle signal a E [0. 1] represent the joint's corresponding activation level for this DoF, we compute the corresponding torque
TMTG MTG(K, A, a) = Tactive Tpassine
(11)
The active torque is further broken down into
Tachive (4) To(2) To (A.33)
(12)
where (4: 7w) models the the active-torque-angle relationship [21 as shown in Fig. 2 These relationships are parameterized by the y coefficients, which are unique for each joint's DoF and direction, and are identified via dynamometry. This joint-dependent parameterization preserves physiological realism (e.g., hip flexion and knee extension should exhibit different peak torque and passive stiffness profiles), unlike uniform torque models that assume identical properties across the body. For this paper, we use the set of y values summarized in [4 533.
relationship [69 67 and To (p 7e) models
(o(A Ve) is the peak isokinetic torque that controls peak MTG output zero joint velocity, which can be measured with a dynamometer. To is estimated in [54 [3 as a linear approximation of the human's 's intrinsic, scaled by certain external factors such the human's fitness activity level. Since these external factors (and some intrinsic properties) are not readily known, we take the mean effects Yi, Ye from [54]] 53 and add regressed offsets We compute To as
To = (7/A + (7) + 3x)
(13)
Furthermore, account for stability, assume each joint is driven by pair of agonist-antagonist MTGs - a flexor (+) and an extensor (-), that corresponds to the movement direction. Hence, for each joint rotational DoF, we regress 2 muscle signals Haz crst J. and the active torque becomes:
Tactive ftee - ffeer flez fler
10 TO
exst cart
TB T
(14)
Tpassive (4: 7p) is the passive torque [I of a joint that arises when the surrounding muscles, tendons, and ligaments are strained and intensifies near anatomical joint limits [1 81]. The joint's viscous damping and nonlinear stiffness are parameterized by Yp. which encourages the joint move within its range of motion, to compute Tw: Tg, Tpaasive can be found in the supplementary material.
large restoring torque is produced otherwise, shown in Fig. 2 Equations
We define the torque loss E, as the absolute difference between the two sets of predicted joint torques, and another regularizing term Le for all regressed residuals:
L TT TMTGl
(15)
C. = ||8112 + 110112 + ||3||2
(16)
Finally, we have dynamic loss with weights Adym
Ldyn Adyn [or fres Ce]
(17)
4 Experiments
4.1
Implementation and datasets
For training, we used the AMASS dataset [46], with feet-ground contact labels from 83 As such, we trained and evaluated on sequences with feet-ground contact only (denoted i), which also the case for many PHPE experiments [37 20, 82). We trained MusclePose end-to-end with a sequence input length of 16 frames, using total loss Liotal = Lkin + Layn for 25 epochs, using the Adam W optimizer [44 with weight decay of 10-4 and an initial learning rate of 10-4 that decreases by 20% every 5 epochs. Following common curriculum learning (2 practices, we split the training into two phases - for the first 20 epochs, we trained using the ground truth as input, followed by epochs using the model's predictions inputs.

---

For evaluation, we assessed positional accuracy on the inference results of the H36M test set 127 and object-occlusion subset 3DPW (3DPWoc) 77.. As with training, we removed input sequences containing non feet-groung contact, the sitting and sitting down actions in H36M, and courtyard laceshoe, flat guitar, outdoors climbing, outdoors freestyle, outdoors parcours, downtown stairs in 3DPWoc. We further assessed kinetic biofidelity from 3 actions walking from H36M, and baseball pitch and golf swing from the PennAction dataset (PA) [84). As with existing large-scale human video datasets, since neither datasets include force-annotations, we compared our inference results with existing biomechanics studies these movements, and commented on overall trends and plausibility, We selected walking because human gait is heavily studied in biomechanics [72 18 76 29 75).
and is a relatively consistent and cyclic movement. We included the latter two actions to evaluate During inference, to promote a closer comparison with the SOTA regression- based physics pose estimator PhysPT [85], used the same kinematic estimator, CLIFF 40), to extract initial kinematic estimates, and the global trajectory predictor in [85] to extract initial root DoFs. The rationale for using CLIFF
faster movements, for which we were able to find published lab measurements [55 80
in (85) is that it produces competitive positional accuracy but lacks in physical plausibility.
4.2 Positional accuracy
We followed standard evaluation protocol and reported the mean per-joint positional error (MJE) and procrustes-alignes MJE (PJE) in Tab. for the 14 LSP [30] keypointss in millimetres. MJE is the root-aligned mean Euclidean distance in millimeters between the predicted and ground truth 3D keypoints. PJE is the MJE after aligning the predicted pose with the ground truth in translation, rotation, and scale using the Procrustes method. For 3DPWoc, since the data is captured using a moving camera with unknown extrinsics, and our method predicts the global root DoFs directly, reported PJE only, with additional ablations in Tab. to show consistency of results.
We see that MusclePose outperforms other PHPE methods on H36M but is slightly worse than PhysPT on 3DP This, along with the overall worse positional accuracy of PHPE compared to purely kinematic methods, could be due to a kinematics-kinetis trade-off, as our method produced lower residual force (Tab. 4.. Specifically, the root DoFs in 3DPW may be harder to estimate due to the moving camera, Icading to higher residual forces, which the model may try to reduce (lower kinetic error) by estimating a set of local joint kinematics slightly different from the original motion (higher kinematic error).
H36M
3DPWoc
Kinetics
MJEL
PJR.
PIE.
Table 1: Positional accuracy H36M and
kinematic
HybrlK
38
55.4
33.6
3DPWoc. "'ptt"" denotes kinetics obtained from an external physics optimizer, and "regr" de- notes regressed by a neural network. denotes sequences with feet-ground contact only.
HybrIK
156.4
136.7
CLIFF
52.2
36.8
CLIFF
146.5
132.4
'24.0
SimPoE 82
opti.
156.7
141.6
PHPP
DiffPhy 20
opti.
181.7
155.6
D&D 37
regr.
152.5
135.5
PhysIT 85
regr.
150.6
135.5
125.9
MusclePosetours))
regr.
148.4
133.5
127.6
4.3 Biofidelity
Since most PHPE methods do not report kinetics results, we mainly compared ours with values we reproduced from PhySPT. For even comparison, for both pose estimators, we computed joint torques in the generalized space Tg via Lagrange's equations (9) from the predicted motion, anthropometries. and GRFM. Unlike the biomechanics studies, we did not apply additional signal post- post-processing or smoothing to Ty. Hence, we see more noise and spikes the pose estimators' results, which could also be amplified by the low frame rate of PennAction.
Qualitative. Since the different biomechanics studies computed torques differently from different datasets, we comment on general trends and evaluate qualitatively. In Fig. 3 we plotted median torques scaled by predicted body weight, with 25-75% quantile band, for select joints the actions, for which we found reference values. Overall, compared to PhysPT (gray), we see that MusclePose (ours, purple) more closely follows the trends and magnitudes of the reference values (greens and yellow).
7

---

Macifoe 16.07
Maskiuu
14.0
446
300
0.0
0.6
141
1981
081
7.4
44
1.4
11
33
30 Lays 3756
0945
2748
Figure 3: Median predicted joint torques scaled by bodyweight, with 25-75% quantile band, for gait cycles, the downswing phase of golf drives, and the arm acceleration phase of baseball pitches, compared to values from biomechanies studies.
The 2 leftmost columns of Fig. correspond to flexion/extension of the hip, knee, and ankle torques for walking, scaled such that the toe-off occurs at 60% of the gait cycle, and compared to reference torques from 773 18 75
We see that MusclePose produced more reasonable trends overall, whereas PhysPT produced torques with low magnitudes but with higher extreme values. For hip ficxion, our results resembled more of the yellow curve that computed from wearable system in (73), where the authors attributed their errors to a lack of shear force measurement, leading to more noticeable errors in the hips than more distal joints due to the increase in moment arms. Since the subjects in H36M walk in a small circle with slower and varying speeds, a discrepancy can arise the generated shear force, leading the discrepancy in hip torque, as the subjects in the biomechanics studies walk in straight line at a consistent pace. This could also contribute to the low magnitude of ankle torque, where different timings hho--offhhee--offoouc---oooo could affect ankle power generation, as explained in (6].
The middle 2 columns of Fig. correspond to lumbar torques during the downswing phase of golf drives, scaled such that the maximum lumbar rotation occurs at 2/3 of the motion, and compared to reference torques from (55) While we see noticeable spikes from both pose estimators, the extreme values for lumbar lateral bending and axial rotation were much higher for PhysPT.
The 2 rightmost columns of Fig. correspond to shoulder lateral/medial rotation torque during the arm acceleration phase of baseball pitches, scaled such that the maximum moment occurs at 80% of the motion, and compared to reference torques from professional pitchers (darker green) in [62), as well as amateurs (lighter green) in [80]. Although the peak of our 75% quantile slightly exceeds the shoulder medial rotation limit reported in (62), our band was able to cover values from both skill groups, whereas PhysPT's band was below the amateurs, even though the pitchers in PA range from teenage amateurs to adult professionals.
Quantitative. In the bottom right of Fig. 3 we reported the mean residual forces (Fres: Tres mean out of range joint torques TooR, and the median value of the sum of GRFs in the direction opposite of gravity GRF, Residual Fres and Tres were computed entries of that correspond to the translational and rotational Dol's of the root. TOOR was computed as the mean absolute amount outside of joint torque limits (red and blue values in Fig. 3 reported in 43 62 These values are further scaled by predicted body weight and denoted (/W)) In the last column, also reported the mean predicted segment mass m /M percentage of body mass M.
the mean L2 norms of the
Overall, we see that MusclePose inferred more reasonable kinetics, indicated by the lower residual forces, less extreme joint torques, and median GRF, closer to body weight. Similar to its joint torques, PhysPT's GRF, values were overall lower magnitude, with occasional spikes. While MusclePose's is maximum GRF, were very large for the golf swing and baseball pitch, our 99% quantiles were more comparable to the maximum values of about 1.3 times body weight for golf reported in [55], and about 2.3 for pitching in 19.
8

---

Figure 4:
Mean predicted musele activations (dashed) for gait cycles compared to EMG data and predictions of pertinent muscles from literature. Min-max scaling applied to all values.
4.4 Ablations and additional evaluation
We reported ablations results in Tab. [2 Row includes results without custom anthropometrics, and instead computed directly from the SMPL mesh, assuming constant density. Row 3 includes results without MTGs. Row includes results using kinematic-based muscle activations, with details the next paragraph. We see that, overall, MusclePose has better positional accuracy, along with lower residual forces (Fres (Fres + /2), and median GRF, closest to bodyweight.
Table Ablations results.
H336
33DWoc F/W
H36M Walk med GRF
Golf swing
PA Pitch med GRF,W
MJEL
F/W
PJE!
F/W
F/W 4
GRF/"
F/W
MusclePose
48.4
0.08
27.6
0.25
0.08
0.98
0.56
0.98
0.99
A
49.5
0.26
27.9
0.35
0.22
0.74
0.61
0.64
0.67
0.39
w/o with á
49.7
0.14
27.2
0.30
0.15
0.94
0.66
0.88
0.74
0.90
50.9
0.16
28.4
0.26
0.13
0.93
0.59
0.97
0.69
0.90
Furthermore, due to the lack of extrinsics information in 3DPWoc, we reported results from using different kinematic estimators in Tab. B to show consistency.
Table 3: 13DPWoc results with different kinematic estimators.
Ours4CLIFF
CLIFF
40
OHAssWHAM
WHAM
Ouss+CoMotion
CoMotion
PIEL F
27.6
24.0
28.5
22,7
32.6
30,7
0.3
not able to find public video datasets with corresponding MTG activation signals to directly compare with, we followed the evaluation procedure in [29] to assess general trends, and overlayed our mean muscle activation predictions (black. dashed) for gait cycles from H36M with EMG data of pertinent muscles from other gait studies [72] 76 g 29] in rows and 3 of Fig. with min-max scaling applied to all values. We also included the predicted activations (dotted) from [29: however, they use different muscle model. While muscle activations can be seen as surrogate representations EMGs, the two not exactly the same. Hence, mismatches in timing and magnitude will exist, and peaks and valleys may be further amplified by the min-max scaling applied. In general, raw EMG values can vary widely due to electrode placement.
Muscle activations. Since
To mimic methods that regress joint torques as linear combination of joint kinematics, we experi- "kinematic effort term" (denoted â). specifically as
mented with estimating the musele activation a joint's 's angular velocity relative to its limit plus an additionally regressed offset term:
ad -d= q² / + 3d
d {flez, ext)
(18)
We plotted á results (gray, dashed) in rows and of Fig. In comparison, MusclePose (rows and 3) seems to better follow literature trends overall, such as having more noticeable hitch (or second peak) for hip adduction, externa rotation, knee flexion, etc. Furthermore, for the á case, ankle

---

plantarflexion seems to be deactivated during the middle of the gait cycle, when it should peak. Row 4 of Tab. 2 also shows that MuselePose quantitatively outperforms the case.
Kinematic plausibility. In addition to the joint positional errors in Sec. metrics such as acceleration loss (ACC), foot skating (FS), and ground penetration (GP) were introduced further evaluate kinematic plausibility. We computed these values for H36M and 3DPWoc in Tab, 4 where ACC is the mean L2 norm in mm/frame" between the predicted and ground truth keypoint accelerations access jitter. We also included mean torque variation (MTV) as the mean absolute change in joint torques over consecutive frames (in dewton=metres/framme FS is the average displacement in mm of vertices in contact with the ground in consecutive frames. GP is the average vertical distance to the ground in mm of vertices below the ground.
to assess torque continuity,
Table 4: Plausibility metrics. P indicates Procrustes aligned. (%f) indicates % of frames.
Kinematic plausibility
Float (%e)
Kinetic plausibility
GRF
GRP (%f)
MJR
ACC|
GP
Hums > (1,10, 10,20Jmm
F
MTV
[mod
q99
max]
10.01, 0.1, 0.5)
13DDWWc 1H30d
CLIFF
46.5
26.3
PhysPT
50.6
13.7
34.7
6.8
[59.0
31.6
8.5]
0.4
5.3
10.4
2.4
10.0)
[7.2
20.8
60.0)
MusclePose
48.4
12,9
37.2
26.0
[8.0
3,0
1.3]
0.1
2.5
[1.0
1.2
3.0]
13.0
3.0
5.2)
CLIFF
24.0"
13.8
PhysPT
25.9P
3.0"
7.8
11.2
[82.9
73.9
57.2]
0.9
27.0
10.5
1.2
3.91
15.3
11.8
52.4)
MusclePose
27.6"
4.3
12.8
30.8
16.0
4.7
3.7)
12.1
11.0
1.6
4.3)
15.3
5,3
6.3)
While we see an improvement in jitter from both physics pose estimators, as indicated by lower ACC compared to CLIFF, lower FS or GP may not be strictly better. For one, foot sliding may occur naturally. And in terms of the latter, while human bodies deform under pressure and contact, the SMPL mesh does not model this deformation, and will instead penetrate the object is in contact with (68). During walking for example, minimizing GP while assuming this rigidity may restrict natural ankle rotation, potentially leading to the smaller ankle torques compared to literature values in Fig. 3 On the other hand, we should also check for floating. We reported the percentage of frames (%f) when the minimum vertex height Mmin is above certain thresholds (1, 10, 20mm). Since removed the con-feet-grount contact sequences, there are very minimal frames where "floating" occurs. We that while PhysPT has less GP, it also includes more floating.
Ground reaction force. We can also characterize floating as when GRF, is small. As such, we also reported the percentage of frames when GRF, is below certain thresholds (1%, 10%, 50% of body weight) in Tab, 4 The results are consistent with Hmins both indicating less floating (lower %f) for MuselePose. In Fig. 5 we plotted the predicted median vertical GRF of foot, divided by body weight, for gait cycles in H36M. Compared to PhysPT (gray), we see that MusclePose's predictions (purple) closer to literature values (greens) from 18. 75
Figure 5: Median vertical GRF divided by body weight, of a foot for gait cycles in H36M, with 25-75% quantile band.
5 Conclusion
In conclusion, we introduced MusclePose as the first PHPE method to simultaneously predict human kinematics, kinetics, muscle signals, and detailed anthropometrics from monocular video. In Sec. 4.3 a.d[4.4] we showed how the additions of muscle-dynamic modeling and detailed anthropometrics predictions improve the kinetic plausibility regression-bass based PHPE, while being competitive with purely-kinemal pose estimators in positional accuracy in Sec. 4.2 Our framework consists of customizable components, does not require an external physies engine, and can be trained end-to-end.
We acknowledge financial support from the Canada Research Chairs Program, Canadian Sports Institute Ontario, and Mitacs grant.
Acknowledgements.
10

---

References
[1] D. E. Anderson, M. L. Madigan, and M. A. Nussbaum. Maximum voluntary joint torque function of joint angle and angular velocity: Model development and application lower limb. Journal of Biomechunics,
2007.
[2] Y. Bengio, J. Louradour, R. Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, page 41 48, 2009.
[3] N. A. Bernshtein. The co-ordination and regulation of movements. 1967.
[4] C. Brown, W. McNally, and J. McPhee. Optimal control of joint torques using direct collocation to maximize ball carry distance in golf swing. Multibody System Dynamics, 50(3), 2020.
[5] C. Brown and J. J. McPhee. Predictive forward dynamic simulation of manual wheelchair propulsion on rolling dynamometer. Journal of biomechanical engineering. 2020.
[6] A. Buchmann, S. Wenzler, L. Welte, and D. Renjewski. The effect of including mobile arch, toe joint, and joint coupling on predictive neuromuscular simulations of human walking. Scientific Reports, 2024.
[7] Y. Cai, L. Ge, J. Liu. J. Cai, T. J. Cham, J. Yuan, and N. M. Thalmann. Exploiting spatial-tempon relationships for 3D pose estimation via graph convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, volume 2019-October, 2019.
[8] T. Castermans, M. Duvinage, G. Cheron, and T. Dutoit. Towards effective non-invasive brain compuer interfaces dedicated to gait rehabilitation systems. Brain Sciences, 4(1):1 2014.
S.-W. Chen, W.-T. Tang, J.-T. Kung, T.-Y. Hung, W.-H. Lin, Y.-... Chen, and D. Burgee. Comparison of ground reaction force among stride types in baschall pitching. Sports Biomechanics, 0(0):-114, 2024.
[10] Y. Cheng, B. Yang, B. Wang, and R. T. Tan, 3D human pose estimation using tempool networks with explicit occlusion training. In AAAI 2020 - 34th AAAI Conference on Artificial Intelligence, 2020.
[11] H. Ci, C. Wang, X. Ma, and Y. Wang. Optimizing network structure for 3D human pose estimation. In Proceedings of the IEFE International Conference Computer Vision, volume 201 9019-October, 2019.
[12] D. J. Cleather and A. M. J. Bull. Lower-cxtremity musculoskelctal geometry affects the calculation of patellofemoral forces in vertical jumping and weightlfttigg Proceedings of the Institation of Mechanical Engineers, Part H: Journal of Engineering in Medicine, 224: 1083, 2010.
[13] B. Danaci and J. McPhee. Model-based acetabular cup orientation optimization based on minimizing the risk of edge-loading and implant impingement following total hip arthroplasty. Journal of biomechanical engineering, 2022.
[14] R. Drillis, R. Contini, and M. Bluestein. Body segment parameters. Artificial limbs, 8(1):44666, 1964.
[15] G. A. Dudley, R. T. Harris, M. R. Duvoisin, B. M. Hather, and P. Buchanan. Effect of voluntary artificial activation the relationship of muscle torque speed. Journal of Applied Physiology, 69(6), 1990.
[16] R. Dumas, L. Chèze, and J. P. Verriest. Adjustments meconville et al. and young al. body segment inertial parameters. Journal of biomechanics, 40 3:543-53, 2007.
[17] R. Featherstone. Rigid Body Dynamics Algorithms. 2008.
[18] C. A. Fukuchi, R. Fukuchi, and M. Duarte. A public dataset of overground and treadmill walking kinematics and kinetics in healthy individuals. PeerJ, 6, 2018.
[19] E. Gartner, M. Andriluka, H. Xu, and C. Sminchisescu. Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Video. In Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, volume 2022-June 2022.
[20] E. Gärtner, M. Andriluka, E. Coumans, and C. Sminchisescu. Differentiable dynamics for articulated 3d human motion reconstruction. In Proceedings of the IEEE Computer Suciety Conference on Computer Vision and Pattern Recognition, 2022.
[21] D. Haering, C. Pontonnier, N. Bideau, G. Nicolas, and G. Dumont. Using Torque-Ane and Torque- Velocity Models to Characterize Elbow Mechanical Function: Modeling and Applied Aspects, Journal of Biomechanical Engineering, 141(8), 2019.

---

[22] N. Haraguchi, A. Nast, K. A. Inkol, K. Hase, and J. McPhee. Human and passive lower-limb exoskeleton interaction analysis: Computational study with dynamics simulation using nonlinear model predictive control. 2023 62nd Annual Conference of the Society of Instrument and Control Engineers (SICE), pages 844-849, 2023.
[23] A. Hill. The heat shortening and the dynamic constants of muscle. Proceedings of The Royal Society B: Biological Sciences, 126: 136-195, 1938.
[24] P. D. Hoang, R. B. Gorman, G. Todd, S. C. Gandevia, and R. D. Herbert, A new method for measuring passive length-tenset properties of human gastroenemius musele in vivo. Journal of Biomechanics, 38(6):1333 1341, 2005.
[25] K. A. Inkol, C. Brown, W. McNally, C. Jansen, and J. McPhee. Muscle torque generatorss in multibody dynamic simulations of optimal sports performance. Multibody System Dynamics, 50(4), 2020.
[26] K. A. Inkol and J. J. McPhee. Using dynamic simulations estimate the feasible stability region of -ee-cce balance recovery for lower-limb exoskeleton users. 2022 9th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), pages 1 1-6, 2022.
[27] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3.6M, Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 36(7), 2014.
[28] C. Jansen and J. J. McPhee. Predictive dynamic simulation of olympic track cycling standing start using direct collocation optimal control. Multibody System Dynamics, 49:53-70, 2020.
[29] C. T. John, F. C. Anderson, J. S. Higginson, and S. L. D. and. Stabilisation of walking by intrinsic muscle properties revealed and Biomedical Engineering, 16(4):451 46211146622200 2013.
diree-dimeniion muscle-driven simulation. Computer Methods in Biomechanics
[30] S. Johnson and M. Everingham. Clustered pose and nonlincar appearance models for human pose estimation. In British Machine Vision Conference, 2010.
[31] A. Kanazawa, M. Black, D. W. Jacobs, and J. Malik. End-to-End Recovery of Human Shape and Pose. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2018.
[32] B. Katz. The relation between force and speed in muscular contraction. The Journal of Physiology, 96, 1939.
[33] M. A. King, C. Wilson, and M. R. Yeadon. Evaluation of a torque-driven model of jumping for height. Journal of Applied Biomechanics, 22(4), 2006.
[34] M. Kocabas, N. Athanasiou, and M. J. Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2020.
[35] N. Kolotouros, Pavlakos, M. Black, and K. Daniilidis. Learning to reconstruct 3d human pose and shape via model-fittig in the loop. In Proceedings of the International Conference on Computer Vision, 2019.
[36] N. Kolotouros, G. Pavlakos, and K. Daniilidis. Convolutional mesh regression for single-image human shape reconstruction. 2019 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 4496 4505, 2019.
[37] J. L., S. Bian, C. Xu, G. Liu, G. Yu, and C. Lu. D &D: Learning Human Dynamics from Dynamic Camera. In ECCV, 2022.
[38] J. Li, C. Xu, Z. Chen, S. Bian, L. Yang, and C. Lu. Hybrik: A hybrid analytical-neural inverse kinematics solution for 3d human pose and shape estimation. Proceedings of the IEÉE Computer Society Conference on Computer Vision and Pattern Recognition, 2021.
[39] W. Li, H. Liu, H. Tang, Wang, and L. Van Gool. MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2022-June, 2022.
[40] Z. Li, J. Liu, Z. Zhang, S. Xu, and Y. Yan. Cliff: Carrying location information in full frames into human pose and shape estimation. In European Conference Computer Vision, 2022.
12

---

[41] Z. Li, J. Sedlar, J. Carpentier. I. Laptev, N. Mansard. and J. Sivic. Estimating 3D motion and forces of person-object interactions from monocular video. In Proceedings of the IEEE Computer Society Conference on Computer Vixion and Puttern Recognition, volume 2019-June, 2019.
[42] K. Lin, L.. Wang, and Z. Liu. Mesh graphormer. 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pages 12919-12928, 2021.
[43] D. M. Lindsay and J. F. Horton. Trunk rotation strength and endurance in healthy normals and elite male golfers with and without low back pain. North American journal of sports physical therapy: NAJSPT, 1 2:80-9, 2006.
[44] Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.
[45] T. Luan, Y. Wang, J. Zhang. Z. Wang, Z. Zhou, and Y. Qiao. Pe-hmr: Pose calibration for 3d human mesh recovery from 2d images/videos. In AAAI Conference Artificial Intelligence, 2021.
[46] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. Black. AMASS: Archive of motion capture surface shapes. In Proceedings of the IEEE International Conference Computer Vision, volume 2019-Octoher, 2019.
[47] J. Martinez, R. Hossain, J. Romero, and J. Little. Simple Yet Effective Baseline for 3d Human Pose Estimation. In Proceedings of the IEEE International Conference on Computer Vision, volume 2017-October, 2017.
[48] W. McNally and J. McPhee. Dynamic Optimization of the Golf Swing Using Six Degrce-of- Freeccom Biomechanical Model. 2018.
[49] McNally and J. J. McPhee. Dynamic optimization of the golf swing using a six degree-of-freedon biomechanical model. 2018.
[50] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotnychenko, W. Xu, and C. Theobalt. Monocular 3D human pose estimation in the wild using improved CNN supervision. In Proceedings 2017 International Conference on 3D Vision, 3DV 2017, 2018
[51] M. Millard, T. K. Uchida, A. Seth, and S. L.. Delp. Flexing computational muscle: modeling and simulation of musculotendon dynamics. Journal of biomechanical engineering, 135 2:021005, 2013.
[52] G. Moon and K. M. Lec. 121-meshnet: Image-to-lixel prediction network for accurate human pose and mesh estimation from single rgb image. ArXiv, abs/2008 0os/2008.03713, 2020.
[53] A. Nasr, A. Hashemi, and J. McPhee. Scalable musculoskeletal model for dynamic simulations of upper body movement. Computer Methods in Biomechanics and Biomedical Engineering, 2023.
[54] A. Nasr and J. McPhee. Scalable musculoskeletal model for dynamic simulations of lower body movement. Computer methods biomechanics and biomedical engineering, pages 1-27, 2024.
[55] S. Nesbit. Development of a full-body biomechanical model of the golf swing. International Journal of Modelling and Simulation, 27(4):392 404, 2007.
[56] A. Newell, P. Hu, L. Lipson, S. R. Richter, and V. Koltun. Comotion: Concurrent multi-person 3d motion. In ICLR, 2025.
[S7] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to-fine volumetric prediction for single-image human pose. In Proceedings 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, volume 2017 2017-January, 2017.
[58] D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli. 3D human pose estimation in video with temporal convolutions and semi-supervised training. In Proceedings of the IEFE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2019-June, 2019.
[59] X. Peng, Abbeel, S. Levine, and M. Van De Panne. DeepMimie: shample-guides deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics, (7)),, 2018
[60] E. D. Pieri, M. Lund, A. Gopalakrishnan, K. P. Rasmussen, D. E. Lunn, and S. J. Ferguson. Refining muscle geometry and wrapping in the tlem 2 model for improved hip contact force prediction. PLOS ONE, 13, 2018
13

---

[61] D. Rempe. L. J. Guibas. A. Hertzmann, B. Russell, R. Viillegas, and J. Yang. Contact and Human Dynamics from Monocular Video. In 19th ACM SIGGRAPHI Eurographics Symposium on Computer Animation 2020, SCA 2020- Showcases, 2020.
[62] M. Sabick, M. Torry, Y.-K. Kim, and R. Hawkins. Humeral torque in professional basehall pitchers. The American journal of sports medicine, 32:892-8, 2004.
[63] S. Shimada, V. Golyanik, W. Xu, P. Pérez, and C. Theobalt. Neural monocular 3D human motion capture with physical awareness, ACM Transactions on Graphics, 40(4), 2021.
[64] S. Shimada, V. Golyamik, W. Xu, and C. Theobalt. Phys Cap. ACM Transactions on Graphics, 39(6), 2020.
[65] S. Shin, J. Kim, E. Halilaj, and M. J. Black. Wham: Reconstructing world-grounder humans with accurate motion, CVPR, 2024.
[66] T. Siebert, C. Rode, W. Herzog, O. Till, and R. Blickhan. Nonlinearities make difference: comparison of two common hill-type models with real muscle. Biological Cybernetics, 98:133-143, 2008
[67] E. J. Sprigings. Simulation of the force enhancement phenomenon in muscle. Computers Biology and Medicine, 16(6), 1986.
[68] S. Tripathi, L. Müller, C.-H. P. Huang, O. Taheri, M. J. Black, and D. Tzionas. 3d human pose estimation via intuitive physics. ArXiv, abs/2303. 18246, 2023.
[69] A. van Soest and M. Bobbert. The contribution muscle properties in the control of explosive movements. Riological Cybernetics, 69(3), 1993.
[70] G. Varol, D. Ceylan, B. C. Russell, J. Yang. Yumer, I. Laptev, and Schmid. Bodynet: Volumetric inference of 3d human body shapes. ArXiv, abs/ (bs//1844004875 2018.
[71] T. von Marcard, R. Henschel, M. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate human pose in the wild using imus and a moving camera. In European Conference on Computer Vision (ECCV), sep 2018.
[72] H. Wang, A. Basu, G. Durandau, and M. Sartori. Comprehensive Kinetic and EMG Dataset of Daily Locomotion with 6 types of Sensors, May 2022.
[73] H. Wang, A. Basu, G. Durandau, and M. Sartori. A wearable real-time kinetic measurement sensor setup for human locomotion. Wearable Technologies, Feb. 2023.
[74] J. Wang, S. Yan, Y. Xiong, and D. Lin. Motion Guided 3D Pose Estimation from Videos. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Rioinformatics), volume 12358 LNCS, 2020.
[75] D. A. Winter. Biomechanics and motor control of human movement. John Wiley & Sons, Hoboken, NJ, USA, 4th edition, 2009.
[76] A. R. Wu, F. Dzeladini, T. J. H. Brug, F. Tamburella, N. L. Tagliamonte, E. H. F. Asseldonk, H. van der Kooij, and A. ljspeert. An adaptive neuromuscular controller for assistive lower-limb exoskcletons: preliminary study on subjects with spinal cord injury, Frontiers in Neurorobotics, Volume 11 2017, 2017.
[77] K. Xie, T. Wang, U. lqbal, Y. Guo. S. Fidler, and F. Shkurti. Physics-bass based Human Motion Estimation and Synthesis from Videos. In Proceedings of the IEEE International Conference on Computer Vision, 2021.
[78] G. Yamaguchi. Dynamic Modeling of Musculoskeletal Motion. 2001.
[79] G. T. Yamaguchi. Dynamic modeling of musculoskeletal motion: vectorized approach for biomechanical analysis in three dimensions. Springer, Boston, MA, USA, edition, 2006.
[80] K. Yoichi, Sato, and T. Yamaji. Biomechanical analysis of the pitching characteristics of adult amateur bascball pitchers throwing standard and lightweight balls. Journal of Physical Therapy Science, 32:816- 822, 12 2020.
[81] Y.S. Yoon and J. M. Mansour, The passive clastic moment at the hip. Journal of Biomechanics, 15(12)9005- 910, 1982.
[82] Y. Yuan, S.-E. Wei, T. Simon, K. Kitani, and J. Saragih. Simpoe: Simulated character control for 3d human pose estimation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2021.
14

---

[83] S. Zhang, B. L. Bhatnagar, Y. Xu. A. Winkler, P. Kadlecek, S. Tang, and F. Bogo. Rohm: Robust human motion reconstruction via diffusion. In CVPR, 2024.
[84] W. Zhang, M. Zhu, and K. G. Derpanis. From actemes to action: A strongly-supervised representation for detailed action understanding. In 2013 IEEE International Conference Computer Vision, pages 2248- 2255, 2013.
[85] Y. Zhang, J. O. Kephart, Z. Cui, and Ji. Physpt: Physics-aware pretrained transformer for estimating human dynamics from monocular videos. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
June 2024.
[86] C. Zheng, S. Zhu, M. Mendieta, Yang. C. Chen, and Z. Ding. Human Pose Estimation with Spatial and Temporal Transformers. In Proceedings of the IEEE International Conference on Computer Vision, 2021.
[87] X. Zhou, Q. Huang, X. Sun, X. Xue, and Y. Wei. Towards 3D Human Pose Estimation in the Wild: A Seukly-Supervieed Approach, In Proceedings of the IEEE International Conference Computer Vision, volume 2017-Octoher, 2017.
[88] Y. Zhou, C. Barnss, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. In Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, 2019.
[89] W. Zhu, X. Ma, Z. Liu, L. Liu, W. Wu, and Y. Wang. Learning human motion representations: unified perspective. In Proceedings of the International Conference Computer Vision, 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: our contributions summarized in the last paragraph of Sec. I with corre- sponding results in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
limitations are discussed in the supplementary material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
all formulas used can be found in Sec. with details in the supplementary
material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: all models implemented can be found in Sec. 3 with details, such as model
coefficients hyperparameters, listed in the supplementary material. Experiment implemen- tation information, for both training and evaluation, can be found in Sec. 4.1 with additional details the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: while code is not yet ready for open source, we included in the supplemen- tary material, all necessary details required to reproduce our model and results, including implementation details of all novel components, well as all the publicly available code repositories we borrowed from to implement and train our model, and to produce and evaluate our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips.com httpsscccpbbbc/////eeeee// tttp://wwiee/eeeeeeee /
wwwisiiiccciiiiicccooocooi for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: training and test information can be found in Sec. 4.1 with additional details in the supplementary material.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
we include quantile bands in Fig.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: computing information is included in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wrrp...ccoooo..../hcccaa
Answer:
[Yes]
Justification: the research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: societal impacts discussed in the supplementary material.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: assets used in this paper cited in the main text, and the license and terms are respected and mentioned in the supplementary material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode cowc datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: the paper introduces MusclePose novel human pose estimation framework, with details on implementation and application.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: this paper does not involve crowdsourcing, and uses human data from publicly available datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: this paper does not involve crowdsourcing, and uses human data from publicly available datasets.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used for research development nor any part of this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

Supplementary material for 3D Human Pose Estimation with Muscles
A Technical appendix
A.1 Human model
We assume a rigid multibody dynamics model of a human with N₁ = 18 joints - pelvis, lumbar joint, thoracic joint, neck, scapulas, shoulders, elbows, wrists, hips, knees, and ankles. The pelvis is set as the root, with 3 rotational and 3 translational degrees of freedom (DoFs). The scapulas have 2 DoFs, corresponding to depression/clevation and protractiom/reeractioon. The elbows have DoFs, corresponding to flexion/extension and forearm pronation/supination. The wrists have 2 DoFs, corresponding to flexion/extension and ulnar/radial deviation. The knees have DoF, corresponding to flexion/ectension. All remaining joints have 3 Dolss for a total of 47 Dolss We selected this configuration as it aligns best with existing biomechanics models that we implemented, such as for anthropometrics estimation [16] and MTGs [5].
Anthropometries estimation. We predict the human's 's anthropometries by combining our regressed residuals& with initial estimates based on literature values A scaled by our predicted human dimen- sions. Specifically, we want to predict A = Ls {ma; Lo, ( X where my is the mass of segment k, with fo.e as its inertia tensor at zero rotation with scaling matrix Ag, and X as its local CoM position relative to its segment length. For the remainder of the section, we assume relevant units to be in seconds, radians, meters, kilograms, Newtons.
From the predicted Go and Eq. (), we can compute the human's volume and all segment lengths La- We further compute the human's 's initial bodymass estimate M its volume multiplied by constant density of 985 kg/ m². For segment k, let SL, = Lx/ H be its segment length relative to height, and Sm.kk mm/MM be its mass relative to bodymass. Let "bar" (?) denote the human values measured by
Dumas et al. in [66. We set our initial estimates as A scaled by 81, 8
(M, mm,ke AR. Xx {M, 8m,h ST,K
Akr xx + E
(19)
LL
s'm.k
my = Mm,M where Sm,k E3 $'m.
(20)
1o = m
(21)
Lastly, we compute the body weight W of the human in Newtons, with g 9.8mm/ s² as
W
9 Σ
mk
(22)
b
A.2 GREM Model
Let J = FF, MJ be the ground reaction forces and moments (GRFM) applied at the CoM of a foot in global cartesian coordinates. Let F = [Fx, FyjFz] where Y is the vertical direction, and z [Zz; Zy: =z]T be the center of pressure (CoP) the foot's local coordinates where æ along the
length of the foot, such that
M R2 Rankte F
(23)
where R2 is joint k's local rotation matrix, and Ro Ro(A)) Rs describes the chain of rotational transformations from the world frame to its local frame. We use lower case æ, y, denote the foot's s local coordinates, and its dimensions {4, 4w, J as shown in Fig. 6
We predict the force in the vertical direction scaled by body weight FW = FW, and CoP along the foot scaled by foot length 2 2z/4, from initial estimates based on the foot's kinematics vi and linear coefficients 7). Furthermore, let l be the coefficient of friction, initialized 0.8. With our regressed residuals 8, and binary contact c, we infer:
{ (FW 2 |1} = (NrY V, 7=2 V. 0.83 + dcy
(24)
Fy = FW body weight
(25)
2, = 2/ 4
(26)
23

---

Specifically, y [1, Pankle.Y, PoppAnkle.. Pankle,y, Pankle.Y 9ankle.s, 9ankle,s, Yankle.2: in- cludes the ankle's linear kinematics the direction opposite of gravity, and angular kinematics corresponding to plantar/dorssifexion, Linear coefficients were fitted on the forceplate data in (72). with nry = [0.3116, 1785 -222963, 0.4151, 0.0088, 0.3374, -0. 020620 -0.0089] and 1/ = [0.68996, -3. 1508, 0.5925, 0.21997, 0.0035, 0. 8502, -0.0331 1, -0.00212).
The remaining ò terms are scaling factors between and I ensure the values in the other directions are physically possible (i.e. Fx + F I 1² 1 and z is within the foot's dimensions)
Fx = Fx
Fz - 6z 12 F
(27)
Fy 16
2= (1m
(28)
AbbeciorsAddaaaii Addiction
Plantan Flesion
Imversiinn Eivevioon
Figure 6: Foot local coordinate system and dimen- sions, with length 4 = L + lo. width ln 2/ and CoM height I,.
Fost Cold
A.3 Muscle torque generators
We compute MTG torque TWTG using the equations below that are parameterized by the y coefficients
that be found in the tables of [54 (33) For each joint rotational DoF g E 9[::]- with angular
velocity 4, let musele signal [0, 1] represent the joint's corresponding activation level for this
DoF, we separate its TMTG into active torque generation Tactive and passive impedance Tpassive
TMTG = Tactive + passive
(29)
We compute the active torque as
Tactive TTT
(30)
where Tu models the aetivetoorue--nguar-peeer relationship 69 67 and is paramterized as a
piecewise function with coefficients 71:3-
(1 7) Wmaar (72 + 1)11'9
Wmaax
(Ø) =L
+ 1g20(
(31)
(1 Wmmar
+ (72 + 1)719
Wmaa + 724
The peak velocity Wmar for each joint we use the values from [54 153. The coefficient 71 is the ratio of the maximum eccentric isokinetic torque over the maximum isometric torque 669 15], 72
the slope of the eccentric and concentrie functions when the angular velocity is zero 69 and
7's is a shape factor that influences the curvature of the hyperbola in the torque-velocity concentric relationship
models the active-torquu--anlee relationship 21 portion of polynomial (2) with coefficients 74.6
33 and is represented by the non-negative
To (q) (74 + 754 + 7
(32)
To is the peak isokinetic torque that controls peak MTG output at zero joint velocity, which can be measured via dynamometry.
Tpassive is the passive torque 1 of joint that arises when the surrounding muscles, tendons, and ligaments are strained and intensifies near anatomical joint limits [ 24 81). A joint's viscous
damping and nonlinear stiffnees commonly described by double exponential function [79
Tpassine 710e
(1 amin) 712e 7a28-- 4max) 714w
(33)
where 710-11 14 are passive coefficients from
48 and 711 is the rotational damping linear coefficient 78 to reflect viscoelasticity. This encourages the joint to move within its range of motion (RoM), large restoring torque is produced otherwise.
24

---

A.4 Inverse dynamics |
We compute Tq using Lagrange's equations derived from d' Alembert's Principle of virtual work
Ty -- +
(34)
We can write the terms on the right hand side as:
28 = Σ J.M
(35)
k
C EJEAMaJ + Jh 10
O
[o MaJajá
(36)
8 F + Teroot Froo FRFoRt
(37)
where I3 is the identity matrix, (1-),) denotes the skew-symmetric form, and
Ig M+ 0
(38)
(gTo,((RR))
To deal with potential energy, we offiset the root acceleration in the direction of gravity by -9.8 m/s? Jacobian matrix J is the mapping from the generalized space to the global Cartesian coordinates, such that for linear and angular velocities Vi, S2 in global Cartesian coordinates, we have:
Jv JR
V2 S2
J&
q
(39)
can be computed analytically using recursive algorithm such as in [17. For segment k we define its parent segment p(k) as its neighboring segment that is closer to the root. Other than the root, each segment has one and only one parent. We define k's children ch(k) as its neighboring segments further away from the root. Let Ta-sa denote the 3D displacement from point to b. For segment k, with linear velocity Vn at its CoM and linear velocity V at its corresponding joint, we have
V foint Vi(a) + 2ppck) x -kiiin = Roo Jv,p(k) [P kjoins (s.pp(k)
(40)
and the velocity at the CoM of segment & becomes:
V2 jointt S2g X Tiioini > Jv.k Jv.n(k) [T [J2,,())) [Megioine sf
(41)
From we can compute the time derivative recursively as:
jv. = Jv.p(k) - (Pp(t))=aoons)Jp.pp -
( [Flssine
(42)
The global angular velocity of k in skew symmetric form is:
[2c] (RQ)) (R) R&) (RO R&)
(43)
(2214) (PP(N))
(Rx (TR)
(44)
= [Sp(ka) + Ro((e) R2(k))ush) [wss] (RR((RR (R)()))
(:' [w)]] = R2 RT)
(45)
Dc Pp(R) + Re(k) Wi
(:: [Ab]] A(b]]AT)
(46)
To avoid confusion of notation, we also write joint k's rotation 0₃ A qu. i.e. we have generalized coordinates
X < RRN where X, is the global root translation, 0o is the global root rotation, O2 describes the local rotation of segment relative to its parent p(k), and [0] 8T I. Let Juik be the local Jacobian such that we = Og. We can compute Q recursively:
22 = -p(kk + RB Ow
(47)
-0 Jw +
R20
( Open + R2 Ju.
(48)
(49)
Derivations in this section are based on C. Karen Liu and Sumit Jain's multibody dynamics notes: https //fab. cba. mit. www.mmttttuutaeeemmmmmmmmm h//cc/aasseee/6/ htt/////gggg/oppttttatttt/////// httprriiiiinnmiiiii.cccmmc
25

---

Let Pk denote the set of all ancestors of l and itself (A E Pk). we split Jn, into N₂ + blocks of size 3 3: Jn [03x3
Jw.o 1 Iep, Ro Fi
N.EP
R P(K) Ju.No]
E R 3xNDoF (50)
For segment k, if we represent rotation O2 = 10x.1 0x.2 O2.3l la angles, removing subscript for notation simplicity, have
B 7 R using Euler
dR
ÖR
0R
OR on
[ RR Σ
RTO,
RT +
R
+
RT4
(51)
00,
da
03
and it remains to compute J's s.t.
0R
0R
dR
Jw
J1
J2 Js],
where Jı]
R
[J2]
R
Jal
R
(52)
da
0
0y
which satisfies [w)] = Σ [.] and = J
(53)
Finally, let segment I Pk: be an ancestor of lo and denote (Jn,k): Roccy Just as the (1 + 2)-th 3 x block Jn from (50). we can compute time derivative as:
OJui A 08,,
(Jnx)
RP. Ja + R Rooos with Juc, Σ
(54)
following
52 remains compute j's s.t.
OJu.J 01.
Jud - [Jt11 Ji. Jr,)
where Jt,5 Σ
(55)
00,,+
1,8
A.5 Neural network
We trained a transformer encoder consisting of 8 layers with latent dimension of 256, using total loss Liotal with weights Apin = [0.5, 10, 1000, 1, 20, 1000] and Adyn [100, 100, 20). We trained on AMASS with sequence input length of 16 frames, after removing sequences containing non feet-ground contact, with contact labels from 83 for 25 epochs. We used the AdamW optimizer 44 with weight decay of 10-1 and an initial learning rate of 10-4 that decreases by 20% every epochs. The entire process can be trained in about 12 hours on single Titan Xp GPU.
26


Reviews:
[{"review_id": "T3VeTko76N", "reviewer": "Reviewer_i28J", "summary": "This paper presents a method to refine the output of existing forward-kinematics pose regressors (SMPL prediction models) by imposing constraints inspired by muscle models such as torque, limb lengths. (more below)", "questions": "L240 says an example consists of 16 frames, but how much real-world time does this correspond to? AMASS has a very high FPS (60-120+ Hz) depending on the subset. I think it is important to know how much motion is captured.\n\nL269 says, the worse performance on 3DPWoc is within reasonable margin compared to SOTA (the best one in the table is CLIFF). But according to L259, CLIFF was used as input to the proposed model. This input-output relationship could be made clear. It's not necessarily a limitation considering other metrics may have improved (Table 3 in supp).\n\nCould you explain the \"with alpha-hat\" ablation in Table 2 more? I found it hard to understand.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "+ I think the proposed novel components are well motivated. It could potentially be used to augment existing inference or evaluation methods.\n\n+ Figure 4 seems to show muscle activations are meaningful.\n\nSome of the muscle-related claims can be misleading, e.g. application to \"whole-body musculoskeletal dynamics of athletes\" (L55, L58) and biofidelity. SMPL joints labels are not consistent with anatomy (e.g. hips; https://files.is.tue.mpg.de/black/talks/SMPL-made-simple-FAQs.pdf). Some of this paper's core designs (e.g. passive and active torque losses) were inspired by and are analogous to muscles, but the interdisciplinary claims do not seem compelling enough. Also, I feel that simpler alternatives could have been explored and compared against.\n\nConsidering the main selling point is biofidelity and physical plausibility, I think more analysis on temporal consistency (e.g. smoothness, jitter, or torque continuity metric) would have been helpful.\n\nCLIFF might be a little dated now. More recent, motion-based approaches such as CoMotion (ICLR 2025) could have been insightful.\n\nComparison with existing energy-minimization approaches in vision (limb length symmetry constraints, foot skating loss) would have been helpful, e.g. hypothetically, results showing those existing techniques should be used with the proposed ones can be directly helpful to other researchers.", "was_revised": true, "final_justification": "The authors have addressed my concerns, particularly regarding the motion-based comparison, so I would like to increase my rating. I also accept that SMPL joints are positioned at the locations where rotation occurs. While I remain uncertain about making claims regarding faithfulness to real biological structures, I believe the experiments are valuable regardless of the specific wording. I defer to the judgment of the other reviewers.", "rebuttal": "We thank you for your thoughtful feedback. We are encouraged you found our method novel, well motivated, compatible with existing methods, and able to infer meaningful muscle activations. We hope our responses to Weakness $\\textbf{W2-4}$ below can improve the significance of our paper, and that our answers to $\\textbf{W1}$ and Questions $\\textbf{Q1-3}$ clear up some of the clarity concerns.\n\n$\\textbf{W1(a),  SMPL joints labels are not consistent with anatomy:}$ \nin your referenced link, Slide11 says, \"SMPL puts the hip joint where the rotation happens,\" which is what we desire to model biomechanically. \nSlide12 depicts the inconsistency of the H36M hip label (not SMPL), and shows its difference with the SMPL hip joint center. \n\n\n$\\textbf{W1(b), interdisciplinary claims not compelling enough, simpler alternatives could have been explored and compared against:}$\nwe chose MTGs as a middle-ground between $\\textit{\\textbf{simple}}$ alternatives to estimate torques that other physics-based human pose estimators (PHPE) currently employ and $\\textit{\\textbf{complex}}$ muscle models, to support our claim of improving biofidelity over the $\\textit{\\textbf{simple}}$ alternatives (with results shown in Sec 4.3), while further allowing us to infer human movement at a musculoskeletal level; at the same time, MTGs are compatible with our learning framework and maintains model parsimony by avoiding detailed musculoskeletal geometry and granular joint contact force computations that complex muscle models require (as discussed in L64-74 and L139-145).\nWe will expand on the MTG details in Sec B.4 of the Supp. A more complete motivation for MTGs can be found in [Inkol et al, 2020, Muscle torque generators in multibody dynamic simulations of optimal sports performance].\n\n\n$\\textbf{W2, more analysis on temporal consistency:}$ \nthank you for this suggestion. We will add the following table to further assess temporal consistency.\nSpecifically, both our method and PhysPT improve kinematically (ACC, JACC) over CLIFF, while our method further improves kinetically (MTV) over PhysPT.\nHere JACC is the mean absolute joint angle acceleration loss (in degrees/frame\\^2), and MTV is the mean torque variation as the mean absolute change in joint torques over consecutive frames (in Newton*metres/frame). Lower is better for both.\nJoint position acceleration loss (ACC) is already reported in Tabs. 3,4 of the Supp.\n\n|        | **H36M** |     |   | **3DPWoc** |      |\n|--------|:--------:|-----|---|:----------:|------|\n|        | JACC     | MTV |   | JACC       | MTV  |   \n| CLIFF  | 2.7      |  -  |   | 1.8        | -    |\n| PhysPT | 1.6      | 5.3 |   | 0.84       | 27.0 |\n| ours   | 1.6      | 2.5 |   | 0.95       | 12.1 |\n\n$\\textbf{W3, using a more recent kinematic model than CLIFF:}$  thank you for this suggestion. We planned on adding results for CoMotion here, but unfortunately ran out of time. $\\textbf{Recap:}$ our goal is a framework that works with most kinematic models, and similar to your suggestion, we have shown consistent results using WHAM in Tab. 4.  $\\textbf{The motivation for using CLIFF}$ during inference (in L256-260) was (1) because it produces competitive positional accuracy but lacks in physical plausibility (for which we demonstrated improvements in), and (2) for an even comparison with PhysPT (which uses CLIFF). However, we will add results for additional kinematics estimators to our Supp.\n\n\n$\\textbf{W4, incorporating existing energy-minimization approaches in vision:}$\nbelow are the results trained with an additional $\\textbf{skating loss}$ [77] to minimize feet velocity when in contact with the ground:\n\n|               | **H36M** |      |      |         |   | **3DPWoc** |      |      |         |\n|---------------|:--------:|------|------|---------|---|:----------:|------|------|---------|\n|               | MJE      | resF | FS   | float\\% |   | MJE        | resF | FS   | float\\% |\n| MusclePose    | 48.3     | 0.08 | 37.2 | 3.0     |   | 27.6       | 0.3  | 12.8 | 4.7     |\n| w/ skate loss |   51.2   | 0.44 | 27.3 | 82.7    |   | 31.2       | 2.6  | 16.8 | 83.5    |\n\n($\\textit{MJE}$: mean joint error in mm. $\\textit{resF}$: residual force scaled by body weight. $\\textit{FS}$: foot skating, average displacement in mm of vertices in contact with the ground in consecutive frames. $\\textit{float\\\\%}$: percentage of frames where the human is more than 10 mm off the ground. Lower is better for all.)\n\nAlthough we see some reduction in foot skating (FS) in H36M for this experiment, the other metrics are all worse off. While these additional kinematic constraints are indeed compatible with our method, and this experiment could be further fine-tuned, they may not be needed for PHPE, as they are already incorporated into the kinetics loss -- these kinematic artifacts may lead to high residual forces or unrealistic ground reaction forces.\n\nFor $\\textbf{limb length symmetry}$, we did not run additional experiments, as it did not seem like a significant issue for SMPL-based human pose estimators. The limb length symmetry error was about 4.5-4.7 mm for our method, PhysPT, and CLIFF, which seemed anatomically reasonable.\n\n\n\n$\\textbf{Q1, amount of motion captured:}$\nduring training, we downsampled AMASS to 10 fps (1.6 seconds of real world time for a sequence). \n\n$\\textbf{Q2(a), input-output relationship:} $\nas discussed in $\\textbf{W3}$, any kinematic model could be used as input, and CLIFF was used during evaluation for even comparison with PhysPT and to better show improvements in physical plausibility. We will clarify better in the paper.\n\n$\\textbf{Q2(b), explanation for ``kinematics-kinetics trade-off\" in 3DPWoc:}$ \nsince this dataset was captured with a moving camera, the root degrees of freedoms are harder to estimate, potentially leading to higher residual forces, which the model may try to reduce (lower kinetic error) by estimating a set of local joint kinematics slightly different from the original motion (higher kinematic error). \n\n\n\n$\\textbf{Q3, $\\hat\\alpha$ ablation:}$ \nwe added this ablation as an additional way to compare with existing methods that estimate joint torques as some linear combination of the joint's kinematics $\\mathcal{K} = \\\\{ \\text{joint angles } q, \\text{angular velocity } \\dot{q},... \\\\}$,\n\n$\n\\text{ours: }\\\\; \\tau_{MTG} = \\alpha \\cdot \\tau_{active}(q, \\dot{q}) + \\tau_{passive}(q)\n$\n\n$\n\\text{existing: }\\\\; \\tau = b_0 + b_1 \\dot{q} + b_2 q + b_3 F(\\mathcal{K}),\\qquad\n$\n(for learned coefficients $b$'s, and some parametric $F$)\n\nSince the muscle signal $\\alpha$ can be interpreted as an ``effort level\" between 0 and 1, for the ablation we estimated it as $\\hat{\\alpha} = (\\dot{q} /\\dot{q}_{limit}) + \\epsilon$,  \n\nwhere $\\dot{q}_{limit}$ is the joint's anatomical angular velocity limit, and $\\epsilon$ is a predicted offset term.\n\n$\\textbf{Results:}$ the worsened results from this initialization could be due to its over-reliance on the initial kinematics estimates."}, {"review_id": "wqA1Cs9Q0y", "reviewer": "Reviewer_m1ia", "summary": "This paper proposed a learning framework for physics-based human pose estimation (PHPE) from monocular videos, taking into consideration of the kinetic constraints. The estimator accounted for individualized anthropometrics, showed improved accuracy compared to purely kinematics-based models, and demonstrated some degree of bio-fidelity.", "questions": "How accurate or realistic are the estimations of ground reaction forces and external forces, since the contact is simplified?\nWhat's the inference efficiency?", "limitations": "The authors discussed some limitations in the supplementary.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The proposed framework incorporated kinetics when estimating joint positions and individualized anthropometrics aiming to improve the pose estimation accuracy.  The estimator didn't need a physics engine nor musculoskeletal simulation and could be trained end-to-end, which makes it a potentially useful tool in computer vision. The model design rationale, architecture and implementations are clearly presented (with more details in supplementary). Evaluations were conducted on several standard 3D human pose estimation benchmarks, showing improved or comparable estimation accuracy, and some degree of bio-fidelity.\n\nThe improvements in pose estimation were not that significant. A MTG module was used for muscle activation-net joint torque estimation, which overlooked the detailed muscle dynamics and the redundancy problem was somehow avoided. Mismatches exist for the estimated human body and biomechanical studies. To better validate the bio-fidelity, simultaneous human behavioral and EMG recordings could be used. The motion included are mainly gross movements, further improvement may demonstrate the performance on fine-motor movements.", "was_revised": true, "final_justification": "I think the authors addressed most of the concerns by reviewers. So I maintain my recommendation for acceptance.", "rebuttal": "We thank you for your thoughtful feedback. We are encouraged you found our method impactful and our paper clearly written. We have responded to your Weakness (W) section and questions (Q) below, and will incorporate them into our submission.\n\n$\\textbf{W1, on modest kinematic improvements:}$ while this is true, the main focus and contribution of our work were to improve the biofidelity of kinetics estimations for regression based physics pose estimators, which we showed in Sec. 4.3 and App. A of the Supp. \n\n$\\textbf{W2, on simplified muscle modeling:}$\nwe used MTGs over complex muscle models to bypass the computational overhead of estimating detailed musculoskeletal geometries and redundant joint contact forces (both of which we don't need), while maintaining biofidelity (L64-74 and L139-145).\n\n$\\textbf{W3, on mismatches with biomechanics studies:}$\nthese additional evaluations were done to qualitatively assess the plausibility of estimated joint torques for certain well-studied human motions, in terms of trends and limits, when ground truth data is unavailable. Since the torques in these biomechanics studies are derived from different data and models, mismatches will exist. \n\n$\\textbf{W4, on comparisons with EMGs:}$ \nour muscle signals are compared to EMG data in Fig. 4. \nNote, the muscles signals can be seen as a surrogate representation of EMGs, and the two are not exactly the same. Hence, mismatches in magnitudes will exist, and this comparison was made to show consistencies in the timing and trends of muscle activations. Furthermore, raw EMG values can vary widely due to electrode placement.\n\n$\\textbf{W5, on evaluation on gross movements over fine-motor movements:}$\nfast and dynamic movements such as baseball pitching and golf swings were chosen deliberately to better assess kinetic estimations, as they are more prone to errors leading to unrealistic force and torque estimations.\n\n$\\textbf{Q1, How accurate or realistic are the estimations of ground reaction forces and external forces, since the contact is simplified?}$ \n$\\textbf{What's the inference efficiency?}$\nGround reaction force (GRFs)  are evaluated in Figs. 3,5. Overall, compared to PhysPT, our median sum of vertical GRFs is much closer to the body weight of the subject. For golf swings and baseball pitching, our 99\\% quantiles are also closer to forceplate values reported in biomechanics studies [50, 59]. For gait, our vertical GRF estimates exhibit the characteristic \"m\" shape. Our method adds about 0.003 seconds to the inference time of the existing kinematics pose estimator used, for a video sequence of 16 frames, on a Titan Xp GPU."}, {"review_id": "sHxxUvJ7Wc", "reviewer": "Reviewer_xJ9Z", "summary": "This paper introduces MusclePose, a 3D human pose estimator that integrates muscle-dynamics modeling and detailed anthropometrics for inferring human dynamics from monocular video. The authors claim that MusclePose achieves competitive positional accuracy while providing more physically plausible kinetic inferences compared to existing methods.", "questions": "(1) The \"kinematics-kinetics trade-off\" is mentioned as a possible reason for slightly worse positional accuracy on 3DPWoc. Could you elaborate on this trade-off in the context of MusclePose? How does the model balance enforcing physical plausibility (kinetics) with maintaining high positional accuracy (kinematics)? Is there a tuning mechanism or a loss weighting that allows for prioritizing one over the other, and if so, how was it determined?\n\n(2) The paper lacks qualitative results showcasing MusclePose's performance on real-world images or videos. Could the authors provide visual examples of predicted 3D poses and body shapes overlaid on input video frames to demonstrate the model's output in a real-world context?\n\n(3) PhysPT, a competing method, employs several physical plausibility metrics (ACCL, VEL, FS, GP) and evaluates on \"all actions\". In contrast, MusclePose's evaluation is limited to sequences with \"feet-ground contact only\". Could the authors justify the omission of a broader evaluation set and these standard physical plausibility metrics, and explain how the presented results still provide a fair comparison with state-of-the-art methods like PhysPT?\n\n\n(4) The \"with \\alpha \" case in the ablations initializes muscle signals based on angular velocity relative to its limit, with an additional offset term regressed. Could you provide more details on why this specific initialization was chosen and what impact it has on the learning process and the resulting muscle signals, especially considering the observed discrepancies in Figure 4 (e.g., for ankle plantarflexion)?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n(1) Novelty in Physics-Infused HPE: The core strength of MusclePose lies in its novel integration of muscle-dynamics modeling and detailed anthropometrics directly into an end-to-end learnable physics-infused 3D human pose estimator. This goes beyond typical physics-based pose estimation methods that often rely on external physics optimizers or treat kinetics as auxiliary predictions.\n\n(2) Improved Biofidelity of Kinetic Inference: The paper demonstrates that MusclePose can infer plausible human kinetics and muscle signals consistent with biomechanics studies, without requiring an external physics engine. This is an improvement over prior regression-based approaches, where kinetic predictions may not be physically plausible. The use of a multihypothesis approach, inferring joint torques via Lagrange's equations and muscle dynamics modeling, contributes to this biofidelity.\n\n(3) Detailed Anthropometric Prediction: MusclePose's ability to predict detailed human anthropometrics based on biomechanics studies is an advancement, contrasting with existing methods that use simpler shape primitives. The ablations confirm that this custom anthropometric prediction leads to better kinetics.\n\nWeaknesses:\n\n(1) Limited Scope of External Force Modeling and the Evaluation Setting: The paper assumes only foot-ground contact for simplicity. While this simplifies the problem and the formulation, it limits the applicability of MusclePose to scenarios involving more complex interactions with the environment or multiple external forces, which are acknowledged as underconstrained problems. Moreover, the paper states that it \"trained and evaluated on sequences with feet-ground contact only\" and explicitly removed non-feet-ground contact sequences from both H36M and 3DPWoc datasets for evaluation. This narrower evaluation scope contrasts directly with PhysPT, which claims to \"adhere to the standard protocol and evaluate our method on all actions,\" implying evaluation on sequences beyond just ground contact. This discrepancy in evaluation protocols makes direct comparisons of generalizability and robustness between MusclePose and methods like PhysPT unfair, as MusclePose's performance on more diverse actions (e.g., sitting, climbing, freestyle movements) remains unassessed.\n\n(2) Qualitative Evaluation of Muscle Signals: While the paper states that MusclePose can infer plausible muscle signals, the evaluation of these signals is primarily qualitative, comparing them to trends from biomechanics studies and EMG data. A more rigorous quantitative assessment, if possible, would strengthen the claim.\n\n(3) Limited Evaluation Metrics: While the paper focuses on MJE/PJE for positional accuracy, it's true that PhysPT, a competing method, utilizes additional physical plausibility metrics such as ACCL (acceleration), VEL (velocity), FS (foot sliding), and GP (ground penetration). The omission of these quantitative physical plausibility metrics in the main evaluation for MusclePose is a notable limitation. This limits a direct and comprehensive comparison of the physical plausibility aspects between MusclePose and PhysPT, especially since MusclePose's core contribution is enhanced biofidelity in kinetics. While the paper includes quantitative metrics like residual forces and out-of-range joint torques (Table 2) , and qualitative comparisons of joint torques and muscle signals against biomechanics studies (Figure 3 and 4), the inclusion of standard physical plausibility metrics like ACCL, VEL, FS, and GP would provide a more complete picture of its physical realism from a kinematic perspective.\n\n(4) Under-discussed Computational Efficiency: While the paper claims inference speed on par with purely kinematic pose estimators, a more detailed analysis of the computational cost and real-time performance of MusclePose compared to its baselines would be beneficial.", "was_revised": true, "final_justification": "All of my concerns are resolved during the rebuttal, therefore I raise my score from 3 to 4.", "rebuttal": "We thank you for your thoughtful feedback. We are encouraged that you found our method novel and an improvement over existing regression-based physics human pose estimators (PHPE). We hope that our responses to Weakness $\\textbf{W1-3}$ below can improve the significance of our paper, and that our answers to $\\textbf{W4}$ and questions $\\textbf{Q1-4}$ resolves the clarity concerns. If you have additional suggestions on how we can improve on overall paper quality and originality, please let us know.\n\n$\\textbf{W1(a), limited scope of external force modeling:}$ \nwe, along with most regression-based PHPE [33,55,57,71], assume the feet-ground contact only scenario due to a lack of large-scale video datasets with contact labels, unlike some optimization-based methods [17, 75] that use a physics simulator. Under this scenario, when both feet are in contact with the ground, the dynamics problem is not simplified as it remains underconstrained. As discussed in L613-615 of the Supp, without loss of generality, we can expand our method to be trained and evaluated on datasets with more contact labels by simply adding contact points to our human model. While PhysPT includes additional contact points, by the same sense it remains a \"simplified\" case, as the contact points don't cover the entire body of the human. Furthermore, as shown in Fig. 7 of PhysPT, feet-ground contact takes up the significant majority of the contact events in their training data.\n\n$\\textbf{W1(b), on the narrower evaluation scope leading to unfair comparisons with PhysPT:}$ \nwe respectfully disagree with this comment. Comparisons on H36M (Tab. 3, Figs. 3,5), 3DPWoc (Tab. 4),  PennAction (Fig. 3), were done with the same data and procedure (after removal of non feet-ground contact). \n\n$\\textbf{W1(c), on limited implications due to the narrower evaluation scope:}$ \nagain, we respectfully disagree.\nThe \"standard protocol\" mentioned in PhysPT refers to the evaluation procedure for purely kinematic pose estimators, while regression-based PHPE often evaluates on the feet-ground contact only scenario, described in \\textbf{W1(a)}. \nWhile certain sequences (e.g. sitting) are removed in the latter case, the majority (77\\%) of the data remains, and contains poses that are kinematically similar (e.g. squatting) to the removed ones.\nWhat remains unevaluated is the estimated kinetics of these removed scenarios, which we do not claim to model, and which PhysPT also did not report.\nHowever, for the feet-ground contact case, which is the majority, we showed improvements in kinetics over PhysPT.\n\n\n$\\textbf{W2, lack of rigorous muscle assessment:}$ \nour current muscle evaluation in Fig. 4 follows the procedure in [25], to show consistencies in the timings and trends of our muscle activation predictions with biomechanics studies, as we are not aware of public video datasets with corresponding muscle activation levels to directly compare with.\nWhile we do compare with EMG data, the predicted muscles signals are more of a surrogate representation of EMGs, and the two are not exactly the same. Hence, mismatches in magnitude will exist. Furthermore, raw EMG values can vary widely due to electrode placement.\n\n$\\textbf{W3, lack of kinematic plausibility metrics:}$ \nthese metrics are not omitted and are reported in Tabs. 3,4 of the submitted Supp, which also includes additional metrics to further investigate GP, by evaluating the inferred ground reaction forces as an alternative, and by evaluating its opposite case -- floating. We showed that our method had similar kinematic plausibility as PhysPT, both of which improves over purely-kinematic methods, and that our method further improves on kinetic plausibility when compared to PhysPT.\n\n\n$\\textbf{W4, on computational efficiency:}$\nas with most regression-based PHPE, inference time is largely based on the existing kinematics estimator used to generate initial estimates, unlike methods that require an external physics engine for optimization, which are often too slow for real-time inference (Tab. 1 of [17]).\nSpecifically, our method adds about 0.003 seconds to the kinematics estimator used, for a video sequence of 16 frames, on a Titan Xp GPU. \n\n\n$\\textbf{Q1, ``kinematics-kinetics trade-off\" in 3DPWoc, and tuning:}$\nsince this dataset was captured with a moving camera, the root degrees of freedom are harder to estimate, potentially leading to higher residual forces, which the model may try to reduce (lower kinetic error) by estimating a set of local joint kinematics slightly different from the original motion (higher kinematic error). \\textbf{Tuning mechanism:} during training, the weights for the kinematic loss can be increased until the residual forces remain within a desired range.\n\n$\\textbf{Q2, lack of qualitative results:}$\nthese are not omitted and are shown in the top row of Fig 3. They were not overlayed on the images to better visualize the joint torques, align with the figures below, and to avoid cluttering, but we can add the images in the Supp.\n\n$\\textbf{Q3:}$\nthis is related to $\\textbf{W1, W3}$. The metrics are not omitted as they are reported in Tabs. 3,4 of the Supp, and we believe that the comparison is fair as both methods were evaluated using the same procedure and data. Please see our responses for $\\textbf{W1, W3}$.\n\n\n$\\textbf{Q4(a), } \\hat\\alpha\\textbf{ ablation:}$ \nwe added this ablation as an additional way to compare with existing methods that estimate joint torques as some linear combination of the joint's kinematics $\\mathcal{K} = \\\\{ \\text{joint angles } q, \\text{angular velocity } \\dot{q},... \\\\}$,\n\n$\n\\text{ours: }\\\\; \\tau_{MTG} = \\alpha \\cdot \\tau_{active}(q, \\dot{q}) + \\tau_{passive}(q)\n$\n\n$\n\\text{existing: }\\\\; \\tau = b_0 + b_1 \\dot{q} + b_2 q + b_3 F(\\mathcal{K}),\\qquad\n$\n(for learned coefficients $b$'s, and some parametric $F$)\n\nSince the muscle signal $\\alpha$ can be interpreted as an ``effort level\" between 0 and 1, for the ablation we estimated it as $\\hat{\\alpha} = (\\dot{q} /\\dot{q}_{limit}) + \\epsilon$,  \n\nwhere $\\dot{q}_{limit}$ is the joint's anatomical angular velocity limit, and $\\epsilon$ is a predicted offset term.\n\n$\\textbf{Impact on learning:}$ the worsened results from this initialization could be due to its over-reliance on the initial kinematics estimates.\n\n\n$\\textbf{Q4(b), muscle signal discrepancies in Fig.4:}$ as discussed in $\\textbf{W2}$, while we see discrepancies in magnitudes between our predicted muscle signals and the EMG values, the two are not the same, and the comparison was made to show consistencies in the timings and trends of our muscle activations. Furthermore, for ankle plantarflexion, the magnitude discrepancy may also be due to differences in shear force generation, as the subjects in H36M walk in a small circle with slower and varying speeds, whereas the subjects in the biomechanics studies walk in a straight line at a consistent pace."}, {"review_id": "50ECFZFQDC", "reviewer": "Reviewer_TsEp", "summary": "The paper introduces MusclePose, an end-to-end learnable physics-infused 3D human pose estimator that integrates muscle dynamics modeling to infer human dynamics from monocular videos. Unlike existing physics-based pose estimators that rely on external physics optimizers or produce implausible kinetics, MusclePose employs a multihypothesis approach to compute joint torques via Lagrange’s equations and muscle torque generators (MTGs). It also outputs detailed human anthropometrics based on biomechanics studies, enhancing biofidelity.", "questions": "As shown in the Weaknesses section, A reasonable explanation will make me raise the score for this work.", "limitations": "Yes, limitations are discussed in the supplementary material.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. Compared with PhysPT, the proposed ground reaction forces and moments (GRFM) have better mathematical modeling.\n2. The introduced MTGs (Muscle Torque Generators) can generate more reasonable joint torques while providing the muscle activation levels.\n\n## Weaknesses\n### Major\n1. PhysPT said that \"The discrete contact status also introduces a non-differentiable process in estimating the forces.\" How does GRFM deal with this issue?\n\n### Minor\n1. Does the MTG model all joints uniformly? Is it currently less effective for joints with 3 DOF (e.g., shoulders)? How does the paper ensure that fitting errors in complex joints do not propagate to the global dynamics estimation?\n2. Could you clarify the specific location of MTG in Reference [46] (cited in lines 63 and 207)? A more detailed explanation of the MTG implementation would be helpful.\n3. Why are Coriolis and centrifugal forces included in the Lagrangian formulation? Is the algorithm's reference frame the global/world coordinate system?", "was_revised": true, "final_justification": "This is a work with a solid foundation in theory. Considering the comments from other reviewers and the author's responses, I maintain my score.", "rebuttal": "We thank you for your thoughtful feedback. We are encouraged you found our method impactful in producing better kinetics estimates. We have responded to the your Weaknesses ($\\textbf{W}$) section below, and will incorporate them into our submission. We hope our explanations can improve the clarity of the paper.\n\n$\\textbf{W1, on ground reaction forces and moments (GRFM) estimation:}$ we followed the same approach as PhysPT (and other regression-based physics pose estimators), and estimated GRFM based on the foot's mesh height to the ground, and other foot kinematics, which are all non-discrete. One main difference is that we made use of existing biomechanics datasets [66] to initialize our GRFM model parameters (L663-667 of the Supp).\n\n$\\textbf{W2(a), does the MTG model all joints uniformly?}$\nNo.\nWhile all DOFs are modeled using Eq 31-35 in the Supp, each DOF has its own set of MTG coefficients $\\gamma$'s derived from different biomechanics studies (values summarized in Tabs. 4,5 of [48] and Tabs 4,5. of [49]). \nThis joint-dependent parameterization preserves physiological realism (e.g., hip flexion and knee extension exhibit different peak torques and passive stiffness profiles), unlike uniform torque models that assume identical properties across the body.\n\n$\\textbf{W2(b), are MTGs less effective for complex joints?}$ \nPossibly. \nExisting evaluations of complex joint such as the shoulders are limited [46]. As a result, we allowed the maximum isokinetic torque and range of motion of all DOFs to be learnable, instead of being fixed or scaled by human dimensions, as in biomechanics studies [48,49]. \n\n$\\textbf{W2(c), limiting effects of errors in complex joints on global dynamics estimation:}$ \nthese errors are more pronounced in distal segments (the diagonal of their inverted mass matrices are larger), hence we moved the DOF for forearm pronation/supination from the wrist to the elbow. DOFs for the ankles are constrained from the GRFM model, as errors can result in unrealistic forces. Lastly, redundant DOFs were removed, such as the one in the scapula that coincides with shoulder flexion/extension.   \n\n$\\textbf{W3, location of MTG in [46] (cited in L63,207):}$\nEq. 4 in [46], which we expanded and rewrote as Eq 31-35 in App. B.4 of the Supp. \nWe used the set of MTG coefficients in Tabs. 4,5 of [48] and Tabs. 4,5 of [49], and allowed the maximum isokinetic torque and range of motion of all DOFs to be learnable.\nWe will expand on the MTG details in App. B.4 of the Supp. \nA more complete explanation of MTGs can be found in [Inkol et al, 2020, Muscle torque generators in multibody dynamic simulations of optimal sports performance].\n\n$\\textbf{W4(a), why are Coriolis and centrifugal forces included:}$ \nthese are components of the equations of motion derived via Lagrange’s method (Eq. 1, 11). These second-order effects become significant when angular velocities are high or when multi-joint interactions occur, such as in baseball pitching or golf swings, where rapid rotations of the torso and shoulders generate substantial inertial coupling effects. Neglecting these terms would lead to systematically biased torque estimates, particularly for proximal joints where interaction forces dominate. \n\n$\\textbf{W4(b), is the algorithm's reference frame the global/world coordinate system?}$ \nYes. Specifically, the root is defined in world coordinates (we used the canonical form in [77], and centered the non-gravity axes at 0 at the start of each input sequence), while joint rotations are represented in the local coordinates consistent with the International Society of Biomechanics (ISB) format.\nThis hybrid representation allows us to maintain physical consistency (e.g., direction of gravity) to better compute ground contact, Coriolis force, etc., while the latter enables accurate relative joint modeling and compatibility with MTG models."}]

Author rebuttal:

Meta-review: This paper introduces a 3D human pose estimator that integrates muscle-dynamics modeling and detailed anthropometrics to simultaneously predict human kinematics, kinetics, muscle signals, and anthropometrics from monocular video. Experiments on benchmark datasets demonstrate improved accuracy and bio-fidelity compared to purely kinematics-based models.  The paper received four reviews: one accept and three borderline accepts. Reviewers acknowledge the novelty of explicitly combining muscle modeling, kinematic modeling, and anthropometrics to produce biologically faithful 3D body pose estimation. They also recognize the experimental results showing improved performance over purely kinematic methods.   However, reviewers raise several concerns. Major issues include the evaluation setting being limited to foot–ground contact, the absence of quantitative evaluation of estimated muscle signals, evaluation restricted to gross body motion, questions about the biological faithfulness of the estimated poses, inadequate evaluation metrics, and the lack of computational efficiency analysis.  The authors provide an extensive and detailed rebuttal with additional experimental results, which effectively addresses most of the reviewers’ concerns.
Decision: Accept (poster)