A Self-Supervised PINN for Inertial Pose and Dynamics Estimations

Authors: Markus Gambietz, Eva Dorschky, Marcel Schöckel, Jörg Miehling, Anne D. Koelewijn
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords: self-supervised learning, biomechanics, physics-informed neural networks

Abstract:
Accurate real-time monitoring of not only movements, but also internal joint moments or muscle forces that cause movement in unrestricted environments is key for many clinical and sports applications. A minimally obstrusive way to monitor movements is with wearable sensors, such as inertial measurement units, using the fewest sensors possible. 
Current real-time methods rely on supervised learning, where a ground truth dataset needs to be measured with laboratory measurement systems, such as optical motion capture, which then needs to be processed with methods that are known to introduce errors. There is a discrepancy between laboratory and real-world movements, and for analysing new motions, new ground truth data would need to be recorded, which is impractical.
Therefore, we introduce SSPINNpose, a self-supervised physics-informed neural network that estimates movement dynamics, including joint angles and joint moments, from inertial sensors without the need for ground truth data for training.
We run the network output through a physics model of the human body to optimize physical plausibility and generate virtual measurement data. Using this virtual sensor data, the network is trained directly on the measured sensor data instead of a ground truth. Experiments show that SSPINNpose is able to accurately estimate joint angles and joint moments at 8.7 degrees and 4.9 BWBH%, respectively, for walking and running at up to speeds of 4.9 m/s at a latency of 3.5 ms. We further show the versatility of our method by estimating movement dynamics for a variety of sparse sensor configurations and inferring the positions where the sensors are placed on the body.

Decision: reject
Reviewer scores: 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730640700342

thread_type:
official_review

<thread_object>
creator: reviewer_nikt

<content>
summary:
The paper proposes SSPINNpose, a self-supervised physics-informed neural network for estimating human movement dynamics using inertial measurement units (IMUs) for real-time applications in uncontrolled environments. SSPINNpose leverages a combination of physics-informed losses and recurrent neural network (RNN) architecture to infer both kinematic and kinetic properties of human motion.

strengths:
- The paper estimates dynamics as a byproduct of kinematic alignment of actual IMU and estimations from a virtual model. I find this as the most interesting aspect of the paper.

- The authors use auxiliary physics-inspired losses for attempting realistic estimations.

weaknesses:
> The authors highlight that most state-of-the-art methods rely on supervised approaches, while their method does not require labeled data, and that, this is one of the highlight of their work.

- However, as such, in this case of joint kinematics and kinetics regression, there is nothing as labeled or unlabeled data because the authors are anyway using full lower-body motion. The question of labeled or unlabeled data may appear in cases of classifying the intent such as walk level ground, climb stairs etc., but in the case where the goal is to regress body kinematics or kinetics, anyway, the recording full body motion is done, thus, there is no question of explicitly labeling the data.

- Thus, both supervised and self-supervised approaches appear to aim at the same objective: minimizing the deviation between experimental and estimated kinematics. So here, the self-supervised goal seems to be only a wording and framing problem. 
It is also evidenced by the main self-supervised objective that the authors use, minimizing the deviation between actual IMU motion and the virtual model’s motion, which closely resembles a supervised objective. Could the authors clarify how their approach differs fundamentally from traditional supervised methods? Further, the fact that dynamics is estimated as a byproduct of minimizing kinematic deviations could be done by posing this as a supervised method also, since the authors are estimating GRF using a ground contact model anyway.  

> I found the objectives, contributions, and methodology of this work weakly positioned and unclear, as the authors highlight multiple objectives to establish the novelty of their method over existing approaches. Could the authors to clarify the primary advantages of their approach and how it complements or differs from existing state-of-the-art tools. Here are some specific points of feedback:

- One stated objective appears to be the estimation of kinematics and kinetics using IMUs rather than optical motion capture systems. Then why go for a self-supervised method, which may be less accurate? Musculoskeletal models, e.g., OpenSim can do this too. AFAIK, OpenSim 4.1 has begun supporting IMU data analysis. Could the authors elaborate on the specific advantages of their self-supervised approach compared to these methods? It appears that the real-time estimation is the only advantage. Then why this highlight of IMUs vs. OMC?

- Another goal mentioned is to facilitate analyses outside laboratory settings. This could theoretically be achieved by collecting data in real-world environments and applying OpenSim or similar tools to calculate inverse kinematics and dynamics. Could the authors clarify the added value of SSPINNpose for these applications, particularly if it avoids preprocessing requirements that are still necessary for lab-based systems?

- The authors also emphasize real-time computation of kinematics and kinetics. This seems particularly relevant to their approach, as OpenSim and related tools typically do not offer real-time capabilities. However, did the authors test this model in real-time by actually doing lower-limb gait analysis in the wild? Could the authors describe the setting in detail and elaborate on the real-time metrics like what was the real-time prediction frequency, lost packets, how the sensor synchronization and failure were handled, read rate etc? Sorry if I missed the information.

> The authors miss many common regression-based works that estimate joint kinematics regression of one joint from other inputs such as [1], [2], [3], [4], etc.
 - Many such works are missing in the related works.
 - Comparison with many such common existing methods also seem to be missing.

> When posed as a self-supervised problem, how do the authors handle collapse, especially, when the virtual model estimation is so perfect that the difference between the actual and virtual model estimations are almost trivial?

> Do the authors evaluate other similar self-supervised methods like [5] on their data?




References:

[1] Lim, Hyerim, Bumjoon Kim, and Sukyung Park. "Prediction of lower limb kinetics and kinematics during walking by a single IMU on the lower back using machine learning." Sensors 20.1 (2019): 130. 

[2] Lower body kinematics estimation from wearable sensors for walking and running: A deep learning approach. Gait & posture, 83, pp.185-193.

[3] A function approximator model for robust online foot angle trajectory prediction using a single IMU sensor: Implication for controlling active prosthetic feet. IEEE Transactions on Industrial Informatics, 2022

[4] Estimation of Lower Limb Joint Angles and Joint Moments during Different Locomotive Activities Using the Inertial Measurement Units and a Hybrid Deep Learning Model. Sensors 23.22 (2023): 9039.

[5] Large-scale Training of Foundation Models for Wearable Biosignals, ICLR 2024

questions:
Please see the weaknesses section
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681219991

thread_type:
official_review

<thread_object>
creator: reviewer_ce42

<content>
summary:
The paper introduces SSPINNpose, a self-supervised physics-informed neural network designed to estimate human movement dynamics from inertial measurement unit (IMU) data without requiring ground truth labels. By integrating a physics model of the human body and leveraging virtual sensor data, the method enforces physical plausibility using Kane's equations and a temporal consistency loss. SSPINNpose is capable of accurately estimating joint angles and joint moments in real-time, even with sparse sensor configurations, and can infer the positions of the sensors on the body. Experiments demonstrate that SSPINNpose achieves comparable performance to supervised methods in estimating lower-body dynamics during walking and running at various speeds.

strengths:
Innovation: Proposes a novel self-supervised approach to estimate human movement dynamics from IMU data without the need for ground truth labels, addressing a significant limitation in current methods.
Real-time Capability: Demonstrates that SSPINNpose can estimate dynamics in real-time with low latency, which is crucial for practical applications in clinical and sports settings.
Physical Plausibility: Incorporates physics-based constraints using Kane's equations and temporal consistency to ensure physically plausible motion estimations.
Sparse Sensor Configurations: Shows the method's versatility by effectively handling sparse IMU setups and inferring sensor placements, enhancing its practicality.

weaknesses:
Limited to 2D Movements: The method focuses on lower-body dynamics in the sagittal plane and does not address full 3D motion estimation, limiting its applicability to more complex movements.
Simplifying Assumptions: Assumes flat ground and non-sliding feet, which may not hold in real-world scenarios and could affect the accuracy of estimations in such conditions.
Model Personalization: Uses a generic multibody dynamics model without personalization, potentially affecting the accuracy due to individual anatomical differences.
Replication of Noise: The model may replicate noise in IMU signals, which could reduce the physical plausibility of the estimated motions.

questions:
How does SSPINNpose perform when applied to movements involving significant out-of-plane dynamics or complex 3D motions?
Can the method be adapted or extended to handle 3D movement estimation, and what challenges would that entail?
How sensitive is SSPINNpose to variations in IMU placement, and how robust is the sensor placement inference in practice?
What impact do the simplifying assumptions (e.g., flat ground, non-sliding feet) have on the method's performance in more variable real-world environments?
How does the model handle or mitigate the replication of noise in IMU signals to maintain physical plausibility?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691803768

thread_type:
official_review

<thread_object>
creator: reviewer_sle6

<content>
summary:
The paper introduces SSPINNpose, a self-supervised approach for estimating gait kinematics only from IMU data. The authors proposed a self-supervised approach to train a recurrent model based on LSTMs. The model is trained in an IMU signal reconstruction task without any labels while regularized by a multitude of physical constraints. Their approach demonstrates promising results in estimating joint angles, torques, and GRFs from IMU data, and the source code for testing/training is provided.

strengths:
- The paper is well-written and easy to follow. 
- The paper shows the potential of combining physics-informed objectives during self-supervision for gait kinematic estimation.

weaknesses:
- **Lack of Comparisons**: One of the main weaknesses of the paper is its lack of direct comparisons with prior approaches on the same datasets and under the same conditions and evaluation criteria. I find the comparisons in the supplementary materials insufficient for judging the contribution and effectiveness of their work, especially given that the proposed method is not evaluated under the same conditions as prior works. Please see my suggestions below to address this limitation.
- **Contributions**: The paper proposes several physics constraints to guide the model's training. However, it is unclear what their contributions are in this combination. I have found several other optimization-based studies using the same physical constraints during pose/motion/gait kinematic estimation. Despite this, the paper's approach of combining physics-informed objectives during model training is somewhat unique. However, there are no experiments that support or highlight the impact of this novelty.
- **Evaluations**: The authors evaluate their approach only on one dataset, which contains only walking and running sequences. Therefore, the impact of this paper is only on some applications and its influence on others is not fully studied.

questions:
The paper tackles an important problem in the pose and gait estimation fields and shows promising results in the same range as prior works on other datasets. However, I have some concerns regarding the evaluations and the lack of comparisons. Therefore, I suggest rejecting this paper in its current form. To improve the paper, I have several questions and suggestions:
1. Can you please provide direct comparisons with state-of-the-art methods on the same dataset and under the same evaluation conditions? In the first paragraph of page 8, the paper mentions that the dataset owners have used another evaluation scheme, and therefore the results are not directly comparable with the authors. To address this issue, my suggestion is to evaluate your approach with both schemes to back up your claims.
2. Can you please elaborate on the paper's contributions, beyond combining and tuning existing losses during self-supervised training? I suggest authors add a recap of their contributions to the last paragraph of the introduction to clarify and highlight their contributions. 
3. Can you please evaluate the computational cost of adding all objective functions during training? Does it increase the training time?
4. Can you please elaborate on the difference between your approach and two-stage solutions, similar to Physical Inertial Poser (PIP)? I would be interested in knowing if there are any merits in a second-stage optimization after you have the initial results. In this case, some physics constraints like GRF may be necessary during the first stage (self-supervised learning) based on Table 8. But how necessary would the other losses be? One advantage of the second stage is that it can generalize well to unseen actions, which I believe your approach lacks. It would be valuable if the authors could provide more details on the advantages and disadvantages of their design vs optimization-based and hybrid (2-stage) solutions.


Some things could be improved, but did not impact the score:
- On some places, like L236 and L268, the authors say "see supplementary A/B/C", but on others, like L377, L382, and L359, they just say "See B/C/D". Please use a more constant formatting.
- On L240, "This loss is applied generalized coordinates q" seems like an incomplete sentence.
- The authors have used a simple LSTM network that is likely to have been fully tuned in prior works. It would be interesting to see a comparison with more modern signal processing architectures or a comparison with CNN approaches from prior works. What is the intuition behind choosing this network? Tables 5,6 and 8 are more informative for belonging to the main paper, while Table 1 is just a network comparison that should be supplementary.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730926495408

thread_type:
official_review

<thread_object>
creator: reviewer_ig7e

<content>
summary:
This paper presents a method to estimate the human pose and dynamics from wearable IMU sensors. Specifically, a self-supervised method is used to reconstruct the input IMU signals and to estimate joint angles and dynamics. Kane’s equations are used to calculate the motion equations. Experiments show promising performances on dynamics and pose estimations.

strengths:
1. Estimating human dynamics from IMU sensors is an important and meaningful research direction.

2. Using a self-supervised learning method to capture information from IMU sensors sounds very interesting.

3. Combining a neural network with a physics model is also reasonable.

weaknesses:
1. In line 199 it is mentioned that "the IMU data consists of 2D acceleration and 1D gyroscope data per sensor". Why is the acceleration 2D and gyroscope data 1D, instead of 3D? Besides, what does the "gyroscope measurement" refer to? Is it angular velocity or estimated orientation?

2. I am also a little bit confused by the proposed method. It is mentioned that this is a self-supervised method, however, in Section 3.1, the output contains root rotation and translations, joint angles and joint torques. I don't quite understand how each of these outputs is predicted (e.g., what is estimated by neural networks, what is calculated from Kane’s equations)

3. Besides, are the motion capture data used for this method? or purely based on IMU data (since it is self-supervised)? If ground truth joint angles are provided to supervise training, it will not be self-supervised learning.

4. I can not find much information about the adopted dataset Dorschky et al., 2024. I am curious about the choice of dataset and suggest using some well established dataset like [1,2].

[1] Zhang et al.: MMVP: A Multimodal MoCap Dataset with Vision and Pressure Sensors, CVPR 2024

[2] Werling et al.: AddBiomechanics Dataset: Capturing the Physics of Human Motion at Scale, ECCV 2024

5. I think it would be very important and beneficial to show the proposed method can benefit the existing 3D pose estimation methods from IMU sensors. However, such a comparison is only provided in the appendix using different datasets, and compared to SMPL models, the joint angle ranges are manually set. There are also recent works that try to make SMPL models or AMASS datasets align with biomechanical models with a meaningful degree of freedom, like [3]. Perhaps similar methods can be applied to enable a fair comparison.

[3] MANIKIN: Biomechanically Accurate Neural Inverse Kinematics for Human Motion Estimation, ECCV 2024

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1732286527274

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Discussion - New Experiments

comment:
Dear Reviewers, 

Thank you for your valuable feedback. We have conducted additional experiments to address your comments. In this thread, we will present the results of these experiments and discuss their implications. Some experiments are still running - we keep this thread updated.

### Comparable Testing

As mentioned by Reviewer SLE6, we have compared our method with baseline methods [1] and [2] on the same evaluation dataset. To enable a fair comparison, we have altered the training and evaluation schemes of our method to match those of [1] and [2]. 

In [1], the authors trained a CNN on ensemble averaged gait cycles from participants 4-10 as well as simulated gait cycles. They evaluated on ensemble averaged gait cycles from participants 1-3. To compare our method with [1], we trained "Ours - [1]" on the raw data of participants 4-10 and evaluated on ensemble averaged gait cycles from participants 1-3. Therefore, our training scheme is different from [1], but the evaluation scheme is the same. This way, we can train our model without further preprocessing steps. 

In [2], the authors solved optimal control problems on averaged gait cycles for each trial individually. To avoid the need for retraining, we trained and evaluated "Ours - [2]" on ensemble averaged gait cycles for all participants and trials. As our approach is fully self-supervised, there is no label leakage in doing so. 

_Table 1: Comparison of our method with [1,2] on the same evaluation dataset._

| Model | JAE [°] | JTE [BWBH%] | GRF [BW%] | Speed [m/s] |
| --- | --- | --- | --- | --- |
| Ours - [1] | 11.2 | 7.2 | 23.1 | 0.12 |
| [1] | 4.9 | 1.4 | 10.7 | - |
| --- | --- | --- | --- | --- |
| Ours - [2] | 8.9 | 6.8 | 23.8 | 0.25 |
| [2] | 6.3 | 2.6 | 17.9 | 0.25 |


For comparison, when testing "Ours - [1]" on emsemble averaged gait cycles from participants 4-10 (training participants), we obtained the following results:
JAE: 11.24°; JTE: 7.05 BWBH%; GRF: 24.3 BW%; Speed: 0.15 m/s. "Ours - [1]" was trained on the raw IMU streams from participants 4-10, and exhibits a drop-off in JAE and JTE when tested on ensemble averaged gait cycles, but not when tested on different participants. Therfore, we conclude that the model is generalizing well over participants. On the other hand, the domain shift from raw IMU streams to ensemble averaged gait cycles does impact the performance. Lastly, the speed error is decreased in this evaluation setting.

For "Ours - [2]", there is no domain shift, but the training data itself only consists of 60 gait cycles. Therefore, we expect that this training run did not generalize as well as the other models. We set $\lambda _ {\text{FS}}$ to 30, as we found the amount of training data to be problematic.

[1] Dorschky et al. "CNN-Based Estimation of Sagittal Plane Walking and Running Biomechanics From Measured and Simulated Inertial Sensor Data", Frontiers in Bioengineering and Biotechnology, 2020. 

[2] Dorschky et al. "Estimation of gait kinematics and kinetics from inertial sensor data using optimal control of musculoskeletal models." Journal of Biomechanics, 2019.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732806715198

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised version of manuscript uploaded

comment:
Dear Reviewers,

thank you for your insights and helpful suggestions. Your comments contributed to a significant improvement, in our opinion. We have uploaded a revised PDF earlier. In summary, we have made the following changes and additions:

1. **Clearer positioning of our work**: We acknowledge that our work is related to self-supervised physics-based methods from other domains, suchs as computer vision and graphics. These similarities are laid out in a new section in the related work section. 

2. **Contribution**: Our contribution is now more clearly stated in the introduction. 

3. **Deep-learning-based methods**: As suggested, we have expanded our related work section on deep-learning-based methods. This section now includes a list of methods that use two-step inference in computer vision and graphics, as well as single-step regression methods that stem from biomechanical applications.

4. **New Experiments**: As already laid out in the discussion thread, we have conducted a number of new experiments to validate our claims. These experiments include: 
    - A fairer comparison with baseline methods on the same evaluation scheme.
    - Sensitivity to IMU misplacements.
    - Recovery of erroneous IMU placements.

    Furthermore, we conducted two experiments that are only briefly or not mentioned in the paper:
    - Testing on a different dataset containing a variety of motions. Here, we encountered challenges with the dataset that make our method only applicable when taking additional assumptions that might not be accurate. 
    - Combination with PIP's second stage. Here, we were unable to show a considerable improvement over our baseline method, however, that might be because our model has already generalized well to the tested motions. 

5. **Minor Additions**: In Section 4.1, we disussed some of the upsides and downsides of our method compared to two-stage dynamics inference methods. Based on your comments, we have added small remarks and clarifications for better understanding throughout the paper.

We believe these changes address your concerns and significantly enhance the manuscript's quality. We look forward to engaging further in the discussion phase. Thanks again for your feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
## Summary
The paper introduces SSPINNpose, a self-supervised method for estimating human pose and dynamics from wearable IMU sensors. The method reconstructs input signals and estimates joint angles and dynamics using Kane's equations. Experiments show promising results in dynamics and pose estimations. SSPINNpose is a physics-informed neural network that accurately estimates joint angles and moments in real-time, even with sparse sensor configurations. It can infer sensor positions and achieves comparable performance to supervised methods in estimating lower-body dynamics during walking and running at various speeds.

## Strengths
* Highlights the importance of human dynamics estimation from IMU sensors and the use of auxiliary physics-inspired losses for realistic estimations.
* Suggests using self-supervised learning to capture information from IMU sensors by combining a neural network with a physics model for gait kinematic estimation.
* Demonstrates real-time capability and physical plausibility of the method.
* Shows versatility in handling sparse IMU setups and inferring sensor placements.

## Weaknesses
* The authors' approach lacks comparisons with state-of-the-art methods and for not providing direct comparisons with other evaluation schemes.
* The paper's contributions are not elaborated on, and the computational cost of adding all objective functions during training is evaluated.
* The authors' approach is limited to 2D movements and does not address full 3D motion estimation.
* The method assumes flat ground and non-sliding feet, which may affect the accuracy of estimations in real-world scenarios.
* The authors use a generic multibody dynamics model without personalization, potentially affecting accuracy due to individual anatomical differences.
* The main self-supervised objective is minimizing the deviation between actual IMU motion and the virtual model’s motion, which closely resembles a supervised objective.
* The authors miss many common regression-based works that estimate joint kinematics regression of one joint from other inputs.

## Conclusions
Based on the reviews and author’s feedback, the paper has potential to be accepted, however there were some concerns about the paper as stated by two reviewers that were not convinced at all. For instance, as mentioned by one of the reviewers *It seems that other reviewers have also pointed out the limited evaluations, limited 2D-only experiments, and limitations for flat-ground assumptions. Other papers, specifically in physics-based 3D character control, already address these problems. Therefore, I have limited my expectations based on the gait estimation field of research. However, the opinions of others are still valid, and I would agree that the paper does not have enough impact.* and other reviewer had some concerns about the definition of self-supervised learning and the amount of data. Therefore, the paper should address all the concerns before accepting the paper.