<figure><img src="image_1.png" /></figure>

HAL
open science

# Generalization of Marker Augmenter Learning Models on a New Motion Database for Inverse Kinematics

Hasnaa Ouadoudi Belabzioui, Charles Pontonnier, Georges Dumont, Pierre Plantard, Franck Multon

► To cite this version:

Hasnaa Ouadoudi Belabzioui, Charles Pontonnier, Georges Dumont, Pierre Plantard, Franck Multon. Generalization of Marker Augmenter Learning Models on a New Motion Database for Inverse Kinematics. 2024 - 49ème congrès de la société de biomécanique, Oct 2024, Compiègne, France. pp.1-2. hal-04727199

HAL Id: hal-04727199

https://inria.hal.science/hal-04727199v1

Submitted on 9 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Generalization of Marker Augmenter
Learning Models on a New Motion
Database for Inverse Kinematics

Hasnaa Ouadoudi Belabzioui<fnref n="1" />,<fnref n="2" />, Charles Pontonnier
<fnref n="1" />, Georges, Dumont<fnref n="1" />, Pierre Plantard<fnref n="2" />, Franck
Multon<fnref n="1" />

*Corresponding author. Email: hasnaa.ouadoudi-belabzioui@ens-rennes.fr

**Keywords:** walking data, fine-tuning, inverse kinematics

## 1. Introduction

Computer vision techniques now offer an alternative to traditional movement analysis by detecting human joints locations in standard videos. One versatile approach involves triangulating 3D body keypoints and tracking them with musculoskeletal models. OpenCap (Uhlrich et al., 2023) is a web-based tool that utilizes this method to compute motion metrics and estimate musculoskeletal forces from smartphone videos. It operates by generating 3D keypoint trajectories using Human Pose Estimation algorithms and then employing human marker augmenter learning models to map sparse joint center positions to additional anatomical markers positions. OpenCap has been trained and tested on various movements like walking and running, but its marker augmenter learning models were trained on a specific motion database. This study aims to adapt OpenCap's models to a new motion database using fine-tuning, a process of adjusting a pre-trained neural network for a specific task (Fu et al., 2023). Given the differences in the number of markers sets between OpenCap and ours, we only considered similar markers for evaluating inference results and for fine-tuning.

## 2. Materials and methods

### Experimental data

Our study the motion database from Schreiber et al. (2019), with 50 participants walking at five speeds (0–0.4 m/s, 0.4–0.8 m/s, and 0.8–1.2 m/s). Each participant completed at least 3 trials per speed, totalling 1145 trials. Motion was recorded using a Qualisys optoelectronic system (23 cameras, 200Hz). Three dimensional trajectories of 52 cutaneous reflective markers were recorded. To better control the experimental set-up, we simulated the input 3D joint centers, based on Schreiber et al. (2019) full-body motion capture data, as a video HPE would provide. To this end, we used the Reed et al. (1999) regression

equations to emulate the 3D joints $J=\{J_1, ..., J_{20}\}$ at each time.

### Pre-processing

The OpenCap's learning model models the function $\psi_0(\mathbf{V}, \mathbf{M}, \mathbf{L}) = \hat{\mathbf{Y}}$ mapping input features $\mathbf{V}$ (human pose modelled as 3D joints), subject weight $\mathbf{M}$, and subject height $\mathbf{L}$, to output 3D marker positions $\hat{\mathbf{Y}}$. To use it on the new dataset, we processed the database and adapted the learning model accordingly. Prior to training, the 3D marker positions were normalized relative to a root marker (the midpoint of hip keypoints), scaled by the subject height, and resampled at 60 Hz. Inverse kinematics is applied to both ground truth data (originally used to estimate the joint centers, as described above), and the resulting estimated 3D marker positions $\hat{\mathbf{Y}}$. The list of markers predicted by the OpenCap Arm Model and Lower Body model was larger than the ones present in the dataset. Therefore, we considered a subset of the markerset to assess the inference and the fine tuning, excluding the following markers from the outputs of OpenCap: ['r_thigh1_study', 'r_thigh2_study', 'r_thigh3_study', 'L_thigh1_study', 'L_thigh2_study', 'L_thigh3_study', 'r_sh1_study', 'r_sh2_study', 'r_sh3_study', 'L_sh1_study', 'L_sh2_study', 'L_sh3_study)]. The OpenCap set of anatomical markers corresponds to what is commonly used for marker-based motion capture.

### Fine-tuning marker augmenter learning models

OpenCap human marker augmenter proposes two LSTM models: the Body Model for body markers and the Arm Model for arm markers. The Body Model uses lower-limb and torso keypoints, while the Arm Model uses arm and torso keypoints. In our fine-tuning strategy, named Output Layers Fine-tuning (OLF), we freezing all layers except the last one, after this level, we added an additional output layer initialized using a normal distribution with a mean of zero and a standard deviation of 0.022. During the fine-tuning stage, a weight decay of 0.01 was applied to all tunable parameters, excluding biases, in accordance with the methodology outlined by Barone et al. (2017).

Our learning algorithms were trained on a NVidia RTXA3000 GPU. The optimal models were achieved using an early stopping technique. Training concluded when the loss failed to decrease with a minimum delta of $1 \times 10^{-4}$, with a patience value of 10 epochs. Adam optimization was employed with a batch size of 64 and a learning rate ($\alpha$) of $6 \times 10^{-6}$.

### Evaluation

The evaluation utilized Leave-One-Out procedure per subject. Anatomical marker accuracy was assessed by mean Euclidean distance $RMSE_{markers}$ per trial, averaged across trials of the same movement type in millimeters. To assess the impact of the augmenter on

<footnote n="1">¹ Univ Rennes, Inria, CNRS, IRISA-UMR 6074, 35000, Rennes, France</footnote>

<footnote n="2">² Moovency, Inc, 35051, Cesson-Sévigné, France</footnote>
<figure><img src="image_3.png" /><figcaption>Figure 1: Comparison of RIAS (Right anterior-superior iliac spine coordinates) marker results: y-axis = Cartesian coordinates (mm), x-axis = frames. Colors: red = ground truth, blue = fine-tuning, black = inference.</figcaption></figure>

the inverse kinematics, a geometric calibration to scale the model and an inverse kinematics step were applied to the inferred, fine-tuned and ground truth data to obtain the joint coordinates (Muller et al., 2019). Joint coordinate accuracy was evaluated using RMSE<sub>joint</sub> coordinates per trial, averaged across trials of the same motion type in degrees.

## 3. Results and discussion

<table><caption>Table 1: (Top) Average root mean square error RMSE<sub>markers</sub> [mm] for the marker's positions and corresponding standard deviation ρ are presented. (Bottom) Average root mean square error. RMSE<sub>joint</sub> coordinates [Degree] for joint coordinates and corresponding standard ρ.</caption><thead><tr><th></th><th>Inference [mm]</th><th>OLF [mm]</th></tr></thead><tbody><tr><td>Body model</td><td>57.86 ± 5.64</td><td>24.92 ± 6.98</td></tr><tr><td>Arm model</td><td>24.09 ± 3.31</td><td>18.54 ± 3.49</td></tr><tr><td></td><td><strong>Inference [°]</strong></td><td><strong>OLF [°]</strong></td></tr><tr><td>Left knee flexion angle</td><td>8,63 ± 3,37</td><td>9,23 ± 2,08</td></tr><tr><td>Right knee flexion angle</td><td>16,26 ± 4,55</td><td>9,94 ± 1,52</td></tr><tr><td>Left ankle flexion angle</td><td>10,15 ± 2,58</td><td>10,48 ± 1,97</td></tr><tr><td>Right ankle flexion angle</td><td>10,40 ± 2,94</td><td>10,58 ± 2,38</td></tr><tr><td>Left foot inversion angle</td><td>25,68 ± 4,35</td><td>9,27 ± 2,84</td></tr><tr><td>Right foot inversion angle</td><td>33,29 ± 2,94</td><td>8,03 ± 2,01</td></tr></tbody></table>

The inference was conducted overall. Then, we fine-tuned both the OpenCap human marker augmenter, and it was tested on an unseen set of 25 subjects. As illustrated in Table 1 (Top) and Figure 1, both fine-tuned models consistently outperformed the inference models, underscoring their efficacy in reducing marker displacement errors across various body parts. Additionally, the standard deviations also decreased after fine-tuning, suggesting that the models' predictions became more consistent. The results presented in Table 1 (Bottom) illustrate that the fine-tuning generally led to a reduction in the mean values and, in some cases, a decrease in the standard deviation. The changes in angles after fine-tuning may suggest better accuracy in the models' predictions.

## 4. Conclusions and perspectives

This study evaluated fine-tuning impact for marker position prediction and angle estimation, which consistently yielding the lowest average RMSE values for marker positions compared to inference results. As the estimation of joint coordinates is conditioned by anatomical markers, as well as geometric calibration and inverse kinematics, exploring the impact of geometric calibration would be an interesting perspective. However, it was noted that part of the residual error may stem from data emulation, highlighting the significant impact of input skeleton topology and marker set on the learning models performance. In a future study, we aim at applying similar strategies to experimental HPE data to enhance the accuracy and generalization of such marker augmenter to be used on field.

## Acknowledgements

This work was partially funded by the Cifre convention N°0936/2021, Moovency company, and the National Agency under the Inverstments for the Future program (PIA) with the grand ANR-21-ESRE-0030 (CONTINUUM project).

## References

* Barone, A. V. M., Haddow, B., Germann, U., & Sennrich, R. (2017). Regularization techniques for fine-tuning in neural machine translation. *arXiv preprint arXiv:1707.09920*.

* Fu, Z., Yang, H., So, A. M. C., Lam, W., Bing, L., & Collier, N. (2023, June). On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 11, pp. 12799-12807).

* Muller, A., Pontonnier, C., Puchaud, P., & Dumont, G. (2019). CusToM: a Matlab toolbox for musculoskeletal simulation. Journal of Open Source Software, 4(33), 1-3.

* Uhlrich, S. D., Falisse, A., Kidziński, Ł., Muccini, J., Ko, M., Chaudhari, A. S., ... & Delp, S. L. (2023). OpenCap: Human movement dynamics from smartphone videos. PLoS computational biology, 19(10), e1011462.

* Reed, M. P., Manary, M. A., & Schneider, L. W. (1999). Methods for measuring and representing automobile occupant posture (No. 1999-01-0959). SAE Technical Paper. Schreiber, C., & Moissenet, F. (2019). A multimodal dataset of human gait at different walking speeds established on injury-free adult participants. Scientific data, 6(1), 111.

* Received date: leave blank

* Accepted date: leave blank

* Published date: leave blank

* Volume: leave blank

* Publication year: leave blank