# Framework for the application of markerless motion capture to biomechanics.

Lucas Mercier, Thierry Cresson, Neila Mezghani, Carlos Vázquez

► To cite this version:

Lucas Mercier, Thierry Cresson, Neila Mezghani, Carlos Vázquez. Framework for the application of markerless motion capture to biomechanics. Multidisciplinary Biomechanics Journal, 2025, 50th congress of the.., 10.46298/mbj.16163. hal-05191715v2

HAL Id: hal-05191715

https://hal.science/hal-05191715v2

Submitted on 23 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Framework for the application of markerless motion capture to biomechanics

Lucas Mercier<sup>a, b*</sup>, Thierry Cresson<sup>a, b</sup>, Neila Mezghani<sup>b, c</sup>, Carlos Vázquez<sup>a</sup>

<sup>a</sup> LIO-ETS, Montréal, Canada

<sup>b</sup> Centre de recherche du CHUM, Montréal, Canada

<sup>c</sup> Teluq, Montréal, Canada

* Corresponding author: lucas.mercier.1@ens.etsmtl.ca

Received date: 06/04/2025

Accepted date: 27/06/2025

Publication date: 27/10/2025

**Keywords:** markerless, inpainting, video, openpose

© 2025 The Authors

Licence CC-BY 4.0

Published by Société de Biomécanique

## 1. Introduction

Pose estimation models can predict joint centers (JCs) from images or video, offering a promising alternative to traditional, marker-based motion capture (MoCap), which is costly and requires anatomical expertise (Jatesiktat, 2024). However, there seems to be no consensus for using these models in biomechanics. Most studies use models like OpenPose trained on COCO, but JC definitions in such datasets often differ from those in marker-based systems, leading to errors. To avoid this, model training and evaluation should use consistent JC definitions. Marker-based MoCap can be used to generate such data (Vafadar, 2021). However, this means markers will remain visible in training and test images. Studies show that training models with visible markers leads to worse performance on markerless images (Jatesiktat, 2024). Since markers won't be used in clinical applications, we propose a framework that allows accurate JC estimation from images without markers.

## 2. Method

To estimate JC from images without relying on markers, the proposed framework involves selecting a pose estimation model, removing markers on each image, and training the model, ideally leveraging transfer learning from COCO for better performance.

### 2.1 Model choice

The first step is to choose which model will be trained using the framework, ideally, the best performer on pose estimation benchmarks.

Although OpenPose remains one of the most widely used models, it is significantly outperformed by HRNet (Sun, 2019), which will be our chosen model.

### 2.2 Marker inpainting

To remove markers from images, we used a GAN-based approach (Suvorov, 2021) that takes an image and a corresponding mask indicating the regions to be removed as input. (Fig. 1). For small markers, masks are created by automatically placing circles at known marker positions (Fig. 1.A), for larger elements like rigid bodies, pre-defined bounding boxes are positioned on the 1st frame based on JC positions (Fig. 1.B). The user can then adjust their size if needed. As the system is supposed to be rigid w.r.t the leg, boxes will then be automatically positioned in the following frames. The masked regions are then inpainted with realistic skin and clothes textures.

### 2.3 Training on MoCap dataset

While JC annotations from public datasets (like COCO) can't be directly used, they contain many subjects and environments, providing a robust initial JC estimation. We thus recommend using transfer learning from the COCO database to initialize the model weights before training.

### 2.4 Evaluation of the proposed framework

We used the proposed framework to train HRNet to estimate JC from images on two datasets, collected using
<figure><img src="image_2.png" /><figcaption>**Figure 1.** Proposed workflows to remove markers and rigid bodies in images</figcaption></figure>

Credit: Lucas Mercier.

different MoCap systems and marker sizes. **RRIS40** is a dataset from (Jatesiktat 2024) containing 10 participants equipped with 40 small reflective markers (Qualisys) used to compute 12 JC from upper and lower limbs and 8 video cameras. We collected the KKGVideo dataset with 28 participants walking on a treadmill for 45 seconds using the KneeKG system (Hagemeister, 2005), which includes videos and joint centers of the right knee, ankle, and hip. To assess the impact of each proposed step, we performed multiple trainings (Tab. 1): 1) train on COCO as the literature does, 2) train on images with markers, 3) train with markers and transfer learning from COCO, 4) train with markers removed and transfer learning from COCO. We then evaluated each model on images with markers removed by computing the Euclidean distance between predicted and ground truth JCs in pixels. For the RRIS40 dataset, evaluation and

training were conducted using a leave-one-out strategy on 10 subjects. For the KKGVideo dataset, 23 subjects were used for training, and 5 separate subjects were used for evaluation.

## 3. Results and discussion

The model trained on COCO (Tab.1.1) performs better than the model trained with markers only (HRNet-markers Tab. 1.2) on markerless images. As the literature suggests (Jatesiktat, 2024), this is because these models learn to identify features from markers that help them estimate JC. Adding pretraining on COCO (HRNet-markers* Tab. 1.3) already shows an important boost in performance because of the additional anatomical information learned. When using the proposed framework (HRNet-inpaint* Tab. 1.4), errors are reduced compared to HRNet-markers* (Tab. 1.3), showing that our method benefits both small markers and rigid bodies. There is a bigger gap between HRNet-markers* (Tab 1.3) and HRNet-inpaint* (Tab 1.4) on the KKGVideo dataset because rigid bodies are easier to spot in images than small markers. Compared to HRNet-COCO (Tab 1.1), HRNet-inpaint* (Tab 1.4) reduces mean error by ~40% and significantly lowers error variability on markerless images across both datasets.

<table><caption>**Table 1.** Mean Euclidean distance ± standard deviation (pixel) for all JC. J represents the number of JC. Results are with markers removed. * means pretrain on COCO</caption><thead><tr><th rowspan="2">#</th><th rowspan="2">Model<br>(-TrainDB)</th><th colspan="2">Eval Dataset</th></tr><tr><th>RRIS40 (J=12)</th><th>KKGVideo (J=3)</th></tr></thead><tbody><tr><td>1</td><td>HRNet-COCO</td><td>8.51 ± 8.41</td><td>8.79 ± 8.14</td></tr><tr><td>2</td><td>HRNet-markers</td><td>9.89 ± 16.88</td><td>15.08 ± 20.14</td></tr><tr><td>3</td><td>HRNet-markers*</td><td>5.70 ± 5.63</td><td>6.11 ± 9.06</td></tr><tr><td>4</td><td>HRNet-inpaint*</td><td>5.09 ± 5.21</td><td>5.35 ± 3.29</td></tr></tbody></table>

Credit: Lucas Mercier.

## 4. Conclusions

The proposed framework improves JC prediction on markerless images, bridging the gap between current research validation, where markers remain visible, and the markerless data used in clinical environments, thereby enhancing real-world applicability.
## Acknowledgements

We would like to thank Bianca Marois for the help during data collection.

## Conflict of Interest Statement

The authors declare that they have no known competing financial interests that could have appeared to influence the work reported in this paper.

## Contributor Roles

LM: Conceptualization, data curation, formal analysis, Software, Investigation, Writing original draft; TC: Conceptualization, supervision, investigation, writing-review; NM: Supervision, investigation, writing-review; CV: Supervision, investigation, writing-review.

## Funding

This study was partly funded by the NSERC Alliance grant: ALLRP 565212 – 21

## References

Hagemeister, N., Parent, G., Van de Putte, M., St-Onge, N., Duval, N., & de Guise, J. (2005). A reproducible method for studying three-dimensional knee kinematics. *Journal of Biomechanics, 38*(9), 1926–1931. https://doi.org/10.1016/j.jbiomech.2005.05.013

Jatesiktat, P., Lim, G. M., Lim, W. S., & Ang, W. T. (2024). Anatomical-marker-driven 3D markerless human motion capture. *IEEE Journal of Biomedical and Health Informatics*, 1–14. https://doi.org/10.1109/JBHI.2024.3424869

Sun, K., Xiao, B., Liu, D., & Wang, J. (2019). Deep high-resolution representation learning for human pose estimation. In *2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, Long Beach, CA, USA, 5686–5696. https://doi.org/10.1109/CVPR.2019.00584

Vafadar, S., Skalli, W., Bonnet-Lebrun, A., Khalifé, M., Renaudin, M., Hamza, A., & Gajny, L. (2021). A novel dataset and deep learning-based approach for marker-less motion capture during gait. *Gait & Posture, 86*, 70–76. https://doi.org/10.1016/j.gaitpost.2021.03.003

Suvorov, R. et al. (2022). Resolution-robust large mask inpainting with Fourier convolutions. In *2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, Waikoloa, HI, USA, 3172–3182. https://doi.org/10.1109/WACV51458.2022.00323