# Evaluation of a markerless data augmentation algorithm

Sandrine Bousigues*, Alexandre Naaim, Antoine Muller, Thomas Robert

Univ Eiffel, Univ Lyon 1, LBMC UMR T9406, F-69622 Lyon, France

* Corresponding author: sandrine.bousigues@univ-eiffel.fr

Received date: 07/04/2024

Accepted date: 28/06/2024

Publication date: 31/01/2025

**Keywords:** Opencap, Openpose, video-based analysis

© 2025 The Authors
Licence CC-BY 4.0
Published by Société de Biomécanique

## 1. Introduction

Markerless movement analysis enables in situ measures, removing the time needed to equip the volunteer. Yet, current pose estimation algorithms like Openpose only return a small set of keypoints. This small set of keypoints strongly limits the 3D movement analyse possibilities compared to the standard approaches using a larger set of skin markers: kinematic assumptions are required to restrict the number of degrees of freedom and ISB recommendations are almost impossible to follow. An alternative was recently proposed within the Opencap pipeline (Uhlrich et al. 2023): two neural networks (Long Short-Term Memory (LSTM)) were trained to predict the location of 43 skin keypoints placed on anatomical landmarks, from 20 keypoints output by Openpose. This keypoint augmentation is unlikely to compensate for the information missing from Openpose, particularly wrist prono-supination, but it could provide interesting information for conducting a biomechanical study while controlling its limitations. Although promising, this approach still lacks of validation: to our knowledge, it has only been assessed once by the team who developed it (Uhlrich et al. 2023). Therefore, additional and alternative assessments seem necessary.

In this study, we propose an evaluation of the keypoint augmentation algorithm on 3D point estimation on a dataset composed with markerless videos and marker-based acquisitions.

## 2. Methods

### 2.1 Data

Data collected in a previous study (Chaumeil et al. 2024) were used with 16 participants (9 men and 7 women) with a mean age of 25.1 ± 3.0 years old, (mean height 1.7 ± 0.1 m and mean weight 68.4 ± 13.5 kg). had participated. Each participant was equipped with 46 reflective markers. Ten optoelectronic cameras (Qualisys Miquis M3 300 Hz) and ten videos (Qualisys Miquis video 60 Hz) cameras were used during the data collection. This setup enabled us to synchronize marker-based and marker-less data, and both systems were calibrated using Qualisys Track Manager (QTM). The following tasks from the dataset were selected: walking at a self-selected speed (Walk) and sit-to-stand transfers.

### 2.2 Data processing

The marker-based data were processed using QTM with steps of labelling and gap-filling of marker trajectories. Videos were processed using Openpose (v1.7.0) with default settings (resolution of 208x368 pixels) using the model `body_25B` to obtain the position of the different keypoints. These 2-D points were then triangulated. Finally, the LSTM network deep-learning algorithm (https://github.com/stanfordnmbl/opencap-core/tree/main) was applied to get the 43 skin keypoints (Uhlrich et al. 2023). Volunteer's mass and height were given as input to the algorithm. All the data were processed using `pose2sim` for the 3D reconstruction and
keypoint augmentation procedure (https://github.com/perfanalytics/pose2sim).

On each frame, the distance between the marker from the marker-based data and the keypoint inferred with the deep-learning algorithm was computed (Figure 1 illustrates the markerset). The mean Euclidean distance and the standard deviation were calculated for each task and each marker with its corresponding keypoint. For simplicity's sake, on the figure, the two sides were averaged so that the comparison would be smoother between the two tasks.

## 3. Results and discussion

The global mean distances (mean [min-max]) on all the markers was 32 mm [17 – 48 mm] for the walk task and 46 mm [34 – 62 mm] for the sit-to-stand task. The walk task presents lower distances between the inferred keypoint and the real marker position than the sit-to-stand task (Figure 1) which may be explained by the training dataset of the algorithm. However, in comparison with the results given in the supplementary materials of (Uhlrich et al. 2023), there is more inter-task variability in the present. Hence, in their work, the global mean error was 36 mm [29 – 43 mm] for the Walk and 29 mm [19 – 40 mm] for the sit-to-stand (using Openpose default settings and five cameras for each task).

In the walk task, the errors were smaller on the thigh, the shank and the upper arm (Figure 1). However, on the wrist, the influence of a smaller error can have a significant impact on the segment orientation estimations. The difference was 21 mm and 37 mm (respectively for RSP and USP) which is large in relation to the width of the wrist.

A major limitation of this study is that the knowledge of the Euclidean distance between the real marker and the inferred keypoint does not give much information on the variability of the direction of this error. Indeed, if there was a bias in the same direction, it may be possible to correct it once it has been quantified. With regards to the inter-task variability of the Euclidean distance, it would appear that this bias cannot be corrected. A study of the projection of these distances in anatomical coordinate systems would corroborate this hypothesis. A perspective of this work would be to analyse how these errors affect the kinematics results. However these results would be dependent on different choices (filtering, kinematics algorithm used,...) and we

have therefore chosen to start by looking at the rawest possible results from keypoint augmentation.

<figure><img src="image_1.png" /><figcaption>**Figure 1.** Mean distance (circle size, proportional to the legend) and its standard deviation (color) between the inferred keypoint and the real marker for each marker for the walk and sit to stand tasks.</figcaption></figure>

(C7: 7th cervical vertebra spinous process; SAT: Most anterior lateral point of the acromion; H-ME/LE: Humeral medial/lateral epicondyle; RS: Radial styloid; US: Ulnar styloid; A/PSIS: Anterior/Posterior superior iliac spine; M/L-FE: Medial/Lateral femoral epicondyles; L/MM: Lateral/Medial malleolus; FCC: Calcaneum; FM1/5: First/Fifth metatarsal head)

## 4. Conclusions

The differences found in the present study are in line with the walk task previous results but seems more important in the sit-to-stand task. This could be explained by a
difference in range of motion and angular velocity between the tasks, which can create different occlusions and soft tissue artefacts. The keypoint augmentation algorithm provides access to a wealth of additional information, but its use requires an understanding of its limitations and its accuracy.

## Funding

This study was partly funded by the ANR PPR STHP 2020 (project PerfAnalytics, ANR 20-STHP-0003).

## Conflict of Interest Statement

The authors declare that they have no known competing financial interests that could have appeared to influence the work reported in this paper.

## References

Chaumeil A., B. K. Lahkar, R. Dumas, A. Muller, and T. Robert. (2024). Agreement between a Markerless and a Marker-Based Motion Capture Systems for Balance Related Quantities. *J. Biomech.* 165:112018. doi: 10.1016/j.jbiomech.2024.112018.

Uhlrich, S. D., A. Falisse, Ł. Kidziński, J. Muccini, M. Ko, A. S. Chaudhari, J. L. Hicks, et S. L. Delp. 2023. OpenCap: Human Movement Dynamics from Smartphone Videos, édité par A. L. Marsden. *PLOS Comput. Biol.* 19(10):e1011462. doi: 10.1371/journal.pcbi.1011462.