# Implementing Real-Time Markerless Motion Capture in Smartphone Exergames

Mathieu Phosanarack, Laura Wallard, Sophie Lepreux, Christophe Kolski, Eugénie Avril

<figure><img src="image_1.png" /></figure>

## To cite this version:

Mathieu Phosanarack, Laura Wallard, Sophie Lepreux, Christophe Kolski, Eugénie Avril. Implementing Real-Time Markerless Motion Capture in Smartphone Exergames. Mardi des Chercheurs 2025, Apr 2025, Valenciennes, France. hal-05026604

HAL Id: hal-05026604

https://hal.science/hal-05026604v1

Submitted on 9 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

# Implementing Real-Time Markerless Motion Capture in Smartphone Exergames

Mathieu Phosanarack, Laura Wallard, Sophie Lepreux, Christophe Kolski, Eugénie Avril

LAMIH UMR CNRS 8201, Univ. Polytechnique Hauts-de-France, Valenciennes, France

## Highlights

* Smartphone cameras using AI-powered pose estimation enable affordable and accessible exergames for rehabilitation and physical activity.

* Balancing accuracy and latency is crucial, lightweight AI models improve responsiveness but may reduce movement precision.

* User positioning affects visibility and tracking: solutions include larger UI elements, auditory feedback and upper-body-focused exercises.

* Optimization of AI models and user studies are needed to improve usability in mobile exergames.

## Introduction

* **Gamifying** physical exercises fosters user engagement and motivation [1].

* Physical exercises which track user movement often rely on specialized hardware [2].

* Mobile health has gained interest and smartphone cameras could be leveraged to perform real-time markerless Motion Capture (MoCap) [3].

* However, pose estimation models are heavy for mobile devices [4] and motion based interaction is not optimal on a mobile screen.

<figure><img src="image_5.png" /></figure>

## Markerless Motion Capture

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /><figcaption>Result</figcaption></figure>

Once the pose estimated, the position of the joints can be exploited to perform motion based controls.

## Challenges and Trade-offs

### AI models performances:

Many pose estimation models for markerless MoCap are too heavy to run on mobile devices altering performances with high processing time [4].

Lightweight models for better responsiveness. Risks of lower accuracy.

Reduce the number of calculations per second by avoiding pose estimation at every frame.

<figure><img src="image_8.png" /></figure>

Off-loading the pose estimation to a server or cloud service.

Combining previous solutions.

### Interaction challenge:

Distance induced by full body tracking reduces visibility.

Perform upper-body-focused exergames allowing the user to stay closer to the screen.

Use large user interface elements and alternative feedback mechanisms such as auditory feedback.

## Future work

<figure><img src="image_9.png" /></figure>

Advancing lightweight AI models will improve motion capture **accuracy** while maintaining real-time responsiveness.

<figure><img src="image_10.png" /></figure>

Intuitive design like voice, gesture-based interactions, and alternative feedback to **improve** exergame **interactions**.

<figure><img src="image_11.png" /></figure>

Improve **accessibility** through **alternative interaction** and **feedback** mechanisms for users with **special needs**.

<figure><img src="image_12.png" /></figure>

User studies to **evaluate** and **validate** interaction design for motion-based exergames on mobile devices.

## References

[1] Phosanarack, M., Avril, E., Lepreux, S., Wallard, L., & Kolski, C. (2025). User-centered personalized gamification: an umbrella review. User Modeling and User-Adapted Interaction. https://doi.org/10.1007/s11257-024-09423-z

[2] Meulenberg, C. J. W., De Bruin, E. D., & Marusic, U. (2022). A Perspective on Implementation of Technology-Driven Exergames for Adults as Telerehabilitation Services. Frontiers in Psychology, 13.

[3] Bazarevsky, V., & Grishchenko, I. (2020). On-device, Real-time Body Pose Tracking with MediaPipe BlazePose. Accessed february 6, 2025 at https://research.google/blog/on-device-real-time-body-pose-tracking-with-mediapipe-blazepose/

[4] Jo, B., & Kim, S. (2022). Comparative Analysis of OpenPose, PoseNet, and MoveNet Models for Pose Estimation in Mobile Devices. Traitement du Signal, 39(1), 119-124. https://doi.org/10.18280/ts.390111

<figure><img src="image_13.png" /></figure>