DOI: 10.5281/zenodo.10408306
title: Feasibility of 3D Body Tracking from Monocular 2D Video Feeds in Musculoskeletal Telerehabilitation

description: Musculoskeletal conditions affect millions of people globally, however, conventional treatments pose challenges concerning price, accessibility, and convenience. Many telerehabilitation solutions offer an engaging alternative but rely on complex hardware for body tracking. This work explores the feasibility of models for 3D Human Pose Estimation (HPE) from monocular 2D videos (MediaPipe Pose) in a physiotherapy context, by comparing its performance to ground truth measurements. MediaPipe Pose was investigated in eight exercises typically performed in musculoskeletal physiotherapy sessions, where the Range of Motion (ROM) of the human joints was the evaluated parameter. This model showed the best performance for shoulder abduction, shoulder press, elbow flexion, and squat exercises (MAPE ranging between 14.9% and 25.0%, Pearson’s coefficient ranging between 0.963 and 0.996, and cosine similarity ranging between 0.987 and 0.999). Some exercises (e.g. seated knee extension and shoulder flexion) posed challenges due to unusual poses, occlusions and depth ambiguities, possibly related to a lack of training data. This study demonstrates the potential of HPE from monocular 2D videos, as a markerless, affordable and accessible solution for musculoskeletal telerehabilitation approaches. Future work should focus on exploring variations of the 3D HPE models trained on physiotherapy-related datasets, such as the Fit3D dataset, and post-preprocessing techniques to enhance the model's performance.
description: An experimental study was designed to evaluate the performance of MediaPipe Pose on eight exercises typically performed in musculoskeletal physiotherapy sessions by comparing it to a gold-standard motion tracking system, namely the Qualisys Motion Capture system. The selected exercises were: Shoulder Flexion/Extension, Shoulder Abduction/Adduction, Elbow Flexion/Extension, Shoulder Press, Hip Abduction/Adduction, Squat, March, and  Seated Knee Flexion/Extension.

The eight healthy participants (seven females and one male, aged between 19 and 21 years old) performed eight exercises displayed on a screen to guide them through the execution of the movements. Each participant performed two sets of seven exercise repetitions, with a 10-second resting period between sets. A total of 64 acquisitions (eight participants × eight exercises) were collected, of which only 63 were studied due to a technical issue during the shoulder flexion exercise performed by Subject 5. For each acquisition, the following data was collected simultaneously: (1) ground truth data, i.e. 3D coordinates (x, y, z) of six anatomical landmarks placed on the right side of the participant, using the Qualisys Motion Capture system (operating at 100 FPS) from the laboratory and (2) monocular 2D video recordings, using a Nikon Coolpix A10 camera (operating with 1280 × 720 resolution at 30 FPS). From this data, two datasets were built:

(1) The ground truth dataset (data_pos_qualisys) was collected using Qualisys, a gold-standard motion capture system. Six markers were placed in anatomical regions on the right side of the participant's body close to the shoulder, elbow, wrist, hip, knee, and ankle. Qualisys measured the 3D position of the six markers.

(2) The predicted dataset (data_pos_mediapipe) was estimated by MediaPipe Pose, a state-of-the-art 3D HPE algorithm. The 3D position of twelve human joints (shoulders, elbows, wrists, hips, knees, and ankles) was estimated by the algorithm from monocular 2D videos.

Files are named according to the following identification criteria: xy.txt, where x is the subject ID and y is the exercise number. For example, 13.txt is the acquisition of subject 1 performing exercise 3 (i.e. elbow flexion).