SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Experiments We trained our ML model on 109,000 pairs of MoCap and wearable sensor output frames and validated and tested on 13,000 frames of data. We evaluated the results with quaternion distance  $(D_q)$  to compare the estimated 3D joints' angles with corresponding values from MoCap ground truth data, as shown in Figure 5 (a). The calculation of  $D_q$  (see Equation 1 for details) is performed under the scale of normalized quaternions [58]. Figure 5 (b) illustrates an example output of quaternions for the squat exercise separately derived for the left and right legs. To enhance comprehension, the evaluation results expressed in Euler angles is incorporated into the Figure A6. The motion recorded by the pressure sensors aligns well with the changes of quaternions, as displayed in Figure 4. Table 1 summarizes the root-mean-square error (RMSE) of each joint in degrees converted from quaternion distance results. We report average errors of 9.16, 9.75, and 6.64 degrees for the knee, hip, and ankle angles, respectively. The toe has a relatively low margin of error because it is not a primary joint used in squats and is not significantly involved in activities during exercise.
Figure 6 Device's generation evaluation (unit: normalized quaternions). Top (a-c): Qualitative performance extracted from unseen tasks for squat, hamstring curl, and leg raise exercises. Middle (d-f): Error of joints for predictions of seen tasks. The model sees all the data from participants and tasks. Bottom (g-i): The occurrence of prediction joint quaternion error in new tasks and individuals. The training was conducted on a partitioned dataset that excluded tested actions to exam the generalization of our model to unseen tasks.
Our assessment involves measuring the device's ability to estimate joint angles for activities that have not been previously observed. As displayed in Figure 6 (g-i), the model performs well on different unseen tasks. Our dataset is roughly categorized into three types of exercises: squat, hamstring curl, and leg raise. Despite the various forms of squatting available (stepwise squat, tired squat, etc. See Table B7 for details), we view them as the same movement when it comes to training. In our trials of unseen exercises, we exclusively evaluated the bent squat Figure 6 (d, g) because other variations of squatting produced very similar results. To estimate the bent squat, we trained solely on exercises involving leg raises and hamstring curls, excluding all other types of squats from the training process. In theory, regardless of the type of exercise performed, the pressure sensor and IMUs should exhibit comparable patterns as long as there is similar muscle contraction and extension since the human pose is essentially linked to muscle activation.
Our smart knee sleeve generalizes to unseen poses with slightly increased errors as for the case of hamstring curls Figure 6 (e, h). The reasonable degradation of performance in unseen tasks can arise from the mildly distinctive patterns in leg raise. Leg raise in Figure 6 (f, i) is the only pose in our dataset that starts from a sitting position. Since we are measuring the pressure sensor and IMUs with relative values to avoid the sensor and marker displacement influence, the start point for both strain sensors and IMU data is zero. This is reasonable for the poses that start with the standing position. However, for the sitting position, the supervised labels provided by the MoCap are actually 90 degrees, and the pressure sensors will also have initial values with stress applied. To eliminate the effect, we rotate all the quaternions of leg raise from IMUs 90 degrees before training. The pressure sensor data will also have a relatively influenced pattern due to the initial sitting position. The inconsistency between IMUs, sensors, and ground truth data induces confusion and errors in the model inference process. If we let the model see only  $10\%$  of the leg raise data in the training process, the performance will be improved with less error (see Figure A5 of Appendix A).
Our wearable Knee Sleeves are customized to fit each individual perfectly. Except for poses that have not yet been encountered, it is possible that the wearable electronics, markers, and MoCap system calibration positions may shift when tested on different dates and individuals. We have conducted tests under these conditions and have depicted the results in Figure 7. No discernible decrease is observed in the outcome and accuracy of the model.
Figure 7 Quaternion distance for unknown individual exercises (a) and unseen dates (b). The model's performance remains consistent when trained with unknown individuals and dates, with only a minimal rise in the quaternion distance error, indicating its strong generalization capabilities.
5 Discussions and Limitations
Our attempts to obtain precise angle measurements from the lower body's anatomical pose have encountered multiple challenges that can compromise measurement accuracy during exercises. These challenges include soft tissue movement and sensor displacement during prolonged exercises, which can result in potential errors. Moreover, the accuracy of our model inference is compromised when testing the leg raise pose, which is the only pose starting from a seated position. To overcome these challenges and enhance our system, we will include additional scenarios that involve transitioning from sitting to standing or lying down to examine the impact of the starting position on our smart Knee Sleeves measurements. We plan to modify the starting position from relative to absolute values or add a calibration period to ensure accurate measurements across all poses by aligning sensor values at 0. Furthermore, when measuring errors from various joints, we noticed that the toe angles consistently demonstrate low error rates. This is likely due to minimal movement in the toe angle during these poses. To better evaluate and compare the model's performance on joints, it would be preferable to use percentage error measuring systems or add poses that include obvious toe movement.
In addition, we discovered that MoCap systems can encounter occlusion problems, which can affect the accuracy of ground truth labels in our dataset. As a result, we plan to thoroughly distill the ground truth data and ensure that as supervised information, MoCap feedback is accurately interpreted to achieve improved accuracy in the future. What's more, we have mentioned using Fourier resampling
to make the recordings from wearable knee sleeves more smooth in section 3, but Fourier transform can't thoroughly solve the uneven time interval problem, which may cause drifted predictions in the test as in Figure A3. Future methods will be recommended to include more specific and complex algorithms focusing on lost time points to address Bluetooth issues.
We provide a comprehensive dataset and baseline model for 3D human pose estimation with a pair of durable, stretchable, wearable sensors. We demonstrate our ML model pipeline's effectiveness across various scenarios including generalization to unseen tasks and individuals. We collected a synchronized dataset that comprised time-series data from our smart Knee Sleeves and corresponding ground truth labels from MoCap system. By utilizing these perception outcomes as guidance, our system was able to generate 3D human models solely based on the wearable sensor-integrative apparel readings of individuals performing diverse activities. We achieved an average RMSE of 7.21 degrees across eight joints in the lower body compared to commercially available MoCap tools. Our work offers a novel sensing modality that complements traditional vision systems and enables human pose estimation without being impacted by visual obstructions in a seamless and confidential manner. This innovation has potential applications from home fitness to sports analysis, personalized healthcare, and physical rehabilitation focusing on pose and movement estimation.
The smart Knee Sleeves and related app and software for data readout are provided by Texavie Technologies Inc. Texavie collects all wearable sensor data we analyzed in this paper. We express our gratitude to the volunteers who participated in the data collection experiment, as well as to the anonymous reviewers for their valuable comments and discussions. This work received partial support from the University of British Columbia. The opinions, findings, conclusions, and recommendations presented in this paper belong to the authors and do not necessarily represent the views of the funding agencies or the government.
[1] Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. Deep learning-based human pose estimation: A survey. arXiv preprint arXiv:2012.13392, 2020.
[2] Dangwei Li, Xiaotang Chen, Zhang Zhang, and Kaiqi Huang. Pose guided deep model for pedestrian attribute recognition in surveillance scenarios. In 2018 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2018.
[3] Amarjot Singh, Devendra Patil, and SN Omkar. Eye in the sky: Real-time drone surveillance system (dss) for violent individuals identification using scatternet hybrid deep learning network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1629-1637, 2018.
[4] Arvin Tashakori, Wenwen Zhang, Z Jane Wang, and Peyman Servati. Semipfl: personalized semi-supervised federated learning framework for edge intelligence. IEEE Internet of Things Journal, 2023.
[5] Zhihao Zhou, Sean Padgett, Zhixiang Cai, Giorgio Conta, Yufen Wu, Qiang He, Songlin Zhang, Chenchen Sun, Jun Liu, Endong Fan, et al. Single-layered ultra-soft washable smart textiles for all-around ballistocardiograph, respiration, and posture monitoring during sleep. Biosensors and Bioelectronics, 155:112064, 2020.
[6] Keyu Meng, Shenlong Zhao, Yihao Zhou, Yufen Wu, Songlin Zhang, Qiang He, Xue Wang, Zhihao Zhou, Wenjing Fan, Xulong Tan, et al. A wireless textile-based sensor system for self-powered personalized health care. Matter, 2(4):896-907, 2020.
[7] Md Atiqur Rahman Ahad, Anindya Das Antar, and Omar Shahid. Vision-based action understanding for assistive healthcare: A short review. In CVPR Workshops, pages 1-11, 2019.
[54] Tanvir Alam, Fadoua Saidane, Abdullah Al Faisal, Ashaduzzaman Khan, and Gaffar Hossain. Smart-textile strain sensor for human joint monitoring. Sensors and Actuators A: Physical, 341:113587, 2022.
[55] Subramanian Sundaram, Petr Kellnhofer, Yunzhu Li, Jun-Yan Zhu, Antonio Torralba, and Wojciech Matusik. Learning the signatures of the human grasp using a scalable tactile glove. Nature, 569(7758):698-702, 2019.
[56] Chengong Liu, Yilin Zhang, and Huanyu Zhou. A comprehensive study of bluetooth low energy. In Journal of Physics: Conference Series, volume 2093, page 012021. IOP Publishing, 2021.
[57] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[58] Du Q Huynh. Metrics for 3d rotations: Comparison and analysis. Journal of Mathematical Imaging and Vision, 35:155-164, 2009.