SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Post-Processing and Data Loader. We unify the coordinate system for the point cloud data from LiDAR and mmWave radar according to the right-hand rule. To ensure the synchronized frame for all modalities, we construct the frame of MM-Fi dataset with a uniform sampling rate of  $10\mathrm{Hz}$ . Figure 2 shows one MM-Fi data frame. All the data except the temporal segments are saved using the numpy array format, i.e., "numpy" files, while the temporal segments are saved using a single ".csv" file. The PyTorch data loader is provided to load one or multiple modalities conveniently.
Keypoints Quality To evaluate the quality of our 3D keypoints, we first re-project them onto 2D keypoints and then manually annotate 100 frames for each action category, resulting in a total of 2,700 video frames. We compute the re-projection error as the percentage of correctly located keypoints using a threshold of  $50\%$  of the head segment length (denoted as PCKh@0.5). The analysis reveals that the re-projection PCKh@0.5 is  $95.66\%$ , indicating the high quality of our annotation.
Intended Uses MM-Fi opens up a wide range of applications in human sensing tasks, encompassing both existing and novel areas. By harnessing diverse modalities, it facilitates robust 2D/3D human pose estimation (HPE) across various modalities, paving the way for multi-sensor HPE through effective multi-modal learning techniques. Moreover, due to the inclusion of multiple subjects and environments, MM-Fi empowers new sensing and recognition tasks in wireless sensing. It enables self-supervised multi-modal sensing, revolutionizing the way data is leveraged across different modalities without relying on explicit annotations. MM-Fi empowers cross-domain HPE by developing new domain adaptation and domain generalization techniques, contributing to pose estimation across different domains and environments. Lastly, it enables few-shot action recognition by providing a rich dataset encompassing various actions and scenarios, empowering efficient and accurate recognition of actions with limited training samples. The versatility and richness of MM-Fi make it a valuable resource for researchers and practitioners in the field of human sensing.
5 Benchmark and Evaluation
In this section, we introduce the benchmark setup, evaluation metrics, and baseline methods for single-modal and multi-modal human pose estimation based on the proposed MM-Fi dataset. The results are analyzed to show the merits and drawbacks of these modalities. We also include a benchmark on skeleton-based action recognition [38, 9] using our dataset as a supplement in the Appendix.
5.1 Benchmark Setup
Protocol. We provide three protocols that are tailored to the various scenarios, based on the categories in Section 4.2. Protocol 1 (P1) includes 14 daily activities that are performed freely in space, e.g., picking up things and raising arms. Protocol 2 (P2) includes 13 rehabilitation exercises that are performed in a fixed location, e.g., limb extension. Protocol 3 (P3) includes all 27 activities. By using these three protocols, we evaluate the performance of the models on both free and fixed actions.
Data Splits. To evaluate the model, we provide three data split strategies. Setting 1 (S1 Random Split) involves a random split of all video samples into training and testing sets, with a split ratio of 3:1. Setting 2 (S2 Cross-Subject Split) splits the data by subject with 32 subjects for training and 8 subjects for testing. Setting 3 (S3 Cross-Environment Split) randomly selects 3 environments for training and 1 environment for testing.
Evaluation Metrics. We assess the performance of the models using two widely-used metrics in human pose estimation: Mean Per Joint Position Error (MPJPE) and Procrustes Analysis MPJPE (PA-MPJPE) [16]. MPJPE measures the difference between the ground truth and prediction for all joints by Euclidean distance after aligning the pelvis of the estimated and true 3D pose. PA-MPJPE refers to the MPJPE after adopting the Procrustes method [55] for alignment, which conducts a similarity transformation including translation, rotation, and scaling before the MPJPE calculation. MPJPE measures both the quality of joints and the spatial position, while PA-MPJPE focuses on the quality of joints. We provide the mean and standard deviation of these metrics across 3 runs.
Baseline Methods. We evaluate the model of 3D human pose estimation using a single modality including RGB image (I), LiDAR (L), mmWave radar (R), and WiFi (W) using the recent models. These methods are briefly introduced as follows:
- RGB: We use the RGB-based pose estimation model from [29] that transforms a sequence of 2D keypoints into 3D pose using a convolutional neural network. In the benchmark, we directly use its pre-trained model and evaluate it on our test sets.
- LiDAR and mmWave: For mmWave radar, we adopt the data processing technique from [2] that aggregates the consecutive frames into one data sample, while we directly use the vanilla LiDAR point cloud at each frame. Then we upgrade the neural network in [2] using the PointTransformer [51]. The model is trained from scratch using our training split.
- WiFi: We employ the data processing and model from MetaFi++ [56] with a convolutional network and some transformer blocks trained to regress WiFi CSI data to a root joint. The model is trained from scratch on our dataset.
We evaluate the performance of combining multiple modalities for various scenarios, including a robotics setting  $(\mathrm{I} + \mathrm{L})$ , low-cost setting  $(\mathrm{R} + \mathrm{W})$ , privacy-preserved setting  $(\mathrm{R} + \mathrm{L} + \mathrm{W})$ , and all modality fusion  $(\mathrm{R} + \mathrm{L} + \mathrm{W} + \mathrm{I})$ . We adopt the simple result fusion method in the benchmark following the Least Mean Square (LMS) algorithm to learn the fusion weights of multiple sensors through linear combination [28] and leave space for future multi-modal fusion research. Instead of directly solving for the optimal weighting coefficients for each sensor, we reformulate the problem as a search for the optimal mean square estimation error, which allows for automatic parameter adjustment without the need for a correlation matrix.
5.2 Results and Analytics
Random Split Results (S1). Table 3 presents the evaluation results for human pose estimation using a single modality, including RGB, LiDAR, mmWave radar, and WiFi, under three settings and three protocols. Under random split (S1), LiDAR achieves 98.1, 94.9, and  $92.5\mathrm{mm}$  MPJPE for P1, P2, and P3, respectively, stably outperforming other modalities. The mmWave results are better than other modalities regarding the PA-MPJPE metric, achieving 55.6, 55.3, and  $57.3\mathrm{mm}$  PA-MPJPE for P1, P2, and P3, respectively. Due to the resolution limit of WiFi CSI, the results of WiFi modality are the worst. The RGB-based results are not satisfactory due to the domain gap, i.e., the distribution shift between the dataset for the pre-trained model and our dataset.
Cross-Subject Results (S2). In S2, the focus of the evaluation is on testing the model's robustness against subject differences. Results in Table 3 show that the LiDAR and mmWave radar models show good generalization ability under S2. The PA-MPJPE results of LiDAR and mmWave radar models only vary within  $3\mathrm{mm}$  when they are compared to the S1 for three protocols. Nevertheless, the 3D HPE model via WiFi CSI shows a significant decline in performance thanks to the limited resolution which fails to fully capture human subtle motions and differences, thereby restricting the model's generalization ability across different subjects.
Cross-Environment Results (S3). In the cross-environment setting, the performances of all modalities significantly deteriorate. It is observed that 3D HPE based on mmWave radar achieves 166.2, 168.0, and  $161.6\mathrm{mm}$  MPJPE for P1, P2, and P3, respectively, significantly outperforming other modalities. A likely reason is that mmWave point cloud reflects the moving objects in space [32] and thus it is the least affected. The LiDAR MPJPE and PA-MPJPE drop a lot because the LiDAR data also captures many points on the ground and nearby objects. The WiFi CSI reflects the multi-path effect of the propagated signals, so the environment changes lead to decreasing performance for WiFi HPE model [58, 49].
Protocol 1	Protocol 2	Protocol 3
Modality	Setting	MPJPE (mm)	PA-MPJPE (mm)	MPJPE (mm)	PA-MPJPE (mm)	MPJPE (mm)	PA-MPJPE (mm)
RGB	S1	263.3	80.0	291.6	83.6	279.0	81.2
S2	267.7	81.2	304.3	82.6	285.3	81.9
S3	276.6	83.0	301.5	85.3	288.6	84.1
LiDAR	S1	98.1±2.2	65.2±0.7	94.9±1.1	60.4±1.8	92.5±0.6	61.5±1.3
S2	110.1±2.9	66.2±1.2	103.9±0.6	60.3±2.3	103.8±1.5	61.5±1.2
S3	192.3±30.4	100.4±5.4	186.0±2.9	103.5±11.9	303.8±11.6	133.0±3.3
mmWave	S1	109.8±2.7	55.6±1.4	124.3±2.2	55.3±2.2	117.0±3.7	57.3±1.8
S2	128.4±6.9	58.7±4.3	142.2±0.6	57.4±2.3	129.7±2.2	60.0±1.7
S3	166.2±4.5	73.9±2.7	168.0±0.9	73.0±3.9	161.6±1.8	73.7±0.6
WiFi	S1	186.9±0.1	120.7±0.9	213.5±0.5	121.4±0.1	197.1±0.6	121.2±0.5
S2	222.3±0.8	125.4±2.3	247.0±0.2	122.7±1.3	231.1±0.4	124.0±0.5
S3	367.8±0.9	121.0±2.2	360.2±1.3	117.2±0.9	369.5±0.3	116.0±1.8
Table 3: 3D human pose estimation results for RGB, LiDAR, mmWave radar, and WiFi. The mean and standard deviation of MPJPE are reported under 3 settings and 3 protocols.
Protocol 1	Protocol 2	Protocol 3
Modalities	Setting	MPJPE (mm)	PA-MPJPE (mm)	MPJPE (mm)	PA-MPJPE (mm)	MPJPE (mm)	PA-MPJPE (mm)
I+L	S1	94.3±1.5	63.9±0.6	90.1±1.8	58.9±2.1	87.5±0.8	59.8±1.2
S2	109.4±1.6	65.8±0.6	103.5±0.6	60.2±2.4	104.6±0.6	62.1±0.3
S3	159.6±25.9	84.5±1.8	153.5±3.37	87.1±10.3	192.1±21.8	89.6±3.9
R+W	S1	94.2±3.7	55.2±6.3	92.8±13.2	51.8±4.2	113.2±4.2	75.6±2.8
S2	127.4±2.8	72.9±4.0	151.5±4.4	64.6±1.9	127.5±5.8	77.3±5.1
S3	104.9±9.3	47.6±2.4	110.5±10.1	54.2±1.7	116.1±7.8	57.2±4.9
R+L+W	S1	74.1±1.7	46.7±1.4	72.7±6.8	42.7±1.1	72.7±6.8	42.7±1.1
S2	111.9±3.6	83.0±3.6	106.5±1.1	63.5±2.2	99.6±1.5	72.0±1.5
S3	119.0±3.8	66.8±5.8	111.1±7.8	61.8±6.8	126.3±17.9	61.0±10.8
R+L+W+I	S1	72.9±1.0	47.7±1.2	69.5±3.4	43.1±1.9	89.8±2.0	63.2±1.9
S2	112.0±4.7	82.8±4.0	114.8±2.32	67.0±2.73	99.2±1.0	73.0±1.4
S3	98.1±10.4	58.3±8.8	92.7±11.65	65.0±7.59	116.5±23.6	70.6±13.5
Table 4: 3D human pose estimation results of multi-sensor fusion, where the modalities are denoted as I (Image), L (Lidar), R (mmWave Radar) and W (WiFi CSI).
Modality Fusion. We search for the optimal weights and obtain the improved multi-modal HPE results in Table 4 compared to the single-modal results in Table 3. For example, the fusion of RGB images and LiDAR (I+L) outperforms LiDAR significantly for MPJPE under three settings. For all the activities (P3), the best PA-MPJPE results come from the modality fusion of  $\mathrm{R + L + W}$ , achieving 42.7, 72.0, and  $61.0\mathrm{mm}$  for S1, S2, and S3 respectively. This demonstrates that in real-world scenarios, fusing multiple modalities in some ways for 3D HPE can be more robust and accurate.
6 Limitations of MM-Fi
The MM-Fi dataset currently has limitations regarding annotations and benchmarks. Firstly, the annotation process is manual and the quality is limited. Due to the resolution of wireless sensing data, the current sensing tasks are restricted to activity level and keypoint level, which have been validated with high quality. However, for tasks that require higher resolution, such as dense pose estimation, we only provide annotations obtained from algorithms that have not been validated. These annotations are included to facilitate new tasks and inspire further research. In the upcoming MM-Fi V2.0, we plan to use a motion capture system to annotate the dense pose to address this limitation. Secondly, as the first dataset to offer mmWave radar, LiDAR, RGBD, and WiFi data simultaneously, there are tasks that have not been extensively studied yet. Therefore, some baseline methods developed by us may not perform optimally without careful design. Our intention is to inspire researchers to explore these unexplored fields and contribute to the future benchmarking of the MM-Fi dataset. Thirdly, the current dataset is collected in a controlled condition, i.e., 3m away and same facing direction, with a single person. We plan to include multi-orientation, multi-location, and multi-user scenarios in MM-Fi 2.0. We acknowledge these limitations and aim to improve the dataset by addressing them in future versions. Our objective is to provide a comprehensive resource that encourages research and advancements in the non-intrusive wireless sensing research community.
In this paper, we present MM-Fi, a novel non-intrusive multi-modal 4D human pose dataset for wireless human sensing. It consists of four sensing modalities: RGB-D, LiDAR, mmWave radar, and WiFi, and is currently the most comprehensive benchmark for wireless human pose estimation. Our MM-Fi is composed of  $320.76k$  synchronized frames, and 27 categories of poses performed by 40 subjects. The collection process and the four data modalities are introduced, followed by extensive experiments to produce a single-modal and multi-modal benchmark. Our proposed dataset contributes to the multi-modal human pose estimation research and helps researchers choose suitable sensor modalities according to their advantages and disadvantages. We hope that our proposed MM-Fi dataset and benchmarks can facilitate future research in machine learning, ubiquitous computing, computer vision, mobile computing, and healthcare.
Acknowledgment. This work is supported by NTU Presidential Postdoctoral Fellowship, "Adaptive Multimodal Learning for Robust Sensing and Recognition in Smart Cities" project fund, at Nanyang Technological University, Singapore.
[1] An, S., Li, Y., Ogras, U.: mri: Multi-modal 3d human pose estimation dataset using mmwave, rgb-d, and inertial sensors. arXiv preprint arXiv:2210.08394 (2022)
[2] An, S., Ogras, U.Y.: Mars: mmwave-based assistive rehabilitation system for smart healthcare. ACM Transactions on Embedded Computing Systems (TECS) 20(5s), 1-22 (2021)
[3] An, S., Ogras, U.Y.: Fast and scalable human pose estimation using mmwave point cloud. arXiv preprint arXiv:2205.00097 (2022)
[4] Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: New benchmark and state of the art analysis. In: Proceedings of the IEEE Conference on computer Vision and Pattern Recognition. pp. 3686-3693 (2014)
[5] Antunes, J., Bernardino, A., Smailagic, A., Siewiorek, D.P.: Aha-3d: A labelled dataset for senior fitness exercise recognition and segmentation from 3d skeletal data. In: BMVC. p. 332 (2018)
[6] Ar, I., Akgul, Y.S.: A computerized recognition system for the home-based physiotherapy exercises using an rgbd camera. IEEE Transactions on Neural Systems and Rehabilitation Engineering 22(6), 1160-1171 (2014)
[7] Cai, Z., Ren, D., Zeng, A., Lin, Z., Yu, T., Wang, W., Fan, X., Gao, Y., Yu, Y., Pan, L., et al.: Humman: Multi-modal 4d human dataset for versatile sensing and modeling. In: European Conference on Computer Vision. pp. 557-577. Springer (2022)
[8] Chen, A., Wang, X., Zhu, S., Li, Y., Chen, J., Ye, Q.: mmbody benchmark: 3d body reconstruction dataset and analysis for millimeter wave radar. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 3501-3510 (2022)
[9] Chen, Y., Zhang, Z., Yuan, C., Li, B., Deng, Y., Hu, W.: Channel-wise topology refinement graph convolution for skeleton-based action recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 13359-13368 (2021)
[10] Cong, P., Xu, Y., Ren, Y., Zhang, J., Xu, L., Wang, J., Yu, J., Ma, Y.: Weakly supervised 3d multi-person pose estimation for large-scale scenes based on monocular camera and single lidar. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 461-469 (2023)
[11] Dai, Y., Lin, Y., Lin, X., Wen, C., Xu, L., Yi, H., Shen, S., Ma, Y., Wang, C.: Sloper4d: A scene-aware dataset for global 4d human pose estimation in urban environments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 682-692 (2023)
[12] Ding, G., Sener, F., Yao, A.: Temporal action segmentation: An analysis of modern technique. arXiv preprint arXiv:2210.10352 (2022)
[13] Geng, J., Huang, D., De la Torre, F.: Densesepose from wifi. arXiv preprint arXiv:2301.00250 (2022)
[14] Ghorbani, S., Mahdaviani, K., Thaler, A., Kording, K., Cook, D.J., Blohm, G., Troje, N.F.: Movi: A large multi-purpose human motion and video dataset. Plos one 16(6), e0253157 (2021)
[53] Zhao, M., Tian, Y., Zhao, H., Alsheikh, M.A., Li, T., Hristov, R., Kabelac, Z., Katabi, D., Torralba, A.: Rf-based 3d skeletons. In: Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication. pp. 267-281 (2018)
[54] Zheng, J., Shi, X., Gorban, A., Mao, J., Song, Y., Qi, C.R., Liu, T., Chari, V., Cornman, A., Zhou, Y., et al.: Multi-modal 3d human pose estimation with 2d weak supervision in autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4478-4487 (2022)
[55] Zhou, X., Zhu, M., Leonardos, S., Daniilidis, K.: Sparse representation for 3d shape estimation: A convex relaxation approach. IEEE transactions on pattern analysis and machine intelligence 39(8), 1648-1661 (2016)
[56] Zhou, Y., Huang, H., Yuan, S., Zou, H., Xie, L., Yang, J.: MetaFi++: Wifi-enabled transformer-based human pose estimation for metaverse avatar simulation. IEEE Internet of Things Journal (2023)
[57] Zou, H., Yang, J., Prasanna Das, H., Liu, H., Zhou, Y., Spanos, C.J.: Wifi and vision multimodal learning for accurate and robust device-free human activity recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 0-0 (2019)
[58] Zou, H., Yang, J., Zhou, Y., Xie, L., Spanos, C.J.: Robust wifi-enabled device-free gesture recognition via unsupervised adversarial domain adaptation. In: 2018 27th International Conference on Computer Communication and Networks (ICCCN). pp. 1-8. IEEE (2018)