SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As mentioned earlier, the stacking choices for these tasks are different. With the present framework, the stack will pile up contiguous frames both before and after the current frame. Since our frame rate is 24 frames per second, the action classification task naturally requires a higher stacking value  $s$ .
The results in Figure 4 demonstrate that there is a plateau effect, indicating that when the stacking number  $s$  reaches a certain limit, it ceases to contribute to the network's final performance. As indicated in Figure 4, we found that PointNet++ performs the best when  $s = 5$  and  $s = 50$  on identification and action classification, respectively. Following a few manual experiments, we found that  $s = 5$  produces optimal results for both keypoint estimation and identification tasks, while  $s = 50$  is superior for action classification. It is worth noting that higher  $s$  values require more computing and memory resources when training the network, so we summarise all the 'turning point' for the plateaus and report them in Table 4 in our Appendix.
In this paper, we introduce a novel mmWave radar dataset composed of three distinct tasks related to human activity sensing. Our high-quality dataset is expected to be a valuable training and evaluation resource for further research into point-based deep learning methods. We hope it will prove to be an
(a) Keypoint estimation
(b) Action classification
Figure 4: An illustration of the plateau effect with stacking more frames  $(s)$ . We show this effect on two tasks, keypoint estimation (measured in MLE loss) and action classification (measured in Top1 accuracy). We generally pick the turning point as the optimal stacking, as this offers a good balance between performance and run-time efficiency. The detail is discussed in Section 4.2.
instrumental asset to the field. However, in Section 5.2 we would like to comprehensively address the limitations and discuss potential future work in Section 5.2 that can resolve these limitations.
5.1 Why mmWave Sensing?
We have briefly explained the particular advantages of using mmWave radar in Section 1 and a table to compare different sensors in Table 1. We detail this comparison here by comparing it to camera based systems and lidar based systems.
When compared to camera-based or infrared systems, mmWave and lidar technology provide an attractive solution due to their non-intrusiveness and robustness under varying lighting and atmospheric conditions. The non-intrusive nature of these sensors provides a greater guarantee for user privacy. Atmospheric conditions, such as dust, smoke, and fog, present a formidable obstacle to visual sensors such as cameras. To contend with these issues, lidar and mmWave technologies provide a reliable solution [13] – explaining why lidar has become also a popular choice for autonomous driving applications.
mmWave radar is an attractive option because it is relatively low-cost and is able to fit within smallform devices. Moreover, with regard to resolution, mmWave radar provides a higher quality of resolution compared with other options such as microwave radar for the same range.
5.2 Limitations
A major limitation of our dataset is that the data collection experiment was conducted using only one mmWave radar, whereas in reality, multiple radars can be implemented for the same task [6]. On the other hand, introducing additional radars into the data collection process would bring significant complications. The relative positions and angles between the radars can have a significant impact on the sensing quality, but also there is a risk that the radars may potentially interfere with one another.
The concern of interference naturally brings up another issue: our data collection is predominantly conducted in a relatively stable indoor environment. It is entirely possible that the outside world may contain more complex scenarios which can produce signals that significantly interfere with our sensor signal, thus compromising the quality of the sensing.
Another major limitation is that we only consider a limited range of human movements, primarily those that focus on the limbs, such as hands and legs. As a result, we do not capture more complex postures that a human might take, such as sitting or lying down.
Another particular problem with radar sensing is the multi-path effect, where multiple reflection paths of the RF signal cause noises and ghost targets in radar imaging. However, this issue is less significant in the mmWave frequency band when compared with the traditional UWB bands, making mmWave less sensitive to location or distance changes given that all the experiments are conducted in a clear line of sight and without any neighboring clutters. Meanwhile, the question of how to mitigate
the multipath effect in a complex and diverse environment is left as future work, as this on its own can be a huge research topic.
Finally, the radar we used has three transmitters and four receivers that were originally designed for automotive driving applications, which has more azimuth antennas than elevation ones. This can potentially result in poor elevation resolution and affect the performance of certain actions when the height information is critical. Nevertheless, the sensor presently employed is the most widespread within the sector, in other words, it can be treated as a standardised sensor currently in this domain. A variation in the number of transmitters and receivers or their relative positioning necessitates rigorous cooperation and re-engineering on the device side, which is beyond the scope of this paper.
5.3 Future Work
One research direction is using the raw IF signal as the dataset input (See Figure 1). While the point cloud is an effective spatial representation of the subject motion, it is a high-level data representation derived from the IF signal, where a large proportion of information may have been discarded. This also brings up the research question that whether the data processing chain in Figure 1 is optimal or other signal processing techniques, like Capon beamforming rather than angle-FFT, can increase the accuracy of the radar point cloud and, hence, the performance of the proposed tasks. However, capturing and processing the IF signal requires a significantly higher data bandwidth and computation resources, and, therefore, is left as future work.
Another area of potential future research involves utilising multiple radars for estimating human activity. Such a cooperative system would enable the collection of more comprehensive data; however, it is also sensitive to relative positions of each radar, creating the potential for interference resulting from the mmWave transmissions. The dataset currently focuses on a single-radar case, so the implications of a multi-radar system are left as an area for future exploration.
In this paper, we introduce MiliPoint, a dataset designed to systematically evaluate the performance of DNNs for point-based mmWave radar. The goal of MiliPoint is to bridge the gap between the accessible mmWave sensor and various downstream tasks, by providing a diverse yet systematic mmWave radar dataset.
MiliPoint is the largest mmWave radar dataset assessed to date in terms of the number of frames collected, and it holds three primary downstream tasks: identification, action classification and keypoint estimation, with a diverse set of associated actions labelled. With this dataset, the research community can delve deeper into applying deep learning to advance the function of mmWave radars.
[21] W. Shi and R. Rajkumar. Point-gnn: Graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1711-1719, 2020.
[22] A. D. Singh, S. S. Sandha, L. Garcia, and M. Srivastava. Radhar: Human activity recognition from point clouds generated through a millimeter-wave radar. In Proceedings of the 3rd ACM Workshop on Millimeter-Wave Networks and Sensing Systems, 2019.
[23] H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pages 945-953, 2015.
[24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[25] L. Wang, Y. Huang, Y. Hou, S. Zhang, and J. Shan. Graph attention convolution for point cloud semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10296-10305, 2019.
[26] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[27] C. Wu and H. Aghajan. Model-based human posture estimation for gesture analysis in an opportunistic fusion smart camera network. In 2007 IEEE Conference on Advanced Video and Signal Based Surveillance, pages 453-458. IEEE, 2007.
[28] H. Xie, H. Yao, X. Sun, S. Zhou, and S. Zhang. Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2690-2698, 2019.
[29] H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259-16268, 2021.
[30] P. Zhao, C. X. Lu, B. Wang, N. Trigoni, and A. Markham. Cubelearn: End-to-end learning for human motion recognition from raw mmwave radar signals. IEEE Internet of Things Journal, 2023.
[31] P. Zhao, C. X. Lu, J. Wang, C. Chen, W. Wang, N. Trigoni, and A. Markham. mid: Tracking and identifying people with millimeter wave radar. In 2019 15th International Conference on Distributed Computing in Sensor Systems (DCOSS), pages 33-40, 2019.
Figure 5: Location heatmap of the participants during the data collection.
A Dataset characteristics
We show the statistics with the datasets used in this section. In Table 4, we demonstrate the used stacking number  $s$  and label domain. Table 5 and Table 6 demonstrate the label statistics of each dataset respectively.
We instructed all participants to face the radar direction and carry out specific actions within a pre-defined area measuring  $1\mathrm{m} \times 1\mathrm{m}$ , located approximately  $0.65\mathrm{m}$  from the radar. The instructional video provided for the participants was a cardio workout, ensuring a reasonable variation of movements across the entire  $1\mathrm{m} \times 1\mathrm{m}$  space. These movements included adjustments (relatively small) in position, such as stepping leftwards, rightwards, forwards, or backwards. Essentially, our dataset incorporates slight variations in positions while keeping the direction or angular changes fixed. We hypothesized that increasing the angular and positioning variations would require gathering significantly more (or maybe orders of magnitude more) data to achieve the same level of accuracy, and thus is left as future work. In order to provide a glimpse of the diversity captured in our collected dataset, we have created a 'heatmap' visualization in Figure 5. This heatmap encompasses all the data points obtained for each of the 18 human keypoints in our samples.
Table 4: Suggested stacking number and label domain for each task in MiliPoint.
Identification	Action Recognition	Keypoint Detection
9 point	18 point
Suggested s	5	50	5	5
Label Domain	R11	R39	R9×3	R18×3
B Stacking styles
Section 3.4 explained why we set an upper limit  $k$  to the mmWave radar sensor: the number of points in each data packet depends on the scene and can vary from a few points to a few hundreds. There are obviously two padding strategies we can follow:
- Pad each data packet to a fixed size and stack them (This is what we have used)
- Stack each data packet to a batch, and pad the whole batch to a fixed size.
Table 5: Label statistics for the identification task.
Label ID	Number of Samples
0	91,566
1	91,369
2	61,879
3	29,456
4	61,752
5	32,054
6	24,921
7	28,943
8	60,799
9	31,911
10	30,409
Total	545,059
Table 6: Label statistics for the action classification task.
Label ID	Number of Samples	Label ID	Number of Samples	Label ID	Number of Samples
0	4,986	17	4,870	34	4,284
1	5,119	18	5,096	35	4,292
2	5,174	19	5,117	36	4,337
3	5,122	20	4,602	37	4,577
4	5,150	21	4,299	38	4,580
5	5,148	22	4,282	39	4,578
6	5,118	23	4,241	40	4,291
7	5,129	24	4,558	41	4,284
8	5,145	25	4,525	42	2,176
9	5,088	26	4,299	43	2,234
10	5,149	27	4,556	44	2,232
11	5,142	28	4,535	45	2,038
12	5,172	29	4,337	46	2,015
13	5,153	30	4,580	47	1,951
14	4,893	31	4,321	48	968
15	5,135	32	4,346
16	5,117	33	4,579
Total	212,920
Table 7: Different padding styles, tested with PointNet++ on the identification task, batch size is 128 and  $s = 5$ .
Pad per data packet	Pad per batch
87.30%	72.31%
We experimented the two different padding strategies in Table 7, and our results suggest that padding per data packet shows a significantly better performance. Intuitively, padding at a per data packet level provides a better data alignment for the DNN to deal with.
C Instruction and Compensation to Participants
Each participant was given an information sheet that explains the purpose of this research, what tasks they were required to complete, what data would be collected and how they would be processed. We specifically emphasized that all data would be anonymized and it would not be possible to identify the participants from the data. Participants were then requested to sign a consent form for using and
publishing the data for research purposes. A copy of the information sheet and the consent form are attached as supplementary materials. Each participant was given a £10 Amazon voucher for entering the study. The study was approved by the Faculty of Engineering Research Ethics Committee at the University of Bristol (ethics approval reference code 12802).
D Dataset Documentation, Intended Uses and Maintenance
Our dataset is publicly available on GitHub (https://github.com/yizzfz/MiliPoint/) with raw data on Google drive (https://drive.google.com/file/d/1rq8yyokrNhAQryx7trpUqKenDnTI6Ky/), where we not only provide the dataset, but also baseline point-based methods and accompanying training and evaluation code for full reproducibility. To further increase accessibility, we have also open-sourced a number of pre-trained baseline models. We created a detailed Readme file to facilitate user onboarding and to guide users through the entire flow, and through running each model with varying downstream tasks. We are responsible for upkeep and maintenance, mainly through GitHub Issues. An MIT license is used for the dataset, as one can check on the GitHub project. We declare that we bear all responsibility in case of violation of rights, and have participants signed the consent forms before conducting this experiment, as explained in Section 3.1.
Table 8: Open source assets used for the MiliPoint benchmarks and the corresponding licenses.
Name	License	Link
PyTorch	Modified BSD	https://github.com/pytorch/pytorch
PyTorch Geometric	MIT	https://github.com/pyg-team/pytorch-geometric
PointMLP	Apache 2.0	https://github.com/ma-xu/pointMLP-pytorch
The dataset itself does not rely on any existing code assets, but to demonstrate the dataset and the benchmarks, we used several open source projects as listed in Table 8.