MiliPoint: A Point Cloud Dataset for mmWave Radar
Authors: Han Cui
Shu Zhong
Jiacheng Wu
Zichao Shen
Naim Dahnoun
Yiren Zhao
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: mmWave radar
point cloud
keypoint estimation
identification
action classification
human activity recognition
mmWave dataset
TL;DR: Our work, termed MiliPoint, provides a large-scale, open dataset for the community to explore how mmWave radars can be utilised for human activity recognition.

Abstract: Millimetre-wave (mmWave) radar has emerged as an attractive and cost-effective alternative for human activity sensing compared to traditional camera-based systems. mmWave radars are also non-intrusive, providing better protection for user privacy. However, as a Radio Frequency based technology, mmWave radars rely on capturing reflected signals from objects, making them more prone to noise compared to cameras. This raises an intriguing question for the deep learning community: Can we develop more effective point set-based deep learning methods for such attractive sensors? 
  
To answer this question, our work, termed MiliPoint, delves into this idea by providing a large-scale, open dataset for the community to explore how mmWave radars can be utilised for human activity recognition. Moreover, MiliPoint stands out as it is larger in size than existing datasets, has more diverse human actions represented, and encompasses all three key tasks in human activity recognition. We have also established a range of point-based deep neural networks such as DGCNN, PointNet++ and PointTransformer, on MiliPoint, which can serve to set the ground baseline for further development.

MiliPoint: A Point Cloud Dataset for mm Wave Radar
Han Cui Shu Zhong Jiacheng Wu 1 Zichao Shen ', Naim Dahnoun Yiren Zhao
School of Computer Science, Electrical and Electronic Engineering, and Engineering Maths, University of Bristol
DDeparteent of Computer Science, University College London 'Department Electrical and Electronic Engineering, Imperial College London
Abstract
Millimetre- (mmWave) radar has emerged attractive and cost-effective alternative for human activity sensing compared to traditional camera-based sys- tems. mm Wave radars also non-intrusive, providing better protection for user privacy. However, as Radio Frequency (RF) based technology, mm Wave radars rely on capturing reflected signals from objects, making them more prone to noise compared to cameras. This raises an intriguing question for the deep learning community: Can we develop more effective point set-based deep learning methods for such attractive sensors?
To answer this question, our work, termed MiliPoin delves into this idea by providing a large-scale, open dataset for the community to explore how mm Wave radars can be utilised for human activity recognition. Moreover, MiliPoint stands out as larger in size than existing datasets, has more diverse human actions represented, and encompasses all three key tasks in human activity recognition. We have also established a range of point- point-based deep neural networks such as DGCNN, PointNet++ + and Transrrannfrmerr MiliPoint, which can serve to set the ground baseline for further development.
Introduction
In modern systems, sensors play vital role in allowing intelligent decision- making 13 5
Millimetre- Wave radar (mm radar) is often employed in automotive, industrial and civil applica- tions. This type of sensor particularly advantageous it offers good balance between resolution, accuracy, and cost 7 15]. In this work, we focus on exploring the potential of mm Wave radars as sensors for human activity sensing. Despite the high accuracy of camera-based systems demonstrated for various tasks in this domain [27]3] their intrusive nature has raised considerable concerns in terms of user privacy. The utilization of Radio-Frequency (RF) signals for human activity analysis presents an attractive alternative due to their non-intrusive nature.
When compared with traditional low frequency RF sensors, like WiFi and Bluetooth, mm Wave radars can utilize a much higher bandwidth and achieve a finer resolution. Together with the multiple- tinpiplee-nput multiple-output (MIMO) technique, mm wave radars advanced human activity recognition tasks to be performed. Meanwhile, the short wavelength of mm Wave signals facilitates the development of a small-factor and low-cost sensor. However, as a RF-based technique, mm Wave radars rely on the reflected signal phase from an object to detect its spatial feature, which be prone noise and is less accurate than cameras and lidars. A comparison between mm Wave radars and other commonly seen sensors is shown in Table [ As shown, mm Wave radar cost-effective, non-intrusive sensing solution that can be advantageously used in various sensing scenarios.
as 3D imaging sensors and enable
Work done while the author was at University of Bristol. Available at htttp://giihub..com/iizzzzz
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Table A comparison of different sensors, mm Wave radar is cost-effective, non-intrusive sensor compared to other solutions.
Sensor type
3D camera
Lidar
Traditional RF
mm Wave Radar
Cost
Medium
High
Low
Low
Intrusiveness
High
Medium
Low
Low
Resolution
High
High
Low
Medium
Viewing condition requirement
High
Medium
Low
Low
Researchers have demonstrated the effectiveness of mm Wave radar in many human activity sensing tasks. However, the varying operation conditions and task specifications of radar-based human pose estimation make comparisons between existing methods and evaluations of their generalizability challenging. For instance, single person identification is the focus of Zhao et al. [31], while Pegoraro et al. [16] show mm Wave radars can concurrently identify up to three people. Sengupta et al. [20] concentrate on differentiating human arm motions with fixed-location subjects. whereas An al. [1] cover 12 actions which showcase variety of human postures, and the number of samples can span from a few thousand approximately 160k. terms of hardware, single-chip GHz radar with an integral transmitter and receiver is used in various research [i];; nevertheless, two radars 6 2|20 or GHz radar with separate transmitters and receivers (1O) are also evaluated by researchers. Furthermore, parameters like the radar chirp configuration, which can have major impact on the detection result, have been neglected by many existing studies.
This study presents the development of MiliPoint, a standardised dataset, designed for the facilitation of future research in this domain, enabling researchers to make cross-comparisons in a uniformed framework. In this paper, we make the following contributions:
We introduce the MiliPoint dataset, which includes three main tasks in human activity recognition: identification, action classification and keypoint estimation.
MiliPoint offers more comprehensive view of human motion than existing datasets, featuring 49 distinct actions 4..08x more than the most action-diverse dataset and 545K frames of data, 3.26 greater than the largest dataset in existence.
We implemented and tested the performance of existing point-based DNNNs MiliPoint, and found that action classification is a particular challenging task, compared to identity classification and keypoint estimation.
2
Related Work
We begin by introducing the mechanics of millimeter wave sensing in Section 2.1 Section 2.2 surveys existing mmWave datasets and elucidates how MiliPoint differs from them. Following this, Section 2.3 outlines the popular deep neural network (DNN) models proposed for 3D point sets.
2.1 Millimeter Wave Sensing
A mm Wave signal refers to electromagnetic signal between 30 GHz to 300 GHz that has wavelength of sub 1 cm. Signals at this frequency band can have a much larger bandwidth (a few gigahertz) than the traditional RF signals, which make them very suitable for short-range radar applications as the resolution the short wavelength allows many antennas to be integrated into a single small-factor platform, enabling it to determine the angle-of-incident of the signal refection and depict the 3D spatial feature of the scene. Although it is less accurate than 3D cameras and lidars, mmwave radars still offer several distinct advantages such as cost-efffetiveensss, non-intrusiveness, and lack of reliance on various viewing conditions. All these features give mmwave radar increased popularity in human activity sensing.
radar is directly determined by its signal bandwidth. Meanwhile,
mmWave radars often use frequency modulated continuous wave (FMCW) to detect objects in the scene. [ the workflow of a typical mm Wave Radar. The radar transmits millimeter wave signals. The object in front of the sensor then reflects the signal back and the signal is picked

---

Rodler
HTT dor wehdttp
Thoree Sier
TX
Magi Er
Fifte
elpertin
coondrutes
Raciener
Fot signll
Moral SP
PPy Cie
Figure I: An illustration of how typical maweve Radar work. The radar has several transmitters (TX) and Receivers (RX) for transmitting and collecting the reflected signals. These signals are then mixed and filtered to form an Intermediate Frequency (IF) signal. Subsequent to this, three Fast Fourier Transforms (FFTs) are implemented on the range, velocity, and angle domains. A Constant False Alarm Rate (CFAR) algorithm is also utilized to detect potential peaks from the FFT outputs. Eventually, the (x, y: 2) coordinates of the objects in the metric space are acquired.
Table 2: A comparison to existing mm datasets (A=Action classification, I=Identification, K=Keypoint estimation). Our dataset, Milipoint, is far more diverse in both tasks and actions, and also has much larger dataset size.
Dataset
Task
Participants
Dataset size
Action involved
mmPose 20
K
15k
MARS 12
K
4
40k
10
HuPR 12
141k
mRI 1
K
20
160k
12
CubeLearn [30]
A
1k
RadHAR [22]
A
2
167k
5
MiliPoint
A.I.K
11
5451
49
up by the receiver. The distance and angle of the object would be encoded in the frequency and phase of the reflected signal. Following this, the on-chip data processing unit mixes and applies a low pass filter to the signal produce Intermediate Frequency (IF) signal. Two Fast Fourier Transforms (FFTs) are then applied on this mixed signal, before a Constant False Alarm Rate algorithm used for peak detection. This, together with the FFT for the angle, provides the user with the data packet that contains the 3D coordinates of the object in the scene.
2.2 Existing mm Wave Datasets
Although many mm Wave radar frameworks have been proposed in the human activity recognition literature, only a few researchers have released their datasets publicly. These are summarized in Table 2 Existing datasets focus primarily
single task, with majority being devoted to keypoint estimation. Meanwhile, CubeLearn (30] and RadHAR 22 are two datasets specifically designed for action classification. Previous datasets have limited the number of frames collected, with the largest datasets, mRI [ and RadHAR [22). containing meagre 160K frames. Additionally, the range of human actions included is not extensive, with the greatest total being 12 in the mRI dataset [.
Our work the first mave dataset that includes all three main tasks in human activity recognition: identification, action classification, and keypoint estimation. It also fills critical gap in terms of size and diversity, with 11 participants performing a total of 49 different actions across 545k frames. This provides a more comprehensive picture of human movements than has ever before been possible for mm Wave radar sensing.
2.3 Point-based Neural Networks
Point clouds, composed of 3D points representing an object's shape, are commonly used in computer graphies and 3D sensing [19 Graph neural networks (GNNs) process point clouds directly as individual points, rather than voxels 4 or multi-view images [3 28 The unordered point
TTe action dataset has 213k filtered frames after excluding warm-up time and breaks.

---

Receivers
Transmittors
Front view
(b) Back view
(c) Radar board
Figure 2: Front and back view of the data collection setup. (1) mm Wave radar, (2) 7ed 2 stereo camera, (3) a monitor displaying movement for the participant to follow, and (4) is the designated area for the participant to stand. An overview of the mmWave Radar board is shown in (c).
sets can be treated as nodes and used inputs for machine learning system. PointNet [17 and PointNet++ [18], which use sampling to reduce high-dimensional unordered points in the metric space into fixed-length feature vectors, and Deep Neural Networks (DNNs) to process these features. Given
is a natural abstraction to view set of points as graph 25 25 26), DGCNN employs EdgeConv to derive local neighbourhood information, which then be stacked to comprehend global features [26]. With the increased use of self attention modules for Natural Language Processing [24). Zhao et al. applied this computation pattern point clouds in their method, Point Transformer 29 Ma et al. shed a new perspective on this problem; rather than constructing a complex network architecture, they developed simple residual Melti-Laayer-Pecepttite requires no intricate local geometrical extractors yet yields very competitive results [14. In this work, we create benchmark utilising several representative networks including PointNet++ and PointMLP (purely point-based), DGCNN (GNN based), and Poin(Transformer (Transformer-basee))
(MLP) network- termed PointMLP that
3 Dataset
In this section, we provide a detailed overview the dataset collection and construction process. Section 3.1 outlines the data collection measures from the participants, Section 3.3 discusses the associated tasks and their respective specifications and Section 3.4 3.4eexlainns the data processing.
3.1 Data Collection
We conducted an in-person data collection, when participants were asked to perform series of low-intensity cardio-burning fitness movements f The exercise video was chosen with meticulous consideration given to factors such intensity, diversity of movements, and movement speed. The video lasts around 30 minutes with 49 different actions; each action lasts 30 seconds with a 10 seconds kept anonymous to protect their privacy, and our released
break in between. The participants data consists purely of point clouds from our nave sensor and ground truth keypoints, with no imagery contents. The information captured by the camera is used calculate the keypoints, and the original video is immediately discarded to ensure the continued protection of participant privacy.
We present the physical data collection setup in Figure which shows how the mm Wave radar, Zed 2 Stereo camera, and monitor are assembled to form the setup. The human participants are instructed to stand in front of both the mm nave and stereo camera sensors, and follow the movements displayed on the monitor. The mm Wave radar is connected to the power and its output data is then transmitted through serial port our work station. The stereo camera is placed behind the mm Wave radar, but configured to be at a different height. This setup has been verified to yield a high quality streams of frames.

---

As illustrated in Figure we use an on-the-fly data processing approach with the mm Wave radar chip to obtain data packets at our workstation. These data packets contain information about the tarassee
points (Ir, U. 2). which are represented by dataset d E RR where N indicates the number of points. We used the 'TI IWR 1843 mm mawe radar, a commercial off-the-shelf radar that has received great popularity among researchers due to its 3D imaging capability and processing power. The radar operates between 77 GHz to 81 GHz, has three transmitters and four receivers that operate in a time-division multiplexing mode, and has on-chip DSP processor that applies the described data processing and outputs point clouds to the workstation. The radar was configured to have a chirp time of 100 us and chirp slope of 40 MHz/us, /us, to utilize the full 4 GHz available bandwidth and achieve a range resolution of 4 cm. The ADC sampling rate was set to 5 MHz. The CFAR threshold was empirically set to dB in both the range and Doppler direction, which gives reasonable number of points per frame in our experimental environment.
We utilized the Zed 2 Stereo Camera for producing ground truth data on the keypoint estimation task. The stereo camera calculates the disparity between two views to give a depth map of the scene, and applies a posture estimating neural network to get 3D skeleton models of people in the scene. Given the camera parameters, the 3D coordinates of the skeleton with respect to the camera can be calculated through simple trigonometry.
The Zed Camera System offers an impressive depth accuracy of less than 1% up to 3 meters and less than 5% up to 15 meters. While high-end industrial level optical tracking systems, such as the OpticTrack system and their Motion Capture Suits, may provide a more precise baseline, we found that the Zed 2 Camera already offers a very strong performance.
Figure Suune@shhone shows the exact experimental setup for the data collection. A mm Wave radar (1) is placed
in front of the participant's 's designated area (4) and behind the radar is monitor displaying the
movement for the participant to follow (3), and a Zed 2 Stereo camera (2). The area (4) is set to 1 m by 1 m. The distances from the radar and camera to the area centre are 0.65 m and 3 m, and the heights are m and 0.7 m, respectively. The positions are chosen avoid occluding as much as possible. During data collection, the radar data and camera images are timestamped and synchronized based on their time-of-and ofrivvl to the workstation, at 24 frames per second. After data acquisition, the camera data is calibrated to the radar coordinate system to serve as the ground truth.
3.2 Participant Recruitment
A total of 11 participants were recruited through university emailing lists and word-of-mouth with 4 females and 7 males. The average height and weight of the participants were 171.84 84 cm + 10.41 and 67.73 kg = 13.08, respectively. All participants had none mobility impairments. All participants were given information explaining the nature and purpose of the procedures involved in this study and signed a consent form before starting the experiment. The study was approved by the Faculty of Engineering Research Ethics Committee, University of Bristol
3.3 Tasks
The next step after data collection is to design tasks. In this case, three tasks are established, which are: identification, keypoint estimation, and action classification; an overview is presented in Figure B
The process of identification involves analyzing the collected data in order to discriminate between unique individuals. This requires making comparisons between various characteristics. In doing so, the DNN model is expected to be capable of recognizing specific traits that associated with particular individuals. In our identification task, the output labels are numerical numbers ranging from 0 to 10 which correspond to the 11 unique participants.
Action classification requires the recognition of behaviour patterns. Our raw data gathered by mmWave radar can be broken down into sets of frames, each of which is annotated with an action, that is detailed in Appendix. This segmentation of data greatly facilitates the recognition of actions.
Finally, keypoint estimation involves detecting interest points or key locations in the input data, which typically involve identifying various keypoint landmarks in a human body. The detection labels each individual image's 's points according to their position, size, and orientation, allowing for the development of a better understanding of human posture from the input data. We designed two tasks for keypoint estimation with varying levels of difficulty. The first task requires detection of 9
5

---

Radar point cloud
Walding Arm swing Jmb extersion,
Alice, Bob,
Action
Identification
Keypoint estimation
classification
Figure 3: The three tasks: identification, keypoint estimation, and action classification. We show the raw radar point cloud the first row and expected predictions on the second row.
keypoints from the human body, including 'Right Shoulder', 'Right Elbow", 'Leff Shoulder', 'Left Elbow', 'Right Hip', 'Right Knee', 'Left Hip', 'Left Knee* and 'Head'. The second task presents challenge by requiring detection of additional keypoints, namely 'Nose", 'Neck', 'Left Wrist', 'Left Ankle', 'Left Eye', 'Left Ear', 'Right Wrist', 'Right Ankle', 'Right Eye' and 'Right Ear'. Notably, 'Head' is excluded due to the finer granularity of facial keypoints.
3.4 Data Processing Pipeline
The mm Wave radar produces data packets in the form of point clouds that encode the spatial shape of the subject. The number of points in each data packets depends on the scene and can vary from few points to a few hundred. The number of points at each frame is not constant since it depends on the instantaneous signal reflection from the subject. To make the input size consistent across frames, set an upper limit to the point cloud population in each packet. Point clouds with more points will be randomly sampled
and point clouds with less points will be zero-padded. This is equivalent to a data frame d E ø To create a single data point, we then stack consecutive frames, forming data point d E Rexkx3
We process the collected data for each participant, thus providing labels for the identification task. The ground truth for both keypoint estimation tasks is derived from the Zed 2 detection results, which serves as the reference for the mm Wave radar sensor. The action labels at each timestamp are derived from the video content and synchronized the collected data, the participants were instructed to always follow the action in the video. We also manually scrutinized and discarded incorrect labels when the participants failed to follow the video.
Evaluation
We first explain our setup in Section Section 4.2 | shows how various point-based DNN models perform MiliPoint and Section 4.2 explains how important hyperparameter, the number of stacking, is picked for each task MiliPoint.
4.1 Experiment Setup
To assess the usability of our dataset, we ran several representative point-based deep neural networks (DNNs) with split of 80%, 10% and 10% for training, validation, and testing partitions, respectively. All the models shown in the evaluation are implemented in Pytorch and Pytorch Geometric [8]. These models trained with mainly two hardware systems. System one has NVIDIA RTX2080TI cards, where system two has 2 NVIDIA RTX3090TI cards. Running all networks on all downstream tasks cost around 300 GPU hours.
The Adam optimizer
is used together with a Conealnn learning rate scheduling 19, and the learning rate is set to -5 Each data point is run three times with different random seeds to

---

Table 3: Accuracy (Ace f) and mean localization error (MLE values for different point-based
DNN methods running on our MiliPoint dataset. Iden, Action and Keypoint mean Identification, Action classification and Keypoint estimation respectively.
Action (Ace% t)
Keypoint (MLE cm {)
Model
Iden (Acc% 1)
Top1
Top3
point
18 point
Random
7.69
2.59
7.69
155.74 + 1.32
161.64 + 2.11
DGCNN
77.65 + 0.92
13.61 + 2.09
34.59 + 2.74
16.53 + 0.11
18.51 + 0.03
Pointformer
83.94 + 0.81
29.27
0.55
50.44
1.18
14.99 + 0.03
17.03 0.13
+ointNet++
87.30 + 0.27
34.45 + 0.80
54.96 1.21
13.55 + 0.03
14.94 + 0.03
PointMLP
95.88 + 0.40
18.37 = 0.08
3.194
13.12 = 0.30
14.11 0.22
calculate its average and standard deviation values. We set the stacking to for identification and keypoint estimation, but 50 for action classification. We futher justify hyperparameter choices in Section 4.2and and also our Appendix.
4.2 Results
We present the results of different point-based methods on the MiliPoint in Table A row labelled Random is also included to show the random guess accuracy for the various classification tasks. is noteworthy that the keypoint estimation is evaluated by means of Euclidean distances the ground truth, and thus a lower value signifies better performance. The Random results for identification and action classification are calculated from the number of labels. For keypoint estimation, we employ models using randomised weights and record their results across three distinct random seeds.
We report Topl accuracy for identification, both Topl and Top3 accuracy for action classification, and mean localization error (MLE) for keypoint estimation. We evaluate four different point- point-based DNN methods, namely DGCNN (26] Pointformer 21 PointNet++ [8] and PointMLP 29
The results presented in Table indicate that point-based methods can perform quite effectively for identity classification, achieving an accuracy of greater than 75% across all DNNs evaluated. Conversely, action classification appears to be much more challenging, with the highest accuracy recorded being below 40%. Action classification is challenging task, it requires a construction of semantic meaning from a sequence of frames, and this is especially challenging when the point cloud data is sparse and noisy. We chose to stack 50 frames for this task, since an action typically takes one to two seconds, and our frame rate is 24 frames per second. is apparent that certain point-basee based methods perform better than others; this is evident in Table where PointNet++ and PointMLP have outperformed the other methods in the MiliPoint benchmark.
As mentioned earlier, the stacking choices for these tasks are different. With the present framework, the stack will pile up contiguous frames both before and after the current frame. Since our frame rate is 24 frames per second, the action classification task naturally requires higher stacking value
The results in Figure 4 demonstrate that there is plateau effect, indicating that when the stacking number 8 reaches a certain limit, it ceases to contribute to the network's 's final performance. As indicated in Figure 4 found that PointNet++ performs the best when = and 50 identification and action classification, respectively. Following a few manual experiments, we found that 5 produces optimal results for both keypoint estimation and identification tasks, while 50 is superior for action classification. It is worth noting that higher s values require more computing and memory resources when training the network, so we summarise all the 'turning point' for the plateaus and report them in Table 4 in our Appendix.
5 Discussion
In this paper, we introduce a novel mmWave radar dataset composed of three distinct tasks related to human activity sensing. Our high-quality dataset expected be a valuable training and evaluation resource for further research into point-based deep learning methods. We hope it will prove to be an
7

---

Keypaint Estimation using +ointNet+++
Action classification using PointNet++
20
(a) Keypoint estimation
(b) Action classification
Figure 4: An illustration of the plateau effeet with stacking more frames (s). We show this effect on two tasks, keypoint estimation (measured in MLE loss) and action classification (measured in Topl accuracy). We generally pick the turning point as the optimal stacking, as this offers a good balance between performance and run-time efficiency. The detail is discussed in Section 4.2
instrumental asset to the field. However, in Section we would like to comprehensively address the limitations and discuss potential future work in Section 5.2 that can resolve these limitations.
5.1 Why mm Wave Sensing?
We have briefly explained the particular advantages of using mawavv radar in Section | and table to compare different sensors in Table [ based systems and lidar based systems.
We detail this comparison here by comparing it to camera
When compared to camera-based or infrared systems, mm Wave and lidar technology provide an attractive solution due to their ion-inrrusiveness and robustness under varying lighting and atmo- spheric conditions. The non-intrusive nature these sensors provides greater guarantee for user privacy. Atmospheric conditions, such as dust, smoke, and fog, present formidable obstacle to visual sensors such as cameras. To contend with these issues, lidar and mm nave technologies provide reliable solution [ explaining why lidar has become also popular choice for autonomous driving applications.
mm Wave radar is an attractive option because it is relatively low-cost and is able to fit within small- form devices. Moreover, with regard resolution, mm Wave radar provides a higher quality of resolution compared with other options such as microwave radar for the same range.
5.2 Limitations
major limitation of our dataset is that the data collection experiment was conducted using only one mmWave radar, whereas in reality, multiple radars can be implemented for the same task [6]. On the other hand, introducing additional radars into the data collection process would bring significant complications. The relative positions and angles between the radars can have a significant impact on the sensing quality, but also there is risk that the radars may potentially interfere with one another.
The concern of interference naturally brings up another issue: our data collection is predominantly conducted in relatively stable indoor environment. It is entirely possible that the outside world may contain more complex scenarios which can produce signals that significantly interfere with our sensor signal, thus compromising the quality of the sensing.
Another major limitation is that we only consider a limited range of human movements, primarily those that focus the limbs, such hands and legs. As result, we do not capture more complex postures that a human might take, such as sitting or lying down.
Another particular problem with radar sensing is the multi-pate effect, where multiple reflection paths of the RF signal cause noises and ghost targets radar imaging. However, this issue less significant in the mm Wave frequency band when compared with the traditional UWB bands, making mm Wave less sensitive location or distance changes given that the experiments are conducted in clear line of sight and without any neighboring clutters. Mcanwhile, the question of how to mitigate

---

the multipath effect in a complex and diverse environment is left as future work, as this on its own can be huge research topic.
Finally, the radar we used has three transmitters and four receivers that were originally designed for automotive driving applications, which has azimuth antennas than elevation ones. This can potentially result in poor elevation resolution and affect the performance of certain actions when the height information is critical. Nevertheless, the sensor presently employed the within the sector, in other words, it can be treated as standardised sensor currently in this domain. A variation the number of transmitters and receivers or their relative positioning necessitates rigorous cooperation and re re-engineering on the device side, which is beyond the scope of this paper.
widespread
5.3 Future Work
One research direction is using the raw IF signal the dataset input (See Figure TT. While the point cloud is an effective spatial representation of the subject motion, it is a high-level data representation derived from the IF signal, where large proportion of information may have been discarded. This also brings up the research question that whether the data processing chain in Figure T is optimal or other signal processing techniques, like Capon beamforming rather than angle-FFT, can increase the accuracy of the radar point cloud and, hence, the performance of the proposed tasks. However, capturing and processing the IF signal requires significantly higher data bandwidth and computation resources, and, therefore, is left as future work.
Another area of potential future research involves utilising multiple radars for estimating human activity. Such a cooperative system would enable the collection of more comprehensive data; however, it is also sensitive to relative positions of each radar, creating the potential for interference resulting from the mm Wave transmissions. The dataset currently focuses on single-radar case, so the implications of multi-radar system are left as area for future exploration.
6 Conclusion
In this paper, we introduce MiliPoint, a dataset designed to systematically evaluate the performance of DNNs for point-based mm Wave radar. The goal of MiliPoint is bridge the gap between the accessible mm Wave sensor and various downstream tasks, by providing diverse yet systematic mmWave radar dataset.
MiliPoint is the largest mm Wave radar dataset assessed to date in terms of the number of frames collected, and it holds three primary downstream tasks: identification, action classification and keypoint estimation, with diverse set of associated actions labelled. With this dataset, the research community can delve deeper into applying deep learning to advance the function of mm Wave radars.

---

References
[1] An, Y. Li, and U. Ogras. mRI: Multi-modal 3d human pose estimation dataset using minwave, RGB-d, and inertial sensors. In shith Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[2] S. An and U. Y. Ogras. Mars: Mawave-hased assistive rehabilitation system for smart healthcare. ACM Trans. Embed. Comput. Syst., 20(5s), sep 2021.
[3] Biswas and Veloso. Depth camera based indoor mobile robot localization and navigation. In 2012 IEEE International Conference on Robotics and Automation, pages 1697-1702. IEEE,
2012.
[4] C. Choy, J. Gwak, and Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1075-3084, 2019.
[5] H. Cui and N. Dahnoun. High precision human detection and tracking using millimeter-wave radars. IEEE Aerospace and Electronic Systems Magazine, 36(1):22-32, 2021.
[6] H. Cui and N. Dahnoun. Real-time short-range human posture estimation using mmwave radars and neural networks. IEEE Sensors Journal, 22(1):535-543, 2022.
[7] S. H. Dokhanchi, B. S. Mysore, K. V. Mishra, and B. Ottersten. A mmwave automotive joint cadar-communications system. IEEE Transactions on Aerospace and Electronic Systems, 55(3):1241-1266 2019.
[8] M. Fey and J. E. Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arxiv: /903.02422 2019.
[9] A. Gotmare, N. S. Keskar, C. Xiong, and R. Socher. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv preprint arXiv: 130.133,1 2018.
[10] Gu, Z. Fang, Z. Yang, P. Hu, and P. Mohapatra. Mmsense: Multi-person detection and identification via mmwave sensing. In Proceedings of the 3rd ACM Workshop on Millimeter- Wave Networks and Sensing Systems, mmNets 19, page 45-50, New York, NY, USA, 2019. Association for Computing Machinery.
[11] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.69880 2014.
[12] S.-P. Lee, N. P. Kini, W.-H. Peng, C.-W. Ma, and J.-N. Hwang. Hupr: A benchmark for human pose estimation using millimeter wave radar. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 5715-5724, January 2023.
[13] Y. Li and J. trends for automotive lidar and perception systems. IEEE Signal Prucessing Magazine, 37(4):50- 61, 2020.
Lidar for autonomous driving: The principles, challenges, and
[14] X. Ma, C. Qin, H. You, H. Ran, and Y. Fu. Rethinking network design and local geometry in point cloud: A simple residual mlp framework. arXiv preprint arXiv:2202.077 022122,, 2022.
[15] W. Menzel. Millimeter-wave radar for civil applications. In The 7th European Radar Conference, pages 89-92. IEEE, 2010.
[16] Pegoraro and M. Rossi. Real-time people tracking and identification from sparse mm-wave radar point-clouds, 2021.
[17] C. R. Qi, H. Su, K. Mo, and L. J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660- 2017.
[18) C. R. Qi. L. Yi, H. Su, and L. Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30, 2017.
[19] R. B. Rusu and S. Cousins. 3d is here:
Point cloud library (pcl). In 2011 IEEE international conference on roboties and automation, pages 1-4. IEEE, 2011.
[20] A. Sengupta and S. Cao. mmpose-nlp: A natural language processing approach to precise skeletal pose estimation using mmwave radars, 2021.
10

---

[21] W. Shi and R. Rajkumar. Point-gnn: Graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1711-1719, 2020.
[22] A. D. Singh, S. S. Sandha, L. Garcia, and M. Srivastava. Radhar:
Human activity recognition from point clouds generated through millimeter-wave radar. In Proceedings of the 3rd ACM Workshop on Millimeter-Wan Wave Networks and Sensing Systems, 2019.
[23] H. Su, S. Maji, E. Kalogerakis, and E. Learned--Min Millerr. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference computer vision, pages 945-953, 2015.
[24] A. Vaswani, Shazeer, N. Parmar, Uszkoreit, L., Jones, N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[25] L. Wang, Y. Huang, Y. Hou, S. Zhang, and J. Shan. Graph attention convolution for point cloud semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10296-10305, 2019.
[26] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[27] C. Wu and H. Aghajan. Model-based human posture estimation for gesture analysis in an opportunistic fusion smart camera network. In 2007 IEEE Conference on Advanced Video and Signal Based Surveillance, pages 453-458. IEEE, 2007.
[28] H. Xie, H. Yao, X. Sun, S. Zhou, and S. Zhang. Pix2vox: Context-aware 3d reconstruction from single and vuet images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2690-2699 2019.
[29] H. Zhao, L. Jiang, J. Jia, H. Torr, and V. Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259-16268, 2021.
[30] Zhao, C. X. Lu, B. Wang, N. Trigoni, and A. Markham. Cubelearn: End-to-end learning for human motion recognition from raw mmwave radar signals. IEEE Internet of Things Journal, 2023.
[31] Zhao, X. Lu, J. Wang. Chen, W. Wang, Trigoni, and Markham, mid: Tracking and identifying people with millimeter wave radar. In 2019 15th International Conference Distributed Computing in Sensor Systems (DCOSS), pages 33-40, 2019.

---

1.00
0.75
0.50
0.25
0.00
-0.25
-0.50
0.75
-1.00
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Figure 5: Location heatmap of the participants during the data collection.
A Dataset characteristics
We show the statistics with the datasets used this section. In Table 4 stacking number and label domain. Table 5 and Table demonstrate the label statistics of each dataset respectively.
we demonstrate the used
We instructed all participants to face the radar direction and carry out specific actions within pre-defined area measuring Im x 1,, located approximately 0.65m from the radar. The instruc- tional video provided for the participants was cardio workout, ensuring a reasonable variation of movements across the entire 1m x space. These movements included adjustments (relatively small) in position, such as stepping leftwards, rightwards, forwards, or backwards. Essentially, our dataset incorporates slight variations in positions while keeping the direction or angular changes fixed. We hypothesized that increasing the angular and positioning variations would require gathering significantly more (or maybe orders of magnitude more) data to achieve the same level of accuracy, and thus is left as future work. In order to provide glimpse of the diversity captured in collected dataset, we have created a 'heatmap' visualization Figure [ This heatmap encompasses all the data points obtained for each of the 18 human keypoints in our samples.
Table 4: Suggested stacking number and label domain for each task in MiliPoint.
Keypoint Detection
Identification
Action Recognition
9 point 18 point
Suggested s Label Domain
50
R
39
9x3
118x
B Stacking styles
Section 3.4
explained why we set an upper limit & to the mm mave radar sensor: the number of points in each data packet depends on the scene and can vary from few points to few hundreds. There are obvisouly two padding strategies we can follow:
Pad each data packet to fixed size and stack them (This is what we have used)
Stack each data packet to a batch, and pad the whole batch to fixed size.
12

---

Table 5: Label statisties for the identification task.
Label ID
Number of Samples
91,566
91,369
61,879
29,456
61,752
32,054
24,921
28,943
60,799
31,911
30,409
Total
545,059
Table 6: Label statistics for the action classification task.
Label ID
Number of Samples
Label
Number of Samples
Label ID
Number of Samples
0
4,986
4,870
34
4,284
1
5,119
18
5,096
35
4,292
2
5,174
19
5,117
36
4,337
3
5,122
20
4,602
37
4,577
4
5,150
21
4,299
38
4,580
5
5.148
22
4,282
39
4,578
6
5,118
23
4,241
40
4,291
7
5.129
24
4,558
41
4,284
8
5,145
25
4,525
42
2,176
9
5,088
26
4,299
2,234
10
5,149
27
4,556
44
2,232
11
5,142
28
4,535
2,038
12
5,172
29
4,337
46
2,015
13
5,153
4,580
1,951
14
4,893
31
4,321
48
968
15
5.135
32
4,346
16
5,117
33
4,579
Total
212,920
Table 7: Different padding styles, tested with PointNet++ the identification task, batch size is 128
and 5.
Pad per data packet
Pad per batch
87.30%
72.31% %
We experimented the two different padding strategies in Table and our results suggest that padding per data packet shows a significantly better performance. Intuitively, padding at a per data packet level provides a better data alignment for the DNN to deal with.
C Instruction and Compensation to Participants
Each participant was given an information sheet that explains the purpose of this research, what tasks they were required to complete, what data would be collected and how they would be processed. We specifically emphasized that data would be anonymized and it would not be possible identify the participants from the data. Participants were then requested to sign consent form for using and
13

---

publishing the data for research purposes. A copy of the information sheet and the consent form are attached as supplementary materials. Each participant was given f10 Amazon voucher for entering the study. The study was approved by the Faculty of Engineering Research Ethics Committee at the University of Bristol (ethics approval reference code 12802).
D Dataset Documentation, Intended Uses and Maintenance
with raw data where we not only pro- vide the dataset, but also baseline point-based methods and accompanying training and evaluation code for full reprodueibility. To further increase accessibility, we have also open-sourced a number of pre-trained baseline models. We created detailed Readme file to facilitate user onboarding and to guide users through the entire flow, and through running each model with varying downstream tasks. We responsible for upkeep and maintenance, mainly through GitHub Issues. An MIT license is used for the dataset, as one can check on the GitHub project. We declare that we bear all responsibility in case violation rights, and have participants signed the consent forms before conducting this experiment, as explained in Section
Our dataset is publicly available on GitHub http:p///gsttu..ooo////////////s
Google drive (attpe://árive google. con/file/ /t/il//////////// http://wwwyy.ssraarrrrrrrrrrrrrr
Table 8: Open assets used for the MiliPoint benchmarks and the corresponding licenses.
Name
License
Link
PyTorch
Modified BSD
http:://gtthuu..cco/pppttdrdrrrr
PyTorch Geometric
MIT Apache 2.0
PointMLP
The dataset itself does not rely on any existing code assets, but to demonstrate the dataset and the benchmarks, used several open source projects listed in Table
14


Reviews:
[{"review_id": "MQ9vaybnAD", "reviewer": "Reviewer_UdR9", "summary": "", "questions": "", "limitations": "1. The dataset's collection environment lacks sufficient diversity.\n2. In the evaluation, the performance of action classification was not particularly good, but the authors did not analyze or provide an explanation for it.\n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. This paper introduces a large-scale dataset aimed at systematically evaluating the performance of point-based Deep Neural Networks (DNN) for millimeter-wave radar applications.\n2. The dataset comprises diverse data and labels, encompassing various downstream tasks.\n3. The authors utilized this dataset to implement and evaluate various DNN models.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "jWSnZqKI4g", "reviewer": "Reviewer_k6RA", "summary": "", "questions": "", "limitations": "The authors provide enough analysis on limitations  and negative societal impact of the work. Their self-analysis is adequate and can be informative to readers.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset is unique and large in size, which can provide the wider research community a chance to explore new ways to utilize mmWave radar. The baseline models can also server as good starting point for other researchers to compare the performance. The uniformed data collection setup can make life easier for trouble shooting and compare performance on different activities. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "zkJ52cbt8z", "reviewer": "Reviewer_hrVa", "summary": "", "questions": "", "limitations": "- multi-device (and multi-modal) datasets are more useful for ML research. The authors commented that multiple radars could interfere. However, this reviewer believes interference can be mitigated using careful designs, e.g., different bands (60GHz & 77GHz) or via frequency multiplexing.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- dataset size is good\n\n- baselines are representative of relevant methods from literature", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "9w3YZWKJRh", "reviewer": "Reviewer_zF3S", "summary": "", "questions": "", "limitations": "They carefully discussed the limitations of their work and provide a plan for managing the dataset.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The new mmWave radar dataset outperforms existing radar datasets by including a diverse range of participants, a wide variety of actions, and various tasks. The study shows that mmWave signals can effectively recognize human actions without visual sensors, offering privacy-preserving solutions for action recognition in sensitive environments. \n2. Benchmarking results: The metrics and used deep learning models are appropriate for showing their contribution. Also, despite complex actions, the model trained with the proposed dataset can be used to precisely estimate human actions.\n3. The authors have generously included a helpful guide for data loaders. They have thoughtfully provided a code skeleton tailored for PyTorch users, making it easier for researchers and practitioners to load and utilize the dataset effectively.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "OFLW2yQbl9", "reviewer": "Reviewer_Q9DZ", "summary": "", "questions": "", "limitations": "Noise and interference: mmWave radar signals are susceptible to noise and interference from various sources, such as electromagnetic interference and multipath effects, which can degrade data quality and introduce inaccuracies in activity recognition. However, in the proposed dataset by the authors, the inclusion of such noise cases is not observed.\n\nLack of standardized annotation: Annotating mmWave radar-based human activity datasets can be subjective and challenging, as there is often a lack of standardized annotation protocols and guidelines. This can lead to inconsistencies and potential bias in the labeling process. In the authors' work, per limb or per body-based annotation is not present.\n\nLimited sensor coverage: mmWave radar sensors have a limited field of view and can only capture activities within their specific coverage area. This limitation can result in incomplete or partial observations of human activities, leading to potential gaps in the dataset. In this work, the author only utilized one radar in a perfect view condition, further limiting the sensor coverage.\n\nComplexity of data interpretation: Interpreting mmWave radar signals and extracting meaningful features for human activity recognition can be complex. It requires specific signal processing and feature extraction techniques tailored to mmWave radar data, adding an additional layer of complexity to the analysis process. Unfortunately, in this work, the authors did not showcase raw or interpreted mmWave radar perception images.", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "Larger dataset size: The new dataset is significantly larger, providing a more extensive collection of data for analysis.\n\nIncreased test subjects: The dataset incorporates a greater number of test subjects, enhancing diversity and representation.\n\nMulti-modality: The dataset encompasses multiple modalities, allowing for a more comprehensive understanding of the subject matter.\n\nIntegration of different tasks: Unlike previous datasets, this new dataset combines various tasks into a single comprehensive dataset, facilitating holistic analysis and exploration.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper contributes a dataset of Millimetre-wave (mmWave) radar point cloud data for benchmarking human activity detection inside interior environments using such mmWave radar sensors.  The reviewer responses were mostly positive, recognizing the value and relative scarcity of such data, and the technical soundness of the presented results.  The one negative reviewer expressed concerns regarding the clarity of the writing and loss of some useful data due to post-processing.  Similarly, some of the positive reviewers saw weaknesses in the lack of true "raw" radar data.  The author rebuttal alleviated some concerns expressed by the reviewers and led to upgrading of scores.  The initially negative reviewer did not express an updated opinion or oppose the paper being accepted.  The AC thus finds no basis to overrule the majority reviewer opinion and recommends that the paper be accepted.
Decision: Accept (Poster)