MVDoppler: Unleashing the Power of Multi-View Doppler for MicroMotion-based Gait Classification
Authors: Soheil Hor
Shubo Yang
Jae-Ho Choi
Amin Arbabian
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: micro-Doppler analysis
Gait analysis
multi-view sensor Fusion
Doppler-based Perception
mmWave Sensing
TL;DR: A novel dataset and corresponding baseline models for multi-view Doppler-based gait classification

Abstract: Modern perception systems rely heavily on high-resolution cameras, LiDARs, and advanced deep neural networks, enabling exceptional performance across various applications. However, these optical systems predominantly depend on geometric features and shapes of objects, which can be challenging to capture in long-range perception applications. To overcome this limitation, alternative approaches such as Doppler-based perception using high-resolution radars have been proposed. 
Doppler-based systems are capable of measuring micro-motions of targets remotely and with very high precision. When compared to geometric features, the resolution of micro-motion features exhibits significantly greater resilience to the influence of distance. However, the true potential of Doppler-based perception has yet to be fully realized due to several factors. These include the unintuitive nature of Doppler signals, the limited availability of public Doppler datasets, and the current datasets' inability to capture the specific co-factors that are unique to Doppler-based perception, such as the effect of the radar's observation angle and the target's motion trajectory.
This paper introduces a new large multi-view Doppler dataset together with baseline perception models for micro-motion-based gait analysis and classification. The dataset captures the impact of the subject's walking trajectory and radar's observation angle on the classification performance. Additionally, baseline multi-view data fusion techniques are provided to mitigate these effects. This work demonstrates that sub-second micro-motion snapshots can be sufficient for reliable detection of hand movement patterns and even changes in a pedestrian's walking behavior when distracted by their phone. Overall, this research not only showcases the potential of Doppler-based perception, but also offers valuable solutions to tackle its fundamental challenges.

MVDoppler: Unleashing the Power of Multi-Vien Doppler for MicroMotion-Basel Gait Classification
Soheil Hor, Shubo Yang, Jaeho Choi, and Amin Arbabian
Department of Electrical Engineering
Stanford University
Stanford, CA 94305
(soheilh 1h, sauuh,,hhbuy... jaabbo,,joochoiiarbbbabo @stanford edu
Project page: httpps://mvdopplerr. github.
Code and dataset toolbox:
Abstract
Modern perception systems rely heavily on high-resolution cameras, LiDARs, and advanced deep neural networks, enabling exceptional performance across various applications. However, these optical systems predominantly depend on geometric features and shapes of objects, which can be challenging to capture in long-range perception applications. To overcome this limitation, alternative approaches such Doppler-based perception using high-resolution radars have been proposed. Doppler-bas -baser systems are capable of measuring micro-motions of targets remotely and with very high precision. When compared geometrie features, the resolution of micro-motion features exhibits significantly greater resilience to the influence of distance. However, the potential of Doppler-based perception has yet be fully realized due to several factors. These include the unintuitive nature of Doppler signals. the limited availability of public Doppler datasets, and the current datasets' inability to capture the specific co-factors that are unique to Doppler- based perception, such as the effect of the radar's observation angle and the target's s motion trajectory. This paper introduces a new large multi-view Doppler dataset together with baseline perception models for micro-motion- based gait analysis and classification. The dataset captures the impact of the subject's 's walking trajectory and radar's observation angle on the classification performance. Additionally, baseline multi-view data fusion techniques provided to mitigate these effects. This work demonstrates that sub-second micro-motion snapshots can be sufficient for reliable detection of hand movement patterns and even changes in pedestrian's 's walking behavior when distracted by their phone. Overall, this research not only showcases the potential of Doppler-bsss based perception, but also offers valuable solutions to tackle its fundamental challenges.
Introduction
Today's perception systems typically benefit from the integration of high-resolution optical sensors and advanced deep neural networks surpassing even human perception in various applications. However, such systems also suffer from inherent limitations of optical system design and the physics of energy propagation in 3D space, which cannot be solved even using the most advanced perception neural networks. As example, camera-based perception systems typically rely extracting geometrical (spatial) features of the environment with the effective observed resolution decreasing quadratically a function of distance. This results in significantly lower performance in long- range perception applications. Alternatively, there exist perception approaches that do not rely on geometrical features and therefore offer enhanced resilience against iettance-relttd information loss.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

One such approach is Doppler-based perception, enabled by the new generation of commercially available, high-resolution, mm-Wave coherent radar systems. Unlike camera- based systems that primarily capture an object's s shape, location, and color information (such as the shape and pose of a human body, the color of a car, or the dimensions of a helicopter), Doppler-hased systems detect the motions exhibited by subjects and their components (such as the motion of human hands or legs during walking, the movements of a car's wheels, or the rotation a helicopter's 's rotors).
One application of micro-Doppler analysis methods that has been less investigated is using automotive radar sensors, already present in cars, for predictive accident prevention. Such approaches involve extracting high-level information that can potentially be used to predict pedestrians' traffic behaviour and their interactions with other pedestrians and cars on the road. Examples include, analyzing a pedestrians' hand or leg micro-movr movement patterns to determine whether the pedestrian is accompanied by a child [1] or, as investigated in this paper for the first time, detecting pedestrians distracted while texting on their phones. While Doppler-based perception is relatively new compared to vision-based systems, it offers fundamentally novel approach to perception and activity recognition, particularly in long-range applications.
Although there are several advantages to Doppler-based perception, also comes with its set of unique challenges, which have limited its real-world applications. One challenge is that Doppler measurements are "directional", meaning that the micro-motien information captured by a Doppler sensor depends on the radar's observation angle (a function of subject's 's relative location to the radar) and the relative direction of the movement of the subject. This challenge is well-known in the radar community, but as it will be shown in the next section, is rarely addressed or mitigated in the literature. Another significant challenge in Doppler-based perception that micro-motion data is not human-readable. This is in contrast to optical sensors that generate 2D or 3D geometrical (spatial) intuitive representations. This has made it impractical to design hand-crafted micro-motion features, especially for more complex perception tasks. Deep learning methods have addressed this issue by eliminating the need for hand-engineered features, but they require access to large and representative datasets. Publicly available Doppler datasets today are typically very limited in size, and more importantly, lack diversity in capturing co-factors unique to Doppler- based perception, such as the effect of the relative location and trajectory of subjects discussed earlier.
In this paper, we release one of the largest public multi-view micro-Doppler datasets to date, along with single-radar and multi-view baseline perception models. Our work showcases the potential of off-the-shelff mm- Wave radars for complex Doppler-basee perception tasks. In specific, this paper's contributions are summarized as follows:
The first multi-view Doppler dataset for micro-motion-bee gait classification
The published dataset is one of the largest public micro-Dopple datasets in context of human gait analysis and the only multi-view micro-Doppler dataset publicly available in this context to date.
Baseline neural networks for real-time hand movement and distracted pedestrian detection We introduce baseline neural networks for a standard task of hand movement detection, and task never explored before: distracted pedestrian detection. The baseline models provided in this paper make predictions in less than second. This highlights the potential of such methods for virtually real-time Doppler analysis applications.
A new benchmark for the effect of location and walking trajectories on gait Doppler signatures The published dataset is unparalleled in capturing the effect of different relative locations and walking directions on the walking Doppler signature of subjects. Combining the multi-view aspect of the published dataset and its comprehensive coverage of different locations and walking trajectories, also provide the first maajectory-agnostic micro-Doppler based classifier in the context of gait Mhcro-Doppler analysis.
2
Relevant Work
2.1 Gait miero-Doppler Datasets
The overview of relevant gait miero-Doppler datasets is shown in Table One main factor separating different datasets in the context of gait classification is the definition of classes. In most Doppler datasets today, the gait classes are limited to simple actions (e-g.. fast vs slow walking, or crawling vs jumping) which can be casily classified using the macro-motions of the body (mainly torso). In

---

contrast, hand micro-motion detection is a much harder task, especially in presence of other sources of micro-motions, like movement the legs while walking. However, most existing works focus on detecting hand while standing 2 k 4 Our work is one of the few public datasets that include the non-trivial task of hand movement detection for a walking subject, and the first dataset to investigate changes in the walking pattern of pedestrians while distracted by their phones.
Another factor highlighting this proposed dataset the classification latency of the baseline methods. While most papers aim to classify motion snapshots of around three seconds [S] 2 4 we aim to achieve real-time Doppler- based perception by reducing the size of the micro-motion snapshots to less than one human walking cycle 1000m))0.ms.....
Moreover, trajectory and location coverage often overlooked in existing datasets while Doppler measurements are very sensitive on radar's observation angle. There are three critical factors to consider in this context: range coverage, azimuth coverage, and the coverage of participants motion trajectories. Nevertheless, none the existing datasets have considered these three aspects simultancously. Vishwakarma et al. [2] select Region of Interest (Rol) spanning 1-10m in range, but instead of complete coverage of different ranges, they measure random walking patterns of the participants within the Rol. Chakraborty et al. proposed the dataset containing activities with discrete operation ranges and motion directions (i.e., 0º, + 1300 :1450) from the radar. The dataset presented in this paper enjoys a comprehensive coverage of all of these factors, providing a new benchmark for examining the effect of location and walking trajectories on gait Doppler signatures.
Table 1: Comparison of public gait micro- -Dopper datasets.
Location and Trajectory Coverage
Hand Micro-motion
Classification Snapshot longth
Classification Latency
Participants Number
Dataset
Paper
Across Range
Across Azimuth
Motion Direction
Gurbuz
38
Fast
Vishwakarma
x
Ws
2.78
Fast
/
M
Yang et al.
WS
Gambi al. x
WW
16s
Slow
V
L
Bhavanas ct al. ?
3,78
Fast
R
R
L
Chakraborty et
3s
Fast
MVDoppler (Ours)
WW
0.64s
R
M
LB
WS: While Standing
WW: While Walking
not specified Realtime: <ls
Slow: >5s
Fast: 1-5s
Full coverage
S: <10 people <10,000s
M: 10-20 people 10,000-20,000s
L >00 people >00,000s
2.2 Multi-Radar Fusion
As mentioned earlier, Doppler-based perception using radar is dependent on the subject's relative location and movement direction. One approach to deal with this issue is to fuse information from multiple radars. Multi-radar fusion techniques can typically be categorized into two approaches: fusion of co-located perpendicular radars for improved angular resolution in both elevation and azimuth dimensions
[ 19, and fusion of spatially distributed radars with the goal of achieving
better coverage over the Rol [10 I 12 I3 [4]. Our work falls under the later group of spatially distributed radars. Early approaches demonstrated the efficacy of fusion from distributed radars at the simulation level, particularly in terms of through-the-we wall sensing [10, [1] and ghost mitigation
[2]. More recently, the focus has been shifted towards woal-world multi-radar setups, with [3 integrating signals from two off-the-shelf radars based on graph matching techniques, while [T5 leveraged ensemble learning for improved activity perception within a small Rol. Building upon prior work, this study proposes multi-view approach to Doppler-based gait analysis. We have specifically designed a multi-radar setup that not only eliminates the blind spots of each radar but also ensures
'MVDoppler includes over 10.5 hours (37,800 seconds) of simultaneous data capture from two radars with complementary views resulting in over 54,000 multi-view snapshots. For single-radar classification tasks, this hodar-hours or over 108,000 single-radar recorded snapshots. 'Latency in the context of gait analysis is typically determined by the minimum observation window size required by the perception model to make an accurate classification.
translates to an equivalent
3

---

that the information from both radars is complementary with the goal of achieving truly trajectory and location-age -aniosoii sensing system. To the best of our knowledge, this goal has not been achieved by any single-radar multi-radar system to date.
3 Dataset
3.1 Classes
To showcase the power of Doppler-based perception for real-time analysis of complex pedestrian gait patterns, we defined four classes and two tasks as shown in Figure [[a. A total of 13 volunteers were recruited, All providing informed consent per IRB guidelines and agreeing to have their data included in a publicly available datase The four classes defined in this study include normal walking ("Normal"), phone call ("Phone Call"), hands in pockets ("Pockets")) and texting ("Texting") Each class represents common activity undertaken by pedestrians in typical traffic scenario. Normal class intended to capture the natural walking patterns of subjects, allowing for complete freedom of movement in their hands and legs. The Phone Call class is similar to Normal, except for the assumption that the participants are holding their phone to their ear with one hand, leaving only one hand free. Pockets class assumes that the participants hands are in their pockets, restricting hand movement entirely. Lastly, Texting class assumes the participants are using both hands to text on their phones while walking. To better capture the effects of distracted walking on gait patters,, asked participants to keep their eyes fixed on their phones rather than maintaining their typical attentive walking behavior.
We defined two classification tasks based on the four classes. The first task is hand movement detection ("Hand"), which aims to distinguish classes with no hand movement (Pockets and Texting) from classes with one or more moving hands (Phone Call and Normal). The second classification task ("Distract"), which aims to detect changes in the walking patterns of individuals when they are distracted by their phones. In absence of hand-movement signatures, we expect the perception models to pick up more subtle cues in the distracted pedestrian's 's walking pattern, making this task significantly harder comparing to the "Hand" task.
3.2 Data Capture Setup
Figure (c) illustrates the multi-radar setup and its positioning with respect to the Rol. For experiments, we used two off-the-shelf FMCW radars (AWR 1843 from Texas Instruments), and a HD stereo camera (ZED from Stereolabs). The two radars were designed operate at two different frequencies (77-78GHz for radarO and 79-80GHz for radarl) I), to ensure the simultaneous operation of both radars at high chirping duty-cycle (>95%) without any risk of interference.
As it will be shown, symmetrical coverage of the Rol by both radars critical this study. This is achieved by selecting rectangular area with 10m extent in both range and azimuth of each radar, and utilizing the entire width of each radar's Field of View (FoV) in azimuth (#45)). Assuming the average height of a human to be less than 2m, both radars Doppler signatures of both hands and legs can be properly captured. Considering the FoV of each radar in elevation (+15 we chose the minimum distance of the Rol to each radar to be at 5m. This guarantees the full utilization of the vertical beam width of each radar while eliminating the effect of the beam shape on received signals in close ranges B
set at height of 1m to ensure the
3.3 Experiment Design
To cover all possible walking directions and trajectories within the Rol, we defined seven walking patterns shown in Figure (b). Participants were instructed to walk at their natural speed and cover the entire Rol while repeating each walking pattern for all activity classes. Note that these patterns were carefully selected to be complementary from the perspective of each radar, ensuring a natural balance between the distribution of data across radars and walking patterns.
Moor information in the supplementary materials

---

Texeie
Dopler Snapehat
madenwak
Cw RM
Tas Nearr
(a) Activity classes
(b) Walking patterns
15m
RadarO
5m
Rol
15m
Rol
5m
Radarō Camera
(c) Experiment layout
Figure 1: Experiment Design. (a) The activity classes and corresponding classification tasks. High- resolution Doppler snapshots human-readable, but can be sufficient for real-time motion-based activity classifications. (b) The selected walking patterns. (c) Rol and its position relative to the multi-radar setup.
3.4 Dataset Statistics
In Figure 2],,, we present a histogram showing the number of snapshots captured at each location within the Rol. With uniform coverage of samples and over 90 snapshots at each location bin, the dataset provides sample size to accurately measure the effect of relative location on classification results.
Figure 2 illustrates the distribution of snapshots captured for different walking velocities. It can be observed that the dataset exhibits a symmetrical coverage within average walking speeds of our subjects (1 0.25m// indicated by the white dashed lines) with over 90 snapshots per bin for the primary linear walking directions (0", 45%, 90%, and 135", and their counterparts). This is evident that the dataset provides sufficient sample-size size to accurately measure the effect of walking trajectories on the gait classification performance.
Figure shows the distribution of average walking velocity over the Rol. The velocity is mostly uniform within the Rol, with the exception of low-speed regions around the edges of the Rol where participants stop and turn avoid exiting the field. Minimizing the correlations between location and walking velocities of subjects in the training data is a critical component in ensuring generalization power of Doppler-basee based classifiers, factor commonly overlooked in the capture and analysis of micro-Dopple datasets.
4
Single-Radar Baseline Neural Networks
4.1
Pre-Processing
The initial step involves extracting the micro- Doocolr signatures from each radar's received raw signal. We employ the CFAR algorithm [6] to detect the participant locations, and then use short-time Fourier transform on the slow-time axis to extract the gait micro- micro-Doppler spectrograms. Then, the spectrogram's 's magnitude is fog-rransorrmed and normalized as grayscale input to deep neural n
5

---

10
(a) Number snapshots captured at each location.
(b) Number of snapshots captured for different motion directions and velocities.
(c) Average walking velocity over the Rol.
Figure 2: Dataset statistics. The captured dataset provides comprehensive coverage across all walking directions and locations within the Rol.
The dataset is collected in the form of 10-second walking episodes, which are further divided into ..28-second clips with 50% overlap. To avoid the classifiers being sensitive to the relative start phase in walking cycles, augment each training batch by randomly cropping each clip into snapshot 640ms. Our experiments show that this simple augmentation technique is highly effective in enhancing the generalization of Doppler-based gait classifiers unseen data.
4.2 Gait Classification
Figure Ma) displays examples of micro-Doppler snapshots captured for the four classes. These snapshots cannot easily be distinguished by untrained human eyes, but contain high nigh-dimensional micro-motion signatures of human gait patterns that can be leveraged by Deep neural networks for complex classification tasks. Hence, utilize MobileNetV2 [17] as the backbone model for both Hand and Distract tasks. We report the 4-fold volss-aaiiationn accuracy across different subjects in Table Remarkably, the classifier achieves reliable hand-movement detection accuracy 85.25% across subjects by analyzing Doppler snapshots in window as short as single walking step. Moreover, Figure demonsttates that using longer windows for inference increases the accuracy to 97.9%13.07% (accuracy + standard deviation).
The performance for the more challenging Distract task is naturally lower, yet still statistically significant across subjects (accuracy of 62.55%=5.004 across two radars). This task aims to demon- strate the feasibility of using micro-mort motion signatures to capture effects beyond simple hand or leg movements, and pose challenge to encourage advanced micro- siopper analysis techniques. As shown in Figure similar to the Hand task, increasing the observation window length leads to an accuracy of 72.05% 10.05%.
0.9
0.7
Accuracy
0.8
0.7
Test
Test
Train window size
Train window size
0
8
Window (saconds)
Window (saconds)
(a) Hand task
(b) Distract task
Figure Hand and Distract tasks results across different inference window sizes. The baseline neural networks trained for real-time operation can also be used for slower but more accurate classification.

---

Table 2: Classification accuracies (accuracy % = standard deviation).
Single-Radar
Multi- Radar Fusion
Tasks
Radar 0 Radar
Input-level
Late-level Average
Multi-level Average
Late-level Transformer
Multi-level
Transformer
Hand Distract
86.0++4.6
85.7+4.3
85.24-5.1 60.0=6.9
87.7+44.8 65.8+7.5
88.5=4.5 66.9 #6.4
88.6+ 4.8 67.4=5.6
90.2-+4.6
63.5=6.0
61.5=4.9
66.9-16.3
5
Multi-Radar Baseline Neural Networks
5.1 Multi-Radar Fusion Methodology
In this section, we provide baseline models for the semantic fusion of multi-view Doppler signals, to address their inherent dependency on subject's location and trajectory.
Let f (+ 0) denote a baseline embedding network with its internal parameters 0, designed to project an arbitrary micro-Doppler snapshot X E IR into the latent space f (X ; 0) {m 7/2 ..... , }. where mh is the representation from the I-th encoding layer. The multi-view representations 77 R1] and f (X 0) {n R2 my
(X RI )) { (1 m2
7L f2) obtained
from the multi-view micro-Door Poopler modalities X dal and X Ir2 respectively, can be integrated across range of different fusion levels and methods.
We demonstrate multi-view fusion across three different embedding levels: input-level fusion, where the inputs X and X R2 are channel-wise concatenated and embedded via the same single- view baseline network, 2) late-level fusion, where the features are combined solely under the last layer L. and 3) multi-level fusion, contemplates feature embeddings across all levels wherever the features are resized, e.g., every representation level right after pooling.
One approach for multi-view fusion is to simply combine the corresponding representations from the two radars by summing them together, i.c., (7/ + 77 R2) However, given that the radar signals captured from different spatial nodes pose inequivalent significance with respect to the heading direction of target, it may be more reasonable sequences. This intuition leads us to introduce a transformer-base fusion pipeline that merges the tokenized form of Rl and 17. R2 via token-witn wise concatenation [19] to achieve gross-attention between the multi-vier inputs as well as context-award temporal attention.
assign dynamic weights across the multi-view
[18), followed by self-attention layers
The combinations across different fusion levels and methods yield five multi-view radar fusion variants: input-level fusion, late-level average-based fusion, multi-level average-based fusion, late- level transformer-bant based fusion, and multi-level wansformer-basee fusion.
5.2 Experimental Analysis
Table 2 summarizes the classification performances for each fusion method. The results demonstrate that the multi-view setting has holistic performance improvements in comparison with a single-radar system. Of particular note is the trend that the of adaptive fusion strategies at multiple-levels leads to greater performance gains. Comparing with single-radar baselines, the proposed multi-level wansformer-basee fusion baseline yields in significant improvements of around 5% for both for Hand and Distract tasks.
Results and Discussion
is worth noting that the accuracies shown in Table the average classification performance across the entire Rol and all velocities and directions. In this section, we provide an in-depth analysis of single-viev and multi-view classification results (for the best models presented in the previous section) with respect to the effect of location and velocity. Figure F iillustrates that the single-radar classification accuracy decreases towards the edges the Rol for both radars. This accuracy loss partly due to the reduced velocities at the edges of Rol, and in part due to the decreased signal-to-noise ratio (SNR), which can be mitigated using beamforming techniques or increasing radar's 's transmit power. Figure 5 illustrates the accuracy distribution of the Hand task across the two-dimensional
7

---

walking speed and direction of the subjects. The results highlight the significant sensitivity of single- radar classification such factors. The accuracy substantially drops for subjects with tangential directions from the radar, owing to Doppler's s radial sensing nature. This effect is also clear in the accuracy vs walking angle plot in Figure
In contrast to single-radar baselines, Figure ((c) and Figure show that multi-view sensing has great robustness at the edges of the Rol well as on the tangential velocities. This evidence that multi-view radar sensing techniques can compensate for the limitations of single-radar approach, and eliminate single-radar location and trajectory dependence. This result further elucidated Figure O representing the average accuracy of the Hand and Distract tasks as a function of walking direction relative RadarO. Notably, the two radars exhibit complete complementary behavior specially for the blind spots of each radar (i.e., 9 for RadarO and 0% and 180 for Radarl). The multi-radar fusion consistently achieves higher accuracy than the best performance either sensor alone, irrespective of the directions.
Itans
Pladd)
Pubbn Handi
009
155
225
648
125
030
089
361
135
35.55
(a) Radar
(b) Radar
(c) Fusion
Figure 4: Classification accuracy across the Rol for Hand task. The classification performance of the multi-view fusion approach is uniform across the Rol.
brand
Fusion
1.-
16-
2.-
XX-
075
000
690
da
(a) Radar
(b) Radar
(c) Fusion
Figure Hand task classification accuracy across walking velocity magnitudes and directions. Single-radar classification performance is sensitive to the subjects direction of motion while the multi-view approach is independent of the subjects moving direction.
7 Limitations
In this study, focused exploring the influence of subject's location and trajectory co-factors gait analysis. However, there exist several other co-factors that we did not investigate. For instance, did not examine the impact of multi-path and clutter in different environments the effect of movements the radar itself on the Doppler signature.
Another limitation of the presented dataset is related to the sample size and diversity of participants. Although the presented results demonstrate statistical significance as indicated by the confidence intervals in Table 2 the dataset could enjoy larger and more diverse participants population.

---

0.72
0.92
0.70
0.88
0.68
0.64
0.84
0.62
0.80
135
45
135
180
0.56
45 Walling Angls Rolative to Rodant (degreesi
160
We king Angl: Re orwe Rariar (Sogress)
Hand
(h) Distract
Figure 6: Accuracy of Hand and Distract tasks across walking angles. Radars complement each other in their sensitivity to effect of different walking directions. iusion-based approach outperforms both single radars across all walking directions.
Moreover, it is important for readers to note that the classes defined this paper do not fully encompass the breadth of diversity and complexity inherent in the two tasks. The selected classes and the tasks introduced in this study were specifically designed encompass varying levels of difficulty in Doppler-bas bbsee pedestrian gait analysis. They serve initial step towards utilizing Doppler-base based methods for advanced perception tasks.
Conclusion
This study introduces novel micro-Doppler dataset, consisting of multiple views, for Doppler- based pedestrian gait classification. Our research focuses on two challenging tasks: real-time hand movement detection and distracted pedestrian detection. We establish neural network baselines for these tasks and investigate the impact of Doppler directionality on single radar classification performance-a factor often overlooked in existing Doppler-based classification literature.
To address the challenges posed by relative location and velocity, we propose multi-view neural network baselines based fusion transformers. Through this approach, achieve trajectory- agnostic, real-time pedestrian gait classification. The results we present demonstrate generalization across different subjects and various walking trajectories, evidenced by the narrow confidence intervals of 5-7% reported for both tasks.
The availability of our dataset, along with the corresponding bascline neural networks, lays solid foundation for future research in real-time Doppler-based perception. This research avenue offers an alternative to conventional vision-based perception approaches. Overall, our work contributes significantly to the field Doppler-bas pedestrian gait classification and opens up opportunities for further advancements in real-time Doppler-base based perception.
Acknowledgments and Disclosure of Funding
This work has been supported by Samsung Electronics (Samsung Semiconductor USA).

---

References
[1] Soheil Hor, Men Pilanci, and Amin Arbabian. A data-driven waveform adaptation method for mm-wave gait classification at the edge. IEEE Signal Processing Letters, 29:26-30, 2021.
[2] Shelly Vishwakarma and Shobha Sundar Ram. Dictionary learning with low computational complexity for classification of human micro-dopplers across multiple carrier frequencies. IEEE Access, 6:29993-29880 2018.
[3] Shufan Yang. Julien Le Kernec, Olivier Romain, Francesco Fioranelli. Pierre Cadart, Jeremy Fix, Chenfang Ren, Giovanni Manfredi, Thierry Letertre, Israel David Hinostroza Saenz, et al. The human activity radar challenge: benchmarking based on the 'radar signatures of human activities' dataset from glasgow university. IEEE Journal of Biomedical and Health Informatics, 2023.
[4] Mainak Chakraborty, Harish C Kumawat, Sunita Vikrant Dhavale, and Arockia Bazil Raj. Diat-u radhar (micro-doppler signature dataset) & radnet (a lightweight denn)-for for human suspicious activity recognition. IEEE Sensors Journal, 22(7):6851-655 2022.
[5] Sevgi Z Gurbuz, M Mahbubur Rahman, Enre Kurtoglu, Trevor Macks, and Francesco Fio- ranelli. Cross-frequency training with adversarial learning for radar micro-doppll signature classification (rising researcher). In Radar Sensor Technology XXIV, volume I 11408, pages 58-68. SPIE, 2020.
[6] Ennio Gambi, Gianluca Ciattaglia, Adelmo De Santis, and Linda Senigagliesi. Millimeter wave radar data of people walking. Data in brief, 31:105996, 2020.
[7] Geethika Bhavanasi, Lorin Werthen-Braihe Brabants, Tom Dhaene, and Ivo Couckuyt. Patient activity recognition using radar sensors and machine learning. Neural Computing and Applications, 34(18):16033-166 16048, 2022.
[8] Zhi Wu, Dongheng Zhang, Chunyang Xie, Cong Yu, Jinbo Chen, Yang Hu, and Yan Chen. RFMask: simple baseline for human silhouette segmentation with radio signals. IEEE Transactions Multimedia, pages 1-12, 2022.
[9] Shih-Po Lee, Niraj Prakash Kini, Wen- Peng. Ching- Wen Ma, and Nenq Hwang. HuPR: A benchmark for human pose estimation using millimeter wave radar. In JEEE/CVF 2023.
Winter Conference Applications Computer Vision (WACV), pages
[10] Jie Yang, Abdesselam Bouzerdoum, and Moeness G. Amin. Multi-view through-the-wall radar imaging using compressed sensing. In European Signal Processing Conference, pages 1429-1433, 2010.
[11] C. H. Seng, A. Bouzerdoum, F. C. Tivive, and M. G. Amin. Fuzzy logic-based image fusion for mieti-view through-the-war radar. In International Conference on Digital Image Computing: Techniques and Applications, pages 423-428- 2010.
[12] Huquan Li, Guolong Cui, Shisheng Guo, Lingjiang Kong, and Xiaobo Yang. Target tracking and ghost mitigation based on multi-view through-the-wall radar imaging. In IEEE Radar Conference (RadarCouf), pages 1-5, 2020.
[13] Shisheng Guo, Jiahui Chen, Zhenpeng Shi, Huquan Li, Peilun Wu, Songlin Li, Guolong Cui, and Xiaobo Yang. Graph matching based image registration for multi-view through-the the-wall imaging radar. IEEE Sensors Journal, 22(2):1486-1 1494, 2022.
[14] Shaofeng Cai, Yao Shu, and Wei Wang. Dynamic routing networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3588-3597, January 2021.
[15] Pengyun Chen, Xiang Wang. Mingyang Wang, Xiaqing Yang, Shisheng Guo, Chaoshu Jiang, Guolong Cui, and Lingjiang Kong. Multi-view real-time human motion recognition based on ensemble learning. JEEE Sensors Journal, 21 (118))20333-200 20347, 2021.
[16] Mark A Richards. Fundamentals of radar signal processing. McGraw- Hill Education, 2014.
10

---

[17] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[18] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7073-7083, 2021.
[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, 2017.


Reviews:
[{"review_id": "KBR1KTURyH", "reviewer": "Reviewer_cCoP", "summary": "", "questions": "", "limitations": "As mentioned this is highly susceptible to test conditions and the presence of multi subjects or different types of movements can impact the accuracy of the work.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "New dataset for using doppler for looking into gait", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "2eCvR0XNN1", "reviewer": "Reviewer_reAu", "summary": "", "questions": "", "limitations": "1. Although clearly a promising extension on existing doppler-based datasets, this dataset is still from a relatively contrived setup that it's not clear will lead to significant improvements in methodology. \n\n- For example, the radars remain static, though clear use-cases for such systems are from moving radars that are mounted on cars. Given doppler is a signal that involves understanding how objects have changed over time, moving radars on a track might be more directly useful in developing such radar-perception systems.\n\n- A different fusion system that is considerate of real-world applications might have placed two radar setups in stereo with one another, as if they were setup a few feet apart on a car. Such stereo-fusion would be another interesting analysis (and potentially applicable).\n\n2. Coming from only a few hours of recording, the dataset feels more like a proof-of-concept of an improved doppler dataset, and could benefit from more involved actions and benchmarks. Specifically, good datasets are often forward-looking, even if they have insufficient baselines. For example:\n\n- Even if baselines struggle at these tasks, one could have included a few dozen more actions including gaits (running, skipping, biking). \n\n- Even if there's no consideration of analyzing this, one could have recorded multiple people crossing paths with one another. Such a complicated environment is not necessary for nascent first-of-its-kind-contributions, but could be useful for further, more developed methods. Having this data can be what really makes a dataset useful (even when it's unclear how any method might use it). Once the recording setup is established, it appears like it shouldn't be too difficult to record such a diversity and produce a better, more expansive dataset.\n\n- More distractor setups could exist, for example someone walking while reading a book or walking while looking away from the direction of the radars. This last one might be particularly difficult but potentially closer to the essence of what it means to be \"distracted\".", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Compared to existing doppler datasets as described by the authors, their dataset has a better variety of micro-doppler across both range, azimuth, and motion direction. It is also longer, though includes a slightly more limited set of participants than other datasets. It additionally includes motion from hands that occurs while the participant is walking rather than standing. \n\n2. As a promising and under-explored area of perception, this dataset is potentially valuable in enabling the development of more radar/doppler based systems.\n\n3. The paper includes a valuable analysis both of the distribution of data contained (in the form of Figure 2's heatmaps) and the abilities/failure modes of baseline methods on the data (Figure 5). ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "fWzUsCsnzX", "reviewer": "Reviewer_MPVr", "summary": "", "questions": "", "limitations": "As pointed out by the authors (lines 268-271), the dataset is curated under somewhat idealised conditions (e.g., absence of aggressive multipath and clutter). Additionally, the authors remarked on the potential to push the envelope of micro-Doppler analysis for more challenging perception task (lines 275-279); for example, one such very challenging task to to fully identify a subject from their gait signature. Nonetheless, this reviewer believes that the authors have pretty much laid bare the facts for future work.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "- clear, thorough, and well-argued discourse\n\n- rigorous experimental design and dataset statistics\n\n- good variants of models to establish a 1st baseline on multi-view Doppler learning\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Fsk4eW1iso", "reviewer": "Reviewer_EN5b", "summary": "", "questions": "", "limitations": "The authors did not do a very thorough job discussing the societal impact, particularly because the paper presents a method that could be leveraged in the context of surveillance.  It would have been nice to see a little more discussion on this.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset seems to have properties not covered by existing data.  The authors have presented the paper in a clear manner with good figures.  The data was available on Google drive and seems well organized.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents a dataset of multi-view doppler data for gait classification.  The reviewers all agree that this dataset has merits and all reviewers recommended that this paper was above the acceptance threshold.  The strengths of the paper were: 1) that it contributes to an underrepresented domain, that the paper was clear and well presented, and that the dataset has properties that are not covered in other datasets. The setup of the data collection is a little contrived and the scale is a little small; however, on balance the reviewers agree that this dataset is a valuable resource for the community.
Decision: Accept (Poster)