SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2 summarizes the classification performances for each fusion method. The results demonstrate that the multi-view setting has holistic performance improvements in comparison with a single-radar system. Of particular note is the trend that the use of adaptive fusion strategies at multiple-levels leads to greater performance gains. Comparing with single-radar baselines, the proposed multi-level transformer-based fusion baseline yields in significant improvements of around  $5\%$  for both for Hand and Distract tasks.
6 Results and Discussion
It is worth noting that the accuracies shown in Table 2 represent the average classification performance across the entire RoI and all velocities and directions. In this section, we provide an in-depth analysis of single-view and multi-view classification results (for the best models presented in the previous section) with respect to the effect of location and velocity. Figure 4 illustrates that the single-radar classification accuracy decreases towards the edges of the RoI for both radars. This accuracy loss is partly due to the reduced velocities at the edges of RoI, and in part due to the decreased signal-to-noise ratio (SNR), which can be mitigated using beamforming techniques or increasing radar's transmit power. Figure 5 illustrates the accuracy distribution of the Hand task across the two-dimensional
walking speed and direction of the subjects. The results highlight the significant sensitivity of single-radar classification to such factors. The accuracy substantially drops for subjects with tangential directions from the radar, owing to Doppler's radial sensing nature. This effect is also clear in the accuracy vs walking angle plot in Figure 6.
In contrast to single-radar baselines, Figure 4(c) and Figure 5(c) show that multi-view sensing has great robustness at the edges of the RoI as well as on the tangential velocities. This is evidence that multi-view radar sensing techniques can compensate for the limitations of single-radar approach, and eliminate single-radar location and trajectory dependence. This result is further elucidated in Figure 6, representing the average accuracy of the Hand and Distract tasks as a function of walking direction relative to Radar0. Notably, the two radars exhibit complete complementary behavior specially for the blind spots of each radar (i.e.,  $\pm 90^{\circ}$  for Radar0 and  $0^{\circ}$  and  $180^{\circ}$  for Radar1). The multi-radar fusion consistently achieves higher accuracy than the best performance of either sensor alone, irrespective of the directions.
(a) Radar 0
(b) Radar 1
(c) Fusion
Figure 4: Classification accuracy across the RoI for the Hand task. The classification performance of the multi-view fusion approach is uniform across the RoI.
(c) Fusion
Figure 5: Hand task classification accuracy across walking velocity magnitudes and directions. Single-radar classification performance is sensitive to the subjects direction of motion while the multi-view approach is independent of the subjects moving direction.
In this study, we focused on exploring the influence of subject's location and trajectory co-factors on gait analysis. However, there exist several other co-factors that we did not investigate. For instance, we did not examine the impact of multi-path and clutter in different environments or the effect of movements of the radar itself on the Doppler signature.
Another limitation of the presented dataset is related to the sample size and diversity of participants. Although the presented results demonstrate statistical significance as indicated by the confidence intervals in Table 2, the dataset could enjoy a larger and more diverse participants population.
(a) Hand
(b)Distract
Figure 6: Accuracy of Hand and Distract tasks across walking angles. Radars complement each other in their sensitivity to effect of different walking directions. Fusion-based approach outperforms both single radars across all walking directions.
Moreover, it is important for readers to note that the classes defined in this paper do not fully encompass the breadth of diversity and complexity inherent in the two tasks. The selected classes and the tasks introduced in this study were specifically designed to encompass varying levels of difficulty in Doppler-based pedestrian gait analysis. They serve as an initial step towards utilizing Doppler-based methods for advanced perception tasks.
This study introduces a novel micro-Doppler dataset, consisting of multiple views, for Doppler-based pedestrian gait classification. Our research focuses on two challenging tasks: real-time hand movement detection and distracted pedestrian detection. We establish neural network baselines for these tasks and investigate the impact of Doppler directionality on single radar classification performance—a factor often overlooked in existing Doppler-based classification literature.
To address the challenges posed by relative location and velocity, we propose multi-view neural network baselines based on fusion transformers. Through this approach, we achieve trajectory-agnostic, real-time pedestrian gait classification. The results we present demonstrate generalization across different subjects and various walking trajectories, as evidenced by the narrow confidence intervals of  $5 - 7\%$  reported for both tasks.
The availability of our dataset, along with the corresponding baseline neural networks, lays a solid foundation for future research in real-time Doppler-based perception. This research avenue offers an alternative to conventional vision-based perception approaches. Overall, our work contributes significantly to the field of Doppler-based pedestrian gait classification and opens up opportunities for further advancements in real-time Doppler-based perception.
This work has been supported by Samsung Electronics (Samsung Semiconductor USA).
[17] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[18] Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7073-7083, 2021.
[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, 2017.