# Towards Deep Radar Perception for Autonomous Driving: Datasets, Methods, and Challenges
## 7 Challenges
### Fusion in Adverse Weather  
Adverse weather conditions, such as heavy rain, snow, and fog, can be a significant threat to safe driving. Different sensors operate in different electromagnetic wavebands, thus having different robustness to environments. A comparison of the weather effects on different sensors can be found in [217]. Visual perception is susceptible to blur, noise, and brightness distortions [218; 219]. In adverse weather, LiDAR suffers from reduced detection range and blocked view in powder snow [220], heavy rain [221], and strong fog [221]. In contrast, radar is more robust under adverse weather. The effect of weather on radar can be divided into attenuation and backscattering [222]. The attenuation effect decreases the received power of the signal, and the backscattering effect increases the interference at the receiver. Experiments [223; 224; 225; 43] reveal attenuation and backscattering under dust, fog, and light rain are negligible for radar, while the performance of radar degrades under heavy rainfall. [PERSON] et al. [222] summarised the mathematical models for the attenuation and backscattering effects of rain. They suggest the detection range of radar can be reduced by up to 45% under severe rainfall conditions (150 mm/h). For close targets with small RCS, the backscattering effect is more severe and can cause additional performance degradation. Driving in adverse weather can be considered as a corner case [226] for autonomous driving. The concept of the operational design domain (ODD) [227] is proposed to define the conditions under which autonomous vehicles are designed to operate safely. If the monitoring system [228] detects a violation of OOD requirements, control will be handed over to the driver. However, changes in the operational environment are usually rapid and unpredictable. Therefore, the hand-over mechanism is controversial in terms of safety. In the future, a fully autonomous vehicle (SAE Level 5) is expected to work under all environmental and weather conditions. However, most fusion methods are not designed to explicitly consider weather effects. Networks trained in good weather may experience performance degradation in adverse weather. There are some possible ways to adapt the network to different weather conditions. One way is to add a scene switching module [229], then use different networks for different weather. This method is straightforward, but introduces additional computational and memory costs. The other option is to add some dynamic mechanisms into the network. [PERSON] et al. [33] added a two-stage attention block in the fusion module. They first applied self-attention to each modality and then mixed them through cross attentions. Experiments showed that the fusion mechanism performs robustly in foggy weather. They further investigated the domain generalisation problem, i.e., training with a good weather dataset and inference in foggy weather. The result showed a significant accuracy drop compared with training with data in both good and foggy weather, indicating that their model relies on data to generalise. [PERSON] et al. [230] proposed a gating strategy to rank each modality and pick the top three reliable modalities for fusion. They compared three types of gating methods: knowledge-based, CNN-based, and attention-based gating. Knowledge-based gating uses a set of pre-defined modalitywise weights for each weather condition, while CNN-based and attention-based learn the weights from data. Experiments on the RADIATE dataset [44] indicate gating methods outperform fusion methods under adverse weather, and attention-based gating can achieve the best performance. Alternatively, [PERSON] et al. [43] proposed an entropy-steered fusion network, which uses the sensor entropy as modalitywise weights. Specifically, they used a deep fusion architecture that continuously fuses feature maps from different modalities. The pixelwise entropy is used as the attention map for each sensor branch.