RLNet: Adaptive Fusion of 4D Radar and Lidar for 3D Object Detection

Authors: ruoyu  xu
zhiyu  xiang
Venue: thecvf_com_ECCV_2024_Workshop_ROAM
Published: 2024-07-29 18:12:11

Abstract: Lidar based 3D object detection has made great progress in recent years and has become the mainstream configuration for autonomous vehicles. However, Lidar can experience substantial performance degradation in the case of adverse weather or long-distance object detection, due to its short wavelength and the limitation of energy emission. 4D millimeter-wave radar is capable of providing 3D point clouds similar to Lidar, with much more robustness against adverse weather conditions. However, 3D object detection with only 4D radar is less satisfactory due to the high sparsity and flickering nature of the measurements. In this paper, we propose a novel 3D object detection method termed RLNet, which effectively integrates 4D radar and Lidar through adaptive feature fusion. An adaptive radar point speed compensation and a modality dropout training strategy are further introduced to improve the performance. RLNet achieves the state-of-the-art performance in the experiments, outperforming baseline method by 7.35 and 2.76 percent in mAP on the popular VoD and ZJUODset dataset, respectively. The code will be available.

Main paper: 
 introduction: 
 As a popular sensor for autonomous driving, Lidar is well-known for its capability of providing accurate 3D information of surrounding environments. As a result, Many successful Lidar-based 3D object detection methods [1] [2] [3] [4] [5] have been proposed and achieve state-of-the-art performance in a variety of public datasets [6] [7] [8] . However, due to the short wavelength and energy emission limitations, Lidar can experience substantial performance degradation in the case of adverse weather [9] or long-distance object detection [10] .In recent years, 4D millimeter wave radar has received widespread attention. Due to the penetrative nature of millimeter waves, radar can well handle adverse weathers such as rain, snow, and fog, and achieve a longer detection distance [9] . Unlike the traditional 3D radar, 4D radar is capable of providing 3D point clouds similar to Lidar, providing the possibility for accurate 3D object detection. Some public datasets such as Astyx [11] , VoD [12] , Tj4dradset [13] and ZJUODset [10] have been published to boost the research. However, the high sparsity and noisy Fig. 1: A typical scenario on the ZJUODset [10] . The orange and green points in the left image denote the point clouds acquired by Lidar and 4D radar, respectively. The three enlarged bounding boxes on the right show the exampling objects with only radar points, only Lidar points and both of them, respectively. nature of the acquired 4D radar point clouds pose a great challenge to the robust detection.As shown in Fig. 1 , the information provided by Lidar and radar sensors can be complementary for the object detection task. Radar can address the limitations of Lidar in the case of part obstruction and objects in the distance, as well as supplying valuable Doppler velocity to facilitate detection of dynamic targets. Current Lidar and radar fusion methods [14, 15] do not well consider the different characteristics of the two modalities during fusion, and they also don't take into account the extreme case of failure of one sensor. Therefore, it is crucial to explore the differences in modalities and design appropriate fusion approaches. To address these problems, we propose a novel 3D object detection method termed RLNet, which effectively fuses 4D radar and Lidar features through adaptive weighting. A radar point speed compensation and a modality dropout training policy are further introduced to improve the detection performance. The experimental results on the VoD dataset [12] and ZJUODset [10] demonstrate the effectiveness of our method.In summary, our main contributions are as follows:-We introduce a lightweight Lidar-radar fusion network, which fulfills the 3D object detection task in complex environment by adaptively weighting the importance of 4D radar and Lidar features. -We propose an effective speed compensation method for radar point cloud preprocessing. We estimate the ego-speed from the raw Doppler speed and then obtain the compensated radial speed for each point. -We propose a special training method with random modality dropout, which enhances the feature of each single-modality and improves the robustness of the network. -We conduct extensive experiments on the VoD [12] and ZJUODset [10] datasets. Experimental results show that our method achieves state-of-theart performance. 
method: 
 In this section, we introduce our RLNet network in detail, which fuses 4D radar and Lidar point clouds for 3D object detection. The network framework is introduced first, followed by description of each module.We implement our RLNet based on SECOND [2] , as shown in Fig. 2 . Besides the backbone, the framework consists of a speed compensation module, an adaptive feature fusion module, and a random modality dropout strategy. Before inputting to the network, the 4D radar point clouds are preprocessed by the speed compensation module to obtain the absolute radial velocity for each point. After feature extraction by the sparse encoders, the features from both the radar and Lidar branches are fed into the adaptive feature fusion module, which generates suitable weights to each modal feature before concatenation. We design a random modality dropout strategy during training, thereby enhancing the robustness of the network when the failure of one modality happens. The input for each Lidar point is a 4D-vector with [x,y,z,r], where the first 3 components are the 3D coordinates and the last one is the reflection magnitude. For the 4D radar point, the input is regarded as a 6D-vector, with additional speed and timestamp components over the Lidar point.The extra Doppler speed provided by the radar can be valuable for the network to detect and classify the moving objects. However, the raw Doppler speed obtained from 4D radar is the relative radial speed to the ego-vehicle. The unknown motion of the ego-vehicle can greatly influence the measured Doppler speed, causing difficulty to the network when utilizing the speed information. To address this issue, we propose a speed compensation module for the radar points based on the assumption that most of the radar points are static in the environment.The process is illustrated in Fig. 3 and described as follows.1. Assuming that the ego-vehicle moves only on x-y plane and heads along the x direction, we project the raw Doppler speed of the points onto the x-y plane, then project the resulting V r to x-axis to obtain the relative speed V x along the vehicle's motion direction with V x = V r /cosθ; 2. Assuming the majority of points in the scene are from the background, which is static relative to the world coordinate system, we divide the speed space of V x into several bins and count the votes in each bin, and the speed value with the highest count is regarded as the estimated ego-vehicle velocity V e ; 3. Compensate the radial velocity of each point based on the estimated egovehicle velocity, with V comp = V r -V e * cosθ. The resulting V comp can better model the speed field of the scene in that most of the static background objects' speeds are zero.Following the sparse convolution layer on the backbone, the network obtains the Lidar and radar features with each of them having dimensions of B×C×H×W, where B, C, H and W are the batch size, the channel number, the height and width of the feature map, respectively. The simplest way to fuse these two features is concatenation, which combines features along the channel dimension to form a feature of size B×2C×H×W. However, this simple way does not consider the large difference of the sensor characteristics, such as the multi-path noise of the radar data and occlusion sensitiveness of the Lidar data. Simple concatenation of their features can confuse the learning of the network and degrade the performance.To tackle this problem, we propose an Adaptive Fusion module to combine the Lidar and radar features by adaptive weights. Specifically, Lidar feature map F L and radar feature map F R are first processed through a channel average pooling layer, yielding two channel feature maps F LA and F RA as follows:EQUATIONEQUATIONThen these two feature maps are concatenated along the channel dimension and fed into a 3×3 convolution layer, followed by batch normalization and softmax.The result is the weight map of W L and W R , with each having the size of B×H×W:EQUATIONThe weight maps represent the importance of each spatial location of the feature maps. The modality features are then weighted and concatenated by Eq.( 5), yielding the fused feature map F f usion with size B×2C×H×W:EQUATIONThis fused feature map is then fed into the subsequent backbone to perform 3D object detection.The data from multimodal sensors can effectively improve the performance of 3D object detection. However, in some adverse scenarios when a certain sensor is constrained or degraded, the network may receive input from only one sensor.In such case, the full multimodal trained network can experience significant degradation in performance. Inspired by CramNet [16] , we introduce random modality dropout strategy during training. The process is described with Eq.( 6) and (7) , where F ′ L and F ′ R represent the modal features after dropout gate, 1(•)is the indicator function with outputting values 0 or 1, other parameters are explained in the following.EQUATIONEQUATIONFirst a random keeping probability p 1 is generated. If p 1 is greater than the dropping probability threshold P drop , both features of the two modalities are retained and regular training process proceeds. Otherwise, one of the modality feature should be dropped. In this paper, P drop is set to 0.2.When deciding which modality feature to drop, another random keeping probability p 2 is generated. If p 2 is greater than the Lidar's keeping probability threshold P L , the Lidar features are kept and the 4D radar features are dropped; otherwise the Lidar features are dropped in this training epoch. Unlike CramNet [16] which does not consider the difference of the modalities and uses the same dropout probability for the camera and radar features, we set P L to 0.2 in this work considering that Lidar features play a dominant role in the fusion. In addition, the experimental results in Section IV suggest that modality dropout strategy can not only mitigate degradation caused by the sensor failure, but also enhance the feature of each modality, thereby promoting the performance of multimodal 3D object detection. 
dataset and evaluation metrics: 
 We conduct experiments on the popular VoD dataset and ZJUODset.VoD Dataset [12] . The VoD dataset is collected in Delft, which includes a substantial number of car, pedestrian, and cyclist objects to detect. Most of the cars are aligned on the roadside with different extent of occlusions, which pose challenges to the Lidar. We follow the official partition and divide the dataset into training and validation set with 5139 and 1296 frames respectively. Based on the target's distance to the ego-vehicle, VoD separately evaluates the performance on the entire annotated area and the driving corridor. The latter refers to the narrow area within the range of [-4m, 4m] on the x-axis and [0, 25m] on the z-axis (front) in the camera coordinate.ZJUODset [10] . ZJUODset collects data on the real traffic scenes of Hangzhou city, aiming at addressing complex and long distance detection requirements for autonomous driving. It collects point clouds acquired from a solid state Livox Lidar and an Oculii Eagle 4D radar, and evaluates the detection performance up to 150 meters. We define the evaluation area as extreme level. 'Easy', 'Moderate' and 'Hard' levels represent objects within 30, 50 and 80 meters, respectively. Within the 3800 annotated frames, we split the first 2660 frames as the training set and the last 1140 frames as the validation set.AP40 metric is employed for the evaluation. On the VoD Dataset, we use IOU thresholds of 0.5/0.25/0.25 for car, cyclist and pedestrian, respectively, in order to be in line with the previous works. On the ZJUODset, we instead use IOU thresholds of 0.7/0.5/0.5 for car, cyclist and pedestrian respectively, to evaluate the performance with a higher standard than VoD.On the VoD Dataset, the entire detection range is set as (0, 51.2m) on the xaxis, (-25.6m, 25.6m) on the y-axis, and (-3m, 2m) on the z-axis in the Lidar coordinate. We set the voxel size to (0.05m, 0.05m, 0.1m) and the maximum number of points in each voxel to 5. On the ZJUODset, we define the detection range as (0, 158.4m) on the x-axis, (-39.6m, 39.6m) on the y-axis, and (-5m, 3m) on the z-axis. The space is partitioned into voxels of (0.075m, 0.075m, 0.2m) for encoding. We use the random flipping along the x-axis and random global scaling with the scaling factor in [0.95,1.05] for data augmentation. We implement our RLNet based on mmdetection3d [17] and OpenPCDet [18] framework. We employ the Adam optimizer for parameter updates with an initial learning rate 0.001 and a weight decay factor 0.01. The learning rate is updated using a cyclical decay method, with the maximum learning rate being 0.01 and the minimum being 10 -7 . Just as SECOND [2] , the loss for the model comprises three components: classification loss, detection regression loss, and angular loss. Specifically, we adopt Focal Loss for classification, Smooth L1 Loss for location regression and Cross-Entropy loss for angular regression.The experimental results on the VoD dataset are shown in TABLE 1 . Besides comparing to other Lidar+radar methods, we also list methods using other modalities, such as pure radar, pure Lidar and radar+image for reference. As shown in the TABLE 1, our method achieves the best performance among all of the methods, with 3.91% and 3.30% improvements on mAP over the second place method (the feature cascade version of SECOND [2] with Lidar+radar), for the entire annotated area and driving corridor, respectively. Comparing to the pure radar version of SECOND [2] , our method has about 35% improvement on mAP in the entire area, showing the importance of the Lidar in the task. Our RLNet also performs much better than the pure Lidar method, showing the critical role of the 4D radar in detecting some hard occluded objects. The qualitative results shown in Fig. 4 also validate this statement. Meanwhile, our method shows superior performance over the radar+image method like RCFusion [19] and LXL [20] , which should thank to the accurate geometric information provided by the Lidar over the image. Similar phenomenon can be observed on the ZJUODset, as shown in TA-BLE 2. The difference is that the improvements are much harder to achieve than on the VoD dataset, due to more complex environment, much longer detecting distance requirements (3 times longer than VoD) and higher standard of evaluation metrics of the task. Our RLNet still stands 1.61% higher on mAP over the second best method Interfusion [14] , validating the effectiveness of our method. The qualitative results shown in Fig. 5 also reveal less false positive and false negative detections of our method over its counterparts.We conduct ablation study on the ZJUODset and the results are in TABLE 3 .Effects of speed compensation. We employ a speed compensation strategy to rectify the relative speed caused by the ego-motion of the vehicle. Comparing (a) with (b), or (c) with (d) in TABLE 3 , we see that introducing speed compensation can add 0.6∼0.7 percent on 3D mAP, which suggests that removing the influence of ego-motion on the Doppler speed is beneficial for 3D object detection.Effects of adaptive fusion module. The simple feature cascade (FC) cannot fully explore the complementary nature of the Lidar and radar data, resulting in inferior performance. Comparison of (a) with (c) or (b) with (d) in TABLE 3 shows that adaptive feature fusion module (AF) can better fuse the features of the two modalities, with 0.4∼0.7 percent improvements on mAP.Effects of random modality dropout. Comparing (d) with (e) in TA-BLE 3, we see that introducing random modality dropout (RD) effectively improves the performance, with an increase of 3D and BEV mAP by 1.04% and 0.75%, respectively. Finally, the complete configuration of our method (e) obtains 2.1% higher than the baseline (a) on 3D mAP, revealing the effectiveness of our model design. 
conclusions: 
 In this paper we introduce an effective method for 3D object detection by fusing 4D radar point clouds to the popular Lidar. Although both sensors have the similar form of 3D point, the high sparsity and range noise contained in the 4D radar has to be well treated. Based on the popular SECOND [2] backbone, we design speed compensation module for radar points to provide the rectified speed to the network, and propose adaptive fusion module to well balance and enhance the multi-modal features. A special random modality dropout training strategy is further employed to strengthen the robustness of the feature. The experimental results on VoD and ZJUODset datasets demonstrate our success. 
 
 Appendix: The development of 3D object detection based on Lidar has become quite mature to date. VoxelNet [1] divides 3D space into regular voxels for encoding and utilizes 3D convolution to accomplish 3D object detection. However, the employment of 3D convolution introduces substantial computational overhead, making it hard to meet real-time requirements. SECOND [2] proposes sparse 3D convolution, which significantly improves the network's inference speed. PointPillars [3] introduces the concept of pillars and utilizes 2D sparse convolution to further accelerate the network. CVFNet [21] projects point clouds onto the cylindrical view and fuses the original 3D point-level features to accomplish multi-view 3D object detection.Introducing image information to Lidar can effectively improve the object detection performance. MV3D [22] uses different views from Lidar point clouds and RGB images as inputs. Candidate boxes are generated from various views, and object detection results are produced by integrated features across the two sensors. PointPainting [23] projects image semantic segmentation results onto 3D point clouds and achieves data fusion at the point level. Very recently, LogoNet [24] fuses features from point clouds and images globally and locally through the attention mechanism.Due to the lack of height information, 3D radar is seldom used alone in the task of 3D object detection. Most of the works focus on the fusion of camera or Lidar to achieve better performance. CenterFusion [25] fuses the features of 3D radar and camera through the Pillar Expansion and Frustum Association. CRAFT [26] designs a soft association method to match the radar points and the image features and proposes a Spatio-Contextual Fusion Transformer to further interact the information between the two sensors. CRN [27] introduces depth estimation and radar occupancy to facilitate view transformation and employs an attention mechanism to conduct feature fusion in BEV. By fusing Lidar, RadarNet [28] employs a voxel-based network to extract features from Lidar and radar point clouds, improving the detection of dynamic objects.With similar 3D point clouds to the Lidar, Lidar-based networks can be directly applied to 4D radar without much modification. However, due to the high sparsity and noises with the 4D radar data, work with other modalities is more preferable. RCFusion [19] fuses features of 4D radar and images in the BEV perspective via attention mechanism. LXL [20] estimates the depth of image through a depth estimation network and then incorporates radar grid occupancy to fuse radar and image features in the BEV perspective. Due to the similarity in the form of point clouds, there has been relatively less prior work focusing on the fusion of 4D radar and Lidar. Interfusion [14] employs an inter-attention mechanism to facilitate information exchange between 4D radar and Lidar at the pillar level, addressing the issue of information loss caused by single sensor. Building on Interfusion, M 2 -Fusion [29] predicts potential target key points on the feature map, and further refines the pillars around these key points, thereby enhancing the accuracy of 3D object detection. Unlike [14] and [29] , we employ voxel level fusion for the two modalities, with an adaptive weighting to balance the fusion. Moreover, we introduce speed compensation for Doppler speed and random modality dropout strategy to enhance the robustness of the network. With minimal additional overhead, our method achieves much higher performance than the baseline method.The model trained by random modality dropout can also better tackle the failure of one modality. To verify this, we test our model by giving only Lidar or radar input, and compare with the model trained by regular process without random dropout. TABLE 4 exhibits the performance gap when only Lidar data is available. As expected, feeding only Lidar modality to the regularly trained RL-Net experiences drastic performance drop, while the model trained with random dropout could almost maintain the performance close to the pure Lidar-trained SECOND [2] model. TABLE 5 shows the effect of random modality dropout with radar input only. Considering the long range and the high sparsity of the 4D radar point clouds, only the result of the car category at IOU=0.5 is reported in TABLE 5 . Similar phenomenon to TABLE 4 can be observed. However, due to the much lower possibility of feeding radar data only in the RLNet's training process, the gap between RLNet and SECOND becomes larger than that in TABLE 4 . Lidar and radar points are marked orange and green, while the predicted and ground truth bounding boxes are in red and blue, respectively. 


Decision: The average score of the paper given all the reviews received before the deadline was higher than 5.5 (1 is lowest, 10 is highest), therefore the paper is accepted. The Authors are encouraged to consider feedback for the camera ready version of the paper due on August 31st.

Reviews: review: title: title: Review of Submission 6
main_review: main_review: RLNet, the core contributions are an adaptive Fusion module to combine the Lidar and radar features by adaptive weights and a special training method with random modality dropout (Inspired by CramNet when a certain sensor is constrained or degraded). Their experiments achieve better performance than single modality input and some other fusion methods (Lidar, Radar, Radar+Image and Lidar+Radar); their experiments also reveal fewer false positive and false negative detections over its counterparts, however, the results were only displayed in the VoD and ZJUODset datasets, did you use other datasets such as BDD100K or nuScenes? how do you calculate the 0.2 from the dropout probability threshold? (eq. 6 and 7). How are the additional speed and timestamps represented in the 6D-vector input?
title: title: Review for 'RLNet: Adaptive Fusion of 4D Radar and Lidar for 3D Object Detection'
main_review: main_review: In this paper, the authors present an approach for 3D object detection that fuses 4D radar and LiDAR data using adaptive feature fusion. The paper's motivation stems from the complementary nature of these two sensing modalities: LiDAR provides precise 3D spatial information, while 4D radar offers robustness in adverse weather conditions and over longer distances. However, both sensors have inherent limitations that the authors aim to address this through a new fusion technique.

The adaptive feature fusion module, which dynamically weights the importance of radar and LiDAR features, is a significant contribution. This method effectively handles the distinct characteristics of each modality. Additionally, the inclusion of radar point speed compensation and the random modality dropout during training are practical enhancements that contribute to the robustness of the model. The ablation studies, which test the impact of these individual architectural improvements, are particularly insightful.

While the paper demonstrates strong results on two datasets, it would be beneficial to explore how and under what conditions this method generalizes to other datasets.
title: title: review of lidar-radar fusion detection
main_review: main_review: The paper proposes a 3D object detection method termed RLNet, which effectively integrates 4D Radar and Lidar through adaptive feature fusion. The framework is relatively simple and has limited innovation. 

pros:
1.The paper is well organized.
2.The experimental result outperforms baseline method by 7.35 and 2.76 percent in mAP on the VoD and ZJUODset dataset.

cons:
1.Existing experiments cannot reflect the excellent performance of the algorithm. It is recommended to compare with more updated Lidar+Radar 3D detection methods. 
2.The Lidar+Radar version Second implemented by the author has very similar performance with the Lidar version Second, and the baseline seems a bit easy.
It is recommended to analyze the reasons for the poor vehicle detection results on the ZJUODset dataset.
title: title: Lightweight lidar-radar fusion detection network
main_review: main_review: The paper proposes a lightweight 3D detection network based on lidar and rader input. The proposed method has three novel modules: a speed compensation module for radar data preprocessing, a fusion module to learn weights when fusing lidar and radar features, and a random modality drop training strategy to be more robust to real-world driving data where sensor data quality is compromised.

The paper is well-written and clear. The experiments are conducted on two small datasets and show the proposed method has outperforms lidar-only methods and some other lidar-radar detection works. 

I believe that the experiments were not conducted on e.g., nuscenes because the radar data is the traditional 3D radar data that doesn't have height information, and other fusion methods (e.g., CenterFusion) are developed based on the 3D radar instead of the 4D radar that this work focuses on.
title: title: Review of Submission6
main_review: main_review: In this paper, the authors propose a simple yet effective radar/lidar fusion method for 3D object detection. The core contribution is that a weighting module is proposed to mix the feature of lidar and radar. The experiments are conducted on VoD/ZJUODset datasets. The paper is well-written and easy to follow. The experiments show that the proposed RLNet achieve better performance than single modality input and some other fusion methods. 
The main weakness is that the experiments are limited on small, non-mainstream datasets and the compared methods are not SOTA.
title: title: a simple 4D Radar and Lidar fusion based 3D object detection method
main_review: main_review: This paper presents a simple 4D Radar and Lidar fusion based 3D object detection method. The proposed methods introduced three modules, namely, a speed compensation module, an adaptive feature fusion module, and a random modality dropout module.
The overall framework is very straightforward. So the technical novelty is kind of limited.
The proposed method is validated on the VoD and ZJUODset datasets and the results show some improvements over the baselines.
Why didn't you conduct comparative experiments on the nuScenes dataset which is more commonly used?
Besides, the compared methods are kind of out-dated.
title: title: A simple fusion method for lidar and radar sensors
main_review: main_review: This submission presents a novel fusion method of 4D radar and lidar sensor-data. The two modalities have similar 3D representation but are characterized by different point distributions also varying on the environmental conditions. The main message of the submission is clear: the adaptive fusion method proposed by the authors achieves superior results compared to the baseline solutions. The ablation study is clear, it helps evaluating the impact of the ingredients of the final solution.

Feedbacks:
It would be useful to discuss the reasons behind using the two specific datasets for the experiments (VoD, ZJUOD) instead of other publicly available lidar+radar datasets 
A more detailed discussion would be helpful about how and why the speed compensation module helps the detection and whether it is related to the synchronization method used in the paper. How the radar timestamps are represented, and how the network is expected to use this information?