Title: Echoes Beyond Points: Unleashing the Power of Raw Radar Data in Multi-modality Fusion
Venue: NeurIPS 2023
Accepted: yes
Keywords: 4D Radar; Transformer; Multi-modality
Abstract: Radar is ubiquitous in autonomous driving systems due to its low cost and good adaptability to bad weather. Nevertheless, the radar detection performance is usually inferior because its point cloud is sparse and not accurate due to the poor azimuth and elevation resolution. Moreover, point cloud generation algorithms already drop weak signals to reduce the false targets which may be suboptimal for the use of deep fusion. In this paper, we propose a novel method named EchoFusion to skip the existing radar signal processing pipeline and then incorporate the radar raw data with other sensors. Specifically, we first generate the Bird's Eye View (BEV) queries and then take corresponding spectrum features from radar to fuse with other sensors. By this approach, our method could utilize both rich and lossless distance and speed clues from radar echoes and rich semantic clues from images, making our method surpass all existing methods on the RADIal dataset, and approach the performance of LiDAR. The code will be released on https://github.com/tusen-ai/EchoFusion.

Echoes Beyond Points: Unleashing the Power of Raw Radar Data in Multi-modality Fusion

Abstract

Radar is ubiquitous in autonomous driving systems due to its low cost and good adaptability to bad weather. Nevertheless, the radar detection performance is usually inferior because its point cloud is sparse and not accurate due to the poor azimuth and elevation resolution. Moreover, point cloud generation algorithms already drop weak signals to reduce the false targets which may be suboptimal for the use of deep fusion. In this paper, we propose a novel method named EchoFusion to skip the existing radar signal processing pipeline and then incorporate the radar raw data with other sensors. Specifically, we first generate the Bird's Eye View (BEV) queries and then take corresponding spectrum features from radar to fuse with other sensors. By this approach, our method could utilize both rich and lossless distance and speed clues from radar echoes and rich semantic clues from images, making our method surpass all existing methods on the RADIal dataset, and approach the performance of LiDAR. The code will be released on https://github.com/tusen-ai/EchoFusion.

1 Introduction

Robust and accurate perception is a long-standing challenge in Autonomous Driving Systems (ADS). The full perception system usually relies on multiple sensor fusion including camera, LiDAR, and radar. Camera provides rich semantic cues of objects and excellent resolution, while LiDAR is capable of capturing highly accurate spatial information. However, neither camera nor LiDAR can directly measure speed or survive in adverse weather conditions, such as fog, sandstorms, and snowstorms [26]. Although radar can overcome the aforementioned problems, even the point cloud from the latest 4D millimeter wave (mmWave) radars suffers from severe sparsity and low angular resolution, which makes it hard to discriminate objects and backgrounds solely. Fortunately, radar and camera are highly complementary to each other. Fusing these two types of sensors becomes a promising solution.

Radar data are usually represented as point clouds. Technically, the radar point cloud stems from raw Analog-Digital-Converter (ADC) data received by antennas. Range, speed, and angle of arrival (AoA) information of perceivable objects can be explicitly extracted in the frequency domain by consecutively applying FFT along corresponding dimensions. Among these steps, side lobe suppression and constant false alarm rate (CFAR) detector [40] are usually adopted to reduce the noise and false alarms. As a result, all the derived points are equipped with spatial coordinates, speed, and reflection intensity. Though these noise suppression operations can significantly reduce the data size and further computation costs, the resulting radar point cloud is extremely sparse, as shown in Figure 1. Even worse, quite a lot of useful information is lost during CFAR, which is adverse to the accurate perception of environments and subsequent fusion with other sensors [18]. Raw data serves as a

(a)

(b)

2

X

Z

potential solution to alleviate these problems. However, how to efficiently utilize it, especially fusing with other sensors remains an open problem.

Recently, Birds-Eye-View (BEV) based methods are prevalent for driving scenario perception. It integrates features from multiple views and multiple modalities into a unified 3D representation space [4; 17; 24]. Among the varieties of BEV-based methods, PolarFormer [9] divides the 3D space in polar coordinates, which matches the format of raw radar data better. As such, we chose PolarFormer as our baseline model.

In this work, we present EchoFusion, a method designated for fusing raw radar data with images. We observe that a set of 3D positions exactly corresponds to one column in the image and one row in raw radar data. Thereby we propose a novel polar-aligned attention (PAA) technique to efficiently fuse features from these two different modalities. With row-wise cross-attention on radar data and column-wise attention on image, PAA could precisely aggregate essential features from both modalities while maintaining simple and efficient implementation. Finally, a polar BEV decoder refines object queries for accurate bounding box predictions.

The contribution of our paper is three-folds:

We are the first to fuse raw radar data with images in BEV space. Specifically, we propose a novel Polar-Aligned Attention module to guide the network effectively learn radar and image features.

We relabel the RADIal dataset [39] with accurate 3D bounding box annotations. The new annotations is published with our codebase together.

Extensive experiments show that our method has outperformed all the existing methods and achieved promising BEV detection performance, even approaching the LiDAR-based method.

2 Related Work

2.1 Deep Learning on Radar

2.1.1 Radar Data Representation and Datasets

Traditional radar usually only outputs sparse points [2; 5; 42] for use. However, such points are subjective to heavy signal processing methods to reduce noisy observation and alleviate bandwidth limitation. Nevertheless, such hand-crafted pipelines make the subsequent fusion with other sensors intractable. Consequently, some attempts have been made to provide upstream data, such as range-azimuth-Doppler (RAD) tensor [34; 54], range-azimuth (RA) maps [44], range-Doppler (RD) spectrums [29] or even raw Analog Digital Converter (ADC) data [19; 29]. Note that the RAD, RA, and RD tensors are generated from ADC data using several times (almost) lossless FFT. Thus we call all these four types of data "radar raw data" in the sequel.

Since our motivation is to deeply fuse with other sensors, we only consider the datasets providing data beyond point cloud. Among them, both RADIal [39], Radatron [26] and KRadar [35] provide such data. But Radatron only covers a single scenario while lacking camera calibration, LiDAR modality,

and height annotation, while KRadar only provides radar RA map and PCD formats. Taking the factors mentioned above into account, we carry out our experiments mainly on the RADIal dataset, and provide additional results on a subset of KRadar dataset. However, the annotations of RADIal only contain object-level points. For a more comprehensive and objective evaluation, we further annotate the bounding box size and heading of each object based on dense LiDAR points. The annotations can be found in our codebase.

2.1.2 Object Detection on Radar Data

Traditional radar object detection methods that utilize point cloud face challenges of super sparsity due to the low angular resolution. Practical treatments include occupancy grid mapping [59] and separate processing for static and dynamic objects [43]. 4D radar technology improves the azimuth resolution and provides additional elevation resolution, which enables the combination of static and dynamic object detection in a single network, using either PointPillars [37] or tailor-made self-attention modules [1; 51].

The pre-CFAR(Constant False Alarm Rate Detector) data provides rich information of both targets and backgrounds. RAD tensor is fully decoded but brings high demand of storage and computation. Hence, Zhang et al. [54] takes the Doppler dimension as channels and Major et al. [27] projects RAD tensor to multiple 2D views. RTCNet [36] divides RAD tensor into small cubes and applies 3D CNN to reduce computation burden. Besides, [39; 55] take complex RD spectrum as input and apply neural networks to automatically extract spatial information. Networks such as RODNet [48] adopt RA maps for detection, which avoid false alarms caused by extended Doppler profile. Despite the aforementioned research works, the utilization of ADC radar data has recently gained increasing attention within the community [7; 52]. However, their results remain unsatisfactory.

The fusion of different type of sensors provides complementary cues, leading to more robust performance. At input level, radar point clouds are usually projected as pseudo-images and concatenated with camera images [3; 33]. At Region of Interest (RoI) level, some approaches [31; 32] consecutively refine the RoIs by radar and other modalities, while others [12; 14] unify RoIs generated independently by different sensors. At feature level, [50; 56] integrate feature maps generated from different modalities, while [11; 13] use RoIs to crop and merge features across modalities. To the best of our knowledge, we are the first to deeply fuse radar using raw data with other modalities in a unified BEV perspective.

2.2 BEV 3D Object Detector

BEV object detection bursts out a rising wave of research due to its vast success in 3D perception tasks. One thread is to adopt transformations to project 2D image features to 3D BEV space for further detection, such as OFT [41] and LSS [38]. Another thread applies initialized BEV queries [58] or object queries [22; 23] to iteratively and automatically sample features from multi-view images. Based on these advanced techniques, BEVFusion [24] explores the advantages of BEV representation in multi-sensor fusion and achieves impressive performances. Despite that, how to make full use of other information sources also attracts the interest of researchers. BEVFormer [17] and its variant [53] utilize temporal information to enhance detection capability, while BEVStereo [16] and STS

0

100

200

0

100

200

300

400

500

ADC data

0

100

200

0m

10m

20m

30m

40m

50m

60m

70m

80m

90m

100m

Range-Time Data

0

100

200

0m

10m

20m

30m

40m

50m

60m

70m

80m

90m

100m

0

100

200

0m

10m

20m

30m

40m

50m

60m

70m

80m

90m

100m

CFAR result

-20 m-10 m 0 m

10 m 20 m

0 m

20 m

40 m

60 m

80 m

100 m

-40°

-30°

-20°

-10°

0°

10°

20°

30°

40°

0 m

20 m

40 m

60 m

80 m

100 m

Range-Azimuth Map

[49] explore how the estimated depth can benefit BEV-based detection. Besides, PolarFormer [9] introduces the polar grid setting and proves its effectiveness in environment perception, which is closely related to our work.

3 Preliminary

In the automotive industry, multiple-input, multiple-output (MIMO) frequency-modulated continuouswave (FMCW) radars are adopted [59] to balance cost, size, and performance. Each transmitting antenna (Tx) transmits a sequence of FMCW waveforms, also named chirps, which will be reflected by objects and returned to the receiving antennas (Rx). Then the echo signal will be first mixed with the corresponding emitted signal and then passed through a low pass filter, so as to obtain the intermediate frequency (IF) signal. The discretely sampled IF signal, which includes the phase difference of emitted and returned signal, is called ADC data [46]. The ADC data has three dimensions, fast time, slow time, and channel, which have physical correspondence with range, speed, and angle respectively.

Because of the continuous-wave nature, the IF signal of a chirp contains a frequency shift caused by the time delay of traveling between radar and objects. Then the radial distance of obstacles can be extracted by applying Fast Fourier Transform (FFT) on each chirp, i.e. fast-time dimension. This FFT operation is also called range-FFT. The resulting complex signal is denoted as range-time (RT) data . Next, a second FFT (Doppler FFT) along different chirps, i.e. slow-time dimension, is conducted to extract the phase shift between two chirps caused by the motion of targets. The derived data is named range-Doppler (RD) spectrum . We denote N Tx and N Rx respectively as the number of transmitting and receiving antennas. And each grid on the RD spectrum has N Tx × N Rx channels. The small distance between adjacent virtual antennas leads to phase shifts among different antennas, which is dependent on the AoA. Thus a third FFT (angle-FFT) can be applied along the channel axis to unwrap AoA. The AoA can be further decoded to azimuth and elevation based on the radar antenna calibration. The final 4D tensor is referred to as the range-elevation-azimuth-Doppler (READ) tensor .

If point cloud is desired, the RD spectrum will be first processed by the CFAR algorithm to filter out peaks as candidate points. Then the angle-FFT will only be executed on these points. Final radar point cloud (PCD) consists of 3D coordinates, speed, and reflective intensity. In traditional radar signal processing, there is another data format named range-azimuth (RA) map , which is obtained by decoding the azimuth of a single elevation on the RD spectrum and compressing the Doppler dimension to one channel. For comprehensive analyses, we include this modality in our experiments as well. The data formats mentioned above are illustrated in Figure 2, except for READ tensor because it is difficult to visualize.

RGB Image

Image

Encoder

Radar

Encoder

Object

Queries

Multi-level

Image Features

Multi-level

Radar Features

Multi-level

BEV

Features

Multi-scale

Polar BEV

Decoder

Final

Predictions

Cross

Attention

𝑐𝑐𝑐𝑐𝑐𝑐

.

𝑥𝑥

𝐼𝐼

Polar Aligned Attention

H

W

R

A

R

T

Range-Time

Tensor

Cross

Attention

BEV Query

@

(

𝑟𝑟

𝑏𝑏𝑏𝑏𝑏𝑏

,

𝜙𝜙

𝑏𝑏𝑏𝑏𝑏𝑏

)

𝑟𝑟𝑐𝑐𝑟𝑟 𝑟𝑟

𝑅𝑅

Coord.

Mapping

4 Methods

In this section, we will introduce our network design. Firstly we will give a brief introduction of the whole pipeline in Section 4.1. Then our proposed Polar-Aligned Attention module will be elaborated in Section 4.2 and Section 4.3. Finally, the detection head and loss functions are introduced in Section 4.4.

4.1 Overall Structure

The overall architecture of our model is shown in Figure 3. First, we use separate backbones and FPNs [20] to extract multi-level features from RT radar maps and images. Then the polar queries aggregate camera features and radar features by cross-attention. Finally, the multi-scale polar BEV map will be passed through a transformer-based decoder with object queries. The final score and bounding box predictions can be obtained by decoding the output object queries. The core of our EchoFusion is how to fuse the camera and radar slow-time data. We fuse them using a novel strategy named Polar Aligned Attention (PA-Attention). In the next two subsections, we will explain this strategy in detail.

4.2 Column-wise Image Fusion

After obtaining image and radar features from corresponding encoders, we first initialize the l -th level polar queries Q l ∈ R R l × A l × d uniformly in polar BEV space. Here, R l , A l , and d respectively denote the shape of range, azimuth, and channel dimensions. For brevity, we omit l in the following two subsections. Given a query q ∈ R d located at ( r bev , ϕ bev ) , it corresponds to a pillar centered at ( r bev , ϕ bev ) with infinite height. Taking the coordinate system defined in RADIal [39], x R , y R , and z R axes respectively points to the front, left, and upright, while subscript R means radar coordinate. Similarly, the camera coordinates are denoted as ( x C , y C , z C ) , which points to the right, down, and front, respectively. Thus we have correspondence in Cartesian coordinate system as follows:

Using extrinsic matrix and intrinsic matrix, we have the following formulation:

where u 0 , f x are principal point offset and focal length on x -axis in the intrinsic matrix, x I is the image column index that corresponds to the point ( x R , y R , z R ) , while R and T are rotation matrix and translation vector of the calibration matrix between camera and radar, respectively. Under mild distortion conditions and normally adopted sensor setting, x I is determined by pillar position and calibration parameters as follows:

Limited by the number of pages, we leave the detailed proof in the appendix.

However, the height is not specified for a query, while for a pixel ( x I , y I ) on the monocular image, its depth is unknown as well. Thus we cannot precisely associate a query to the image feature, but we can correspond queries of the same azimuth to the same column of images. Enlightened by this observation, we use a versatile cross-attention mechanism to flexibly aggregate required features from images. Namely, all the queries with the same azimuth form the query matrix, while all the features in the corresponding column in the image form the key and value matrix. Formally, these two steps can be expressed as follows:

where f I ( x I , y I ) is the image feature located at x I , y I , while details of cross attention function CrossAttn can be found in [47].

4.3 Range-wise Radar Fusion

We denote updated BEV queries as ˆ Q l ∈ R R l × A l × d . After that, it will be used to sample features from radar. Since the radar feature map shares the same range partition with the BEV queries, the updated query ˆ q ( r bev , ϕ bev ) matches the same range index of the radar feature, i.e. r R = r bev . To aggregate all the features from all chirps in one frame, we perform cross attention for a certain query ˆ q with the row r R in RT map as key and value. Formally,

where R l and T l are the height and width of the radar feature map, and f R ( r R , t R ) ∈ R d is the radar feature of position ( r R , t R ) .

Note that though we don't explicitly decode AoA as in RA map, the AoA has been implicitly encoded in the phase difference of the response of different virtual antenna. Consequently, it can be learned from the features in RT map indirectly.

4.4 Head and Loss

We follow polar BEV decoders [9] to decode multi-level BEV features and make predictions from object queries. Since we don't have velocity annotation on this dataset, we remove the corresponding branch. The regression targets include ( d ρ , d ϕ , d z , log l , log w , log h , sin( θ ori -ϕ ) , cos( θ ori -ϕ ) ), where d ρ , d ϕ , d z are relative offset to the reference point ( ρ , ϕ , z ), l , w , h , θ ori are length, width, height, and orientation of bounding box. The classification and regression tasks are respectively supervised by Focal loss [21] and L1 loss.

5 Experiments

In this section, we conduct thorough experiments to validate the effectiveness of our approach. We first introduce the dataset and metrics used in our experiments, then followed by comparisons with other state-of-the-art methods. Lastly, we ablate some designs and potential input format of raw radar data in our method, and discuss some limitations.

5.1 Experimental Settings

The RADIal Dataset [39] provides 8252 annotated frames with synchronized multi-sensor data. Training, validation, and test sets contain 6231, 986, and 1035 frames, respectively. Each object is annotated with a 3D center coordinate of the visible face of the vehicle. The size and orientation of the objects are pre-defined templates, i.e. they are the same for all objects. In the evaluation, precisions and recalls under ten thresholds are calculated. The average precision (AP), average recall (AR), and F1 score are obtained through the ten precisions and ten recalls as the evaluation metric. Range error (RE) and azimuth error (AE) are also reported to evaluate localization accuracy. The annotation method and metrics are quite different to those in common autonomous driving datasets

[2; 45]. Nonetheless, to compare with state-of-the-art methods, we still list our results using the metrics defined in RADIal [39] as a reference.

Re-annotation and more metrics. Based on the provided LiDAR point cloud, we have refined the cluster-center-based annotation to 3D bounding box form, which includes center position, scale, and heading in 3D space. Capitalizing on these new annotations, we are able to apply metrics that are vastly used in 3D object detection tasks. Specifically, we take AP defined in Waymo dataset [45] as our main metrics. Since the LiDAR used in RADIal is only 16-beam, we cannot label accurate height for the objects. So we mainly utilize BEV IoU for evaluation. We use the threshold of 0.7 for normal IoU and longitudinal error-tolerant (LET) IoU computation [8]. Considering the spatial distribution of ground-truth, the distance evaluation is further broken down into two categories: 0 to 50 meters, 50 to 100 meters. For a more comprehensive evaluation, we also introduce the error metrics defined in nuScenes dataset [2], including ATE, ASE, and AOE for measuring errors of translation, scale, and orientation.

The KRadar Dataset [35] is a recently published 4D radar dataset, which contains 58 sequences, i.e. 35K frames of synchronized multi-modality data. However, we have limited time to obtain the full data via shipping, so we only provide results on the sub-dataset stored on Google Drive, which consists of 20 sequences. This so-called KRadar-20 dataset includes 6493 training samples and 6515 test samples, of which 50 % are nighttime data. The training samples are named trainval split, it is further split into train split of 5190 samples and val split of 1303 samples for ablation. Metrics follow KITTI [6] protocols with 40 recall positions. Different from RADIal, it only contains radar formats of range-azimuth map (RA map) and radar point cloud. But the RA map of KRadar contains an extra height dimension, and it enables us to test 3D prediction ability with 4D radar data.

Radar Formats. As introduced in Section 3, there are multiple formats of radar raw data, such as ADC, RT, RD, READ. These formats can be converted sequentially by FFT, which is a linear operation that can be absorbed into the linear layers. So from the perspective of neural network, these formats are equivalent. However, our proposed Polar-Aligned Attention requires a range dimension for indexing key and value in cross-attention, so we choose RT as a representative in the experiments. We also conduct experiments on RA to investigate the necessity of explicit azimuth.

Implementation Details. All our experiments are carried out on eight 3090 GPUs with the same batch size of 8. For the image backbone, we adopt ResNet-50 with pre-trained weights provided by BEVFormer [17]. For LiDAR and radar point cloud branch, we borrow the voxel encoder from PointPillar [15] and use ResNet-50 with modified strides as the backbone. Other representations of radar data share similar modified ResNet-50 as the backbone, but with an extra 1 × 1 convolution at the beginning for channel number alignment. To prevent overfitting, the image-only variant is trained for only 12 epochs while others are trained for 24 epochs. AdamW [25] is adopted as the optimizer and a learning rate of 5e-5 is shared for all models. Due to the limited size of the dataset, we find that the results are unstable across different runs. To make a fair comparison, each experiment is repeated three times. In modality ablation, we show the mean and variance of each experiment result.

5.2 Comparison to State-of-the-art Methods

5.2.1 Results on RADIal Dataset

We first compare our method with state-of-the-art detectors on AP, AR, range error, and azimuth error defined in RADIal [39], with original object-level point annotations 1 . The results are illustrated in Table 2. Our method significantly improves AR and F1 performance and achieves a remarkable improvement over all existing detectors including CMS [10], which also integrates both radar raw data and camera data. Note that RE and AE are not quite informative because these two metrics are calculated on all the recalled objects of a method, which is unfair to high-recall models like ours.

To evaluate in a more rigorous and informative way, we also conduct experiments on the refined annotations and vastly adopted metrics in 3D object detection. FFTRadNet [39] is lifted up with additional branches to predict the scale and orientation of targets. We can only compare with this FFTRadNet variant since codes of other algorithms [7; 10; 52] are not available yet. The results in Table 3 show that the proposed method outperforms FFTRadNet by a large margin. This significant improvement reveals the huge benefit of unleashing the power of radar data in multi-modality fusion.

5.2.2 Results on KRadar Dataset

The baseline on KRadar dataset is the official RTNH [35], which requires radar point cloud. Here, the proposed EchoFusion is input with image and RA map. To align with the baseline, we take the radius and azimuth range respectively as [0, 72] m and [-20, 20] degrees by the limit of camera field of view. The results are shown in Table 4. With RA map and image fusion, our EchoFusion improves over 10 points at each metric, except for BEV AP of IoU threshold 0.3. And since KRadar has higher vertical resolution and more accurate 3D groundtruth than RADIal, our method achieves more considerable improvements in 3D metrics.

5.3 Ablation Studies

We ablate the input format to the radar feature extraction network in Table 5. It shows that the decomposition of complex input can improve overall detection performance, especially for longrange perception. The magnitude and phase representation encodes a non-linear transformation of the complex representation, which may relate to the final prediction target better. We also tried to remove the pre-training of the image branch. The result drops significantly, which confirms that in this relatively small dataset, a good pre-training is crucial for good performance.

5.4 Comparison of Different Modalities

To further study the effects of different modalities, we change the input format indicated in Figure 2 and corresponding backbones, and test with and without image modality. Table 6 presents the results, more results on KRadar can be found in our appendix, and our findings are listed as follows.

Comparing single-modality results It is not surprising that LiDAR ranks first while the camera is the worst in terms of AP. Though poor in depth estimation, the camera has excellent angular resolution, which is beneficial for LET-BEV-AP. It achieves comparable results with radar in LET-BEV-AP. Besides, without the guidance of image, the radar-only network gets better performance as more hand-crafted post-processing are involved. However, their metrics still lag far from those of LiDAR.

Comparing multi-modality with single modality The power of radar data is released by fusing with image. No matter in what data format, all the radar raw representations gain around 30 points improvement by fusing with images. By combining image and range-time data, we obtain BEV AP that is only 0.03 less than that of LiDAR only method. And the LET-BEV-AP and long-range detection ability are even better. We argue that it is mainly because images provide enough clues to decode the essential information from raw radar representation.

Comparing multi-modality results When integrating with images, the LiDAR-fused method still outperforms other radar-fused methods. In terms of different representations of radar data, both RT and RA outperform traditional cloud points, especially in the 50-100m range, in which the difference is as large as 4 points in AP. We owe this finding to the information loss and false alarm of radar point cloud as shown in Figure 1. The performance gap between RT and RA with camera modality is within the error bar, indicating that it is not necessary to explicitly solve the azimuth, nor necessary to permute the Doppler-angle dimensions as FFTRadNet [39] does.

5.5 Discussion and Limitation

Firstly, it is worth noticing that although the overall performance of radar is improved by fusing with the camera, the performance in the short-range is lower than that in the long-range. We speculate the main reason is inaccurate annotation. The 16-beam LiDAR provided in RADIal is too sparse at

a farther distance for annotators to give an accurate size for the long-range objects. In such cases, they are required to assign a template with a fixed size to these objects. These annotations make the problem much easier for farther distances, which leads to higher performance than near distances.

Secondly, though the radial speed label is included in the original annotations of RADIal, radial speed is not quite useful for downstream modules like planning since they need accurate longitudinal and lateral velocity to predict whether the vehicles will interfere with the ego vehicle. However, the synchronized frames are not consecutive in this dataset, it is hard to label velocities for the objects. Considering these factors, velocity prediction is not included in our task.

Finally, as shown in Table 7, our method still lags far from LiDAR-based methods. The primary limitation can be attributed to imperfect z localization, resulting in relatively high Average Localization Error (ATE). The inferior performance is mainly due to coarse LiDAR annotation and inferior radar elevation resolution. The low elevation distinguishing power of LiDAR makes annotation error much worse at long range. As a result, both LiDAR-based and radar-based methods experience a significant drop in 3D AP within the 50-100m interval. But when equipped with high elevation resolution and accurate 3D groundtruth provided by KRadar, our method shows excellent performance in 3D metrics, as shown in Table 4. This emphasizes the pressing need for a raw radar dataset of higher quality that enables better exploration of radar data usage.

6 Conclusion and Future Work

In this work, we have proposed a novel method for radar raw data fusion with other sensors in a BEV space. Our proposed EchoFusion is concise and effective, which outperforms previous work by a significant margin. We are the first to demonstrate the potential of radar as a low-cost alternative for LiDAR in autonomous driving systems through thorough analyses and experiments.

This work is only a starting point to study how raw radar data can be exploited. However, many attempts are limited by the available datasets. We urge a large-scale and high-quality dataset. However, the acquisition of high-quality multi-modality data with accurate annotation needs great effort and deliberate design for clock synchronization and high storage demand. We will try to build the dataset to facilitate further research.

Societal Impacts Our method can be deployed in the autonomous driving system. Performance loss caused by improper usage may increase security risks.

References

Jie Bai, Lianqing Zheng, Sen Li, Bin Tan, Sihan Chen, and Libo Huang. Radar transformer: An object classification network based on 4d mmw imaging radar. Sensors , 2021.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , 2020.

Simon Chadwick, Will Maddern, and Paul Newman. Distant vehicle detection using radar and vision. In ICRA , 2019.

Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao. Futr3d: A unified sensor fusion framework for 3d detection. arXiv preprint arXiv:2203.10642 , 2022.

Jean-Luc Déziel, Pierre Merriaux, Francis Tremblay, Dave Lessard, Dominique Plourde, Julien Stanguennec, Pierre Goulet, and Pierre Olivier. Pixset: An opportunity for 3d computer vision to go beyond point clouds with a full-waveform lidar dataset. In ITSC , 2021.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition , pages 3354-3361. IEEE, 2012.

James Giroux, Martin Bouchard, and Robert Laganiere. T-fftradnet: Object detection with swin vision transformers from raw adc radar signals. arXiv preprint arXiv:2303.16940 , 2023.

Wei-Chih Hung, Henrik Kretzschmar, Vincent Casser, Jyh-Jing Hwang, and Dragomir Anguelov. Let3d-ap: Longitudinal error tolerant 3d average precision for camera-only 3d detection. arXiv preprint arXiv:2206.07705 , 2022.

Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers. AAAI , 2023.

Yi Jin, Anastasios Deligiannis, Juan-Carlos Fuentes-Michel, and Martin Vossiek. Cross-modal supervisionbased multitask learning with automotive radar raw data. IEEE Transactions on Intelligent Vehicles , 2023.

Jinhyeong Kim, Youngseok Kim, and Dongsuk Kum. Low-level sensor fusion network for 3d vehicle detection using radar range-azimuth heatmap and monocular image. In ACCV , 2020.

Youngseok Kim, Jun Won Choi, and Dongsuk Kum. Grif net: Gated region of interest fusion network for robust 3d object detection from radar point cloud and monocular image. In IROS . IEEE, 2020.

Youngseok Kim, Sanmin Kim, Jun Won Choi, and Dongsuk Kum. Craft: Camera-radar 3d object detection with spatio-contextual fusion transformer. AAAI , 2023.

Youngseok Kim, Sanmin Kim, Juyeb Shin, Jun Won Choi, and Dongsuk Kum. Crn: Camera radar net for accurate, robust, efficient 3d perception. arXiv preprint arXiv:2304.00670 , 2023.

Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR , 2019.

Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo. arXiv preprint arXiv:2209.10248 , 2022.

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV , 2022.

Teck-Yian Lim, Amin Ansari, Bence Major, Daniel Fontijne, Michael Hamilton, Radhika Gowaikar, and Sundar Subramanian. Radar and camera early fusion for vehicle detection in advanced driver assistance systems. In NeurIPS workshop , page 7, 2019.

Teck-Yian Lim, Spencer A Markowitz, and Minh N Do. Radical: A synchronized fmcw radar, depth, imu and rgb camera data dataset with low-level fmcw radar signals. IEEE Journal of Selected Topics in Signal Processing , 2021.

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR , 2017.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV , 2017.

Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In ECCV , 2022.

Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petrv2: A unified framework for 3d perception from multi-camera images. arXiv preprint arXiv:2206.01256 , 2022.

Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. ICRA , 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR , 2019.

Sohrab Madani, Jayden Guan, Waleed Ahmed, Saurabh Gupta, and Haitham Hassanieh. Radatron: Accurate detection using multi-resolution cascaded mimo radar. In ECCV , 2022.

Bence Major, Daniel Fontijne, Amin Ansari, Ravi Teja Sukhavasi, Radhika Gowaikar, Michael Hamilton, Sean Lee, Slawomir Grzechnik, and Sundar Subramanian. Vehicle detection with automotive radar using deep learning on range-azimuth-doppler tensors. In ICCV Workshops , 2019.

Michael Meyer and Georg Kuschk. Automotive radar dataset for deep learning based 3d object detection. In EuRAD , 2019.

Mohammadreza Mostajabi, Ching Ming Wang, Darsh Ranjan, and Gilbert Hsyu. High-resolution radar dataset for semi-supervised learning of dynamic objects. In CVPR Workshops , 2020.

Richard M Murray, Zexiang Li, S Shankar Sastry, and S Shankara Sastry. A mathematical introduction to robotic manipulation . CRC press, 1994.

Ramin Nabati and Hairong Qi. Radar-camera sensor fusion for joint object detection and distance estimation in autonomous vehicles. arXiv preprint arXiv:2009.08428 , 2020.

Ramin Nabati and Hairong Qi. Centerfusion: Center-based radar and camera fusion for 3d object detection. In WACV , 2021.

Felix Nobis, Maximilian Geisslinger, Markus Weber, Johannes Betz, and Markus Lienkamp. A deep learning-based radar and camera sensor fusion architecture for object detection. In SDF , 2019.

Arthur Ouaknine, Alasdair Newson, Julien Rebut, Florence Tupin, and Patrick Pérez. Carrada dataset: Camera and automotive radar with range-angle-doppler annotations. In ICPR , 2021.

Dong-Hee Paek, Seung-Hyun Kong, and Kevin Tirta Wijaya. K-radar: 4d radar object detection for autonomous driving in various weather conditions. Advances in Neural Information Processing Systems , 35:3819-3829, 2022.

Andras Palffy, Jiaao Dong, Julian FP Kooij, and Dariu M Gavrila. Cnn based road user detection using the 3d radar cube. IEEE Robotics and Automation Letters , 2020.

Andras Palffy, Ewoud Pool, Srimannarayana Baratam, Julian FP Kooij, and Dariu M Gavrila. Multi-class road user detection with 3+ 1d radar in the view-of-delft dataset. IEEE Robotics and Automation Letters , 2022.

Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV , 2020.

Julien Rebut, Arthur Ouaknine, Waqas Malik, and Patrick Pérez. Raw high-definition radar for multi-task learning. In CVPR , 2022.

Mark A Richards. Fundamentals of radar signal processing . 2014.

Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic feature transform for monocular 3d object detection. BMVC , 2019.

Ole Schumann, Markus Hahn, Nicolas Scheiner, Fabio Weishaupt, Julius F Tilly, Jürgen Dickmann, and Christian Wöhler. Radarscenes: A real-world radar point cloud data set for automotive applications. In IEEE FUSION conference , 2021.

Ole Schumann, Jakob Lombacher, Markus Hahn, Christian Wöhler, and Jürgen Dickmann. Scene understanding with automotive radar. IEEE Transactions on Intelligent Vehicles , 2019.

Marcel Sheeny, Emanuele De Pellegrin, Saptarshi Mukherjee, Alireza Ahrabian, Sen Wang, and Andrew Wallace. Radiate: A radar dataset for automotive perception in bad weather. In ICRA , 2021.

Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR , 2020.

Shunqiao Sun, Athina P Petropulu, and H Vincent Poor. Mimo radar for advanced driver-assistance systems and autonomous driving: Advantages and challenges. IEEE Signal Processing Magazine , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS , 2017.

Yizhou Wang, Zhongyu Jiang, Yudong Li, Jenq-Neng Hwang, Guanbin Xing, and Hui Liu. Rodnet: A real-time radar object detection network cross-supervised by camera-radar fused object 3d localization. IEEE Journal of Selected Topics in Signal Processing , 2021.

Zengran Wang, Chen Min, Zheng Ge, Yinhao Li, Zeming Li, Hongyu Yang, and Di Huang. Sts: Surroundview temporal stereo for multi-view 3d detection. arXiv preprint arXiv:2208.10145 , 2022.

Zizhang Wu, Guilian Chen, Yuanzhu Gan, Lei Wang, and Jian Pu. Mvfusion: Multi-view 3d object detection with semantic-aligned radar and camera fusion. arXiv preprint arXiv:2302.10511 , 2023.

Baowei Xu, Xinyu Zhang, Li Wang, Xiaomei Hu, Zhiwei Li, Shuyue Pan, Jun Li, and Yongqiang Deng. Rpfa-net: A 4d radar pillar feature attention network for 3d object detection. In ITSC , 2021.

Bo Yang, Ishan Khatri, Michael Happold, and Chulong Chen. Adcnet: End-to-end perception with raw radar adc data. arXiv preprint arXiv:2303.11420 , 2023.

Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird's-eyeview recognition via perspective supervision. arXiv preprint arXiv:2211.10439 , 2022.

Ao Zhang, Farzan Erlik Nowruzi, and Robert Laganiere. Raddet: Range-azimuth-doppler based radar object detection for dynamic road users. In CRV , 2021.

Guoqiang Zhang, Haopeng Li, and Fabian Wenger. Object detection and 3d estimation via an fmcw radar using a fully convolutional network. In ICASSP , 2020.

Jingwei Zhang, Ming Zhang, Zicheng Fang, Yulong Wang, Xian Zhao, and Shiliang Pu. Rvdet: featurelevel fusion of radar and camera for object detection. In ITSC , 2021.

Lianqing Zheng, Zhixiong Ma, Xichan Zhu, Bin Tan, Sen Li, Kai Long, Weiqi Sun, Sihan Chen, Lu Zhang, Mengyue Wan, et al. Tj4dradset: A 4d radar dataset for autonomous driving. In ITSC , 2022.

Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmentation. In CVPR , 2022.

Yi Zhou, Lulu Liu, Haocheng Zhao, Miguel López-Benítez, Limin Yu, and Yutao Yue. Towards deep radar perception for autonomous driving: Datasets, methods, and challenges. Sensors , 2022.

Supplementary Material

A Proof for column and pillar correspondence

In this section, we will provide a detailed proof for the correspondence between pillar in radar coordinate and column in camera coordinate as described in Section 4.2 of our main paper.

Suppose that the rotation matrix and translation vector between radar and camera are:

Then for a point located at ( r bev , ϕ bev , z ) under radar's polar coordinate, it has cartesian coordinate ( x R , y R , z R ) :

and it can be projected to the camera frame using extrinsic parameters:

Using the intrinsic parameters, the camera coordinate ( x C , y C , z C ) can be correlated with image pixel position ( x I , y I ) using:

where f x , f y , u 0 , and v 0 are intrinsic parameters. Our goal is to find a situation that the pillar is projected as a column on the image plane. Under this condition, x I should be irrelevant with z , i.e. in the equation below:

The coefficients of z should be 0, which means by combining the relationship between rotation matrix R with roll α , pitch β , and yaw γ [30], we can derive:

Considering second row of Eq. (S6), there are two solutions β = ± π 2 or α = ± π 2 . Considering the first solution β = ± π 2 , the first row of Eq. (S6) is transformed to:

For another solution α = ± π 2 , the first row of Eq. (S6) is transformed to:

Thus we have:

As a result, the final solution is:

which means:

Thus we have:

In standard ADS, the LiDAR or radar coordinate system usually has a z-axis pointing up, while the camera coordinate system has a y-axis pointing up or down. So a widely adopted practice first exchanges the y and z axes, which means a roll α = ± π 2 . After that, a pitch β around the vertical y-axis is executed, while further rotation around the z-axis is not required, as shown in Figure S1. In other words, the roll β can be any value between -π and π , while yaw γ is set to 0. This is exactly the situation of the third solution of Eq. (S12). And under this condition, we have expression of R according to [30]:

x

z

y

𝛼𝛼

=

𝜋𝜋

2

y

x

z

Camera

x

y

z

𝛽𝛽

pillar@

(

𝑟𝑟

𝑏𝑏𝑏𝑏𝑏𝑏

,

𝜙𝜙

𝑏𝑏𝑏𝑏𝑏𝑏

)

column@

𝑥𝑥

𝐼𝐼

Radar

Thus, using Eq. (S3) and Eq. (S5), we can get the expression of column index x I . If α = π 2 , we have:

For RADIal, the rotation matrix from the radar coordinate system to the camera coordinate system is:

which is the approximation of ( α, β, γ ) = ( π/ 2 , -π/ 2 , 0) . And thus we approximately have:

It can be inferred that x I is almost irrelevant to z , which means one pillar in the polar coordinate of radar corresponds to a column in the image.

B Visualization and Dataset Statistics

To better illustrate the performance of our method, we visualize the predictions and ground truths in Figure S2. As can be seen from the cases in the first and the second column, our method can accurately predict the position of vehicles. But there are several cases in which the accurate prediction does not get high confidence, as shown in the upper case in the third column. On the other hand, the orientation estimation error may be significant when the yaw of the vehicle is large, shown in the inferior case in the third column. The main reason is that the samples with large yaw are relatively rare in the dataset, as shown in Figure S3 (b) and (c).

As shown in Figure S3 (a), the samples of test data are rather rare within 30 meters and out of 80 meters. To comprehensively explore the detection performance of objects, we divide the range into 0-50m and 50-100m, which correspond to short-range and long-range detection performance respectively.

We further visualize scenarios that are corrected by our new labels. It is worth noting that the original annotation and prediction of previous methods are centers of objects' visible faces, and they assign a fixed scale and zero orientation when calculating Average Precision (AP). But vehicles of large angles and scales are normal in daily scenarios, as shown in Figure S4. Our new annotations based on dense LiDAR points correct this, helping a more comprehensive evaluation of existing methods. We also annotate other traffic participants such as pedestrians and cyclists. But due to their limited numbers (4 cyclists, 23 pedestrians in training split, and 17 cyclists, 0 pedestrians in test split), we don't add more fine-grained class labels.

If α = -π 2 , we have:

10m

50m

20m

10m

20m

20m

0m

20m

40m

60m

80m

100m

40m

60m

80m

100m

Om

20m

40m

60m

80m

100m

(c)

(a)

(b)

Range distribution of RADlal test split

Yaw distribution of RADlal train split

Yaw distribution ofRADlal testsplit

70

150*

150°

60

100*

100*

Number of samples

50

50*

50°

40

0a

OE

-50°

-50*

20

-100°

-100*

10

-150°

-150°

0

20m

40m

60m

80m

100m

20m

40m

60m

80m

100m

0m

20m

40m

60m

80m

100m

C Complementary property of radar and camera

To highlight the complementary nature of radar and camera properties, we investigated the performance of LET-BEV-AP with varying longitudinal tolerance ( T l ), as presented in Table S1. Notably, when using a larger T l of 0.3, the camera's performance experiences a significant improvement. This suggests that the relatively lower performance of long-range LET-BEV-AP under T l of 0.1 primarily stems from depth errors. By reducing the impact of longitudinal estimation, the camera's exceptional angular perception accuracy becomes more pronounced. Conversely, the LET-BEV-AP performance of the radar-only variant remains relatively unchanged. This finding indicates that the radar's depth estimation is quite accurate and minimally affected by the value of T l . Similarly, the highly accurate LiDAR-only variant and camera and RT fusion variant don't suffer from performance fluctuation as well, which supports our conclusion.

To further investigate the primary factor affecting radar performance, we conducted a comparison of BEV recall at IoU thresholds of 0.7 and 0.3, as outlined in Table S2. It can be deduced that by using a

Range

A

B

C

D

-Vmax

-Vmax/2

0

Vmax/2

Vmax

less stringent IoU threshold, the recall of the RT-only variant experiences a significant increase. This result suggests that the RT-only variant's limitation mainly lies in its inferior angular localization ability. But by combining the strengths of camera and radar modalities, the model can leverage the advantages of both modalities, leading to a significant performance boost.

D Doppler domain multiplexing and coordinate system

As mentioned in [39], Doppler Domain Multiplexing(DDM) is used to distinguish received radar signals from different transmitters. The mechanism can be best illustrated in Range-Doppler (RD) spectrum format of data, as shown in Figure S5. Without DDM, the valid measurable speed interval is [ -V max , V max ] . Suppose we have four transmitters, Tx 1/2/3/4, and one object with speed in [0 , V max / 2] . The echo signals from Tx 1/2/3/4 will respectively fall into sections C, D, A, B by using DDM, showing a periodical pattern. But now we can only measure speed in [0 , V max / 2] . Similar

phenomenon can be observed in the figure of Range Doppler data, i.e. the third image in Figure 2 in the paper.

We also carried out experiments on coordinate system of BEV space. Though the range dimension is under sphere system, we found that a polar system can achieve on par performance, as shown in Table S4. Thus we apply the polar system in the paper.

E More experimental results

Referring to Table S3, we have included the detection performance when our model consumes the ADC data or range-Doppler spectrum as input. It is worth noticing that we add an extra complex linear layer initialized with FFT coefficients for ADC data to replace the range FFT and the side-lobe suppression. In the case of the single modality version, the performance discrepancy between RT and these two modalities remains within the margin of error. When considering the fusion of image data, the performance of RD slightly lags behind that of RT, while that of ADC is almost the same as RT. These findings suggest that the integration of image data significantly mitigates the importance of conventional radar signal processing in accurate radar-based detection.

Besides, we offer ablation of different modality combinations on KRadar. The results are reported in Table S5. Here, we train all models with training split. The RA map shows considerable superiority over radar point cloud when fusing with the monocular image, which aligns with our research motivation and results on RADIal dataset.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method to perform image and raw radar fusion for object detection from a Mobile platform. The authors explain that the commonly used radar data processing pipelines discard too much data, resulting in very sparse point clouds. In contrast, the proposed method operates on raw data and performs end-to-end filtering within the model with intermodality cross-attention. They consider the RADIal radar dataset, for which they provide more precise and exhaustive annotations to better evaluate their method's added precision.\n\nSTRENGTHS:\n- The motivation and results are convincing, and the analysis interesting\n- The model is natural and exploits the unique properties and structure of radar acquisitions\n- Extensive ablation study and new annotations\n- Not a lot of work on this interesting and promising modality exists\n\nWEAKNESSES:\n- The method section could be more precise and detailed. The CrossAttn operator is not defined with enough details (what are its arguments, Why are arguments 2 and 3 are always the same?), and neither are polar queries. There is a reference to [8] at the end of the method which helps a  little bit, but the paper should be readable on its own. As it stands, the cross-attention method is not reimplemetable.\n\n- While submissions about sensors not typically treated at NeurIPS are welcome, the community is not necessarily familiar with the subtlety of radar data. As it stands, the submission overly relies on jargon and complex facts presented as self-evident. The figures and their caption could be more helpful. The authors should be more didactic.\n\n- The writing could be significantly improved, as it contains many imprecise and clumsy formulations.\n\nQUESTIONS:\nThis submission presents a well-motivated and designed method with promising results. However, the presentation makes it hard to follow the methodology, and the lack of pedagogy makes it hard to understand for non-radar experts. However, the weaknesses should be mostly fixable.\n\nQ1) What are the arguments of CrossAttn? Why are arguments 2 and 3 are always the same?\n\nQ2) What are polar queries? Are they different from object queries?\n\nQ3) \"The polar coordinate is aligned with radar data on the range dimension and aligned with image data on the azimuth dimension\"\nThis sentence is hard to parse. What does this mean for a coordinate system to be aligned with a type of data?\n\nQ4) Figure 1 is unclear and hard to parse. Which one is the proposed method? Whar exactly are the methods compared? What are the red circles supposed to highlight?\n\nS1) Figure 2 could be great for explaining concretely the nature of the data presented. However, it is full of jargon and acronym and not clear at all. Maybe on a simpler example? Or even a toy example?\n\nLIMITATIONS:\ngood", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-22"}

comments: SUMMARY:
This paper proposes a Polar-Aligned Attention module to guide the fusion of radar data and images. The module projects the image and radar features onto a polar coordinate system to calculate the similarity of the associated information. This effectively enhances the direct fusion of radar data and images. Additionally, the authors have restandardized the RADIal dataset, improving its accuracy and standardization.

STRENGTHS:
The authors claim that this paper is the first work to directly fuse radar data with image data, achieving the fusion of relevant features through cleverly designed attention mechanism modules. Additionally, the authors' reannotation of the dataset has played a positive role in the development of this research direction.

WEAKNESSES:
Although this paper proposes a polar-coordinate-based attention mechanism module for the fusion of radar and image data, the contribution of such a structure is limited. Furthermore, due to the lack of dataset support for the research task of radar and image fusion in this study, the experiments were only conducted on an existing dataset with modifications. Therefore, I do not have enough confidence to affirm the sufficiency of the research motivation in this paper.

QUESTIONS:
1. How is the alignment of different modalities ensured when fusing the features of image and radar data?
2. Is it possible to visualize the alignment relationship between the two modalities in the polar-coordinate attention mechanism?

LIMITATIONS:
Due to the limitations of the dataset used in this study, it is not possible to extensively and sufficiently demonstrate the motivation behind the research in this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a method for camera-radar detection and fusion. The authors propose an approach that follows:\n\nYanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers. AAAI, 2023.\n\nand \n\nJulien Rebut, Arthur Ouaknine, Waqas Malik, and Patrick Pérez. Raw high-definition radar for multi-task learning. In CVPR, 2022.\n\nSpecifically, the paper introduces a polar transformer method that cross-attends image and radar features across modalities according to view and sampling geometry. The technique relies on RAW radar measurements as input and, similar to recent works (above), is evaluated on an existing dataset. The approach is validated to be on par with recent methods.\n\nSTRENGTHS:\nThe cross-modal fusion is well-motivated and has shown to be effective. Although a large body of work has proposed similar methods that also rely on RAW radar data, the design of the proposed method is sound and well-executed. Devising methods that operate on RAW data is an exciting opportunity for the field to embrace ideas from computational imaging in vision models.\n\nWEAKNESSES:\nThe proposed method lacks novelty and experimental margins. Specifically, areas of concern are the following:\n\n*Novelty and Technical Contribution: A large body of work has explored BEV-based fusion approaches and, recently, RAW-based detections. Specifically, this work directly follows the core ideas from:\n\nYanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers. AAAI, 2023.\n\nand \n\nJulien Rebut, Arthur Ouaknine, Waqas Malik, and Patrick Pérez. Raw high-definition radar for multi-task learning. In CVPR, 2022.\n\nWhile the specific attention mechanism is new, it also extends the formulation in PolarFormer to the radar measurement formation model. As such, the technical contribution of the method is incremental in comparison to existing work in the field. \n\n*Experimental Margins: As a results of the relatively incremental changes in architecture and training, the margins that the proposed method achieves are also also incremental. The method is on par with FFTRadNet on the original dataset, but improves after labeling. However, it is unclear if the baseline method has been retrained and all other baseline methods are also missing in the dataset. Moreover, without validating the relabeled dataset separately, it is unclear if the relabling biases the evaluation.\n\nQUESTIONS:\nCan the authors include examples of mislabeled objects or bias in the dataset? \nWhat scenarios are corrected and why do they contribute significantly to the evaluation?\n\nLIMITATIONS:\nThe authors discuss the limitations of their work and societal impacts of the method adequately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper proposes a method for fusing raw radar data with images in bird’s eye view (BEV) space. To this end, the method extract multilevel features from images and radar and uses polar-aligned attention mechanism with polar-BEV decoder for bounding box prediction. Evaluation is done on re-annotated RADIal dataset and shows better performance than other comparable methods.

STRENGTHS:
The paper utilizes raw radar data which in general is a better approach than relying on post-processed radar data for object detection.

WEAKNESSES:
The BEV cross attention assumes synchronized radar and image which may not be robust at high speed driving and unsynchronized radar and camera setup.

QUESTIONS:
How robust is the proposed model to spatio-temporal misalignment?

LIMITATIONS:
The method assumes synchronized radar and camera.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method to fuse radar and imagery data for autonomous driving. Instead of using processed radar data, which discards most of the information in the original signal while attempting to filter out noise, this method leverages the raw data directly as input to the network. The method is based on attentional mechanisms cued to range in radar and azimuth in imagery, as these measurements are more accurate in those modalities. This technique and the use of raw data seem quite novel in this domain. The experiments are very thorough, on the only dataset that provides raw radar data. Full 3D bounding box annotations are added to this dataset, which is a useful contribution.\n\nSTRENGTHS:\nThe introduction clearly lays out the motivation of using raw vs. processed radar data, and Fig. 1 provides a clear example of how this improves performance. \n\nLines 44-46 state “This technique is based on an observation that the polar coordinate is aligned with radar data on the range dimension and aligned with image data on the azimuth dimension.” It’s not clear with “aligned with” means here, but it seems you mean to say that radar data is more informative in the range dimension while imagery is more informative in the azimuth direction. It would be helpful to clarify this important point in the paper.\n\nAdding vehicle bounding boxes to RADial is a significant contribution for those looking to study radar in autonomous driving. It would be useful to expand the annotations beyond a single “vehicle” class however, as with most of the other autonomous driving datasets.\nThe related work section is thorough and covers the main areas of claimed contributions.\n\nThe overall architecture, Fg. 3, nicely highlights the key contribution of the cross-attentional module in polar coordinates that emphasizes azimuthal accuracy in imagery and range accuracy in radar. \n\nThe approach in sec. 4 is clearly explained in text, supported by equations but without relying on those equations to encode the concepts, making the paper more readable and impactful.\n\nThe experiments are thorough, albeit on a single dataset as mentioned below. The primary comparative method [36], which operates on the processed radar data, performs considerably worse on the most important test, table 3, against the newly-created 3D bounding box annotations. The proposed method also does very well against all baselines using the standard point annotations on the original dataset.\nThe ablation studies are very thorough, examining performance on each sensor independently in order to show the value of fusion.\n\nWEAKNESSES:\nExperiments are performed on a single dataset, which is a minor limitation but unavoidable because it is the only one that provides raw radar data calibrated with imagery.\n\nThe focus of the paper is autonomous driving, which is a huge topic and is generally sufficient for papers at major AI conferences. Nevertheless it would be compelling to see how the method performs in other domains where radar and imagery could be fused, such as pedestrian detection and tracking, maritime vessel detection and so on. Datasets for these are hard to find, but perhaps the RADial dataset contains pedestrians that might have some motion signature in the radar data.\n\nQUESTIONS:\nAs above.\n\nLIMITATIONS:\nYes, no concerns.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}