Echoes Beyond Points: Unleashing the Power of Raw Radar Data in Multi-modality Fusion
Authors: Yang Liu
Feng Wang
Naiyan Wang
Zhaoxiang Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: 4D Radar; Transformer; Multi-modality
TL;DR: Fusing raw radar data and image under BEV space, achieving performance closing to LiDAR.

Abstract: Radar is ubiquitous in autonomous driving systems due to its low cost and good adaptability to bad weather. Nevertheless, the radar detection performance is usually inferior because its point cloud is sparse and not accurate due to the poor azimuth and elevation resolution. Moreover, point cloud generation algorithms already drop weak signals to reduce the false targets which may be suboptimal for the use of deep fusion. In this paper, we propose a novel method named EchoFusion to skip the existing radar signal processing pipeline and then incorporate the radar raw data with other sensors. Specifically, we first generate the Bird's Eye View (BEV) queries and then take corresponding spectrum features from radar to fuse with other sensors. By this approach, our method could utilize both rich and lossless distance and speed clues from radar echoes and rich semantic clues from images, making our method surpass all existing methods on the RADIal dataset, and approach the performance of LiDAR. The code will be released on https://github.com/tusen-ai/EchoFusion.

Echoes Beyond Points: Unleashing the Power of Raw
Radar Data in Multi-modality Fusion
Yang Liu CASIA
Feng Wang TuSimple
Naiyan Wang TuSimple
Zhaoxiang Zhang MAIS of CASIA & UCAS, HKISI_CAS
(1iuaang2022, zhaoxiang zhang) (ia.ac) accaaaac.o cn
{feng. wff, kinsty)0gmail, com com
Abstract
Radar is ubiquitous autonomous driving systems due to low cost and good adaptability to bad weather. Nevertheless, the radar detection performance is usually inferior because point cloud sparse and not accurate due to the poor azimuth and elevation resolution. Moreover, point cloud generation algorithms already drop weak signals to reduce the false targets which may be suboptimal for the use of deep fusion. In this paper, propose novel method named EchoFusion to skip the existing radar signal processing pipeline and then incorporate the radar raw data with other sensors. Specifically, we first generate the Bird's Eye View (BEV) queries and then take corresponding spectrum features from radar fuse with other sensors. By this approach, our method could utilize both rich and lossless distance and speed clues from radar echoes and rich semantic clues from images, making our method surpass all existing methods the RADIal dataset, and approach the performance of LiDAR. The code will be released on https///gghubbcco/muuuueaaaaa//t
1 Introduction
Robust and accurate perception is long-standing challenge Autonomous Driving Systems (ADS). The full perception system usually relies multiple sensor fusion including camera, LiDAR, and radar. Camera provides rich semantic cues objects and excellent resolution, while LiDAR is capable of capturing highly accurate spatial information. However, neither camera can directly measure speed or survive adverse weather conditions, such as fog, sandstorms, and snowstorms (26). Although radar overcome the aforementioned problems, even the point cloud from the latest 4D millimeter wave (mm Nave) radars suffers from severe sparsity and low angular resolution, which makes it hard discriminate objects and backgrounds solely. Fortunately, radar and camera are highly complementary to each other. Fusing these two types of sensors becomes promising solution.
LiDAR
Radar data are usually represented as point clouds. Technically, the radar point cloud stems from raw Analog- Digital-Conver (AoA) information of perceivable objects can be explicitly extracted in the frequency domain by consecutively applying FFT along corresponding dimensions. Among these steps, side lobe suppres- sion and constant false alarm rate (CFAR) detector (40) are usually adopted to reduce the noise and false alarms. As result, all the derived points equipped with spatial coordinates, speed, and reflection intensity. Though these noise suppression operations can significantly reduce the data size and further computation costs, the resulting radar point cloud is extremely sparse, as shown in Figure I Even worse, quite a lot of useful information is lost during CFAR, which is adverse to the accurate perception of environments and subsequent fusion with other sensors [ Raw data serves
(ADC) data received by antennas. Range, speed, and angle of arrival
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a)
(b)
Figure I: Illustration of sparsity and false alarms of the radar point cloud. The red points and the blue points correspond to data from radar and LiDAR, respectively. We visualize the predicted bounding boxes of different variants of our proposed method. red denotes ground-truth, purple denotes predictions from image only modality, green denotes predictions from image and radar range-time data fusion, and blue denotes predictions from image and radar point cloud fusion. Best viewed on screen.
potential solution alleviate these problems. However, how efficiently utilize it, especially fusing with other sensors remains an open problem.
Recently, Byrds-Eye- View (BEV) based methods prevalent for driving scenario perception. It integrates features from multiple views and multiple modalities into a unified 3D representation space 17 24 Among the varieties of BEV-based methods, PolarFormer [9] divides the 3D space in polar coordinates, which matches the format of raw radar data better. As such, we chose PolarFormer as our baseline model.
In this work, we present EchoFusion, a method designated for fusing raw radar data with images. We observe that set of 3D positions exactly corresponds to one column in the image and one row in raw radar data. Thereby we propose a novel polar-aligned attention (PAA) technique to radar
efficiently fuse features from these two different modalities. With row-wise cross-attentin data and column-wii -wiee attention on image, PAA could precisely aggregate essential features from both modalities while maintaining simple and efficient implementation. Finally, polar BEV decoder refines object queries for accurate bounding box predictions.
The contribution of our paper is threc-folds:
We are the first to fuse raw radar data with images in BEV space. Specifically, we propose novel Polar-Aligned Attention module to guide the network effectively learn radar and image features.
2. We relabel the RADIal dataset 339 with accurate 3D bounding box annotations. The new annotations is published with our codebase together.
3. Extensive experiments show that our method has outperformed all the existing methods and achieved promising BEV detection performance, even approaching the LiDAR-based method.
2 Related Work
2.1
Deep Learning on Radar
2.1.1 Radar Data Representation and Datasets
Traditional radar usually only outputs sparse points 2
for use. However, such points are subjective to heavy signal processing methods to reduce noisy observation and alleviate bandwidth limitation. Nevertheless, such hand-crafted pipelines make the subsequent fusion with other sen- sors intractable, Consequently, some attempts have been made to provide upstream data, such as srne-aazimuth-Doppple (RAD) tensor (34 54], range-azimuth (RA) maps 44 range- -ooplerr (RD) spectrums [29 or even raw Analog Digital Converter (ADC) data [19 29 Note that the RAD, RA,
5 42
and RD tensors generated from ADC data using several times (almost) lossless FFT. Thus we call all these four types of data "radar raw data" in the sequel.
Since our motivation is to deeply fuse with other sensors, only consider the datasets providing data beyond point cloud. Among them, both RADIal [39], Radatron (26] and KRadar [35] provide such data. But Radatron only covers single scenario while lacking camera calibration, LiDAR modality,

---

Table 1: 4D Radar Datasets Comparison. Data: PC, RA, ADC denote to point cloud, range azimuth map and raw ADC data; Sensors: C, C., L, denote camera, stereo camera, LiDAR, and odometer; Scenarios: U, S, H denote to urban (city), suburban, highway; Annotations: 3D, BEV, T, Po. M denote to 3D bounding box, BEV bounding box, track ID, object-level point and segmentation mask; Classes and Size denote the number of classes and the number of annotated frames of each dataset.
Dataset
Data
Other Sensors
Scenarios
Annotations
Classes
Size
Astyx 28
PC
CL
SH
3D
7
500
View-of Delft 671
PC
C,LO
3D,T
13
8693
RADIal 39
ADC.RA,CC
CLO
USH
P..M
8252
TJ4DRadSct
57
PC
CLO
U
3D,T
7757
Radatron 26
ADC,RA
C,
U
BEV
16K
KRadar 35
RA, PC
CLO
USH
3D,T
5
35K
and height annotation, while KRadar only provides radar RA map and PCD formats. Taking the factors mentioned above into account, we carry out our experiments mainly on the RADIal dataset, and provide additional results only contain object-level points. For a more comprehensive and objective evaluation, we further annotate the bounding box size and heading of each object based dense LiDAR points. The annotations can be found in our codebase.
subset of KRadar dataset. However, the annotations of RADIal
2.1.2 Object Detection on Radar Data
Traditional radar object detection methods that utilize point cloud face challenges of super sparsity due to the low angular resolution. Practical treatments include occupancy grid mapping (59) and separate processing for static and dynamic objects [43]. 4D radar technology improves the azimuth resolution and provides additional elevation resolution, which enables the combination of static and dynamic object detection in single network, using either PointPillars 337 or tailor-made
self-attention modules 1 51
The pre-CFAR(Constant False Alarm Rate Detector) data provides rich information of both targets and backgrounds. RAD tensor fully decoded but brings high demand storage and computation. Hence, Zhang et al. [54] takes the Doppler dimension channels and Major et al. 27 projects RAD tensor to multiple 2D views. RTCNet (36) divides RAD tensor into small cubes and applies 3D CNN to reduce computation burden. Besides, (39 (55] take complex RD spectrum as input and apply neural networks to automatically extract spatial information. Networks such as RODNet [48) adopt RA maps for detection, which avoid false alarms caused by extended Doppler profile. Despite the aforementioned research works, the utilization of ADC radar data has recently gained increasing However, their results remain unsatisfactory.
attention within the community 7 52
The fusion different type of sensors provides complementary cues, leading to more robust perfor- mance. At input level, radar point clouds are usually projected pseudo-images and concatenated
with camera images [3] 33 utively refine the Rols by radar and other modalities, while others 12: 14]
Region of Interest (Rol) level, some approaches [31 32 consec-
unify Rols generated independently by different sensors. At feature level, [50; [56] integrate feature maps generated from different modalities, while [II: [3] use RoIs to crop and merge features across modalities. To the best of our knowledge, we are the first to deeply fuse radar using raw data with other modalities in a unified BEV perspective.
2.2 BEV 3D Object Detector
BEV object detection bursts out a rising wave of research due its vast success 3D perception tasks. One thread is to adopt transformations to project 2D image features to 3D BEV space for
further detection, such as OFT 41 and LSS [8] Another thread applies initialized BEV queries (58]
or object queries [22 23 to iteratively and automatically sample features from multi-view images.
Based on these advanced techniques, BEVFusion (24) explores the advantages BEV representation in multi-sensor fusion and achieves impressive performances. Despite that, how make full use of
other information sources also attracts the interest of researchers. BEVFrrmer [17 and its variant
[53 utilize temporal information to enhance detection capability, while BEVStereo [16] and STS
3

---

Range-Tme
Rangle: Doopler
Posnt Gouds
Range-Arimuth Hap
544
50m-
10m
50m
20 -44
50
50m
50m
204
11e
20m
20m
26m
20m
2
290
300
201
302
200 390
230
Ere 389
293
D10
Figure 2: Radar data formats used in this paper. All the axes that indicate spatial location are labeled with units. For ADC data, the vertical unit is the sampling cycle. And for ADC data, Range- Range-Time data, the horizontal unit is the chirp cycle. Due to Doppler domain multiplexing (DDM), the real speed is periodic on the Doppler dimension, which is hard to label (More details of DDM can be found in the appendix). We use the index number of the Doppler dimension as horizontal unit instead in Sange-Doppler and CFAR result.
49 explore how the estimated depth can benefit BEV-based detection. Besides, PolarFormer [|] introduces the polar grid setting and proves its effectiveness in environment perception, which is closely related to our work.
3 Preliminary
In the automotive industry, multiple-input, multiple-output (MIMO) frequency-mmodulated continuous- wave (FMCW) radars are adopted [59] to balance cost, size, and performance. Each transmitting antenna (Tx) transmits a sequence of FMCW waveforms, also named chirps, which will be reflected by objects and returned to the receiving antennas (Rx). Then the echo signal will be first mixed with the corresponding emitted signal and then passed through low pass filter, so as obtain the intermediate frequency (IF) signal. The discretely sampled IF signal, which includes the phase difference of emitted and returned signal, called ADC data [46.. The ADC data has three dimensions, fast time, slow time, and channel, which have physical correspondence with range, speed, and angle respectively.
Because of the continuous-waye nature, the IF signal of a chirp contains a frequency shift caused by the time delay of traveling between radar and objects. Then the radial distance of obstacles can be extracted by applying Fast Fourier Transform (FFT) on each chirp, i.c. fast-time dimension. This FFT operation is also called range-FFT. The resulting complex signal denoted range-time (RT) data. Next, a second FFT (Doppler FFT) along different chirps, i.c. slow-time dimension, is conducted to extract the phase shift between two chirps caused by the motion of targets. The derived data is named range-Doppler (RD) spectrum. We denote Ny. and Nat respectively as the number of transmitting and receiving antennas. And each grid on the RD spectrum has Nr. x NRt channels. The small distance between adjacent virtual antennas leads to phase shifts among different antennas, which is dependent on the AoA. Thus a third FFT (angle-FFT) can be applied along the channel axis to unwrap AoA. The AoA can be further decoded to azimuth and elevation based on the radar antenna calibration. The final 4D tensor is referred to as the tensor.
(READ)
If point cloud desired, the RD spectrum will be first processed by the CFAR algorithm to filter out peaks candidate points. Then the angle-FFT will only be executed on these points. Final radar point cloud (PCD) consists of 3D coordinates, speed, and reflective intensity. In traditional radar signal processing, there is another data format named range-azimuth (RA) map. which is obtained by decoding the azimuth of single elevation on the RD spectrum and compressing the Doppler dimension to one channel. For comprehensive analyses, we include this modality in our experiments as well. The data formats mentioned above are illustrated in Figure 2 except for READ tensor because it is difficult to visualize.

---

col.x/
Coord. Mapping
Predictions
Image Encoder
Cross Attention
Multi-scale Polar BEV Decoder
Multi-level
Image Feat.
RGB Image
Image Features
Radar Encoder
row
BEV Query Pbes. Poev)
Cross Attention
Multi-level
Radar Feat.
Multi-level BEV Features
Object Queries
Range-Time Tensor
Radar Features
Polar Aligned Attention
Figure 3: Overall pipeline of the proposed EchoFusion. H and W are the height and weight of the image, while R, A, and are the bin number of range, azimuth, and Doppler dimension, respectively, The Coord. Mapping in the figure is explained in section 4.2
Methods
In this section, will introduce network design. Firstly will give brief introduction of the whole pipeline in Section 4.1 Then our proposed Polar-Aligned Attention module will he elaborated
in Section 4.2 and Section 4.3
Finally, the detection head and loss functions are introduced in Section
4.4
4.1
Overall Structure
The overall architecture of our model is shown in Figure First, separate backbones and FPNs [20) to extract multi-level features from RT radar maps and images. Then the polar queries aggregate camera features and radar features by tross-attention Finally, the multi-scale polar BEV map will be passed through a transformer-bast bounding box predictions
decoder with object queries. The final score and be obtained by decoding the output object queries. The core of our EchoFusion how to fuse the camera and radar slow-time data. We fuse them using a novel strategy named Polar Aligned Attention (PA-Attention). In the next two subsections, will explain this strategy in detail.
4.2 Column-wise Image Fusion
After obtaining image and radar features from corresponding encoders, we first initialize the 1-th level polar queries Q₁ E I ft Axxxx uniformly in polar BEV space. Here, R₁, A,, and respectively denote the shape of range, azimuth, and channel dimensions. For brevity, omit in the following two subsections. Given a query q E Rd located at (Tber, Óbeu), it corresponds to a pillar centered at (Tbev: (bev) with infinite height. Taking the coordinate system defined in RADIal [39, ZR. Vn. and zR axes respectively points to the front, left, and upright, while subscript R means radar coordinate. Similarly, the camera coordinates denoted (xc, VC, 2c), which points the right, down, and front, respectively. Thus we have correspondence in Cartesian coordinate system as follows:
x R Theu cos (obev): ;
Tbev sin (obev).
Using extrinsic matrix and intrinsic matrix, have the following formulation:
3C
JR
dg
F1 uo
DC
= R
yR
+ +T.T
d2
fz
2C
2C
ZR
d3
where up. fz are principal point offset and focal length on æ-axis in the intrinsic matrix, Øy is the image column index that corresponds to the point (æR, yR, VR,zR). ZR), while R and T are rotation matrix and translation vector the calibration matrix between camera and radar, respectively. Under mild distortion conditions and normally adopted sensor setting, 2, is determined by pillar position and calibration parameters as follows:
I7 RJ uo + fo Tbev sin Ubeu + d1
(3)
Then COS Obev + d3
5

---

Limited by the number of pages, we leave the detailed proof in the appendix.
However, the height is specified for a query, while for pixel (It, yr) on the monocular image, its depth unknown as well. Thus we cannot precisely associate a query to the image feature, but can correspond queries of the same azimuth to the same column of images. Enlightened by this observation, we use a versatile cross-attention mechanism to flexibly aggregate required features from images. Namely, all the queries with the same azimuth form the query matrix, while all the features in the corresponding column in the image form the key and value matrix. Formally, these two steps can be expressed as follows:
(ar,,)) :) Stack (Ifr (x vr)l) R Vyr E [0, H1 1], (Tbeos Obee) = CrossAttn (g (Tbev, Qbev) Fi (FI, F1 (z1, )),
where fI yr) is the image feature located at 27, yr, while details of cross attention function CrossAttn can be found in 47.
4.3 Range-wise Radar Fusion
We denote updated BEV queries Q R RRxxAx After that, it will be used to sample features from radar. Since the radar feature map shares the same range partition with the BEV queries, the updated query i (Ther: (hen) matches the same range index of the radar feature, i.e. ra Then- To aggregate all the features from all chirps in one frame, we perform cross attention for a certain query q with the on in RT map as key and value. Formally,
FR(TR, -) Stack ([fR(TR, (R))) E R Rexd VtR E [0, 11, ② (Tbev: Φbeu) = CrossAttn (ğ (Fbevs Øbev). FR (FR. , FR (FR, +))
(5)
where R2 and Ti are the height and width of the radar feature map, and JR (FR: (R) E Rd is the radar feature of position (rR, fR).
Note that though we don't 't explicitly decode AoA as in RA map, the AoA has been implicitly encoded in the phase difference of the response of different virtual antenna. Consequently, it can be learned from the features in RT map indirectly.
4.4 Head and Loss
We follow polar BEV decoders [ to decode multi-level BEV features and make predictions from object queries. Since we don't have velocity annotation on this dataset, we remove the corresponding branch. The regression targets include (dp, dg, d2, logl, logw, logh, sin(fori Φ). cos(Oori - -)), where dp. dø. du are relative offset to the reference point (p, Φ, 2), l, w. h, fors are length, width, height, and orientation of bounding box. The classification and regression tasks are respectively supervised by Focal loss
21 and L1 loss.
5 Experiments
In this section, we conduct thorough experiments validate the effectiveness of our approach. We first introduce the dataset and metrics used in our experiments, then followed by comparisons with other state-of-the-art methods. Lastly, we ablate some designs and potential input format raw radar data in our method, and discuss some limitations.
5.1 Experimental Settings
The RADIal Dataset (39 provides 8252 annotated frames with synchronized multi-sensor data. Training, validation, and test sets contain 6231, 986, and 1035 frames, respectively. Each object is annotated with 3D center coordinate of the visible face of the vehicle. The size and orientation of the objects are pre-defined templates, i.e. they are the same for all objects. In the evaluation, precisions and recalls under ten thresholds are calculated. The average precision (AP), average recall (AR), and Fl score are obtained through the ten precisions and ten recalls as the evaluation metric. Range error (RE) and azimuth error (AF) are also reported to evaluate localization accuracy. The annotation method and metrics are quite different those in common autonomous driving datasets

---

Table 2: Detection performances on RADIal Dataset test split with original protocol 39 C. RD, and RT respectively refer to camera, mange-Doppler speetrum, and range-time data. Our method achieves the best performance in both average precision (AP), average recall (AR), and F1 F1-score (F1). The best result of each metric in bold.
Methods
Modality
AP(%)T
AR(%)
F1(%)t
RB(m)
AE
FFTRadNet
39
RD
96,84
82.18
88.91
0.11
0.17
T-FFTRadNet 7
ADC
89.60
89.50
89.50
0.15
0.12
ADCNet 52
ADC
95.00
89.00
91.90
0.13
0.11
CMS [10
RD&C
96.90
83.50
89.70
0.45
n/a
EchoFusion
RT&C
96.95
93.43
95.16
0.12
0.18
45
Nonetheless, to compare with state-of-the-art methods, we still list our results using the
metrics defined in RADIal 33 reference.
Re-annotation and more metrics. Based on the provided LiDAR point cloud, we have refined the cluster-center-base annotation to 3D bounding box form, which includes center position, scale, and heading in 3D space. Capitalizing on these new annotations, we are able to apply metries that are vastly used in 3D object detection tasks. Specifically, we take AP defined in Waymo dataset [45] as our main metrics. Since the LiDAR used in RADIal is only 16-beam, we cannot label accurate height for the objects. So we mainly utilize BEV IoU for evaluation. We use the threshold of 0.7 for normal IoU and longitudinal error-tolerant (LET) IoU computation [8] Considering the spatial distribution of ground-truth, the distance evaluation is further broken down into two categories: 0 to 50 meters, 50 to 100 meters. For a more comprehensive evaluation, we also introduce the error metrics defined in nuScenes dataset [2], including ATE, ASE, and AOE for measuring errors of translation, scale, and orientation.
The KRadar Dataset 35 is recently published 4D radar dataset, which contains 58 sequences, Le. 35K frames of synchronized multi-modality data. However, we have limited time to obtain the full data via shipping, so we only provide results on the sub-dataset stored on Google Drive, which consists of 20 sequences. This so-called KRadar-20 dataset includes 6493 training samples and 6515 test samples, of which 50 % are nighttime data. The training samples are named trainval split, it is further split into train split of 5190 samples and val split of 1303 samples for ablation. Metrics follow KITTI (6) protocols with 40 recall positions. Different from RADIal, it only contains radar formats of Lange-azimutt map (RA map) and radar point cloud. But the RA map of KRadar contains an extra height dimension, and it enables to test 3D prediction ability with 4D radar data.
Radar Formats. introduced Section there are multiple formats of radar raw data, such as ADC, RT, RD, READ. These formats can be converted sequentially by FFT, which is a linear operation that can be absorbed into the linear layers. So from the perspective of neural network, these formats are equivalent. However, our proposed Polar-Aligned Attention requires range dimension for indexing key and value in cross-attention, so we choose RT as a representative in the experiments. We also conduct experiments on RA to investigate the necessity of explicit azimuth.
Implementation Details. All our experiments are carried on eight 3090 GPUs with the same batch size of 8. For the image backbone, we adopt ResNet-50 with pre-trained weights provided by BEVFormer [17]. For LiDAR and radar point cloud branch, we borrow the voxel encoder from PointPillar [5] and use ResNet-50 with modified strides as the backbone. Other representations of radar data share similar modified ResNet-50 as the backbone, but with an extra 1 convolution at the beginning for channel number alignment. To prevent overfitting, the image-only variant is trained for only 12 epochs while others are trained for 24 epochs. Adam and learning rate of 5e-5 is shared for all models. Due to the limited size of the dataset, we find that the results are unstable across different runs. To make a fair comparison, each experiment is repeated three times. In modality ablation, we show the mean and variance of each experiment result.
25 is adopted as the optimizer
7

---

Table 3: BEV detection performances on RADIal Dataset test split with refined 3D ground-truth bounding box. C, RD, and RT respectively refer to camera, data. The best result of each metric is in bold. Our method exceeds FFTRadNet by large margin.
spectrum, and range-time
BEV AP@0.7(%))
LET-BEV-AP@0.700)
Methods
Modality
Overall
0 105m
50 100m
Overall
505
50 100m
FFTRadNec3D 39
RD
57.26
68.66
52.28
62.81
75.55
57.98
EchoFusion
RT&C
84.92
87.56
91.06
88.86
92.81
94.81
Table 4: BEV detection performances on KRadar Dataset test split with KITTI protocol. 0.3, 0.5, and 0.7 are IoU thresholds. The inputs of EchoFusion are image and sange-azimuth map. The best result of each metric is in bold.
BEV AP
3D AP(%)*
Training Set
Method
AP@0.3
AP@0.5
AP@0.7
AP@0.3
AP@0.5
AP@0.7
KRadar
RT
58,04
42.60
10.69
49.65
17.87
0.45
KRadar-20 Bondar-20-traiinn 20-trainval (Radar-20-trainwa
RTNH
61.38
46.47
10.47
53.05
17.98
3.03
EchoFusion
69.95
57.28
33.07
68.35
43.87
14.00
5.2
Comparison to State-of-the-art Methods
5.2.1
Results on RADIal Dataset
We first compare our method with state-of-the-art detectors on AP, AR, range error, and azimuth error defined in RADIal [39], with original object-level point annotations The results are illustrated T Our method significantly improves AR and F1 performance and achieves a remarkable improvement over all existing detectors including CMS [IO], which also integrates both radar raw data and camera data. Note that RE and AE are not quite informative because these two metrics are calculated on all the recalled objects of method, which is unfair to high-recall models like ours.
To evaluate in more rigorous and informative way, also conduct experiments the refined annotations and vastly adopted metrics in 3D object detection. FFTRadNet [39] is lifted up with additional branches to predict the scale and orientation of targets. We can only compare with this FFTRadNet variant since codes of other algorithms 77 T0 52 are not available yet. The results in that the proposed method outperforms FFTRadNet by a large margin. This significant improvement reveals the huge benefit of unleashing the power of radar data in multi-modality fusion.
5.2.2 Results on KRadar Dataset
The baseline on KRadar dataset is the official RTNH [35 which requires radar point cloud. Here, the proposed EchoFusion is input with image and RA map. To align with the baseline. we take the radius and azimuth range respectively as [0, 72] m and [-20, 20] degrees by the limit of camera field of view. The results are shown in Table With RA map and image fusion, our EchoFusion improves over 10 points at each metric, except for BEV AP of IoU threshold 0.3. And since KRadar has higher vertical resolution and more accurate 3D groundtruth than RADIal, our method achieves more considerable improvements in 3D metrics.
5.3 Ablation Studies
We ablate the input format to the radar feature extraction network in Table 5 It shows that the decomposition of complex input can improve overall detection performance, especially for long- range perception. The magnitude and phase representation encodes a non-linear transformation of the complex representation, which may relate the final prediction target better. We also tried to remove the pre-training of the image branch. The result drops significantly, which confirms that in this relatively small dataset, a good pre-training crucial for good performance.
'The official implementation of these metrics are different from common implementations. Specifically, AP and AR are averaged on score thresholds from 0.1 to 0.9 with step of 0.1 and IoU threshold of 0.5. And Fl score directly calculated from AP and AR defined above.

---

Table 5: Ablation studies on our EchoFusion. Complex means interpreting input features as real and imaginary (IQ) or magnitude and phase (MP). Pretrain means whether use the pre-trained image backbone. The best result of each metric is in bold.
Conditions
BEV AP@0.7%%)
LET-BEV-APEO.7(0)
Complex
Pretrain
Overall
0 50m
50 100m
Overall
0 50m
50
100m
MP
84.92
87.56
91.06
88.86
92.81
94.81
IQ
W
82.64
87.09
87.81
88.06
92.25
93,24
MP
w/o
74.34
81.26
78.71
78.90
85.44
84.28
Table 6: BEV detection performances of different modality combinations of our algorithm on RADIal Dataset test split. Refined 3D bounding box annotations are applied. C, I., RD, RT, RA, respectively refer to the camera, LiDAR, cange-Dopples spectrum, range-time data, range-azimuth map, and radar point cloud. The best result without LiDAR of each metric are in bold, and the results with LiDAR are gray background.
Modality
BEV AP@).7(%)
LET-BEV-AP@0.706)
RT RA
RP
Overall
0 50m
50 100m
Overall
0- 50m
50 100m
10.07=1.10
21.6813.11
2.21=0.59
56.92-3.91
78.9811141
36.20.1.81
48. 2..22--
62.33++4.00
45.05-+0.98
55.04+ 1,25
68.56+4.11
52.30+ 1.82
50.67=1.03
65.05=1.559
42.98::0.51
58.53=0.13
72.95:0.778
51.41=0.54
53.55 13.5510.70
66.1911..72
4..41.1.006
6..59.1.14
76.45112.06
56.73.11.33
84.47:10.15
86.92.10.61
91.9111.113
86.07-0.44
88,5510.79
93.1711.10
84.92=0.98
87.56
91.06:11.18
88.86=0.62
92.81=1.37
94.81=0052
84.77.10.65
8..9
91.48 0.228
89.54 0.54
03,53
94.87 10.58
82.35=00.3
86.97+1.01
86.39-+0.74
88.35=0.78
93.07+00800
92.77++0.69
16.35
88.81
94.25.11.68
88.44_0.79
91.6311.34
95.31
5.4 Comparison of Different Modalities
To further study the effects of different modalities, we change the input format indicated Figure 2 and corresponding backbones, and test with and without image modality. Table 6 presents the results, more results on KRadar can be found our appendix, and our findings are listed as follows.
Comparing single-modality results It is not surprising that LiDAR ranks first while the camera is the worst terms of AP. Though poor in depth estimation, the camera has excellent angular resolution, which is beneficial for LET-BEV-AP. It achieves comparable results with radar in LET-BEV-AP. Besides, without the guidance of image, the radar-only network gets better performance as more hand-crafted post-processing are involved. However, their metrics still lag far from those of LiDAR.
Comparing multi-modality with single modality The power of radar data is released by fusing with image. No matter in what data format, all the radar raw representations gain around 30 points improvement by fusing with images. By combining image and range-time data, we obtain BEV AP that is only 0.03 less than that of LiDAR only method. And the LET-BEV-AP and long-range detection ability are even better. We argue that it is mainly because images provide enough clues to decode the essential information from raw radar representation.
Comparing multi multi-modality results When integrating with images, the LiDAR- fuseed method still outperforms other radar- radar-fused methods. In terms of different representations of radar data, both RT and RA outperform traditional cloud points, especially in the 50-100m range, in which the difference is as large as 4 points in AP. We owe this finding to the information loss and false alarm of radar point cloud as shown in Figure [ The performance gap between RT and RA with camera modality is within the error bar, indicating that it is not necessary to explicitly solve the azimuth, nor necessary to permute the Doppler-ane dimensions FFTRadNet [39 does.
5.5 Discussion and Limitation
Firstly, it is worth noticing that although the overall performance of radar is improved by fusing with the camera, the performance in the short-range is lower than that in the long-range. We speculate the main reason is inaccurate annotation. The 16-beam LiDAR provided in RADIal is too sparse at

---

Table 7: 3D detection performances on RADIal Dataset test split with refined 3D ground-truth bounding box. C. L,, and RT respectively refer to camera, LiDAR, and radar-time data. The best result of each metric is in bold.
3D AP@0.5(%)
Average Error@
Modality
Overall
0 905mm
50 100m
ATE
ASE
AOE
L
62.69
80.04
54.26
0.243
0.171
0,027
L&C
68.36
84.74
61.54
0.239
0,180
0.020
RT&C
39.81
56.05
31.24
0.301
0.173
0.020
farther distance for annotators give an accurate size for the long-range objects. In such cases, they are required to assign a template with fixed size to these objects. These annotations make the problem much easier for farther distances, which leads to higher performance than near distances.
Secondly, though the radial speed label is included the original annotations of RADIal, radial speed is not quite useful for downstream modules like planning since they need accurate longitudinal and lateral velocity to predict whether the vehicles will interfere with the ego vehicle. However, the synchronized frames
consecutive in this dataset, is hard to label velocities for the objects. Considering these factors, velocity prediction is not included in our task.
Finally, as shown in Table 7 our method still lags far from LiDAR-ban R-based methods. The primary limi- tation can be attributed to imperfect z localization, resulting in relatively high Average Localization Error (ATE). The inferior performance is mainly due coarse LiDAR annotation and inferior radar clevation resolution. The low elevation distinguishing power of LiDAR makes annotation error much worse at long range. As result, both LiDAR-based and radar-based methods experience significant drop in 3D AP within the 50- 10-100m interval. But when equipped with high elevation resolution and accurate 3D groundtruth provided by KRadar, our method shows excellent performance in 3D metrics, as shown in Table This emphasizes the pressing need for raw radar dataset of higher quality that enables better exploration of radar data usage.
Conclusion and Future Work
In this work, we have proposed novel method for radar raw data fusion with other sensors in a BEV space. Our proposed EchoFusion is concise and effective, which outperforms previous work by significant margin. We are the first to demonstrate the potential of radar as a low-cost alternative for LiDAR in autonomous driving systems through thorough analyses and experiments.
This work is only starting point to study how raw radar data can be exploited. However, many attempts are limited by the available datasets. We urge large-scale and high-quality dataset. However, the acquisition of high-quality deliberate design for clock synchronization and high storage demand. We will try to build the dataset facilitate further research.
data with accurate annotation needs great effort and
Societal Impacts Our method can be deployed the autonomous driving system. Performance loss caused by improper usage may increase security risks.
References
[1] Jie Bai, Lianqing Zheng, Sen Li, Bin Tan, Sihan Chen, and Libo Huang. Radar transformer: An object classification network based on 4d mmw imaging radar. Sensors, 2021.
[2] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal datasct for autonomous driving. In CVPR, 2020.
Simon Chadwick, Will Maddem, and Paul Newman. Distant vehicle detection using radar and vision. In ICRA, 2019.
[4] Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao. Futr3d: A unified sensor fusion framework for 3d detection. arXiv preprint 2203:22003.10622 2022.
10

---

[5] Jean-Luc Déziel, Pierre Merriaux, Francis Tremblay, Dave Lessard. Dominique Plourde, Julien Stanguen- nec, Pierre Goulet, and Pierre Olivier. Pixset: An opportunity for 3d computer vision go beyond point clouds with a full-waveform lidar dataset. ITSC, 2021.
[6] Andreas Geiger, Philip Leuz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361. IEEE, 2012.
[7] James Giroux, Martin Bouchard, and Robert Laganiere. T-fftradnet: Object detection with swin vision transformers from raw ade radar signals. arXiv preprint arXiv:2303.16940 2023.
[8] Wei-Chih Hung, Henrik Kretzschmar, Vincent Casser, Jyh-Jing Hwang. and Dragomir Anguelov. Let- 3d-ap: Longitudinal error tolerant 3d average precision for camera-only 3d detection. arXiv preprint arXiv:2206.07770 2022.
[9] Yangin Jiang. Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu- Yu-Gang Jiang. Polar- former: Multi-camera 3d object detection with polar transformers. AAAI, 2023.
[10] Yi Jin, Anastasios Deligiannis, Juan-Carlos Fuentes-Michel, and Martin Vossiek. Cross-modal supervision- based multitask learning with automotive radar raw data. IEEE Transactions Intelligent Vehicles, 2023.
[11] Jinhyeong Kim. Youngseok Kim, and Dongsuk Kum. Low-level sensor fusion network for 3d vehicle detection using radar range-azimuth heatmap and monocular image. In A0CV,2020.
[12] Youngseok Kim, Jun Won Choi, and Dongsuk Kum. Grif net: Gated region of interest fusion network for robust 3d object detection from radar point cloud and monocular image. In IROS. IEEE, 2020.
[13] Youngseok Kim, Sammin Kim, Jun Won Choi, and Dongsuk Kum. Craft: Camera-radar 3d object detection with spatio-contextual fusion transformer. AAAI, 2023.
[14] Youngseok Kim, Sanmin Kim, Juyeb Shin, Jun Won Choi, and Dongsuk Kum. Cm: Camera radar net for accurate, robust, efficient 3d perception. arXiv preprint vrXiv:2304.00670 2023.
[15] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
[16] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang. Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth es- timation in multi-view 3d object detection with dynamic temporal stereo. arXiv preprint arXiv:2209. 200::2288, 2022.
[17] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: learning bird' s-eye-view representation from multi-camera images via spatiotemporal trans- formers. In ECCV, 2022.
[18] Teck-Yian Lim, Amin Ansari, Bence Major, Daniel Fontijne, Michael Hamilton, Radhika Gowaikar, and Sundar Subramanian. Radar and camera early fusion for vehicle detection in advanced driver assistance systems. In NeurlPS workshop, page 7, 2019.
[19] Teck- Yian Lim, Spencer A Markowitz, and Minh N Do. Radical: A synchronized fmew radar, depth, imu and rgb camera data dataset with low-level fmew radar signals. IEEE Journal of Selecied Topics in Signal Processing, 2021.
[20] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[21] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[22] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In ECCV, 2022.
[23] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petrv2: A unified framework for 3d perception from multi-camera images. arXiv preprint arXiv:2206.0/556, 2022.
[24] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task sentor fusion with unified bird secye view representation. ICRA, 2023.
[25] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
11

---

[26] Sohrab Madani, Jayden Guan, Waleed Ahmed, Saurabh Gupta, and Haitham Hassanieh. Radatron:
Accurate detection using multi-resolution cascaded mimo radar. In ECCV, 2022.
[27] Bence Major, Daniel Fontijne, Amin Ansari, Ravi Teja Sukhavasi, Radhika Gowaikar, Michael Hamilton, Sean Lee, Slawomir Grzechnik, and Sundar Subramanian. Vehicle detection with automotive radar using deep learning on cange-azimuth-opphl tensors. ICCV Workshops, 2019.
[28] Michael Meyer and Georg Kuschk. Automotive radar dataset for deep learning based 3d object detection. In EuRAD, 2019.
[29]
Mohammadreza Mostajabi, Ching Ming Wang, Darsh Ranjan, and Gilbert Hsyu. High-resolution radar dataset for semi-supervised learning of dynamic objects. CVPR Workshops, 2020.
[30] Richard M Murray, Zexiang Li, S Shankar Sastry, and Shankara Sastry. A mathematical introduction to robotic manipulation. CRC press, 1994.
[31] Ramin Nabati and Hairong Qi. Radar-camera sensor fusion for joint object detection and distance estimation in autonomous vehicles. arXiv preprint arXiv:2009.08422 2020.
[32] Ramin Nabati Hairong Qi. Centerfusion: Center-based radar and camera fusion for 3d object detection. In WACV, 2021.
[33] Felix Nobis, Maximilian Geisslinger, Markus Weber, Johannes Betz, and Markus Lienkamp. A deep learning-based radar and camera sensor fusion architecture for object detection. In SDF, 2019.
[34] Arthur Ouaknine, Alasdair Newson, Julien Rebut, Florence Tupin, and Patrick Pérez. Carrada dataset: Camera and automotive radar with range-angle-doppler annotations. In ICPR, 2021.
[35] Dong-Hos Hee Pack, Seung-Hyy Kong, and Kevin Tirta Wijaya. K-radar: 4d radar object detection for autonomous driving in various weather conditions. Advances in Neural Information Processing Systems, 35:3819-3829, 2022.
[36] Andras Palffy, Jiaao Dong, Julian FP Kooij. and Dariu M Gavrila. Can based road user detection using the 3d radar cube. IEEE Robotics and Automation Letters, 2020.
[37] Andras Palffy, Ewoud Pool, Srimannarayana Baratam. Julian FP Kooij, and Dariu M Gavrila. Multi-class road user detection with 3+ 1d radar in the view-of-dellt dataset. IEEE Robotics and Automation Letters. 2022.
[38] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020.
[39] Julien Rebut, Arthur Ouaknine, Waqas Malik, and Patrick Pérez. Raw high-definition radar for multi-task learning. CVPR, 2022.
[40] Mark A Richards.
Fundamentals of radar signal processing. 2014.
[41] Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic feature transform for monocular 3d object detection. BMVC, 2019.
[42] Ole Schumann, Markus Hahn, Nicolas Scheiner, Fabio Weishaupt, Julius F Tilly, Jürgen Dickmann, and Christian Wöhler. Radarscenes: real-world radar point cloud data set automotive applications. In IEEE FUSION conference, 2021.
[43] Ole Schumann, Jakob Lombacher, Markus Hahn, Christian Wöhler, and Jürgen Dickmann. Scene understanding with automotive radar. IEEE Transactions on Intelligent Vehicles, 2019.
[44] Marcel Sheeny, Emanuele De Pellegrin, Saptarshi Mukherjee, Alireza Ahrabian, Sen Wang, and Andrew Wallace. Radiate: A radar dataset for automotive perception in bad weather. In ICRA, 2021.
[45] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaík, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Wayme open dataset. In CVPR, 2020.
[46] Shunqiao Sun, Athina P Petropulu, and H Vincent Poor. Mimo radar for advanced driver-assistance systems and autonomous driving: Advantages and challenges. IEEE Signal Processing Magazine, 2020.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
2017.
12

---

[48] Yizhou Wang. Zhongyu Jiang, Yudong Li, Jenq-Neng Hwang, Guanbin Xing, and Hui Liu. Rodnet: A real-time radar object detection network cross-Supervised by camera-radar fused object 3d localization. IEEE Journal of Selected Topics in Signal Processing, 2021.
[49] Zengran Wang, Chen Min, Zheng Ge. Yinhao Li. Zeming Li. Hongyu Yang, and Di Huang. Sts: Surround- view temporal stereo multi-view 3d detection. arXiv preprint arXiv:2208. 10145, 2022.
[50] Zizhang Wu, Guilian Chen, Yuanzhu Gan, Lei Wang, and Jian Pu. Myfusion: Multi-view 3d object detection with semantic-aligned radar and camera fusion. arXiv preprint arXiv:2302. 10511, 2023.
[51] Baowei Xu, Xinyu Zhang, Li Wang, Xiaomei Hu, Zhiwei Li, Shuyue Pan, Jun Li, and Yongqiang Deng. Rpfa-net: 4d radar pillar feature attention network for 3d object detection. ITSC, 2021.
[52] Bo Yang, Ishan Khatri, Michael Happold, and Chulong Chen. Adenet: End-to-end perception with radar ade data. arXiv preprint arXiv:2303.1 114200 2023.
[53] Chenyu Yang. Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Qiao, Lewei Lu, et Bevformer v2: Adapting modern image backbones bird's-eye- view recognition via perspective supervision. arXiv preprint arXiv:2211. 104,,99,, 2022.
[54] Ao Zhang, Farzan Erlik Nowruzi, and Robert Laganiere. Raddet: srnge-azimmth--oog based radar object detection for dynamic road users. In CRV, 2021.
[55] Guoqiang Zhang. Haopeng Li, and Fabian Wenger. Object detection 3d estimation via an fmow radar using fully convolutional network. In ICASSP, 2020.
[56] Jingwei Zhang, Ming Zhang, Zicheng Fang, Yulong Wang. Xian Zhao, and Shiliang Pu. Rvdet: feature- level fusion of radar and camera for object detection. In ITSC, 2021.
[57] Lianqing Zheng, Zhixiong Ma, Xichan 7hu, Bin Tan, Sen Li, Kai Long, Weiqi Sun, Sihan Chen, Zhang, Mengyue Wan, et al. Tj4dradset: A 4d radar dataset for autonomous driving. In ITSC, 2022.
[58] Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmenta- tion. In CVPR, 2022.
[59] Zhou, Lulu Liu, Haocheng Zhao, Miguel López-Beníte, Limin Yu, and Yutao Yue. Towards deep radar perception for autonomous driving: Datasets, methods, and challenges. Sensors, 2022.
13

---

Supplementary Material
A Proof for column and pillar correspondence
In this section, we will provide detailed proof for the correspondence between pillar in radar coordinate and column in camera coordinate as described in Section 4.2 of our main paper.
Suppose that the rotation matrix and translation vector between radar and camera are:
a1
bj
C1
dj
R
=
Q2
b2
C2
T
=
d2
(S1)
a3
b3
C3
d3
Then for a point located at (Thev: Shew: z) under radar's polar coordinate, has cartesian coordinate (En, yn, RR):
FR
Theu COS (dhre)) 1/ = Thea sin (oher) ER = 2,
(S2)
and it can be projected to the camera frame using extrinsic parameters:
FC
J
01 "beo COS Obev + by Tbew sin Øbev + 01 + dj
yc
R
yR
+T
agTbeo COS Ober + berbeu sin Ubeu + C2 d2
(S3)
2C
0gTbbeo COS Pbev + byrbeo sin Obeu + Cz + d3
Using the intrinsic parameters, the camera coordinate (xC, (C) can be correlated with image pixel position (zr.3r) using:
E1 fø
JC
y1
v0
yC
(S4)
2C
fv
DC
where Ja, Jy, Un, and Un are intrinsic parameters. Our goal is find situation that the pillar is projected as a column on the image plane. Under this condition, I2 should be irrelevant with 2, i.e. in the equation below:
FC tbee COS @bev + by Thew sin Obber + 61 + d1
(S5)
20 agTbev &bev + byfbee sin Obea + C3z + d3
The coefficients of z should be O, which means by combining the relationship between rotation matrix R with roll a, pitch 3, and yaw y 30, we can derive:
C1 = sin a + cos y sin & cos a =
(S6)
C3 cos 3 cos 0.
Considering second row of Eq. (S6), there are two solutions / = + or a = Considering the first solution B = + 2. the first row of Eq. (S6) is transformed to:
sin sin a + cos 7 COs = 0,
(S7)
which means:
cos (7 F (x) = O.
(S8)
Thus we have:
π
π
3
2 7 = a
a -T, π|
2
(S9)
π or, B = 2 aa 2. E [- T]
π
For another solution
-. the first row of Eq. (56) transformed
sin y sin cos sin 3 cos sin 0.
(S10)
Thus have:
+ 2' 7 0 or 3 [-
(S11)
As result, the final solution is:
2 7 a + 2 a n, T].
or, 7 a + E |-
(S12)
or, + 2 7
or म. 3 [-
14

---

In standard ADS, the LiDAR or radar coordinate system usually has z-axis pointing up. while the camera coordi- nate system has y-axis pointing up down. So a widely adopted practice first exchanges the y and z axes, which means roll a + After that, pitch / around the vertical y-axis is executed, while further rotation around the z-axis is not required, as shown in Figure ST] In other words, the roll / can any value between -T and r, while yaw y is set O. This is exactly the situation of the third solution of Eq. (S12). And under this condition, we have expression of R according to [30]:
column@xi
y Camera
pillan@
(Tbew (
Radar
COS 8
+ sin 3
di
0
F1 T=
d2
(S13)
Figure $1: Illustration of coordinate sys- tem transformation.
sin B
+ cos
3
dz
Thus, using Eq.
S3 and Eq.
S5
get the expression of column index 21. If 2.4 have:
Tbeu cos (33 (bev) + d1
Theo sin (3 (beu) + d3 x1 = uo +
(S14)
If
we have:
2
They cos (3 + Øbev) + dl
37 uo +
(S15)
Pee sin (3 + (bev) + d3
For RADIal, the rotation matrix from the radar coordinate system to the camera coordinate system is:
0.0465
-0.9989
-0.0051
R
0.0476
0.0029
-0.9989
(S16)
0.9978
0.0467
-0.0474/
which is the approximation of (a: B, = (m/2, 1 0). And thus we approximately have:
Ther sin Pbeo + d1
R2 uo + f Theu 008 @bev + d3
(S17)
can be inferred that æı is almost irrelevant to 2, which means one pillar in the polar coordinate of radar corresponds column in the image.
B Visualization and Dataset Statisties
To better illustrate the performance of our method, visualize the predictions and ground truths in Figure S2 As can be seen from the cases in the first and the second column, our method can accurately predict the position of vehicles. But there several cases in which the accurate prediction does not get high confidence, as shown the upper case in the third column. On the other hand, the orientation estimation error may be significant when the yaw of the vehicle is large, shown in the inferior case in the third column. The main reason is that the samples with large yaw are relatively rare in the dataset, shown in Figure S3 (b) and (c).
As shown in Figure S3 (a), the samples of test data are rather rare within 30 meters and out of 80 meters. To comprehensively explore the detection performance of objects, we divide the range into 0-50m and 50-100m, which correspond to short-range and long-range detection performance respectively.
We further visualize scenarios that corrected by our new labels. It worth noting that the original annotation and prediction of previous methods are centers of objects' visible faces, and they assign fixed scale and zero orientation when calculating Average Precision (AP). But vehicles of large angles and scales are normal in daily scenarios, as shown in Figure[S4 Our new annotations based on dense LiDAR points correct this, helping a more comprehensive evaluation of existing methods. We also annotate other traffic participants such as pedestrians and cyclists. But due to their limited numbers (4 cyclists, 23 pedestrians in training split, and 17 cyclists, pedestrians in test split), we don't 't add more fine-graine class labels.
15

---

Table S1: LET-BEV-AP results with different longitudinal tolerance Ti. The performances of different modality combinations of our algorithm RADIal Dataset test split presented, using refined 3D bounding box annotations. C, L, and RT respectively refer to the camera, LiDAR, and range-time data. The results with LiDAR
gray background.
Modality
Ti 0.1. LET-BEV-APPO.7000
Ti 0.3, LET-BEE-APP00.700
C
RT L
Overall
0 50m
50 100m
Overall
0-50m
50 100m
36.92.13.91
78.98 1.41
36.20.1.881
81.36.11.15
83.30.1..01
77.00.10.43
/
55.04-1:.25
68.56
52.30
55.34-11.91
69.79.14.59
51.02:10.07
86.07-10.44
88.55
93.17-10110
86.41-1047
89.72-10.03
93.22-10.04
V
88.86-10.62
02.81=11.33
94.81 +0.52
88.56-+0.53
92.58-+0.70
94.57+-0.29
Table S2: Recall with different IoU thresholds. The performances of different modality combinations of our algorithm on RADIal Dataset test split presented, using refined 3D bounding box annota- tions. C, L, and RT respectively refer to the camera, LiDAR, and range-time data. The results with LiDAR are on gray background.
Modality
BEV recall @0.7(%)
BEV recall @0.3(()1
C
RT L
Overall
0 50m
50 - 100m
Overall
0 50m
50 100m
62.73+11 1.24 .24
73.89=2.11
61.21=0.82
86.27+0.43
92. 13:10.71
93.40=+0.54
v
86.08
89.11=0.08
93.35=0.08
91.40=0.08
96.13:0.17
96.95=0.16
16.18:0.13
89.13=0.6
92.53=0.56
91.78:+0.85
95.82:40.50
97.67=1.055
Table S3: BEV detection performances of different modality combinations our algorithm on RADIal Dataset test split. Refined 3D bounding box annotations applied. C, L, ADC, RD, RT, RA, RP respectively refer to the camera, LiDAR, ADC data, sunee-Doppler spectrum, range-time data, range-azimuth map, and radar point cloud. The best result without LiDAR of each metric are in bold, and the results with LiDAR are on gray background.
Modality
BEV AP@0,7
LET-BEV-AP@0.7000
ADC
RD RA
Overall
0 50m
50 100m
Overall
50m
50 100m
10.07:11.10
21.68:3.11
2.21:1.599
56.92.13.91
78.98:1.41
36.2011.881
49.6410.43
61.37:2157
47.92=1.48
56.52412.36
68.93=1.87
54.68
48.26110.92
62.33:14.00
45.05
15.04:1:.255
68.56:14.11
12.30.1..22
47.34-0.73
59.31++1.64
42.53+1198
54.96-10.67
67.02+11.7
50.59+0.226
50.67+:1.03
65.05:+1.59
42.98-0.51
58.53-10.13
72.95=0078
51.1100.5
66.19-1..22
56.73+-1.33
84.47=20.15
86, 06:92-10.61
91.91
86.07-t0.44
88.55=20.79
93.17=00110
84.85-10.22
8..6
91.4621.08
88, 88.66110.33
92.43=1..15
94.99-10.66
84.92
87.56:11.58
91.061
88.86:10.62
92.81
94.81-1..22
4
83.31 D-39
87.26-0.56
89.16-0.44
88.24++0.06
18-14-4
93.26-+0.73
84.77-10.65
87.93:10.60
91.48::0.28
89.54:1.554
93.83:10.56
94,57
82.35 +.993
86.97 1.01
66.39-00.4
88.35-+0.78
93.07
92.77 0.669
86.35-11.15
88.8141.40
94.25
88 44-10.77
91.63=11.34
95.31
Table S4: Ablation the coordinate system. Experiments carried on RADIal dataset, with RT data and image as input.
BEV AP@0.7
LFT--EE--AG00.7000
Coordinate
Overall
0 50m
50 100m
Overall
505
50 100m
Polar
84.9211.99
87.5611 1.58
91.06-11.18
88.8611.62
92.81 93.35-+0.22
94.81-1.552
Sphere
85.02-40.60
88.02-L0.59
91.76+00.98
89.97+0.61
96.41+1.21
16

---

Figure S2: Visualization of ground truths and predictions from RT and image fusion. The ground truth bounding boxes are in pink, while the predicted bounding boxes are in green with the confidence score its upper right. The LiDAR points and the radar points are respectively in blue and red. The rear edge of each car is in cyan. We recommend readers zoom in for better detail.
Tungee disi botien bockaal aod
Var bution SADAs train stit
Yae doli of NADM tusi up
20m
6m
10
30m
On
100m
25m
40m1
80
130m
(a)
(b)
(c)
Figure S3: Statistics of RADIal dataset. Image (a) corresponds to sample distribution over range,
while images (b) and (c) correspond to yaw distribution over the range of train and test data splits.
C Complementary property of radar and camera
To highlight the complementary nature of radar and camera properties, we investigated the perfor- mance of LET-BEV-AP with varying longitudinal tolerance (Ti), as presented in Table SS Notably, when using a larger Ti of 0.3, the camera's 's performance experiences a significant improvement. This suggests that the relatively lower performance of long-range LET-BEV-AP under Ti of 0.1 primarily stems from depth errors. By reducing the impact of longitudinal estimation, the camera's 's exceptional angular perception accuracy becomes more pronounced. Conversely, the LET-BEV-AR performance of the radar-only variant remains relatively unchanged. This finding indicates that the radar's depth estimation is quite accurate and minimally affected by the value of T,. Similarly, the highly accurate -iDAR-only variant and camera and RT fusion variant don't suffer from performance fluctuation as well, which supports our conclusion.
To further investigate the primary factor affecting radar performance, we conducted comparison of BEV recall at IoU thresholds of 0.7 and 0.3, as outlined in Table S2 It can be deduced that by using
17

---

Figure S4: The visualization of the old labels and labels under Bird's Eye View. The old labels include 3D center points and fixed seale and orientation, denoted as green boxes. The new labels are denoted red boxes, which we offered real scale and orientation, well objects' accurate center position. The rear edge each bounding box is colored in cyan. The left, middle, and right column respectively denotes the situation of large heading angle, missed traffic participants, and vehicles of large scale. We recommend readers to zoom in for better detail.
Range
B
D
-Vmax
-Vmmxx/2
Vmax/2
Vmax
Doppler
Figure S5: The illustration of Doppler Domain Multiplexing(DDM), This figure shows toy example of how DDM to distinguish four transmitters.
less stringent IoU threshold, the recall of the RT-only variant experiences a significant increase. This result suggests that the RT-only variant's limitation mainly lies its inferior angular localization ability. But by combining the strengths of camera and radar modalities, the model can leverage the advantages of both modalities, leading to significant performance boost.
D Doppler domain multiplexing and coordinate system
As mentioned in Doppler Domain Multiplexing(DDM) is used distinguish received radar signals from different transmitters. The mechanism can be best illustrated in Range-Doppler (RD) spectrum format of data, as shown Figure 55 Without DDM, the valid measurable speed interval is [-Vmas, Vmez). Suppose we have four transmitters, Tx 1/2/3/4, and one object with speed in [0, Vmnn//2). The echo signals from Tx 1/2/3/4 will respectively fall into sections C, D. A, by using
DDM, showing periodical pattern. But now only measure speed in [0, V Vmaz/22 /2). Similar
18

---

Table S5: BEV detection performances of different modality combination on KRadar Dataset test split with KITTI protocol of 40 recall positions. C, RA, RP respectively refer to the camera, sange-azimuth map, and radar point cloud. The best result of each metric is in bold.
BEV AP(%)
AP(%)T
Training
Method
AP@0.3
AP@0.5
AP@0.7
AP@0.3
AP@0.5
AP@0.7
KRadar-200 20-trrn
(choFusionn(P)
55.74
43.94
20.56
53.40
29.21
2.90
(Radar-20-train
EchoFusion RA)
54.45
42.28
17.97
51.75
24.65
3.85
KRadar- 20-t-20--trinn
EchoFusion
66.46
54.17
26.39
58.62
34.67
3.96
CRadar-20-train
EchoFusion(RA+C)
68.70
55.90
29.65
66.68
34.33
5.34
phenomenon can be observed in the figure of Range Doppler data, i.c. the third image in Figure 2 in the paper.
We also carried out experiments on coordinate system of BEV space. Though the range dimension under sphere system, found that polar system achieve par performance, shown in Table S4
Thus we apply the polar system in the paper.
E
More experimental results
Referring to Table S3 we have included the detection performance when our model consumes the ADC data or range-Dopple Doppler spectrum input. It is worth noticing that we add an extra complex linear layer initialized with FFT coefficients for ADC data to replace the range FFT and the side-lobe suppression. In the case of the single modality version, the performance discrepancy between RT and these two modalities remains within the margin of error. When considering the fusion of image data, the performance of RD slightly lags behind that of RT, while that of ADC is almost the same as RT. These findings suggest that the integration of image data significantly mitigates the importance of conventional radar signal processing accurate radar-based detection.
Besides, we offer ablation of different modality combinations on KRadar. The results are reported in Table S5 Here, we train all models with training split. The RA map shows considerable superiority over radar point cloud when fusing with the monocular image, which aligns with our research motivation and results on RADIal dataset.
19


Reviews:
[{"review_id": "qky4l9YCfM", "reviewer": "Reviewer_jS6D", "summary": " The paper proposes a new method to fuse radar and imagery data for autonomous driving. Instead of using processed radar data, which discards most of the information in the original signal while attempting to filter out noise, this method leverages the raw data directly as input to the network. The method is based on attentional mechanisms cued to range in radar and azimuth in imagery, as these measurements are more accurate in those modalities. This technique and the use of raw data seem quite novel in this domain. The experiments are very thorough, on the only dataset that provides raw radar data. Full 3D bounding box annotations are added to this dataset, which is a useful contribution.", "questions": "As above.", "limitations": "Yes, no concerns.\n", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The introduction clearly lays out the motivation of using raw vs. processed radar data, and Fig. 1 provides a clear example of how this improves performance. \n\nLines 44-46 state “This technique is based on an observation that the polar coordinate is aligned with radar data on the range dimension and aligned with image data on the azimuth dimension.” It’s not clear with “aligned with” means here, but it seems you mean to say that radar data is more informative in the range dimension while imagery is more informative in the azimuth direction. It would be helpful to clarify this important point in the paper.\n\nAdding vehicle bounding boxes to RADial is a significant contribution for those looking to study radar in autonomous driving. It would be useful to expand the annotations beyond a single “vehicle” class however, as with most of the other autonomous driving datasets.\nThe related work section is thorough and covers the main areas of claimed contributions.\n\nThe overall architecture, Fg. 3, nicely highlights the key contribution of the cross-attentional module in polar coordinates that emphasizes azimuthal accuracy in imagery and range accuracy in radar. \n\nThe approach in sec. 4 is clearly explained in text, supported by equations but without relying on those equations to encode the concepts, making the paper more readable and impactful.\n\nThe experiments are thorough, albeit on a single dataset as mentioned below. The primary comparative method [36], which operates on the processed radar data, performs considerably worse on the most important test, table 3, against the newly-created 3D bounding box annotations. The proposed method also does very well against all baselines using the standard point annotations on the original dataset.\nThe ablation studies are very thorough, examining performance on each sensor independently in order to show the value of fusion.\n", "weaknesses": "Experiments are performed on a single dataset, which is a minor limitation but unavoidable because it is the only one that provides raw radar data calibrated with imagery.\n\nThe focus of the paper is autonomous driving, which is a huge topic and is generally sufficient for papers at major AI conferences. Nevertheless it would be compelling to see how the method performs in other domains where radar and imagery could be fused, such as pedestrian detection and tracking, maritime vessel detection and so on. Datasets for these are hard to find, but perhaps the RADial dataset contains pedestrians that might have some motion signature in the radar data.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate your affirmation of our work and constructive advice! The responses to the main concerns are as follows:\n\n## Q1: Possibility of expanded annotation and tasks beyond vehicle detection\n\nIn the original dataset there was only one class. In our refined annotation, we provide fine categories for objects such as car, truck, bus, cyclist and pedestrian. In the training dataset, there are 8227 instances, including 213 trucks or buses, 4 cyclists, and 23 pedestrians. In the test dataset, there are 1314 instances, including 63 trucks or buses, 17 cyclists, and 0 pedestrians. Since the number of these classes is quite limited and many of them stem from the same object, it is not sufficient to train a robust multi-class detector or make reasonable evaluation. Thus, we don’t offer fine-grained label annotations and only conduct experiments for one class. A larger and high-quality dataset is necessary for the research of raw radar data.\n\n## Q2: Limitation of single dataset\n\nRecently we found another dataset contains raw radar data with camera and calibration. Although it has some limitations, we believe it would be a good complement to our work. Please refer to “More experiments on KRadar Dataset” part in General Reply.\n\n## Q3: How to understand “align with” in the sentence “This technique is based on an observation that the polar coordinate is aligned with radar data on the range dimension and aligned with image data on the azimuth dimension.”\n\nSorry for the confusing description. Another reviewer also has a similar question. Please refer to “Understanding of “align with” in lines 44-46” part in General Reply."}, {"review_id": "CYv6Tcylyv", "reviewer": "Reviewer_K4kM", "summary": "The paper proposes a method for fusing raw radar data with images in bird’s eye view (BEV) space. To this end, the method extract multilevel features from images and radar and uses polar-aligned attention mechanism with polar-BEV decoder for bounding box prediction. Evaluation is done on re-annotated RADIal dataset and shows better performance than other comparable methods.", "questions": "How robust is the proposed model to spatio-temporal misalignment?\n", "limitations": "The method assumes synchronized radar and camera.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper utilizes raw radar data which in general is a better approach than relying on post-processed radar data for object detection.\n", "weaknesses": "The BEV cross attention assumes synchronized radar and image which may not be robust at high speed driving and unsynchronized radar and camera setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the very professional comments. We are inspired and hope our discussion brings more insights. The responses to the main concerns are as follows:\n\n## Q1: How robust is the proposed model to spatio-temporal misalignment?\n\nBoth spatial and temporal misalignment can be alleviated by attention mechanism [1,3] and receptive field provided by convolution-based backbone [2]. As shown in Figure R1 in the pdf file of general reply, the BEV queries learn to correctly attend to features of interested objects of specific range within corresponding image columns. In RADIal, radar sample interval is 200ms, which is much larger than radar chirp period. Thus, it is hard to simulate temporal misalignment by concatenating radar chirps from nearby frames. So we only simulate spatial misalignment by adding noise to the extrinsic matrix. As shown in Table R2, our method shows strong robustness to the spatial alignment.\n\nTable R2. The detection performance under spatial and temporal misalignment. The spatial alignment is simulated by adding noise to the extrinsic matrix both in training and testing. Specifically, the rotation noises are sampled from a normal distribution with mean equals 0 and variance equals 2 (rotation noise are in degrees), and the translation noises are sampled from a normal distribution with mean equals 0 and variance equals 10 (translation noises are in centimeters, and the noise of each direction is independent).\n\n| Train | Test  | BEV AP/Overall | BEV AP/0-50m | BEV AP/50-100m | LET BEV AP/Overall | LET BEV AP/0-50m | LET BEV AP/50-100m |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| w/o noise | w/o noise | 84.25±0.33 | 86.67±1.17 | 90.48±1.04 | 88.54±0.52 | 92.10±1.15 | 94.45±0.17 |\n| w/o noise | w noise | 80.96 | 87.38 | 84.25 | 86.87 | 93.30 | 89.73 |\n| w noise | w/o noise | 84.65 | 88.27 | 90.84 | 88.29 | 91.99 | 95.68 |\n| w noise | w noise | 84.50 | 88.20 | 91.82 | 88.18 | 91.71 | 95.66 |\n\n> [1] Li Y, Yu A W, Meng T, et al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection, CVPR2022.\n> \n> [2] Liu Z, Tang H, Amini A, et al. Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation, ICRA2023.\n> \n> [3] Li Z, Wang W, Li H, et al. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers, ECCV2022."}, {"review_id": "ppq2fTEsBF", "reviewer": "Reviewer_eUpu", "summary": "This work presents a method for camera-radar detection and fusion. The authors propose an approach that follows:\n\nYanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers. AAAI, 2023.\n\nand \n\nJulien Rebut, Arthur Ouaknine, Waqas Malik, and Patrick Pérez. Raw high-definition radar for multi-task learning. In CVPR, 2022.\n\nSpecifically, the paper introduces a polar transformer method that cross-attends image and radar features across modalities according to view and sampling geometry. The technique relies on RAW radar measurements as input and, similar to recent works (above), is evaluated on an existing dataset. The approach is validated to be on par with recent methods.", "questions": "Can the authors include examples of mislabeled objects or bias in the dataset? \nWhat scenarios are corrected and why do they contribute significantly to the evaluation?", "limitations": "The authors discuss the limitations of their work and societal impacts of the method adequately.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The cross-modal fusion is well-motivated and has shown to be effective. Although a large body of work has proposed similar methods that also rely on RAW radar data, the design of the proposed method is sound and well-executed. Devising methods that operate on RAW data is an exciting opportunity for the field to embrace ideas from computational imaging in vision models.", "weaknesses": "The proposed method lacks novelty and experimental margins. Specifically, areas of concern are the following:\n\n*Novelty and Technical Contribution: A large body of work has explored BEV-based fusion approaches and, recently, RAW-based detections. Specifically, this work directly follows the core ideas from:\n\nYanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers. AAAI, 2023.\n\nand \n\nJulien Rebut, Arthur Ouaknine, Waqas Malik, and Patrick Pérez. Raw high-definition radar for multi-task learning. In CVPR, 2022.\n\nWhile the specific attention mechanism is new, it also extends the formulation in PolarFormer to the radar measurement formation model. As such, the technical contribution of the method is incremental in comparison to existing work in the field. \n\n*Experimental Margins: As a results of the relatively incremental changes in architecture and training, the margins that the proposed method achieves are also also incremental. The method is on par with FFTRadNet on the original dataset, but improves after labeling. However, it is unclear if the baseline method has been retrained and all other baseline methods are also missing in the dataset. Moreover, without validating the relabeled dataset separately, it is unclear if the relabling biases the evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your kind comments on our manuscript. The responses to the main concerns are as follows:\n\n## Q1: Whether the novelty and contribution of this work is significant\n\nIndeed, our work is based on PolarFormer, but the fusion mechanism is newly designed because raw radar data has no explicit location information. Moreover, our efforts on the annotation is nonnegligible. Please refer to “The Contribution of This Paper” section in General Reply for a conclusion of our contribution.\n\n## Q2: Uncertain about if the baseline method has been retrained and why all other baseline methods are missing in the newly labeled dataset\n\nTo make a reasonable comparison with the baseline method, i.e. FFTRadNet, we added extra branches to predict the scale and orientation of targets (mentioned in lines 251-252), and this modified FFTRadNet is then retrained and tested on our relabeled dataset. The performance of this modified FFTRadNet is reported in Table 3 and discussed in Section 5.2 in the paper.\n\nSince the codes of other baseline methods are still not available (lines 252-253), we only compared their reported performance on original point-level annotation. The results are shown in Table 2 in the paper, which demonstrates the superiority of our method. And to facilitate further research, we are planning to open-source our code.\n\n## Q3: What scenarios are corrected in new labels and how do the new labels influence the evaluation\n\nThe original annotation and prediction of previous methods are center points of objects (mentioned in lines 75-77), and they assign a fixed scale and zero orientation when calculating Average Precision (AP). But vehicles of large angles and scales are normal in daily scenarios, as shown in Figure R3 in the PDF of the general reply. Our new annotations based on dense LiDAR points (mentioned in lines 75-77) correct this, helping a more comprehensive evaluation of existing methods. We also annotate other traffic participants such as pedestrians and cyclists. But due to their limited numbers (4 cyclists, 23 pedestrians in training split, and 17 cyclists, 0 pedestrians in test split), we don’t add more fine-grained class labels."}, {"review_id": "vOR13D3Ixz", "reviewer": "Reviewer_w8tA", "summary": "This paper proposes a Polar-Aligned Attention module to guide the fusion of radar data and images. The module projects the image and radar features onto a polar coordinate system to calculate the similarity of the associated information. This effectively enhances the direct fusion of radar data and images. Additionally, the authors have restandardized the RADIal dataset, improving its accuracy and standardization.", "questions": "1. How is the alignment of different modalities ensured when fusing the features of image and radar data?\n2. Is it possible to visualize the alignment relationship between the two modalities in the polar-coordinate attention mechanism?", "limitations": "Due to the limitations of the dataset used in this study, it is not possible to extensively and sufficiently demonstrate the motivation behind the research in this paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors claim that this paper is the first work to directly fuse radar data with image data, achieving the fusion of relevant features through cleverly designed attention mechanism modules. Additionally, the authors' reannotation of the dataset has played a positive role in the development of this research direction.", "weaknesses": "Although this paper proposes a polar-coordinate-based attention mechanism module for the fusion of radar and image data, the contribution of such a structure is limited. Furthermore, due to the lack of dataset support for the research task of radar and image fusion in this study, the experiments were only conducted on an existing dataset with modifications. Therefore, I do not have enough confidence to affirm the sufficiency of the research motivation in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your valuable comments and constructive advice, which helps us a lot to make the paper better. The responses to the main concerns are as follows:\n\n## Q1: Whether the novelty and contribution of this work is significant\n\nIndeed, our work is based on PolarFormer, but the fusion mechanism is newly designed because raw radar data has no explicit location information. Moreover, our efforts on the annotation is nonnegligible. Please refer to “The Contribution of This Paper” section in General Reply for a conclusion of our contribution.\n\n## Q2: How is the alignment of different modalities ensured when fusing the features of image and radar data?\n\nThe misalignment problem may come from numerous factors including calibration, sensor trigger, ego-motion, and other traffic participants’ motion. The calibration and trigger errors are determined by sensors and always ensured to be limited in an acceptable range. Thus the primary source of misalignments is ego-motion and other traffic participants’ motion. Since we didn't remove the distortion caused by motion, the misalignment mainly influence vehicles with high relative speed.\n\nWe further examined misalignment caused by motion by measuring the average IoU of vehicles in the same (|yaw|<30 degrees) or opposite (|yaw|>150 degrees) directions with ego vehicle (yaw=0).  We take the IoU threshold of 0.5 and the score threshold of 0.01. The former situation includes 87 samples, with a recall of 70% and an average IoU of 0.799. The latter situation includes 1221 samples, with a recall of 94% and an average IoU of 0.867. It shows that vehicles with high relative speed lead to more severe misalignment, but our method still achieves relatively high IoU. Some bad cases are depicted in Figure R2 in the PDF of the general reply. Considering that the speed information is encoded in radar features, we believe that with more training data, the network is able to learn to better alleviate the misalignment by applying speed cues from radar.\n\n## Q3: Is it possible to visualize the alignment relationship between the two modalities in the polar-coordinate attention mechanism?\nThanks for your advice! We visualize the column-wise attention distribution in Figure R1 in the pdf file of general reply. As can be observed, the BEV queries learn to correctly attend to features of interested objects of specific range within corresponding image columns. But since we don't explicitly solve the azimuth information in range-time data and range-doppler spectrum, we are not able to get reasonable visualization of attention distribution for radar data.\n\n"}, {"review_id": "aBNxPyLvsY", "reviewer": "Reviewer_mPFG", "summary": "This paper presents a method to perform image and raw radar fusion for object detection from a Mobile platform. The authors explain that the commonly used radar data processing pipelines discard too much data, resulting in very sparse point clouds. In contrast, the proposed method operates on raw data and performs end-to-end filtering within the model with intermodality cross-attention. They consider the RADIal radar dataset, for which they provide more precise and exhaustive annotations to better evaluate their method's added precision.", "questions": "This submission presents a well-motivated and designed method with promising results. However, the presentation makes it hard to follow the methodology, and the lack of pedagogy makes it hard to understand for non-radar experts. However, the weaknesses should be mostly fixable.\n\nQ1) What are the arguments of CrossAttn? Why are arguments 2 and 3 are always the same?\n\nQ2) What are polar queries? Are they different from object queries?\n\nQ3) \"The polar coordinate is aligned with radar data on the range dimension and aligned with image data on the azimuth dimension\"\nThis sentence is hard to parse. What does this mean for a coordinate system to be aligned with a type of data?\n\nQ4) Figure 1 is unclear and hard to parse. Which one is the proposed method? Whar exactly are the methods compared? What are the red circles supposed to highlight?\n\nS1) Figure 2 could be great for explaining concretely the nature of the data presented. However, it is full of jargon and acronym and not clear at all. Maybe on a simpler example? Or even a toy example?", "limitations": "good", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The motivation and results are convincing, and the analysis interesting\n- The model is natural and exploits the unique properties and structure of radar acquisitions\n- Extensive ablation study and new annotations\n- Not a lot of work on this interesting and promising modality exists\n\n", "weaknesses": "- The method section could be more precise and detailed. The CrossAttn operator is not defined with enough details (what are its arguments, Why are arguments 2 and 3 are always the same?), and neither are polar queries. There is a reference to [8] at the end of the method which helps a  little bit, but the paper should be readable on its own. As it stands, the cross-attention method is not reimplemetable.\n\n- While submissions about sensors not typically treated at NeurIPS are welcome, the community is not necessarily familiar with the subtlety of radar data. As it stands, the submission overly relies on jargon and complex facts presented as self-evident. The figures and their caption could be more helpful. The authors should be more didactic.\n\n- The writing could be significantly improved, as it contains many imprecise and clumsy formulations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments and advice. The responses to the main concerns are as follows:\n\n## Q1: Clumsy formulations and jargons should be avoided\n\nThanks for reminding! Our motivation involves some concepts of radar signal processing. We are trying our best to make them intuitive and vivid. We will adopt plainer expressions in the revision for easier understanding.\n\n## Q2: What are the arguments of CrossAttn? Why are arguments 2 and 3 always the same?\n\nThe Cross Attention is formulated as follows:\n\n$$\n\\mathrm{CrossAttn}\\left( q,K,V \\right) =\\mathrm{softmax} \\left( \\frac{\\left( qW_Q \\right) \\left( KW_K \\right) ^T}{\\sqrt{d^*}} \\right) \\left( VW_V \\right)\n$$\n\n$$\nwhere\\,\\,q\\in \\mathbb{R} ^{1\\times d}, K\\in \\mathbb{R} ^{N\\times d}, V\\in \\mathbb{R} ^{N\\times d},W_Q\\in \\mathbb{R} ^{d\\times d^*}, W_K\\in \\mathbb{R} ^{d\\times d^*}, W_V\\in \\mathbb{R} ^{d\\times d^*},\n$$\n\nhere $N$ is the number of key vectors and value vectors. $d$ is the feature channel of input vectors. Matrix  $W_Q$, $W_K$, and $W_V$ are used to transform feature channel from $d$ to $d^*$.  The main idea of cross attention is first calculating similarity among query vectors $q$ and key vectors $K$. Then these similarities will act as weights to sum the value vectors $V$. The output vector is also of shape $\\mathbb{R} ^{1\\times d}$. Take Equation (3) in our paper as an example:\n\n$$\n\\hat{q}\\left( r_{bev},\\phi_{bev} \\right) =\\mathrm{CrossAttn}\\left( q\\left( r_{bev},\\phi _{bev} \\right) , F_I\\left( x_I \\right) , F_I\\left( x_I \\right) \\right).\n$$\n\nThe second and the third arguments are the same means the key vectors $K\\in \\mathbb{R} ^{N\\times d}$ and value vectors $V\\in \\mathbb{R} ^{N\\times d}$  are the same.\n\n## Q3: What are polar queries? Are they different from object queries?\n\nPolar Query and Object Query respectively depict the scene and candidate objects. To be specific, the polar query is of shape $\\mathbb{R}^{R_l \\times A_l \\times d}$, which describes the whole scene covering the Field of View of radar sensor under BEV space, as mentioned in lines 159-161. On the other hand, each object query is described by a feature vector of $\\mathbb{R}^{C}$ and a corresponding 3D position, which represents one candidate object. Thus, these two kinds of queries represent different entities.\n\n## Q4: Understanding of “align with” in lines 44-46\n\nSorry for the confusing description. Another reviewer also has a similar question. Please refer to “Understanding of “align with” in lines 44-46” part in General Reply.\n\n## Q5: How to interpret Figure 1?\n\nHere the main purpose is to show the drawbacks of representing radar data as point cloud. Thus, we compare the detection results of different variants of our proposed method. The purple box denotes the result from the image-only variant of EchoFusion (row 1 in Table 5), the green box denotes the results from the EchoFusion variant that fuses imagery and range time data (row 6 in Table 5), while the blue box denotes result from image and radar point cloud fusion variant of EchoFusion (row 8 in Table 5). The red circles aim to highlight the false alarm radar points from conventional signal processing pipelines, which appear even if there is no object. And these false alarm points can lead to inaccurate estimation of the scale and position of targets.\n\n## Q6: How to interpret Figure 2 and understand the mentioned jargon and acronym?\nSorry for the confusing description. All the jargon and acronyms are explained in Section 3. They are the fundamental concepts in radar signal processing. In Figure 2 in the paper, **ADC** data means Analog-Digital-Converter (ADC) data, which is directly sampled from the received radar signal (mentioned in lines 115-121). **Range-Time** data means the vertical and horizontal axis respectively denotes range and radar signal periods. The generation of Range-Time data is mentioned in lines 124-127. **Range-Doppler** data means the vertical and horizontal axis respectively denotes range and speed. The generation of Range-Doppler data is mentioned in lines 128-130. **CFAR result** means the result of applying CFAR(Constant False Alarm Rate Detector, mentioned in lines 28-30) on the Range-Doppler data. **Radar point cloud** plots derived 3D radar points under Bird’s Eye View. The generation of the Radar point cloud is described in lines 137-139. **Range-Azimuth Map** means the vertical and horizontal axis respectively denotes range and azimuth. For clarity, we plot this map under polar coordinates. The generation of the Range-Azimuth Map is described in lines 139-142.\n\nIn the Figure 2 caption in the paper, both the “Doppler Domain” and “Doppler dimension” in caption mean the horizontal axis, which indicates the speed of objects. We also mentioned Doppler domain multiplexing (DDM), which is used to distinguish received radar signals from different transmitters. As shown in Figure R2 in the pdf material, without DDM, the valid measurable speed interval is [ − *Vmax*, *Vmax*]. Suppose we have four transmitters, Tx 1/2/3/4, and one object with speed in [0, *Vmax*/2]. The echo signals from Tx 1/2/3/4 will respectively fall into sections C, D, A, B by using DDM, showing a periodical pattern. But now we can only measure speed in [0, *Vmax*/2]. Similar phenomenon can be observed in the figure of **Range Doppler data**, i.e. the third image in Figure 2 in the paper."}]

Author rebuttal:
Dear all,

Thanks again for the very constructive comments and for spending your valuable time reading our responses! We are open to any further discussions during the author-reviewer discussion period (Aug 9-16). PDF material contains Figure R1-R4 used in personal replies.

## More Experiments on KRadar Dataset

Table R1. Detection performance on **KRadar-20** test split. AP_BEV and AP_3D are AP under Bird’s Eye View or 3D View. The ioU threshold of each metric is presented after @. Best results are in **bold**.

| Training Dataset | Vehicle AP/Conf@0.3 | AP_BEV@0.3 | AP_3D@0.3 | AP_BEV@0.5 | AP_3D@0.5 | AP_BEV@0.7 | AP_3D@0.7 |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  KRadar | RTHN-Pretrained | 58.04 | 49.65 | 42.60 | 17.87 | 10.69 | 0.45 |
| KRadar-20-trainval | RTHN-Scratch | 61.38 | 53.05 | 46.47 | 17.98 | 10.47 | 3.03 |
| KRadar-20-trainval | EchoFusion-ra+img | 69.95(+8.63) | 68.35(+15.30) | 57.28(+10.81) | 43.87(+25.89) | 33.07(+22.60) | 14.00(+10.97) |
| KRadar-20-train | EchoFusion-rpcd | 55.74 | 53.40 | 43.94 | 29.21 | 20.56 | 2.90 |
| KRadar-20-train | EchoFusion-ra | 54.45 | 51.75 | 42.28 | 24.65 | 17.97 | 3.85 |
| KRadar-20-train | EchoFusion-rpcd+img | 66.46 | 58.62 | 54.17 | 34.67 | 26.39 | 3.96 |
| KRadar-20-train | EchoFusion-ra+img | 68.70 | 66.68 | 55.90 | 34.33 | 29.65 | 5.34 |

Through additional experiments on the recently published 4D Raddar dataset KRadar [1], we validate our motivation, and prove 3D prediction ability of EchoFusion with a high-resolution 4D radar and accurate 3D bounding boxes.

### About Dataset

We conduct more experiments on the recently published 4D Radar dataset [KRadar](https://github.com/kaist-avelab/K-Radar) [1]. However, we come across extremely limited speed when downloading the large-scale data(12T), which remains unsolved (refer to issues: [#2](https://github.com/kaist-avelab/K-Radar/issues/2),[#11](https://github.com/kaist-avelab/K-Radar/issues/11)). Thus we turn to 20 sequence sub-dataset stored on Google Drive (refer to issue [#9](https://github.com/kaist-avelab/K-Radar/issues/9)). This so-called **KRadar-20** dataset includes 6493 training samples and 6515 test samples. The 6 times larger test split than the RADIal dataset and 50% nighttime data make it challenging. The training samples are named as trainval split, it is further split into train split of 5190 samples and val split of 1303 samples for ablation.

### Result Analysis

KRadar dataset contains radar formats of range-azimuth map (RA map) and radar point cloud (radar pcd). Different from RADIal, the RA map of KRadar contains extra height dimension. We evaluate EchoFusion using these two types of data with official KITTI-style metrics of score threshold 0.3. The baseline is the official RTNH[1], which consumes radar pcd. To align with baseline, we take radius and azimuth range respectively as [0, 72] m and [-20, 20] degrees. The results are shown in Table R1. With trainval split and RA map and image fusion, our EchoFusion improves over 10 points at each metrics, except for BEV AP of IoU threshold 0.3. On ablation experiments using train split, the RA map shows considerable superiority over radar pcd when fusing with monocular image, which aligns with our research motivation. 

> [1] Paek D H, Kong S H, Wijaya K T. K-Radar: 4D radar object detection for autonomous driving in various weather conditions, NIPS2022.
> 

## The Contribution of This Paper

- We are the first to fuse raw radar data with images. Raw radar data has no explicit location information, which is significantly different from point cloud. Existing BEV methods cannot be directly applied in this task.
- We present the limitations of existing radar datasets and provide accurate 3D bounding box annotations for the raw radar dataset RADIal.
- We indeed demonstrate the advantage of using radar raw data instead of the converted radar point cloud in real benchmarks and achieve SOTA.

## Understanding of “align with” in lines 44-46

By using “align with” in this sentence “This technique is based on an observation that the polar coordinate is aligned with radar data on the range dimension and aligned with image data on the azimuth dimension.”, we mean that both of Range-Time data, Range-Doppler data, or Range-Azimuth Map have a dimension describing the range, which can be well depicted by the radial axis of polar coordinates. For the image, the horizontal dimension corresponds to azimuth, which can be well described using the azimuth axis of polar coordinates.

Meta-review: Already the initial reviews were rather positive, and after the rebuttal all five recommendations are on the positive side. The AC agrees with them. There was some criticism regarding the clarity of the paper, please make an effort to improve the presentation as promised during the discussion.
Decision: Accept (poster)