SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yang Liu CASIA
Feng Wang TuSimple
Naiyan Wang TuSimple
Zhaoxiang Zhang  
MAIS of CASIA & UCAS, HKISI_CAS
{liuyang2022, zhaoxiang.zhang}@ia.ac.cn {feng.wff, winsty}  $@$  gmail.com
Radar is ubiquitous in autonomous driving systems due to its low cost and good adaptability to bad weather. Nevertheless, the radar detection performance is usually inferior because its point cloud is sparse and not accurate due to the poor azimuth and elevation resolution. Moreover, point cloud generation algorithms already drop weak signals to reduce the false targets which may be suboptimal for the use of deep fusion. In this paper, we propose a novel method named EchoFusion to skip the existing radar signal processing pipeline and then incorporate the radar raw data with other sensors. Specifically, we first generate the Bird's Eye View (BEV) queries and then take corresponding spectrum features from radar to fuse with other sensors. By this approach, our method could utilize both rich and lossless distance and speed clues from radar echoes and rich semantic clues from images, making our method surpass all existing methods on the RADial dataset, and approach the performance of LiDAR. The code will be released on https://github.com/tusen-ai/EchoFusion.
Robust and accurate perception is a long-standing challenge in Autonomous Driving Systems (ADS). The full perception system usually relies on multiple sensor fusion including camera, LiDAR, and radar. Camera provides rich semantic cues of objects and excellent resolution, while LiDAR is capable of capturing highly accurate spatial information. However, neither camera nor LiDAR can directly measure speed or survive in adverse weather conditions, such as fog, sandstorms, and snowstorms [26]. Although radar can overcome the aforementioned problems, even the point cloud from the latest 4D millimeter wave (mmWave) radars suffers from severe sparsity and low angular resolution, which makes it hard to discriminate objects and backgrounds solely. Fortunately, radar and camera are highly complementary to each other. Fusing these two types of sensors becomes a promising solution.
Radar data are usually represented as point clouds. Technically, the radar point cloud stems from raw Analog-Digital-Converter (ADC) data received by antennas. Range, speed, and angle of arrival (AoA) information of perceivable objects can be explicitly extracted in the frequency domain by consecutively applying FFT along corresponding dimensions. Among these steps, side lobe suppression and constant false alarm rate (CFAR) detector [40] are usually adopted to reduce the noise and false alarms. As a result, all the derived points are equipped with spatial coordinates, speed, and reflection intensity. Though these noise suppression operations can significantly reduce the data size and further computation costs, the resulting radar point cloud is extremely sparse, as shown in Figure 1. Even worse, quite a lot of useful information is lost during CFAR, which is adverse to the accurate perception of environments and subsequent fusion with other sensors [18]. Raw data serves as a
Figure 1: Illustration of sparsity and false alarms of the radar point cloud. The red points and the blue points correspond to data from radar and LiDAR, respectively. We visualize the predicted bounding boxes of different variants of our proposed method. red denotes ground-truth, purple denotes predictions from image only modality, green denotes predictions from image and radar range-time data fusion, and blue denotes predictions from image and radar point cloud fusion. Best viewed on screen.
potential solution to alleviate these problems. However, how to efficiently utilize it, especially fusing with other sensors remains an open problem.
Recently, Birds-Eye-View (BEV) based methods are prevalent for driving scenario perception. It integrates features from multiple views and multiple modalities into a unified 3D representation space [4; 17; 24]. Among the varieties of BEV-based methods, PolarFormer [9] divides the 3D space in polar coordinates, which matches the format of raw radar data better. As such, we chose PolarFormer as our baseline model.
In this work, we present EchoFusion, a method designated for fusing raw radar data with images. We observe that a set of 3D positions exactly corresponds to one column in the image and one row in raw radar data. Thereby we propose a novel polar-aligned attention (PAA) technique to efficiently fuse features from these two different modalities. With row-wise cross-attention on radar data and column-wise attention on image, PAA could precisely aggregate essential features from both modalities while maintaining simple and efficient implementation. Finally, a polar BEV decoder refines object queries for accurate bounding box predictions.
The contribution of our paper is three-folds:
1. We are the first to fuse raw radar data with images in BEV space. Specifically, we propose a novel Polar-Aligned Attention module to guide the network effectively learn radar and image features.
2. We relabel the RADIal dataset [39] with accurate 3D bounding box annotations. The new annotations is published with our codebase together.
3. Extensive experiments show that our method has outperformed all the existing methods and achieved promising BEV detection performance, even approaching the LiDAR-based method.
2.1 Deep Learning on Radar
2.1.1 Radar Data Representation and Datasets
Traditional radar usually only outputs sparse points [2; 5; 42] for use. However, such points are subjective to heavy signal processing methods to reduce noisy observation and alleviate bandwidth limitation. Nevertheless, such hand-crafted pipelines make the subsequent fusion with other sensors intractable. Consequently, some attempts have been made to provide upstream data, such as range-azimuth-Doppler (RAD) tensor [34; 54], range-azimuth (RA) maps [44], range-Doppler (RD) spectrums [29] or even raw Analog Digital Converter (ADC) data [19; 29]. Note that the RAD, RA, and RD tensors are generated from ADC data using several times (almost) lossless FFT. Thus we call all these four types of data "radar raw data" in the sequel.
Since our motivation is to deeply fuse with other sensors, we only consider the datasets providing data beyond point cloud. Among them, both RADIal [39], Radatron [26] and KRadar [35] provide such data. But Radatron only covers a single scenario while lacking camera calibration, LiDAR modality,
Table 1: 4D Radar Datasets Comparison. Data: PC, RA, ADC denote to point cloud, range azimuth map and raw ADC data; Sensors: C,  $C_s$ , L, O denote to camera, stereo camera, LiDAR, and odometer; Scenarios: U, S, H denote to urban (city), suburban, highway; Annotations: 3D, BEV, T,  $P_o$ , M denote to 3D bounding box, BEV bounding box, track ID, object-level point and segmentation mask; Classes and Size denote the number of classes and the number of annotated frames of each dataset.
Dataset	Data	Other Sensors	Scenarios	Annotations	Classes	Size
Astyx [28]	PC	CL	SH	3D	7	500
View-of-Delft [37]	PC	CsLO	U	3D,T	13	8693
RADIal [39]	ADC,RA,PC	CLO	USH	Po,M	1	8252
TJ4DRadSet [57]	PC	CLO	U	3D,T	5	7757
Radatron [26]	ADC,RA	Cs	U	BEV	1	16K
KRadar [35]	RA, PC	CLO	USH	3D,T	5	35K
and height annotation, while KRadar only provides radar RA map and PCD formats. Taking the factors mentioned above into account, we carry out our experiments mainly on the RADlal dataset, and provide additional results on a subset of KRadar dataset. However, the annotations of RADlal only contain object-level points. For a more comprehensive and objective evaluation, we further annotate the bounding box size and heading of each object based on dense LiDAR points. The annotations can be found in our codebase.
2.1.2 Object Detection on Radar Data
Traditional radar object detection methods that utilize point cloud face challenges of super sparsity due to the low angular resolution. Practical treatments include occupancy grid mapping [59] and separate processing for static and dynamic objects [43]. 4D radar technology improves the azimuth resolution and provides additional elevation resolution, which enables the combination of static and dynamic object detection in a single network, using either PointPillars [37] or tailor-made self-attention modules [1; 51].
The pre-CFAR(Constant False Alarm Rate Detector) data provides rich information of both targets and backgrounds. RAD tensor is fully decoded but brings high demand of storage and computation. Hence, Zhang et al. [54] takes the Doppler dimension as channels and Major et al. [27] projects RAD tensor to multiple 2D views. RTCNet [36] divides RAD tensor into small cubes and applies 3D CNN to reduce computation burden. Besides, [39; 55] take complex RD spectrum as input and apply neural networks to automatically extract spatial information. Networks such as RODNet [48] adopt RA maps for detection, which avoid false alarms caused by extended Doppler profile. Despite the aforementioned research works, the utilization of ADC radar data has recently gained increasing attention within the community [7; 52]. However, their results remain unsatisfactory.
The fusion of different type of sensors provides complementary cues, leading to more robust performance. At input level, radar point clouds are usually projected as pseudo-images and concatenated with camera images [3; 33]. At Region of Interest (RoI) level, some approaches [31; 32] consecutively refine the RoIs by radar and other modalities, while others [12; 14] unify RoIs generated independently by different sensors. At feature level, [50; 56] integrate feature maps generated from different modalities, while [11; 13] use RoIs to crop and merge features across modalities. To the best of our knowledge, we are the first to deeply fuse radar using raw data with other modalities in a unified BEV perspective.
2.2 BEV 3D Object Detector
BEV object detection bursts out a rising wave of research due to its vast success in 3D perception tasks. One thread is to adopt transformations to project 2D image features to 3D BEV space for further detection, such as OFT [41] and LSS [38]. Another thread applies initialized BEV queries [58] or object queries [22; 23] to iteratively and automatically sample features from multi-view images. Based on these advanced techniques, BEVFusion [24] explores the advantages of BEV representation in multi-sensor fusion and achieves impressive performances. Despite that, how to make full use of other information sources also attracts the interest of researchers. BEVFormer [17] and its variant [53] utilize temporal information to enhance detection capability, while BEVStereo [16] and STS
Figure 2: Radar data formats used in this paper. All the axes that indicate spatial location are labeled with units. For ADC data, the vertical unit is the sampling cycle. And for ADC data, Range-Time data, the horizontal unit is the chirp cycle. Due to Doppler domain multiplexing (DDM), the real speed is periodic on the Doppler dimension, which is hard to label (More details of DDM can be found in the appendix). We use the index number of the Doppler dimension as horizontal unit instead in Range-Doppler and CFAR result.
[49] explore how the estimated depth can benefit BEV-based detection. Besides, PolarFormer [9] introduces the polar grid setting and proves its effectiveness in environment perception, which is closely related to our work.
In the automotive industry, multiple-input, multiple-output (MIMO) frequency-modulated continuous-wave (FMCW) radars are adopted [59] to balance cost, size, and performance. Each transmitting antenna (Tx) transmits a sequence of FMCW waveforms, also named chirps, which will be reflected by objects and returned to the receiving antennas (Rx). Then the echo signal will be first mixed with the corresponding emitted signal and then passed through a low pass filter, so as to obtain the intermediate frequency (IF) signal. The discretely sampled IF signal, which includes the phase difference of emitted and returned signal, is called ADC data [46]. The ADC data has three dimensions, fast time, slow time, and channel, which have physical correspondence with range, speed, and angle respectively.
Because of the continuous-wave nature, the IF signal of a chirp contains a frequency shift caused by the time delay of traveling between radar and objects. Then the radial distance of obstacles can be extracted by applying Fast Fourier Transform (FFT) on each chirp, i.e. fast-time dimension. This FFT operation is also called range-FFT. The resulting complex signal is denoted as range-time (RT) data. Next, a second FFT (Doppler FFT) along different chirps, i.e. slow-time dimension, is conducted to extract the phase shift between two chirps caused by the motion of targets. The derived data is named range-Doppler (RD) spectrum. We denote  $N_{Tx}$  and  $N_{Rx}$  respectively as the number of transmitting and receiving antennas. And each grid on the RD spectrum has  $N_{Tx} \times N_{Rx}$  channels. The small distance between adjacent virtual antennas leads to phase shifts among different antennas, which is dependent on the AoA. Thus a third FFT (angle-FFT) can be applied along the channel axis to unwrap AoA. The AoA can be further decoded to azimuth and elevation based on the radar antenna calibration. The final 4D tensor is referred to as the range-elevation-azimuth-Doppler (READ) tensor.
If point cloud is desired, the RD spectrum will be first processed by the CFAR algorithm to filter out peaks as candidate points. Then the angle-FFT will only be executed on these points. Final radar point cloud (PCD) consists of 3D coordinates, speed, and reflective intensity. In traditional radar signal processing, there is another data format named range-azimuth (RA) map, which is obtained by decoding the azimuth of a single elevation on the RD spectrum and compressing the Doppler dimension to one channel. For comprehensive analyses, we include this modality in our experiments as well. The data formats mentioned above are illustrated in Figure 2, except for READ tensor because it is difficult to visualize.
Figure 3: Overall pipeline of the proposed EchoFusion. H and W are the height and weight of the image, while R, A, and T are the bin number of range, azimuth, and Doppler dimension, respectively. The Coord. Mapping in the figure is explained in section 4.2.
In this section, we will introduce our network design. Firstly we will give a brief introduction of the whole pipeline in Section 4.1. Then our proposed Polar-Aligned Attention module will be elaborated in Section 4.2 and Section 4.3. Finally, the detection head and loss functions are introduced in Section 4.4.
4.1 Overall Structure
The overall architecture of our model is shown in Figure 3. First, we use separate backbones and FPNs [20] to extract multi-level features from RT radar maps and images. Then the polar queries aggregate camera features and radar features by cross-attention. Finally, the multi-scale polar BEV map will be passed through a transformer-based decoder with object queries. The final score and bounding box predictions can be obtained by decoding the output object queries. The core of our EchoFusion is how to fuse the camera and radar slow-time data. We fuse them using a novel strategy named Polar Aligned Attention (PA-Attention). In the next two subsections, we will explain this strategy in detail.
4.2 Column-wise Image Fusion
After obtaining image and radar features from corresponding encoders, we first initialize the  $l$ -th level polar queries  $\pmb{Q}_l \in \mathbb{R}^{R_l \times A_l \times d}$  uniformly in polar BEV space. Here,  $R_l$ ,  $A_l$ , and  $d$  respectively denote the shape of range, azimuth, and channel dimensions. For brevity, we omit  $l$  in the following two subsections. Given a query  $q \in \mathbb{R}^d$  located at  $(r_{bev}, \phi_{bev})$ , it corresponds to a pillar centered at  $(r_{bev}, \phi_{bev})$  with infinite height. Taking the coordinate system defined in RADIal [39],  $x_R$ ,  $y_R$ , and  $z_R$  axes respectively points to the front, left, and upright, while subscript  $R$  means radar coordinate. Similarly, the camera coordinates are denoted as  $(x_C, y_C, z_C)$ , which points to the right, down, and front, respectively. Thus we have correspondence in Cartesian coordinate system as follows:
$$
x _ {R} = r _ {b e v} \cos (\phi_ {b e v}), \quad y _ {R} = r _ {b e v} \sin (\phi_ {b e v}). \tag {1}
$$
Using extrinsic matrix and intrinsic matrix, we have the following formulation:
$$
\frac {x _ {I} - u _ {0}}{f _ {x}} = \frac {x _ {C}}{z _ {C}}, \quad \left( \begin{array}{l} x _ {C} \\ y _ {C} \\ z _ {C} \end{array} \right) = R \left( \begin{array}{l} x _ {R} \\ y _ {R} \\ z _ {R} \end{array} \right) + T, T = \left( \begin{array}{l} d _ {1} \\ d _ {2} \\ d _ {3} \end{array} \right), \tag {2}
$$
where  $u_0, f_x$  are principal point offsets and focal length on  $x$ -axis in the intrinsic matrix,  $x_I$  is the image column index that corresponds to the point  $(x_R, y_R, z_R)$ , while  $R$  and  $T$  are rotation matrix and translation vector of the calibration matrix between camera and radar, respectively. Under mild distortion conditions and normally adopted sensor setting,  $x_I$  is determined by pillar position and calibration parameters as follows:
$$
x _ {I} \approx u _ {0} + f _ {x} \frac {- r _ {b e v} \sin \phi_ {b e v} + d _ {1}}{r _ {b e v} \cos \phi_ {b e v} + d _ {3}}. \tag {3}
$$
Limited by the number of pages, we leave the detailed proof in the appendix.
However, the height is not specified for a query, while for a pixel  $(x_{I},y_{I})$  on the monocular image, its depth is unknown as well. Thus we cannot precisely associate a query to the image feature, but we can correspond queries of the same azimuth to the same column of images. Enlightened by this observation, we use a versatile cross-attention mechanism to flexibly aggregate required features from images. Namely, all the queries with the same azimuth form the query matrix, while all the features in the corresponding column in the image form the key and value matrix. Formally, these two steps can be expressed as follows:
$$
F _ {I} (x _ {I}, \cdot) = \operatorname {S t a c k} \left(\left[ f _ {I} \left(x _ {I}, y _ {I}\right) \right]\right) \in \mathbb {R} ^ {H _ {l} \times d}, \quad \forall y _ {I} \in [ 0, H _ {l} - 1 ],
$$
$$
\hat {q} \left(r _ {b e v}, \phi_ {b e v}\right) = \operatorname {C r o s s A t t n} \left(q \left(r _ {b e v}, \phi_ {b e v}\right), F _ {I} \left(x _ {I}, \cdot\right), F _ {I} \left(x _ {I}, \cdot\right)\right), \tag {4}
$$
where  $f_{I}(x_{I},y_{I})$  is the image feature located at  $x_{I},y_{I}$ , while details of cross attention function CrossAttn can be found in [47].
4.3 Range-wise Radar Fusion
We denote updated BEV queries as  $\hat{Q}_l \in \mathbb{R}^{R_l \times A_l \times d}$ . After that, it will be used to sample features from radar. Since the radar feature map shares the same range partition with the BEV queries, the updated query  $\hat{q}(r_{bev}, \phi_{bev})$  matches the same range index of the radar feature, i.e.  $r_R = r_{bev}$ . To aggregate all the features from all chirps in one frame, we perform cross attention for a certain query  $\hat{q}$  with the row  $r_R$  in RT map as key and value. Formally,
$$
F _ {R} \left(r _ {R}, \cdot\right) = \operatorname {S t a c k} \left(\left[ f _ {R} \left(r _ {R}, t _ {R}\right) \right]\right) \in \mathbb {R} ^ {R _ {l} \times d}, \quad \forall t _ {R} \in [ 0, T _ {l} - 1 ],
$$
$$
\tilde {q} \left(r _ {b e v}, \phi_ {b e v}\right) = \operatorname {C r o s s A t t n} \left(\hat {q} \left(r _ {b e v}, \phi_ {b e v}\right), F _ {R} \left(r _ {R}, \cdot\right), F _ {R} \left(r _ {R}, \cdot\right)\right), \tag {5}
$$
where  $R_{l}$  and  $T_{l}$  are the height and width of the radar feature map, and  $f_{R}(r_{R},t_{R})\in \mathbb{R}^{d}$  is the radar feature of position  $(r_R,t_R)$ .
Note that though we don't explicitly decode AoA as in RA map, the AoA has been implicitly encoded in the phase difference of the response of different virtual antenna. Consequently, it can be learned from the features in RT map indirectly.
4.4 Head and Loss
We follow polar BEV decoders [9] to decode multi-level BEV features and make predictions from object queries. Since we don't have velocity annotation on this dataset, we remove the corresponding branch. The regression targets include  $(d_{\rho}, d_{\phi}, d_z, \log l, \log w, \log h, \sin(\theta_{ori} - \phi), \cos(\theta_{ori} - \phi))$ , where  $d_{\rho}, d_{\phi}, d_z$  are relative offsets to the reference point  $(\rho, \phi, z), l, w, h, \theta_{ori}$  are length, width, height, and orientation of bounding box. The classification and regression tasks are respectively supervised by Focal loss [21] and L1 loss.
In this section, we conduct thorough experiments to validate the effectiveness of our approach. We first introduce the dataset and metrics used in our experiments, then followed by comparisons with other state-of-the-art methods. Lastly, we ablate some designs and potential input format of raw radar data in our method, and discuss some limitations.
The RADial Dataset [39] provides 8252 annotated frames with synchronized multi-sensor data. Training, validation, and test sets contain 6231, 986, and 1035 frames, respectively. Each object is annotated with a 3D center coordinate of the visible face of the vehicle. The size and orientation of the objects are pre-defined templates, i.e. they are the same for all objects. In the evaluation, precisions and recalls under ten thresholds are calculated. The average precision (AP), average recall (AR), and F1 score are obtained through the ten precisions and ten recalls as the evaluation metric. Range error (RE) and azimuth error (AE) are also reported to evaluate localization accuracy. The annotation method and metrics are quite different to those in common autonomous driving datasets
Table 2: Detection performances on RADial Dataset test split with original protocol [39]. C, RD, and RT respectively refer to camera, range-Doppler spectrum, and range-time data. Our method achieves the best performance in both average precision (AP), average recall (AR), and F1-score (F1). The best result of each metric is in bold.
Methods	Modality	AP(%)↑	AR(%)↑	F1(%)↑	RE(m)↓	AE(°)↓
FFTRadNet [39]	RD	96.84	82.18	88.91	0.11	0.17
T-FFTRadNet [7]	ADC	89.60	89.50	89.50	0.15	0.12
ADCNet [52]	ADC	95.00	89.00	91.90	0.13	0.11
CMS [10]	RD&C	96.90	83.50	89.70	0.45	n/a
EchoFusion	RT&C	96.95	93.43	95.16	0.12	0.18
[2; 45]. Nonetheless, to compare with state-of-the-art methods, we still list our results using the metrics defined in RADIal [39] as a reference.
Re-annotation and more metrics. Based on the provided LiDAR point cloud, we have refined the cluster-center-based annotation to 3D bounding box form, which includes center position, scale, and heading in 3D space. Capitalizing on these new annotations, we are able to apply metrics that are vastly used in 3D object detection tasks. Specifically, we take AP defined in Waymo dataset [45] as our main metrics. Since the LiDAR used in RADial is only 16-beam, we cannot label accurate height for the objects. So we mainly utilize BEV IoU for evaluation. We use the threshold of 0.7 for normal IoU and longitudinal error-tolerant (LET) IoU computation [8]. Considering the spatial distribution of ground-truth, the distance evaluation is further broken down into two categories: 0 to 50 meters, 50 to 100 meters. For a more comprehensive evaluation, we also introduce the error metrics defined in nuScenes dataset [2], including ATE, ASE, and AOE for measuring errors of translation, scale, and orientation.