# Towards Deep Radar Perception for Autonomous Driving: Datasets, Methods, and Challenges
## 3 Datasets, Labelling, and Augmentation
### Data Labelling  
Before introducing the labelling process, we first discuss the time synchronisation problem. Different sensors can be synchronised using pulse-per-second (PPS) triggering signals from the GNSS receiver [71]. However, in most of the radar datasets, sensors differ in their triggering time and sampling frequency. Some of them select one sensor as the  
Figure 5: Two types of radar calibration targets [64; 65]. The front board is made of styrofoam. The red triangle is a radar corner reflector.  
lead and choose the closest frames from other modalities for synchronisation. Assuming a tolerable time offset of 50 ms, a vehicle with a relative speed of 20 m/s will lead to an offset of 1 m. Therefore, it is necessary to compensate for synchronisation errors in high-speed scenarios. [PERSON] et al. [72] designed a pose chain method to interpolate inter-frame measurements. The translational and rotational transformations were determined by a constant velocity model and spherical linear interpolation (SLERP) [73], respectively.  
Labelling radar data is a difficult task. Both radar point clouds and pre-CFAR data are hard to interpret by human labellers. To reduce the labelling efforts, most of the datasets adopt a semi-automatic labelling framework, which includes two steps: cross-modality pre-labelling and fine-tuning.  
In the first step, a well-trained detector on other modalities is leveraged for radar labelling. For 3D tasks, radar point clouds can be annotated by 3D boxes predicted by the detector trained with images and LiDAR point clouds [2]. If we want to obtain pointwise annotations for the radar point cloud, we can first predict a dense semantic map for the corresponding image using a visual segmentation network, such as mask R-CNN [74] or DeepLab V3 [75]. To avoid scale ambiguity, it is better to project the masked image to radar frames using depth measured by LiDAR [72] or a stereo camera [46; 47]. Then, each radar point can be associated with the corresponding semantic labels. Pointwise annotation of RAD tensors or RA maps is a similar process. We can firstly use CFAR to detect peaks as detections, then annotate these point detections with the aligned visual semantic map. The CRUW dataset [47] proposes a postprocessing method to obtain pointwise object annotations for RA maps. The authors define an object location similarity (OLS) metric, which jointly considers the similarities in distance, scale, and class. Then, they proposed a location-based non-maximum suppression (NMS) method that selects one object point out of the adjacent points based on the OLS metric. Compared to the RA map, the RD map alone is much more difficult to label. It needs both depth and radial velocity to associate an RD cell with a pixel or a LiDAR point. Radial velocity can be estimated by visual scene flow [76] or by tracking [45].  
In the second step, manual inspection is required to correct pre-labelling errors. Identifying radar errors involves domain knowledge and, therefore, requires hiring of radar experts. As a result, building a high-quality, large-scale radar dataset is both time-consuming and financially expensive. To improve labelling efficiency, one way is to reduce the amount of data to be labelled. [PERSON] et al. [77] leveraged a tracking algorithm to interpolate annotations between key frames. The intermediate position is estimated by a Kalman filter with optical flow as measurements. [PERSON] et al. [2] adopted an active learning [78] framework to reduce labelling efforts in building the Astyx dataset. The core idea is to only label the most informative data. Specifically, they first labelled a small number of frames and trained a detector with this data subset. The trained detector is then used to make predictions on the remaining unlabelled data. Next, the top N uncertain data are again manually labelled and added to the training subset. This process is repeated until convergence of the validation performance.