# Towards Deep Radar Perception for Autonomous Driving: Datasets, Methods, and Challenges
## 3 Datasets, Labelling, and Augmentation
### Radar Datasets  
Different radar datasets use different types of radar. We can classify radar sensors into low resolution (LR) and high resolution (HR). There are different technical routes to achieve high resolution, such as polarimetric radar [29], cooperative radars [30], multi-chip cascaded MIMO radar [13], synthetic aperture radar (SAR) [31], and spinning radar [32]. Most off-the-shelf radars can output a point cloud with range, azimuth angle, Doppler velocity, and RCS. Next-generation 4D radar can also measure elevation angle. Some radar prototypes can be configured to output radar raw data, including ADC data, RA/RD maps, and RAD tensors. The role of radar in autonomous driving can be divided into localisation and detection. Although this paper focuses on radar detection, we also introduce the localisation datasets in this section. Since these datasets usually provide synchronised LiDAR and images along with radar data, it is possible to annotate them for detection purpose, as done in [33]. There are various levels of label granularity for radar data. For radar point clouds, it is possible to provide 2D bounding boxes, 3D bounding boxes, or pointwise annotations. The 2D bounding boxes are labelled in bird's eye view (BEV) and with orientation information; hence, they are sometimes referred to as pseudo-3D boxes. The 3D bounding boxes further capture height information and pitch angle. If properly annotated, pointwise annotation can provide semantic information at a finer granularity than bounding boxes. In fact, radar detections within the bounding box could also be ghost detection or clutter. Therefore, pointwise annotation is a better way to capture the noisy nature of the radar point cloud. Similarly, radar pre-CFAR data, including RA/RD maps and RAD tensors, are also annotated pointwise. Some works dilate the annotated points into a dense mask or a bounding box. However, these dilated patches do not necessarily reflect the shape information. Some techniques for precise dilation will be introduced later in Section 4. There are some large-scale datasets for autonomous driving that include off-the-shelf 2D radars in their sensor suites. NuScenes [34] is the most popular dataset for its large-scale and diverse scenarios. The capturing vehicle is equipped with a 32-beam LiDAR, 6 cameras, 5 long-range multi-mode radars, and a GPS/IMU system. It provides 3D annotations of 23 classes of road users in 1000 scenes, with a total of 1.3 million frames. However, this dataset is not a good choice for studying the role of radar in perception, because its radar point clouds are too sparse. The PixSet dataset [35] also aims at 3D object detection. The vehicle is equipped with a colocated sensor platform consisting of a solid-state LiDAR, a 64-beam LiDAR, a TI AWR1843 radar, and a GPS/IMU system. The FoVs of different modalities are largely overlapped and, hence, are well suited for evaluating sensor fusion algorithms. The RadarScenes dataset [36] is a diverse large-scale dataset for instance segmentation of radar point clouds. It uses four 77 GHz radars with overlapping FoVs in the front of the vehicle. Each radar is in middle-range mode with a maximum range of 100 m and a 60\({}^{\circ}\) FoV. Compared to the NuScenes dataset, its radar point clouds are much denser. The dataset contains 100 km of driving in 158 different scenarios. It provides both pointwise annotations and track IDs for 11 classes of moving road users. All points with zero velocity are labelled as static. Pointillism [37] leverages a multi-radar setup to improve the resolution. Two TI IWR1443 radars were placed at the front of the car, facing forward, at a distance of 1.5 m. The aim is to study the effect of coherently integrating point clouds from two radar sensors. In order provide the ground truth of radar point clouds, the sensor suite also include a 16-beam LiDAR and a camera with overlapping FoVs. The dataset contains 54K synchronised frames for five typical driving scenarios under different weather conditions.