Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a new dataset called K-Radar. It includes three common sensors for autonomous driving applications, i.e., camera, radar, and lidar. The radar they are using is more advanced than the hardware in most existing datasets, which is very impressive. Besides, the variety of weather conditions is also very competitive. Although some limitations exist, it would still be a good dataset and helpful for related research. 
strengths: - Scale of the dataset is good with a variety of different weather conditions. 
- The radar sensor they use is advanced with 4D tensor representation for many radar perception tasks. 
weaknesses: - Baseline and benchmarking are not sufficient. More radar-based baselines can be added.
- Some other baselines are also recommended to be added like camera-based methods. 
- Some related dataset comparison is missing (see below). 
correctness: The dataset construction is sound.
clarity: Paper writing is clear. However, the calibration part should be more detailed described, especially for camera and lidar extrinsics calibration.
relation to prior work: Missing related dataset: 
[1] Wang, Yizhou, et al. "Rethinking of Radar's Role: A Camera-Radar Dataset and Systematic Annotator via Coordinate Alignment." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
additional feedback: N/A
review_title: Good dataset for autonomous driving applications

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper introduces a 4DRT-based 3D object detection dataset and benchmark, K-Radar. Unlike conventional 3DRT, K-radar contains power measurements along the Doppler, range, azimuth, and elevation dimensions. The paper also shows the baseline results on this dataset for the 3D object detection task. Moreover, the paper shows that height information is crucial for 3D object detection.
strengths: - The new dataset provides 3D bounding box labels and tracking ID for 93.3K objects of five classes. 
- The new dataset also provides boxes in diverse conditions and times.
- Unlike conventional 3DRT, K-radar contains power measurements along the Doppler, range, azimuth, and elevation dimensions, enabling precise 3D object detection.
- The paper shows that height information is crucial for 3D object detection.
- The 4DRT-based baseline outperforms the old lidar-based baseline in the 3D object detection task, especially in adverse weather conditions.
- Excellent documentation
weaknesses: I am willing to raise my score if the authors provide response to my questions.

- Table 1 in the paper does not quantify the statistics of existing datasets. Please quantitatively compare the datasets keeping the following columns in the table.

    - #Images
    - #Labelled Images
    - #3D boxes
    - #Train images
    - #Val images
    - #Test images
    - #Drive (hours)
    - #Area covered (km^2)
    - #Cities
    - Multiple Times of the day (Yes/No)
    - Multiple Lanes (Yes/No)
    - Hilly regions (Yes/No)
    - Videos (Yes/No)
    - Rainy images (Yes/No)
    - Snowy images (Yes/No)
    - Foggy images (Yes/No)

- Most bigger benchmarks compare the object detection performance over multiple classes. This paper compares the detection performance of the sedan class (L257). Please quantitatively report the performance of the baseline and proposed model on all the five classes rather than the sedan class.

- The pointpillars is a pretty old baseline for lidar-based object detection. Why did the authors choose this baseline? Please quantitatively report the results of using newer baselines from 2021.

- Since other datasets do not have the height in the radar modality, it would be good to quantitatively compare K-Radar and Radiant as a pre-training dataset. If the K-Radar is indeed diverse, we should see a strong improvement after training with the K-Radar dataset.

- Next, quantitatively report the performance of the model trained on K-Radar without height and evaluated on RADIATE. Similary, quantitavely report the performance of RADIATE dataset trained model on K-Radar (without height).

correctness: - Most bigger benchmarks compare the object detection performance over multiple classes. This paper compares the detection performance of the sedan class (L257). Please quantitatively report the performance of the baseline and proposed model on all the five classes rather than the sedan class.

- The pointpillars is a pretty old baseline for lidar-based object detection.One can not use an old baseline to claim that 4DRT-based baseline outperforms lidar based methods.
clarity: Yes
relation to prior work: Not completely. The authors did describe the prior work, but do not quantify the statistics of existing datasets.
additional feedback: NA
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In order to overcome the challenges of autonomous driving in adverse weather conditions, the authors collected and organized a new dataset with radar data. They introduce the radar sensor which is unaffected by poor illumination and capable of penetrating snow or rain drops due to its longer wavelength compared with the commonly used LiDAR sensor. The authors also introduced the signal processing for frequency modulated continuous wave radar signal. They carefully introduced their full sensor suite and the distribution of data collected under different weather conditions and lighting scenarios. The object’s type and distance distribution within the dataset has been analyzed in the paper. Visualization and annotation tools have also been introduced in the paper. The authors compare their dataset with other datasets designed for autonomous driving scenarios. In the end, they also provided a baseline neural network for 3D object detection trained using their dataset and compare the detection performance with the lidar detection model. The main contribution of this paper is the proposed dataset which enables 3D object detection using radar signals for autonomous driving research.
strengths: The proposed dataset contains raw radar tensor data which preserves more details that can be studied by future works to advance the detection performance in adverse weather conditions. They are the first work to provide raw radar tensor data with elevation information which is critical for 3D object detection. The proposed dataset, visualization tools, and annotation tools can bring impact to the research community.
weaknesses: The calibration process for the sensor suite is not very clear. The paper has mentioned the calibration between radar and lidar but did not explain other extrinsic calibrations for cameras. The camera extrinsic calibration is important as they show the dash-camera image as a supporting image for annotation in the appendix. The authors mentioned the calibrations between radar and lidar by adjusting the 2D translation and yaw angle, but the error of pitch angle introduced by the two sensors was not calibrated. The authors should consider elaborating more on the calibration process.
correctness: The paper claims to be a dataset paper and was constructed in a sound way. It has the introduction, related work, dataset collection and analysis. They also provided baseline with experiments to verify their dataset.
clarity: Overall, the paper was well written and organized. But I found most of the details about sensor calibration was in the appendix. The authors can consider move the sensor calibration details into the main page.
relation to prior work: The paper compared their dataset with other autonomous driving datasets and clearly state their advantages against others. The authors should consider including another radar object detection dataset RODNet [1] in their related works.

[1] Wang, Yizhou, et al. "RODNet: A real-time radar object detection network cross-supervised by camera-radar fused object 3D localization." IEEE Journal of Selected Topics in Signal Processing 15.4 (2021): 954-967. 
additional feedback: I have a question about the performance of RTNH versus PointPillars under different weather conditions. Why the general AP of the normal condition is lower than the overcast condition? And why the RTNH bird's-eye view AP of the light-snow conditions is lower than heavy-snow conditions while 3D APs are pretty much the same? Perhaps authors can explain more about their experiment.
review_title: A new radar dataset capable of training 3D detection model for autonomous driving

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: - This paper presents a 4D Radar tensor (4DRT)-based dataset (K-Radar) for 3D object detection. It also provides diverse scenarios with challenging illumination, time, and weather conditions.
- A 3d detection baseline method is introduced and proves that 4DRT is essential for autonomous driving.

strengths: - The paper is well-written. Figures and tables present the dataset clearly.
- Comparisons with other datasets are solid. 
- The annotation/visualization tool could benefit future research or customized labels.
weaknesses: - One concern regarding privacy is if the vehicles and pedestrians in the dataset are processed with appropriate anonymization.
- It would be better if more methods are compared in the benchmarking. And discussions in failure patterns.

correctness: yes.
clarity: yes.
relation to prior work: yes, it is clearly discussed.
additional feedback: please see the weakness section.
review_title: Good dataset but weak in benchmarking

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents the first large-scale dataset containing four-dimensional radar tensor (4DRT), Lidar,  stereo camera, IMU, and RTK-GPS data for the perception module in autonomous driving. 
The authors collected and annotated the actual driving data under various weather conditions and road types, especially for adverse weather conditions such as heavy snow.
Additionally, they provide software for the dataset letting users calibrate Lidar and radar sensors, annotate the 3d bounding box and visualize annotation interactively.
Finally, they conducted two experiments to validate the importance of 4DRT in the perception module of autonomous driving.
strengths: 1. The authors release the first public large-scale dataset for autonomous driving incorporating 4DRT data.
2. The data distribution is well organized over different collection times, weather conditions, and road types.
3. Experimental results show that height information is essential for the baseline 4DRT neural network (NN) and that performance of the baseline 4DRT NN under adverse weather conditions surpasses that of Lidar.
weaknesses: 1. The authors do not mention the calibration process of other sensors to Lidar except radar. The yellow bounding boxes shown in https://youtu.be/b_9TtOxaN1w at 1:47 seem not aligned with cars on the image due to the camera and lidar calibration issue.

2. In section D.1 of the Appendix, annotators can verify the height and size information of the 3D bounding box with BFS-2D. However, the strong reflection regions of the side view on the heatmap correspond to vehicle wheels, as shown in Fig.6 (b). How do annotators verify the size information during annotation without the Lidar point cloud?

3. The authors used IoU over 0.3 as a true positive in their evaluation metric. Similar to other public datasets, results using stricter IoU criteria such as 0.5 or 0.7 should be reported.

correctness: Yes, the dataset is constructed in a sound way.
clarity: The paper is well written and provides sufficient supplementary material in the appendix.
relation to prior work: Yes.
additional feedback: 1. Could the authors also include the visualization of the 3D bounding box projected on the image planes during annotation? It would be helpful to verify the object class under most weather conditions.

2. In Fig.11, I wonder if the dash cam is calibrated with Lidar so that the bounding boxes shown on the dash cam video and the radar heat map are programmatically associated.
review_title: K-Radar: 4D Radar Object Detection Dataset and Benchmark for Autonomous Driving in Various Weather Conditions

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a 4D Radar object detection dataset and benchmark for autonomous driving in various weather conditions, including normal, sleet, rain, heavy snow, fog, light snow, and overcast. And the dataset is the first dataset with  4DRT, including Doppler, range, azimuth, and elevation. It provides the baseline method for 3D object detection.
strengths: 1. A 4D Radar dataset and benchmark for autonomous driving in various weather conditions.
2. This is the first dataset with 4DRT for Radar object detection.
weaknesses: 1. The dataset has tracking IDs. However, the paper does not provide any experiments about tracking performance.
2. It lacks reference of the proposed baseline method. Even if the baseline method is proposed by authors, it is still built on existing works. Besides that, it should be better to adapt some existing methods to 4DRT input data for comparison.
correctness: Most are correct. As mentioned in the weakness, the benchmark can be also extended to 3D tracking benchmark. The paper can also provide the evaluation for 3D tracking.
clarity: Yes.
relation to prior work: Yes. It compares with previous datasets and benchmarks.
additional feedback: No
review_title: Review of K-Radar

Meta-review: metareview: 5 out of 6 reviewers are positive about accepting this work. The meta reviewer agrees with the reviewers that the K-Radar dataset is a clear contribution to the community and thus recommends acceptance. The meta reviewer believes that the authors have answered the negative reviewer's question in a fair enough way. The authors are suggested the authors polish their paper writing and exposition following reviewers' comments.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept