Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors of this paper propose a platform, SafeBench, for the safety evaluation of AV's. They claim this platform is the first of its kind, testing 8 different safety critical scenarios. Additionally, different AV algorithms are benchmarked on this simulation platform based on over 2000 safety scenarios. Results are generally promising, tested on simple RL algorithms, demonstrating tradeoffs between different AV metrics such as speed and running red lights. 
strengths: The first platform of its kind as far as I know which is a huge contribution, the paper is very clear and well written, this paper has use-cases ranging from other researchers to actual AV companies, this platform could improve safety of AV's.
weaknesses: Central focus on just safety evaluation of AD makes the platform weaker (also labeling just the performance measures of safebench as true safety is misleading), traffic scenarios are not the only causes of unsafe driving, limited amount of AD algorithms tested, coming up with a single safety evaluation metric can be detrimental to learning/evaluation.
correctness: Yes, NA, yes.
clarity: Yes, the paper is very clear.
relation to prior work: Yes, the related works section is small, but as this is a novel platform, there are limited prior works.
additional feedback: I am curious as to how a person ports a model into the evaluation platform,
review_title: Great Idea to increase AV Safety

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors propose a simulator environment that can be used to evaluate algorithms for autonomous driving according to 10 metrics. 
This benchmark addresses the problem of testing AD algorithm in a standardized manner and seems to be very promising and useful for future research. 
 
strengths: The benchmark software seems very useful for the autonomous vehicle research community and I can see the benchmark gaining a lot of traction. The software seems to be well written and the source code looks reasonable clean and well documented. It is also licensed under the MIT license and available via github.   
The comparison of different algorithms is interesting and the drop in performance in the adverse scenarios invites future investigations.  

weaknesses: My main problem with the paper is that is to me not clear what exactly it wants to be. The simulator is introduced, but I miss a detailed discussion of design decisions and an evaluation of the simulator itself.  For example, I miss a rational why the chosen scenarios are useful or better than some other scenarios. Why does it make sense to only keep scenarios that cause collision with at least two algorithms? Similarly, I miss an evaluation of the metric. Why is the metric good? How does it compare to other metrics? 
Another point, I feel that should be elaborated is if simulator results are actually useful in real world system. So is an algorithm tested on this simulation better in a real world scenario? However, I am aware that such an evaluation is probably very complex and time consuming. 

This is compounded by the excessive use of the appendix. It is ok to push some parts or background into the appendix, but at times the paper reads like a table of contents and most paragraphs seem to reference the appendix somehow. 

It feels like the authors tried to do cram two papers into one; I am not sure if two papers are a better solution, but perhaps a longer journal paper would be an option? Another option might be to move the tables into the appendix and use the additional space to evaluate the simulator (and not the results of algorithms using the simulator!) and provide some more details on the design rational. 
correctness: The simulator seems to be working and the source code is reasonably clean. The documentation seems complete and helpful. 

The test results of the AD algorithms seem correct, however I do wish the authors would have provided a better rational for their metric. 
clarity: The paper is clear and easy to read. 
relation to prior work: Prior work is referenced. I just miss references to docker and ROS on line 128. 
additional feedback: line 127: a space after 'Figure 1'
line 128: references to docker and ros; also the acronym is never introduced
review_title: A useful simulator benchmark, but an unfocused paper

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes SafeBench, the first platform as far as I know for the safety evaluation of auto-driving systems. In this platform, the author introduced several scenarios, auto-driving algorithms, and evaluation metrics. Based on the platform, the author conducted extensive experiments and offered some preliminary insights.
strengths: 1. This paper proposes SafeBench, the first platform as far as I know for the safety evaluation of auto-driving systems.
2. The benchmark is modularized design and easy to use, which would be highly beneficial for researchers in this area.
3. This platform contains several safety-critical scenarios for auto driving and multiple evaluation metrics.
4. The author studied and evaluated several auto-driving algorithms.
weaknesses: 1. Could the users flexibly add new vehicle models to the platform? If yes, how?
2. Could the users add additional testing scenarios?
3. As for the benchmarking results, I suggest the author offer more analysis on these results and provide more insights into the different auto-driving algorithms.
4. Minor: I suggest the author include and add more adversarial attack methods in their platform, for example [1,2].

References

[1] Dual Attention Suppression Attack: Generate Adversarial Camouflage in Physical World. CVPR 2021.
[2] CAMOU: Learning physical vehicle camouflages to adversarially attack detectors in the wild. ICLR 2019.
correctness: Appropriate.
clarity: Yes.
relation to prior work: Yes.
additional feedback: Please see the questions in the [Weakness].
review_title: A good benchmark for safety evaluation of autodriving systems

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents SafeBench, a unified benchmarking platform for evaluating safety violations of autonomous vehicles. The proposed framework considers 8 safety-critical testing scenarios following National Highway Traffic Safety Administration and develops four different scenario generation algorithms. The benchmark includes four different deep reinforcement learning algorithms with four types of input. The benchmark results suggest that the generated scenarios of SafeBench are more challenging for current autonomous driving practices.
strengths: ++ This paper presents a novel benchmark for autonomous driving.

++ The benchmark results and findings are interesting.

++ The paper is well-written and easy fo follow.
weaknesses: -- Some related work is missing.

-- Some experimental settings are not well-justified.

(See details below)
correctness: Overall I think the paper makes a good contribution to both the research community and industry. The benchmark seems useful, and the results are somehow interesting. I think the evaluation methods and experiments are also designed appropriately, but I also think some of the experimental settings need to be justified more carefully.

For instance, the choice of [Etiquette level] metrics. I think the evaluation from such a perspective is interesting, while I also believe it would be better if authors could justify the choice of three different metrics (any references?)

The weight selection for the overall score. While I didn't find the detailed selection criteria for these weights either in the paper or the supplementary materials. It would be nice if authors could add some details about it.
clarity: The paper is well-written.

But the first paragraph of the introduction seems confusing to me. Authors introduce a lot of adversarial sample generation for images/point clouds. While these are not strongly correlated to the paper itself, I would suggest authors shorten this part.
relation to prior work: I think this would be another concern. There have been a lot of attempts ([1]) regarding testing autonomous vehicles (finding safety violations) in the software engineering community. I believe they are also related to the proposed benchmark and should be discussed.

[1] Li, Guanpeng, et al. "AV-FUZZER: Finding safety violations in autonomous driving systems." 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2020.

[2] Gambi, Alessio, Tri Huynh, and Gordon Fraser. "Generating effective test cases for self-driving cars from police reports." Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 2019.

[3] Abdessalem, Raja Ben, et al. "Testing autonomous cars for feature interaction failures using many-objective search." 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2018.
additional feedback: Overall, I believe this is a good paper considering its novelty and contribution. I do have some questions to ask.

1. Can you add more details about your experimental settings, such as weight selection? 

2. Despite the current four RL algorithms, have you ever considered some commercial practices of autonomous driving, such as Baidu Apollo and OpenPilot?
review_title: Important topic and interesting benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper suggests a unified framework to integrate different types of safety critical testing scenarios and scenario generation algorithms with the aim to ease the comparison and the understanding of their effectiveness. Four different RL algorithms are also compared. 
strengths: - Unified test scenarios and scenario generation algorithms.
- The modularity of the platform including ego vehicle, agent, scenario and evaluation nodes, where each module could be customized based on the requirements. 

weaknesses: - Lack of computation complexity analysis for scenario generation algorithms and RL based methods. 
- Not enough documentation to be able to reproduce the provided results.
- Lack of generalization analysis on real-world data.

correctness: No substantial factual errors were identified.
clarity: The paper is mostly clear and well written, though a few items could be improved (identified below).
relation to prior work: The related work section provides a categorization of state-of-the-art work on the basis of how the testing driving scenarios are generated. However, it lacks a more comprehensive comparison with other similar works to precise which algorithms or functionalities are integrated in the proposed uniform platform (a summary table could be helpful). 
additional feedback: - The performances for RL algorithms are only reported for the selected scenarios with high collision rates. Are we assume that they work very well for the scenarios with low collision rate? 
- Any preprocessing step is done before learning the RL based method? This should be discussed in greater details.
- It seems that all the tests are performed using synthetic data. How to ensure that the learned algorithms on synthetic data could generalize as well on real data? 
- In Table 1, various metrics are used to evaluate the performance of scenario generation algorithms. However, less details are provided on how these metrics are computed? Are the metrics could be used to evaluate both scenario generators and RL algorithms? If it is not the case, what overall score (OS) include for each of these steps?
- In the caption of Table 1, it is noticed that the SR (scenario rate) is reported to evaluate the transferability of generation algorithms across AD agents. More explanation should be provided.
- In section 4.3, the results are interpreted by indicating WHICH RL method works better than the others. However, it will be helpful to explain WHY one algorithm shows a better performance with regard to a specific scenario (is the reward function is defined in a way that it encourages more safety than functionality for example?)
- The authors should provide details concerning the algorithmic complexity and computation time of scenario generation algorithms as well as the RL based methods. There could be a trade-off between the complexity and precision. As it has been mentioned among the limitations of this work.
- The Nvidia-docker is used on top of the Docker environment. It implies having access to Nvidia GPU cards. Some clarification should be provided at this level. Which graphic cards are compatible and which algorithms use Cuda capability (only Deep RL methods?). The specification of the docker container, library versions, etc (this information could be added in Github readme or other online documentations). 
- The authors should add a section “Datasheet for Dataset” as it has been mentioned in submission instructions for new datasets. It could clarify the nature of the used data and facilitate its comprehension.  
- No indication on how to propose new RL algorithms (with a different architecture) and specific reward functions.

> I have raised the score to 7 after more in depth investigation concerning my comments
review_title: Safebench: A Benchmarking platform for safety evaluation of autonomous vehicles

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper presents a unified framework for the assessment of the safety levels of autonomous driving algorithms against 8 different safety-critical scenarios. The scenarios are generated on the CARLA simulator with 4 different generation methods and are evaluated with 10 metrics. The code and the leaderboard is provided at the dedicated website.
strengths: The paper is well written and the contributions are clearly presented. 
The work is novel and could represent a step forward in the assessment of the safety of autonomous driving.
The benchmark is overall well designed and fairly comprehensive, and the main project page includes additional videos and the learderboard.

weaknesses: The paper presents a few weaknesses that could be easily solved:
- there is no description of how the traffic scenarios are generated during training of the agent. Additional details could be reported in the supplementary material;
- there is no description of how the BEV images are obtained. In the real world, these are typically obtained with neural networks or other perception pipelines processing images/pointclouds. If this pipeline is considered as "perfect", and the BEV images are collected directly in CARLA, then it is better to highlight this point;
- I feel like the Overall Score could be misleading, depending on the weights that are applied. Maybe it could be an idea to also consider 3 different overall scores to sum up the safety, functionality, and etiquette levels. 
- The choice of the 8 scenarios selected from the NHTSA documents should be justified, since the scenarios listed are way more. Were these scenarios selected by statistical frequency on the roads or by other metrics?
- It is not clear how to submit additional results to the leaderboard.
correctness: Yes
clarity: The paper is well written. There are only a few sentences that require improvements:
- lines 53-54: "previous work can be realized"
- line 246: "trains"
- line 271: "9 \sim 10" (is it 9 or 10?)
relation to prior work: Yes
additional feedback: In my opinion, this is a paper that is worth acceptance for technical contributions, clarity, and presentation.
There are a few minor issues that can be discussed to improve the benchmark, such as summary metrics for each level, or additional clarifications.
Also, additional safety-critical scenarios should be added progressively to obtain an exhaustive assessment.
review_title: Minor comments for a well-written paper

Meta-review: metareview: The paper addresses a much-needed benchmark for the safety of autonomous driving. All reviewers recognize the significance of the paper's contributions. Most of the questions have been addressed satisfactorily by the rebuttal. Overall, all reviewers are positive about the paper and recommend acceptance.
confidence: 5: The area chair is absolutely certain
Decision: Accept