Towards Scalable Coverage-Based Testing of Autonomous Vehicles
Authors: James Tu
Simon Suo
Chris Zhang
Kelvin Wong
Raquel Urtasun
Venue: CoRL 2023 Poster (2023)
Keywords: Self-Driving
Coverage
Testing

Abstract: To deploy autonomous vehicles(AVs) in the real world, developers must understand the conditions in which the system can operate safely. To do this in a scalable manner, AVs are often tested in simulation on parameterized scenarios. In this context, it’s important to build a testing framework that partitions the scenario parameter space into safe, unsafe, and unknown regions. Existing approaches rely on discretizing continuous parameter spaces into bins, which scales poorly to high-dimensional spaces and cannot describe regions with arbitrary shape. In this work, we introduce a problem formulation which avoids discretization — by modeling the probability of meeting safety requirements everywhere, the parameter space can be paritioned using a probability threshold. Based on our formulation, we propose GUARD as a testing framework which leverages Gaussian Processes to model probability and levelset algorithms to efficiently generate tests. Moreover, we introduce a set of novel evaluation metrics for coverage-based testing frameworks to capture the key objectives of testing. In our evaluation suite of diverse high-dimensional scenarios, GUARD significantly outperforms existing approaches. By proposing an efficient, accurate, and scalable testing framework, our work is a step towards safely deploying autonomous vehicles at scale.

Towards Scalable Coverage-Based Testing of Autonomous Vehicles
James Tu
Simon Suo 1.2
Chris Zhang 1.2 Kelvin Wong Raquel Urtasun! 1.2
Waabi UUnieersity of Toronto
iju czhang, kwong, urtasun @waabi ai tuo@cs.toronto..
Abstract: To deploy autonomous vehicles(AVs) in the real world, developers must understand the conditions in which the system can operate safely. To do this in scalable manner, AVs are often tested in simulation scenarios. In this context, it's important to build a testing framework that par- titions the scenario parameter space into safe, unsafe, and unknown regions [1]. Existing approaches rely on discretizing continuous parameter spaces into bins, which scales poorly to high-dimensional spaces and cannot describe regions with arbitrary shape. In this work, we introduce a problem formulation which avoids discretization - by modeling the probability of meeting safery requirements ev- erywhere, the parameter space can be paritioned using a probability threshold. Based formulation, we propose GUARD) leverages Gaussian Processes to model probability and levelset algorithms to ef- ficiently generate tests. Moreover, we introduce set of novel evaluation metries for coverage-based testing frameworks to capture the key objectives of testing, In our evaluation suite of diverse nigh-dimensional scenarios, GUARD significantly outperforms existing approaches. By proposing an efficient, accurate, and scal- able testing framework, our work is a step towards safely deploying autonomous vehicles at scale.
parameterized
testing framework which
Keywords: Testing, Coverage, Self-Driving
Introduction
Autonomous vehicles (AVs) will soon become a staple in ground transportation -nnioraatiionnnttttttttnn with billions of people everyday. At this scale, AVs can drastically reduce accidents, relieve traffic gestion, and provide mobility for those who cannot drive. order to realize this future, developers must first ask the question: "Is the AV safe enough to be deployed in the real world?" To understand how safe the AV is, it's important to identify in which scenarios the AV is safe or unsafe with respect requirements defined by safety experts [1, 2]. Towards this goal, it's important to build testing framework which covers the wide range of scenarios in the AV's operational domain and classify them as safe, unsafe, or unknown.
To cover the wide range of real-world scenarios in scalable manner, the AV industry often builds testing frameworks in simulation [3, 4, 5] where the traffic environment is fully controllable and long-tail events can be synthesized. A popular strategy deseribe real world events in simulation involves designing parameterized scenarios [6]. These scenarios describe semantics of the environ- ment (e.g. truck merging from on-ramp) and its parameters specify low-level characteristics (e.g. velocity of traffic participants). Each parameter configuration then corresponds to a concrete test which can be executed simulation to determine if the AV complies with functional safety require- ments [1]. This is typically captured as a binary pass or fail determined by regulatory demand [7]. For example, an AV could fail if it violates a safety distance threshold.
However, directly covering all variations in a scenario's 's parameter space can be infeasible. Contin- uous parameters imply infinitely many variations and testing frameworks can only execute finite number concrete tests which is limited by computation budget. As a result, to cover the parameter space the testing framework must leverage observed test results to estimate if the AV will pass or fail on unseen test configurations. Furthermore, the testing framework must also efficiently choose concrete tests to execute to maximize coverage and accurately estimate AV performance.
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

Parameterized Socnario Testing
Probubilistic Model of Test Outcome
Testing Process
Sckec kot
Adte
Brodent
W
Pass
Unestain
Fail
Peforsmanc Britaat
Uplac
Adoe edecky
Uncortve
Usortinnty
Pacc
Fasl
Probebilistic Model
Prodict Outcomn of Unsezn Tesls
Figure 1:
Left: Parameterized scenario testing. Middle: Probabilistic model estimates pass/fail outcomes in parameter space. Right: Testing process-Sampp sample and execute tests, then update model.
Existing approaches rely heavily on discretization to reduce the infinite continuous parameter space into discrete bins, making the assumption that tests the same bin yield the same result. While effective for simple 2-dimensional scenarios [8, 9]. they scale poorly to high dimensional scenarios due to exponentially growing number bins. Alternatively, another common approach involves generating adversarial examples [10, 11, 12] to discover failures. While revealing failure cases is useful, these methods forgo covering the parameter space and cannot provide an understanding of AV performance across the parameter space.
In this work, formulate the testing process sequencing finite number of tests to estimate the probability of passing or failing across the entire parameter space, thereby avoiding discretization altogether. To start, we can leverage observed test results model the probability of passing or failing across the entire parameter space. Intuitively, if the AV easily passed particular test, it can pass similar tests with high probability. Moreover, for unseen test with little information from similar tests, the pass/fail outcome should be ambiguous. By modeling the probability everywhere, follows that probability threshold can partition the parameter space into pass regions, 2) failure regions, and 3) uncertain regions, which is the output prescribed by standard safety guidelines [1, 2].
Based on our formulation, we propose our testing framework GUARD. Specifically, we leverage Gaussian Processes (GPs) to model the probability of passing and failing. To achieve high sample efficiency, we adopt levelset algorithms which adaptively generates tests that balance exploration (of high uncertainty regions) and exploitation (by sampling near the pass/fail boundary). More- over, GUARD leverages online GP kernel learning, allowing it to scale to different parameterized scenarios with minimal hyperparameter tuning.
To evaluate testing frameworks, we propose a suite of novel evaluation metrics which capture the key objectives of testing: achieving high coverage, evaluating safety of the AV, and exposing failure cases for autonomy development. Under these metrics, GUARD able to outperform existing test- ing frameworks across diverse of complex and high-dimensional driving scenarios. Moreover, we find that by avoiding discretization, our approach scales much more efficiently as the dimension- ality of test scenarios grow. Finally, GUARD can also benchmark different iterations of autonomy models while identifying specific examples of scenarios where regression occurs.
Related Work
2.1 Coverage-Based Testing
AVs often tested in simulation using parameterized scenarios [6, 13, 14] where it receives pass or fail based on safety requirements [7]. Each parameter configuration corresponds to a concrete test and testing frameworks execute concrete tests cover the parameter space. Since continuous scenario parameters imply an infinite amount of scenario variations, existing works discretize the pa- rameter space into bins assuming all pass/fail outcomes in bin are identical. This discretization introduces error in understanding where the AV passes and fails, which grows with the bin size.
Exhaustively testing all parameter bins [14, 15] is widely adopted approach in industry. How- ever, this approach suffers from coarse discretization when testing high-dimensional scenarios. To increase sample efficiency, (-way testing [16] samples points such that for every subset of pa- rameters, every combination covered. This affords finer discretization but leads to inaccuracies due to assuming that tests which share subset of t parameters yield the same result. Similarly,

---

Algorithm 1: Algorithm for GUARD
Input: Threshold 7, threshold a, budget N, initial budget M, kernel params Ko, update freq K
G GP((k))
// Initialize GP // Initial exploration batch
for t in I.. M do
0, - a (0)
G + Update@PSampless
0r, ((.))
RM f argmax, (() .f f())) 01....0;KK)
// Fit kernel on initial batch
G + Update@PKernel(C..r.
for t in M+ ..... do
// Levelset sampling 1/ Explore or close to boundary
0, - argmaxx (0) Iu(0)
G < UpdateCPSamples G, 0r (0.))
if mod K then
// Fit kernel every steps
K₁ + argmax, x (() ((0)) 0, .1..0k:k))
+ Update@PKernel
P+ - (0 E θ 1 2 }}
// Pass region
F - (
(( o (((1(())
2 a)
// Fail region // Unknown region
u θ P
return P,F,U
levelset estimation [17, 18, 19] has also been applied to increase sample efficiency in discretized coverage-base testing [8]. On the other hand, [9] leverages adaptive discretization resolutions.
2.2 Adversarial Example Generation
Adversarial example generation is a widely studied approach to generate difficult exaples [20, 21, 22, 23, 24, 25], which have been applied to image classification [26], natural language processing [27], and self-driving [10]. In parameterized scenario testing, this is done by optimizing parameters via black-box optimization algorithms such Bayesian Optimization [10, 28, 29, 30, 31] Reinforce- ment Learning [32, 33] to generate challenging safety-critical scenarios. These methods optimize for the most adversarial and do cover the parameter space. On the other hand, [34] showed that BO with the GP-LCB acquisition function can validate that an AV passes all scenario variations the LCB is positive everywhere and failure isn't discovered. This achieves coverage but only in the case where the AV is perfect. Acquisition functions for adversarial example generation can also be used sample tests our framework. However, this oversamples severe failures, whereas oversampling on the pass/fail boundary is more efficient for coverage-based testing.
3 Scalable Coverage-Bare Based Testing
In this section, we introduce our formulation which leverages probabilistic models to partition the parameter space into pass, fail, and unknown regions. Following our formulation, we introduce our testing framework GUARD that leverages Gaussian Processes (GPs), levelset estimation, and learnalle GP kemells. An overview can be found in Figure I and the algorithm in Algorithm 1.
3.1 Problem Formulation
The AV software stack is often tested on parameterized scenarios commonly known as logical sce- narios [13, 6], which are the standard in the self-driving industry. Each logical scenario features a set of d configurable parameters 0 E Rdd where each specific configuration of the parameters results a concrete test, The scenario parameters are bounded [13, 15), i.e., Og E [a,, by]. making the entire parameter search space closed set O in Rdd We assume that simulation test outputs a scalar mea- sure of safety (e.g. minimum distance to another agent). Mathematically. let f O x A - R. be the test function which takes as input test parameters 0 autonomy system and outputs real-valued scalar f (0: A). We omit A to use (*) for brevity. In compliance with regulatory demand [7], binary pass or fail y = 1f (0) 2 7 is computed using threshold y. It follows that we can denote regions in the parameter space O where the system passes and fails
p' (0 E (0) 27); 7}.
F = (0 (0) <7).
(2)
3

---

Way
3-May
Conconal Inforace
Onid
GUARD
Hiddennaama
0185
Balanced Accuracy
0.80
0.6
False Positive Rate
Entrr Recall
0.5
0:15
0.4
0.20
0.3
0.65
02
0.00
0.1
0.06
6.7
3.9
0.9
0.9
Coverage
Covenge
Coverage
Figure 2: GUARD significantly outperforms other testing frameworks. GUARD also sweepable confidence threshold to a trade-off between coverage and the other metrics.
Then, the testing process can be thought of as sequencing a finite set of tests (O)...., Or observ- ing their outcomes ( } and estimating P R P F Rs F Estimating pass/fail across a continuous parameter domain using finite set of test points inevitably leads to estimation errors, especially if the number of concrete tests are limited (e.g. due to testing resource constraints). At the same time, estimation errors and false positive passes in particular can be detrimental to safety.
Thus, we design an uncertainty-aware formulation where the framework quantifies the confidence of its estimation. This allows the framework to say it is uncertain instead of predicting a pass/fail outcome with insufficient information. Specifically, we compute the probability that the system will pass fail at any point in the parameter space. To achieve this, model (0) with a random variable f(0). Under this random variable, the estimated pass and fail regions can be defined as
P= (0 O P(f(O) 2 7) 2 a). F= (0 0, < 7) a).
(3)
with respect to confidence threshold It follows that the unknown region be obtained as = P F, where more information is required to make a prediction. Note that the size of P and FF increase monotonically as decreases. The configurable confidence threshold allows the testing framework to control the tradeoff between the quality and quantity of predictions.
3.2 Gaussian Process
As outlined in Equation our formulation requires a probabilistic estimate f()) for the ground truth test function (0). We adopt GPs as they perform well in low-data regimes, making them suitable in our setting the number of concrete tests are limited. Additionally, GPs non-parametric and make less assumptions about the function being modeled, which is important in allowing our testing framework scale to a wide variety of logical scenarios.
A GP is collection of random variables in which every subset is assumed distributed from multivariate Gaussian. Let X = [0) (0....e. 0x] E Raad ~ NN,,, Σ) be random variables from N test samples and Y (0 ... f [0)) be the corresponding scalar outputs of f*. To estimate test outcomes, we model the distribution over the real-valued output of the metric P(f(0).. Specifically, the value of an unseen datapoint is estimated by the conditional posterior distribution
((f Y) ~ ((()
(4)
(1))
(K
(5)
o2 (0) k(0,,) - k(0, (K ((,))
Here kc(:.)) is the kernel function which provides a similarity measure hetween two GP variables, x(0,X) X) [k(00 ",), Nx)], K2j k,0,,0,), and models noise in the observed values Y. Under the posterior distribution of f()), the probability of observing a value above or below a certain threshold can be computed. Specifically, the probability of observing value greater than the pass-fail threshold 7 is
P() 2 7) = ( (
(1())
where + is the CDF of the normal distribution. The probability of observing a value under ry can be computed in similar fashion. An illustration of this can be seen in Figure 1.

---

Method
(overage(%)
Bal. Acc(S)
Pos Acc(%)
Neg Acc(%)
Ert. Recall(%)
FPR(%)
Grid
87.7 + 0.4
62.3 0.1
97.4 0.03
27.2 + 0.2
25.0 0.3
18.9
0.2
2-way [16]
92.2 = 0.2
60.3 = 0.6
9..0
24.6 = 1.2
24.3 + 1.2
18.2
+ 0.4
3-way [16]
92.4 1 1.0
59.7 1.7
96.2 0.3
23.2 1.7 3.7
22.5 1: 3.6
20.9 1 1.3
HiddenGems [8]
72.8 1 0.6
66.2 2.2
98.3
33.7 4.4
14.8 2.4
16.0
1 0.4
Conformal Inference [9]
89.6 +
61.8 + 0.8
97.0
26.7 +
16.6 +
23.8
+ 0.9
GUARD
94.3
84.2 = 5.3
99.2 0.2
70.3 10.6
5.3
5,89 + 3.1
Table I: Benchmarking GUARD against baselines
3.3 Test Generation
Note that precisely modelling f* when the output is far from adds little value since it won't I affect the pass or fail outcome. In contrast, it's important to accurately model regions near the y-levelset. Hence, we leverage levelset algorithms to efficiently upsample points near the boundary. Specif- ically, we adopt Straddle [35] as it directly integrates GPs, and alternative GP-Ievelset algorithms operate on finite input domains [17, 8, 18]. Straddle iteratively queries test points based on an explo- ration incentive promoting points with high variance and an exploitation incentive promoting points close to the -llleelset. Specifically, these two incentives captured by the acquisition function
14 (0) = (0) (0) -
where B is a weighting coefficient balancing the first exploration term and the second exploita- tion term. At each iteration, we solve a lightweight maximization problem to find the next query point O, = argmaxg (O). We start by sampling /' initial random points from @ Then since (0) is differentiable with respect to 0, the top Q candidates are improved via gradient-based opti- mization. This optimization is inexpensive since the cost of running the GP model and performing backpropagation to evaluate p(0). (0) is negligible compared to running simulation evaluate (0). Finally, the concrete test corresponding to 0, is executed in simulation to obtain f (0,). The observation (0.)) is then added to the GP model to update the GP posterior.
3.4 Automatic Kernel Tuning
In addition to observed test points, the posterior distribution outlined in Equation and 6 also depend heavily on the kernel function A(;,)) Thus, the estimates P and F are sensitive to the kernel function parameters which we denote as K. At the same time, tuning kernel parameters can be tedious and time consuming [8]. Moreover, in our task, each individual scenario has different parameters with varying scales and effects on the final output, requiring different kernels to accurately model different scenarios. This makes manually tuning kernels infeasible for large scale testing.
To circumvent this issue, we first normalize the scenario parameter space C to the unit hypercube [0, 1]d to address parameters with different scales. Furthermore, instead of manually tuned and fixed kernel parameters, optimize the kernel parameters to maximize the marginal likelihood of the observations. In particular, at iteration t, we update the kernel parameters x towards
argmax (() f(0.) ........)))
The marginal likelihood is differentiable with respect to the kernel parameters K2 and we perform gradient- based optimziation every K iterations. The kernel learning process can be prone to overfit- ting a small initial batch of observations. Therefore, we perform initial sampling step where we query M tests using only the variance term o, (0) of the acquisition function outlined in Equation 8.
Experiments
We first introduce the test scenarios, evaluation metrics, and relevant baselines. Then, we demon- strate the effectivness of GUARD and ablate our design choices. Finally, we show how GUARD can be used in practice to benchmark autonomy systems and catch regressions. Additional experi- ments are included in Appendix A.
4.1 Experimental Setup
Simulation Test Scenarios: We use a suite of 10 logical scenarios designed by expert testing engi- neers and based on guidelines [36] prescribed by the National Highway Traffic Safety Association (NHTSA). The scenarios are executed high-fidelity closed-loop simulator and use combina- tion of scripted and reactive actors. The scripted actors execute parameterized maneuvers to stress
5

---

Pass
GLARD
Grid Soarch
Hidduatems
Confamual Inderence
Uncestain
Fail
Pass
Uncertain
Figure 2D slice of 5D parameter space. GUARD performs better by avoiding discretization.
test the AV e.g. cut-ins. The reactive actors realistic background traffic and controlled using the Intelligent Driver Model [37]. Each scenario has d = 5 parameters and descriptions of the scenarios are provided in Appendix B.
We evaluate the PLT motion planner [38] using minimum distance(m) to collision as the safety metric and choosing the safety threshold as = 1.0. We use N = 1000 test samples for experiments unless otherwise specified. In the following experiments, we repeat trials for each scenario to collect mean and standard deviation of metrics. We report the average mean and standard deviation across 10 scenarios. To support large scale experimentation, we construct an offline dataset by evaluating each scenario space over fine grid to create an approximation via linear interpolation.
Metrics: To evaluate performance, propose several metrics which capture important aspects of testing for safety and autonomy development.
Coverage is the percentage of the search space 2 that is covered by the estimates P and F:
|P F
coverage
(10)
|
Balanced Accuracy evaluates how accurate the estimates P and F are and also addresses class imbalance due to the fact that the scenario space is dominated by passes:
PPPPI
|FFFI
balanced acc
(1I)
P P'In(UUU
F (I((PUF)
positive
negative
Error Recall measures the percentage of the failures the testing framework can recall, which is very useful for autonomy development. This computed as:
error recall |FFF
(12)
F
False Positive Rate is the percentage of predicted passes that are incorrect, which is crucial to safety as incorrectly predicting the system to be safe can lead to accidents. This is computed as:
PN
false positive rate
(13)
P
Note that some of these quantities such as | cannot be computed exactly. Therefore, we randomly sample 20, 000 points in the scenario parameter space to estimate the terms in the above equations.
Baselines: We compare GUARD with several baselines outlined in Section 2. These include tradi- tional testing methodologies [16, 14, 15] and works proposed in the literature [8, 9].
Grid Search [15, 14] discretizes continuous parameters, and uses discrete test point repre- sent each bin. Given budget N test samples which cannot cover all bins, we sample subset of the bins random. our scenarios with d dimensions, we use discretization resolution of 4,
T-way testing [16] leverages the assumption that most faults are caused by interactions of at most I parameters. This method draws test samples such that for every I-parameter subset, all possi- ble combinations of parameter values can be found in least one test. our experiments, we consider f 2, and choose discretization resolution of 30 for f 2 and 12 for t = 3.

---

Conforod Grid
2-Way
Confomal
Grid
HiblenOenn
Thlancal Actunsy
Ermr Rocall
Coverage
Bullncal
04
04
Dinendors
Dimaniius
Num Sumples
Nam Samplex
Figure 4: Scaling with scenario dimensionality.
Figure 5: Scaling with testing budget.
HiddenGems [8] employs levelset estimation algorithms to more efficiently infer pass/fail out- comes of the bins. The authors use 33x33 discretization for 2-D parameter space. However, this resolution becomes infeasibly expensive with d = 5 parameters. We adopt a resolution of and further analyze the exponentially increasing runtime in Appendix A.
Conformal Inference [9] uses conformal inference samples in each bin to determine the out- come. Bin sizes are not fixed, since this method chooses how the parameter space is partitioned.
Implementation Details: The initial exploration batch has size M 200. Our GP kernel is product composition of separate Matern32 kernels [39] for each parameter dimension. The kernel lengthscale is initialized to 0.02 and variance initialized to the variance of the initial exploration batch. The kernel is fitted every 50 samples using Adam [40] with learning rate 0.01 for 100 steps. Our acquisition function uses 3 0.5. During acquisition maximization, we sample P 600 candidates improve the top Q = 30 candidates using Adam with learning rate 0.01 for 30 steps.
4.2 Experiment Results
Testing Performance: We demonstrate the effectiveness of GUARD in Figure 2 where we plot test coverage versus balanced accuracy, error recall, and false positive rate (FPR). First, note that we can adjust the trade-off between coverage and the other metrics GUARD by adjusting the confidence threshold a outlined in Equation 3. This is useful in controlling how conservative are in testing. Along the trade-off curve, we can observe how reliable the predictions are at different probability thresholds. Compared other methods, GUARD performs better across all metrics. We provide numerical results in Table I where we set a 0.7 for direct comparison.
To illustrate how GUARD achieves better metrics, we visualize the outputs in Figure 3, using the technique proposed in [41] to visualize a 2D slice of the 5D parameter search space. Specifically, we sample one boundary point and multiple slices on the boundary point, choosing the slice with the highest variance. On each slice we visualize the ground truth and predicted pass/fail regions. Compared to other methods, GUARD is more accurate as it's not limited by discretization.
Scalability: We evaluate GUARD against baselines on lower dimensional functions to analyze parameters
scalability with the number of scenario parameters. The 10 test scenarios have We increase the discretization resolution of baselines at lower
fixed to reduce the dimension resolutions, choosing the resolutions to maintain a similar amount of bins as the 5-D scenarios. Figure 4 shows that baselines perform well for 1-D and 2-D functions where they can afford a high resolution. As dimensionality increases the baselines degrade significantly compared to GUARD.
Sample Efficiency: In Figure 5, we evaluate the models using a varying number of test samples N [200, 400, 600, 800, 1000) and show that GUARD outperforms the alternative methods with fewer test samples as well. With increasing testing budget, GUARD gradually increases in accuracy and coverage. Other approaches which rely on discretization grow in coverage. However, their accuracy is heavily limited by the discretization resolution and improves slowly or not at all.
Ablations: We ablate some design choices of GUARD in Table 2. First, without the initial explo- ration phase, there is noticeable drop across the metrics. Next, we consider removing online kernel learning, parameter normalization, and both. Removing either leads to drops in performance while removing both leads degenerate solutions which predict pass everywhere. This is expected as without normalization or learnable lengthscales, a fixed lengthscale cannot model parameters with different scales (i.e. velocity& E [20, 30mm/ss road curvature & |-0.002, 0.002)). Finally, we ablate the choice of the GP kernel. As described in Section 4.1, GUARD uses product of Matern32 kernels on each parameter because each parameter affects the function output differently. Using a single kernel which doesn't 't consider each parameter individually degrades performance as expected.
7

---

Exp
Norm
Leurn
Prod
Coverage(S6) (6)
Bal. Acc(S)
Pos. Acc(%)
Neg. Acc(%)
Err. Recall(%)
FPR (%)
94.3 + 0.9
84.2 + 5.3
99.2 = 0.2
70.3 = 10.6
58.9 = 5,3
5.9 3.1
92.2 +
82.4
4.9
99.4 = 0.2
65.4 10.0
52.1 = 8.8
7.5
V
93.2 +
83.7 2.3
0.2
68.3 4.6
56.3 - 4.4
58 1.6
8
92.3 + 0.8
84.1 = 1.
98.7 = 0.2
69.7 = 3.8
48.6 3.9
6.2 = 1.0
0.04 + 0.03
8.3
8.1
Table 2: Ablating design choices of GUARD
4.3 Using GUARD In Practice
Different AV systems may have different failure modes and find different scenarios challeng- ing. Since GUARD is agnostic to the system under test, it can evaluate any AV system to discover where it passes and fails across. This is useful during development to iden- tify regressions between two iterations reference planner and introduce variation adjusted First, we evaluate both planners on our test scenarios to measure the aggregate pass and fail rates. As shown in Table 4.3, GUARD correctly identifies the aggressive planner as less safe. Further- more, triage the regressions by visualizing the pass/fail landscape which we show in Figure 6 below.
the AV. To demonstrate this, we use PLT as
make the planner more aggressive.
Planner Pass Rate
Fail Rate
Ref
69.8%
18.5%
Age
66.9%
24.0%
Table Comparing planners.
Agpresivve
Regresioo Regions
Fail
Paos
Uncertain
Regression
Figure 6: Identifying regression regions refer-
Figure 7: Regression example. Top: Refer- ence. Bottom: aggressive planner collides.
ence planner passes, aggressive planner fails.
To aid autonomy development, we can sample any point from the regression region to yield a con- crete test. We show one example of such test Figure change into a neighboring lane while actors from an on on-ramp are merging. The aggressive planner does not lane change safely which ultimately leads to collision. Revealing all of these regressions in an automatic and scalable way is invaluable to autonomy development.
this test, the ego truck attempts to
Limitations
The scope of testing is limited since we assume the pass/fail result is thresholded on a single measure of safety, whereas can be combination of multiple measures. We also did consider discrete scenario parameters which can be categorical (e.g. vehicle type) or numerical (e.g. number of lanes). Future works can incorporate these considerations to build a more complete testing framework. In addition, GPs relies on the assumption that the landscape of the safety measure is smooth enough to be modelled by the GP kernel. Investigating alternative probabilistic models such as neural GPs [42] and probabilistic SVMs [43] is another exciting direction. Finally, the impact of our work and simulation testing general highly dependent on the fidelity of the simulator. Reducing the sim2real gap remains an open problem [44, 45, 46, 47].
Conclusion
This paper tackles coverage-based testing of AVs in parameterized scenarios. We formulate the problem as sequencing finite set of tests to estimate the probability passing or failing across the entire parameter space. Based on this formulation, efficiently samples tests to cover the parameter space and accurately evaluate the AV's performance. This framework can be used in practice with functional safety experts defining a comprehensive set safety requirements and parameterized operational design domain (ODD). GUARD can automatically generate concrete tests and validate the AV meets these requirements across the ODD. Our work contributes streamlined autonomy development, safety validation, and is ultimately step towards safely deploying autonomous vehicles.
propose testing framework GUARD that

---

Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable feedback and sugges- tions to improve the paper. We would also like thank Paul Spriesterbach and Andre Strobel for their input on how our work relates to functional safety guidelines. Finally we would also like to thank Wenyuan Zeng and Sean Segal for their contributions in brainstorming and suggestions improving the manuscript.
References
[1] Iso 21448:2022 road vehicles - safety of the intended functionality. attps://www.iis org/standard/7740 htmm, 2022.
[2] Iso 26262-1:2018 road vehicles - functional safety. https://www.iss iso. wwtp.iii..or///taaard////////// 68383. html. 2018.
[3] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[4] Kaur, S. Taghavi, Z. Tian, and W. Shi. A survey on simulators for testing self-driving cars. In 2021 Fourth International Conference Connected and Autonomous Driving (MetroCAD), pages 62-70. IEEE, 2021.
[5] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, et al. End to end learning for self-driving cars. arXiv preprint arXiv; 1604. 06316, 2016.
[6] H. Weber, J. Bock, J. Klimke, C. Roesener, J. Hiller, R. Krajewski, A. Zlocki, and L. Eckstein. A framework for definition of logical scenarios for safety assurance of automated driving. Traffic injury prevention, 20(sup1): 2019.
[7] (grva) proposal for the 01 series of amendments to regulation no. 157 (automated lane keeping systems).
[8] A. Petrov, C. Fang, K. M. Pham, Y. H. Eng, J. G. M. Fu, and S. D. Pendleton. Hiddengems: Efficient safety boundary detection with active learning. arXiv preprint wxXiv22210113956 2022.
[9] C. Fan, X. Qin, Xia, A. Zutshi, and Deshmukh. Statistical verification of autonomous systems using surrogate models and conformal inference. arXiv preprint arXiv:2004.00279, 2020.
[10] J. Wang, A. Pun, J. Tu, S. Manivasagam, A. Sadat, S. Casas, M. Ren, and R. Urtasun. Advsim: Generating safety-critical scenarios for self-driving vehicles. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9909-99118 2021.
[11] A. Nonnengart, M. Klusch, and C. Müller. Crisgen: Constraint-based generation of critical scenarios for autonomous vehicles. In International Symposium on Formal Methods, pages 233-248. Springer, 2019.
[12] M. O'Kelly, A. Sinha, H. Namkoong, R. Tedrake, and J. C. Duchi. Scalable end-to-end au- tonomous vehicle testing via rare-event simulation. Advances in neural information processing systems, 31, 2018.
[13] Menzel, G. Bagschik, and Maurer. Scenarios for development, test and validation of automated vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1821-1827. IEEE, 2018
[14] MS Windows NT kernel www.oooooooo-----n----eee-eee wtwp:rriiii..aaaaaaaaaaaaa-a-aae -tt- sarragg-arveen--vrificaaee Accessed: 2022-07-31.

---

[15] T. Zhao, E. Yurtsever, J. Paulson, and G. Rizzoni. Formal certification methods for automated vehicle safety assessment. IEEE Transactions on Intelligent Vehicles, 2022.
[16] D. R. Kuhn, R. N. Kacker, Y. Lei, et al. Practical combinatorial testing. NIST special Publica- lion, 800(142):142, 2010.
[17] A. Gotovos. Active learning for level set estimation. Master's thesis, Eidgenössische Technis- che Hochschule Zürich, Department of Computer Science,, 2013.
[18] A. Zanette, J. Zhang, and M. Kochenderfer. Robust super-level set estimation using gaussian processes. In Machine Learning and Knowledge Discovery in Datahases: European Confer- ence, ECML PKDD 2018, Dublin, Ireland, September 10-14, 2018, Proceedings, Part 18, pages 276-291. Springer, 2019.
[19] N. Paragios and R. Deriche. Geodesic active regions and level set methods for motion estima- tion and tracking. Computer vision and image understanding, 97(3):259-282, 2005.
[20] D. Rempe, J. Philion, L. J. Guibas, S. Fidler, and O. Litany. Generating useful secident-prons driving scenarios via a learned traffic prior. In Proceedings of the IEÊE/CVF Conference on Computer Vision and Pattern Recognition, pages 17305-17315, 2022.
[21] C. Yan, W. Xu, and J. Liu. Can you trust autonomous vehicles: Contactless attacks against sensors of self-driving vehicle. Def Con, 24(8):109, 2016.
[22] J. Tu, M. Ren, S. Manivasagam, M. Liang, B. Yang. R. Du, F. Cheng, and R. Urtasun. Physically realizable adversarial examples for lidar object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13716-13725, 2020.
[23] A. Boloor, K. Garimella, X. He, C. Gill, Y. Vorobeychik, and X. Zhang. Attacking vision- based perception in end-to-end autonomous driving models. Journal of Systems Architecture, 100:101766, 2020.
[24] R. Duan, X. Ma, Y. Wang, J. Bailey, A. K. Qin, and Y. Yang. Adversarial camouflage: Hiding physical-world attacks with natural styles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1000-1008. 2020.
[25] Tu, H. Li, Yan, M. Ren, Y. Chen, M. Liang, E. Bitar, E. Yumer, and R. Urtasun. Explor- ing adversarial robustness of multi-senson perception systems in self driving, arXiv preprint 10Xiv:2101.0 066844 2021.
[26] N. Shukla, A. K. Sahu, D. Willmott, and Z. Kolter. Simple and efficient hard label black-box adversarial attacks in low query budget regimes. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pages 1461-1469, 2021.
[27] D. Lee, S. Moon, J. Lee, and H. O. Song. Query-efficient and scalable black-box adversarial attacks on discrete sequential data via bayesian optimization. In International Conference Machine Learning, pages 12478-1222 12497. PMLR, 2022.
[28] B. Gangopadhyay, S. Khastgir, S. Dey, P. Dasgupta, G. Montana, and P. Jennings. Identifica- tion of test cases for automated driving systems using bayesian optimization. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 1961-1967. IEEE, 2019.
[29] Y. Abcysirigoonawarden.. F. Shkurti, and G. Dudek. Generating adversarial driving scenarios in high-fidelity simulators. In 2019 International Conference on Robotics and Automation (ICRA), pages 8271-8277. IEEE, 2019.
[30] S. Silvetti, A. Policriti, and L. Bortolussi. An active learning approach to the falsification of black box cyher-physical systems. In Integrated Formal Methods: 13th International Confer- ence, IFM 2017, Turin, Italy, September 20-22, 2017, Proceedings 13, pages 3-17. Springer, 2017.
[31] Dreossi, A. Donzé, and S. A. Seshia. Compositional falsification of cyber-physical systems with machine learning components. Journal of Automated Reasoning, 13:10111 2019.
10

---

[32] R. Lee, J. Mengshoel, A. Saksena, R. W. Gardner, D. Genin, J. Silbermann, M. Owen, and M. J. Kochenderfer. Adaptive stress testing: Finding likely failure events with reinforcement learning. Journal of Artificial Intelligence Research, 69:1 1665112-2,, 2020.
[33] M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer. Adaptive stress testing for autonomous vehicles. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1-7. IEEE, 2018.
[34] S. Ghosh, F. Berkenkamp, G. Ranade, S. Qadeer, and A. Kapoor. Verifying controllers against adversarial examples with bayesian optimization. In 2018 JEEE International Conference on Robotics and Automation (ICRA), pages 7306-7313. IEEE, 2018.
[35] B. Bryan, R. C. Nichol, C. R. Genovese. J. Schneider, C. J. Miller, and L. Wasserman. Ac- tive learning for identifying function threshold boundaries. Advances in neural information processing systems, 18, 2005.
[36] E. Thorn, S. C. Kimmel, M. Chaka, B. A. Hamilton, et al. A framework for automated driving system testable cases and scenarios. Technical report, United States. Department of Trans- portation. National Highway Traffic Safety ..... 2018.
[37] A. Kesting, M. Treiber, and D. Helbing, Enhanced intelligent driver model to access the impact of driving strategies on traffic capacity. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, (5551198)4588544665 2010.
[38] A. Sadat, M. Ren, A. Pokrovsky, Y.-C. Lin, E. Yumer, and R. Urtasun. Jointly learnable behavior and trajectory planning for self-driving vehicles. In 2019 JEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3949-3956. IEEE, 2019.
[39] D. Duvenaud. Automatic model construction with Gaussian processes. PhD thesis, University of Cambridge, 2014.
[40] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[41] H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[42] J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165, 2017.
[43] V. Franc, A. Zien, and B. Schölkopf. Support vector machines as probabilistic models. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 665-672, 2011.
[44] J. Wang, S. Manivasagam, Y. Chen, Z. Yang. I. A. Bårsan, A. J. Yang, W.-C. Ma, and R. Urta- sun. Cadsim: Robust and scalable wild 3d reconstruction for controllable sensor simu- lation. In 6th Anmual Conference Robot Learning, 2022.
[45] Z. Yang, Y. Chen, J. Wang, S. Manivasagam, W.-C. Ma, J. Yang. and R. Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1389-1399, 2023.
[46] W. Zhao, J. P. Queralta, L. Qingqing, and T. Westerlund. Towards elosing the sim-to-real gap in collaborative multi-robot deep reinforcement learning. In 2020 5th International conference on roboties and automation engineering (ICRAE), pages 7- IEEE, 2020.
[47] Suo, K. Wong, J. Xu, J. Tu, A. Cui, S. Casas, and R. Urtasun. Mixsim: A hierarchical framework for mixed reality traffic simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9622-9631, 2023.
11

---

A Additional Experiments
While GUARD requires minimal tuning of GP kernel parameters due to online kernel learning. the rype of kernel still design choice. GUARD we employ Matern32 kernels to model each parameter and take the product of individual kernels to obtain the final kernel. In Table we conduct an ablation that also considers 1) RBF kernels, 2) Using sum instead of product, and 3) composition, using one RBF or Matern32 kernel for all parameters. We first find that without modeling each parameter separately, the results are poor. Also, using a sum operation to aggregate kernels is ineffective well. The best results are obtained from product of RBF or Matern32 kernels. We choose product of Matern32 as the lower false positive rate is critical for ensuring safery. Furthermore, this choice also exhibits much lower variance in the metrics, making it more reliable for testing.
Ablation on GP Kernels:
BO Acquisition Functions: As mentioned in Section 2 of the main manuscript, many adversar- ial example generation methods use Bayesian Optimization(BO) and employ acquisition functions that minimize some adversarial objective. Since BO also leverages GPs and active learning sim- ilar to GUARD, we can adopt these acquisition functions. In particular we use lower confidence bound (LCB), expected improvement (EI), and probability of improvement (PI). For EI and PI we adjust them to decrease the objective (expected decrease and probability of decrease). The results in Table 5 show that these alternatives can achieve strong results but is not as effective using the Straddle acquisition function. This because cost minimization oversample very negative points and Straddle oversamples boundary points, but the former is more useful in partioning pass / fail regions. The performance is however very close, and we hypothesize this because the parameter space is dominated by positive (passes). With very small negative regions, severe negatives are close to boundary points.
Runtime Comparison A major implementation detail regard- ing baselines is the discretization resolution. For grid search and (-way testing, there a direct tradeoff between coverage and balanced accuracy, error recall, false positive rate. Thus, lect a resolution which can cover most of the parameter space. For HiddenGems, the resolution does not affect coverage since it uses a levelset estimation algorithm to determine which bins covered. The limitation for our choice of resolution of is due to how the computation budget increases exponentially with resolution. This because the method requires running the GP on all discretized bins at every iteration. And while running the GP a single time is neglible compared running simulation, the exponential scaling makes the GP queries bottleneck at higher resolutions. As we show in Figure A, at this resolution the run- time roughly equals GUARD.
GUARD
25000
20000
15500
10000
Resolution
Figure 8: Runtime
Kernel Type
Composition
Coverage ())
Acc(%)
Pos Acc(S)
Neg Acu(%)
Em Recall(S)
[PR((%)
RBF
96.1 = 0.8
80.2 5.7
98.1 : 0.5
62.4 11.2
56.0 + 10.6
10.6
Matern22
87.3
78.5 4.2
99.3 = 0.2
57.7 844
36.8 1 7.7
6.15
RBF
Sum
88.0 + 6.7
58.6 3.2
89.1 + 11.4
28.2 11.5
21.4 12.0
20.2 5.7
Matern32
Sum
90.4 + 3.3
58.4 3.3
88.0 10.7
28.7 + 13.7
23.6 + 12.7
21.0 3.1
RBF
Product
96.8 = 0.7
83.3 7.0
97.9 0.6
68.7 13.9
62.6 13.1
7.83
Matern32
Product
84.2
+ 0.2
70.3
10.6
58.9
5.89
Table 4: Ablation of different kernel choices for GUARD
Aquisition Fn.
Coverage(%)
Bal. Acc(S)
Pos Acc(%)
Neg Acc(%)
Err. Recall(()
FPR(%)
90.7 1.6
82.2 3.9
99.4
64,9 7.8
1:00: 7.5
8.23 2.4
91.8 1.3
82.6 2.3
99.4 = 0.2
65.7 1 0.5
50.7 5.2
6.99 1.4
LCB
91.5 1.5
83.4 + 1.8
99.6 - 0.1
67.3 + 3.6
52.4 4.7
5.99 1.4
Straddle
94.3 + 0.9
84.2 + 5.3
99.2 = 0.2
70.3 = 10.6
58.9 1.3
5.89 + 3.1
Table 5: Comparison of cost minimization acquisition functions versus Straddle acquistion function.
12

---

B Test Scenarios
We describe the 10 logical scenarios used in experiments here:
Actor cut-in: An actor starts in an adjacent lane to the SDV and performs a cut-in maneu-
2. Shoulder actor cut-in: The SDV drives next to the road shoulder. An actor on the shoulder cuts in front of the SDV.
3. Actor overtake cut-in: An actor starts behind the SDV, then rapidly accelerates overtake the SDV and cut-in front of it.
4. Actors merging: The SDV is driving on a merge lane and there are multiple actors merging in.
5. Lead actor braking: An actor starts in front of the SDV and then brakes.
6. Lane change: The SDV attempts to lane change when there an actor in ithe target lane.
7. Lane change beside merge: The SDV attempts to lane change into a merge lane and actors are also merging into that lane.
8. Lane merge: The SDV is merging from on-ramp, and there is an actor in the merge lane that is very close to the SDV.
9. Lane merge multiple actors: The SDV is merging from on-ramp and there are multiple actors in the merge lane,
10. Lane merge parallel on-ramp: The SDV merging from a parallel on-ramp and there is actor in the merge lane that is very close to the SDV.
13


Reviews:
[{"review_id": "ky6GGBQB4j", "reviewer": "Reviewer_YzeP", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors are tackling a problem in the field of autonomous driving or autonomous cars that drive on open streets. A problem for autonomous cars is the testing and verification of the system in order to prove the reliability and quality of the AVs software and hardware. The authors are describing the problem of virtual testing which is necessary to evaluate and verify AVs. The authors of the paper are proposing a \"new\" testing framework called GUARD which relies on Gaussian Process to generate tests for the AV. Everything displayed in the paper is done in simulation only.", "summary_of_recommendation": "In summary, the paper is adressing the critical problem of virtual testing for autonomous vehicles. By proposing to apply the method of Gaussian processes to generate a new testing framework/ new test scenarios in simulation, the authors want to display that their framework can evaluate more efficiently  and accurately. Currently the simulation results and statistical results do not display this claim thoroughly since the result and discussion section is missing critical content and explanations.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "- Although the paper describes the theoretical foundations of GUARD, it does not provide sufficient implementation details or practical guidelines, potentially making it challenging for practitioners to replicate and apply the framework effectively. Right now the paper is missing the context of FuSa, SOTIF and the connection to other simulation frameworks. Where do the authors see their method?\n- Right now, the Generalizability to Various AV Systems is quite unclear. The paper lacks an exploration of how the GUARD framework may adapt to different types of autonomous vehicle systems or the challenges that could arise in diverse AV architectures. This focuses especially on different hardware and software architecture setups as wells specific ODDs", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality:\n- The State of the Art of the paper is quite poor and does not cover the extensive work and papers that were published in the field of scenario based testing or virtual testing for AVs. It is recommend to further read similar papers in order to create a clear contribution for the own paper.\n- Currently, some statements are wrongly made in the paper e.g. \"Intuitively, if the AV easily passed a particular test, it can\n53 pass similar tests with high probability.\" -> This is not true since specific scenario parameters can lead to catastrophic AV behavior e.g. changing the weather in the same scenario and same paramter setup\n\n\nClarity:\n- The paper lacks of clarity and is confusing many times with the goals, objectives and content of the paper\n- The authors use the terms \"testing\" and \"Testing framework\" similarly, unfortunately there are clear differences between the individual terms and the topcis. The authors need to state clearly what they are doing e.g. right now its a scenario generation framework. There is no new simulation or similarly invented.\n- Currently i am missing a clear definition from a functional safety (FuSa - ISO 26262) and Safety Of The Intended Functionality (SOTIF -ISO/PAS 21448) point of view how the approach can be applied and how it is justified. Right now existing approaches need to go over certain scenarios and parameter sets in order to cover everything to be correct from a SOTIF and FuSa perspective. Yes, the author cited both ISOs but miss where the approach does fit into those.\n\nOriginality:\nApplying the method of Gaussian process in order to leverage observed test results to model the probability of passing or\nfailing across the entire parameter space is a neat approach. I think the authors demonstrate on their toy problem and simple \"Pass-uncertain-fail\" setup that the approach can be quite useful\n\n\nSignificance:\nWhile the paper exhibits promising problem formulation and methodological approaches, its lack of comprehensive evaluation, results, and discussion, particularly in the context of Robotics and Autonomous Vehicles (AV), diminishes its significance for the field, highlighting the missed opportunity to showcase the potential and contribution of the research. Additionally, the absence of a demonstration of the paper's integration with other simulation and testing environments, such as CommonRoad, further limits its relevance and applicability within the broader research landscape.\n\n\nStrength:\n-  The paper addresses a crucial and timely challenge in the field of autonomous vehicles - understanding system safety conditions. With the rapid advancement of AV technology, the need for robust testing frameworks becomes increasingly critical, making this research highly relevant.\n- The paper introduces a novel problem formulation to partition the scenario parameter space without relying on discretization. By using a probability-based approach, the authors avoid the limitations of traditional binning methods, enabling a more accurate representation of safe, unsafe, and unknown regions.\n\n\nWeaknesses:\n- The paper primarily focuses on simulation-based testing, but there is limited discussion or validation of the framework's effectiveness in real-world scenarios. Without empirical validation in real-world environments, the applicability and reliability of GUARD might remain uncertain.\n- the paper briefly introduces the use of Gaussian Processes for probability modeling, but it lacks a detailed explanation of the underlying assumptions and limitations of this approach. A more comprehensive discussion would help readers understand the reliability of the probability estimates. Additionally it is recommend to enhance the limitation and discussion section to show the reader where and why the approach can create significance and impact.\n-The paper suffers from unclear simulation experiments and results, which significantly hinders its overall clarity and impact. The lack of a well-defined and transparent description of the simulation setup and execution raises questions about the validity and reproducibility of the findings.\n- Additionally, the presentation of the results is inadequate, with insufficient data visualization or statistical analysis to support the authors' claims effectively. As a result, readers may find it challenging to interpret or draw meaningful conclusions from the simulation outcomes\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "_IWQGS4rZT", "reviewer": "Reviewer_QJej", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper focuses on testing autonomous vehicle software in simulation. The key idea is to infer the probability of a test output (e.g., minimum distance between road agents in the scenario) given some scenario parameters in a scalable but high-quality way. The paper uses a Gaussian Process to infer this probability and uses a previously developed sampling scheme to prioritize data points at the boundary of a pass / fail of the test (where the pass / fail boundary is determined a priori by, for example, a regulatory agency). The paper compares the proposed approach to an exhaustive discretization approach and three other baseline for a 5D scenario parameter space. It shows favorable performance in terms of accuracy, error, and coverage metrics.", "summary_of_recommendation": "While I think this paper focuses on a valuable problem and has an interesting approach, I have two concerns which lead me to recommend a weak reject.  First, the paper's approach of using Gaussian Processes is not benchmarked against SOTA learning-based approaches. This is specifically relevant for two reasons: 1) it is well-known that GPs scale poorly with the number of observations, and one of the major claims of this paper is a more scalable framework, and 2) other SOTA methods mentioned in my review that, for example, use Deep RL, have better potential for scalability and are highly relevant to this conference. Secondly, the clarity of the writing and experimental section should be improved to shed light on this paper's approach and relevance for autonomous driving (e.g, simulator used, models for the road agents during testing, what the scenario parameters model, how are these scenario params chosen, etc.).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* In section 3.2, the writing and notation made it hard to tell if the goal was to infer a distribution over test parameters \\theta (i.e., $P(\\theta)$), a distribution of pass-or-fail (i.e. $P(y)$), or a distribution over the real-valued output of the metric (i.e. $P( f(\\theta) )$). Which distributions are being learned by the GP?  \n* How does this work compare (in terms of scalability, ability to find failure scenarios, etc.) to adaptive stress testing approaches like [1] and [2] which utilize RL and monte carlo tree search? \n[ 1 ] Lee, Ritchie, et al. \"Adaptive stress testing: Finding likely failure events with reinforcement learning.\" Journal of Artificial Intelligence Research 69 (2020): 1165-1201.\n[ 2 ] Koren, Mark, et al. \"Adaptive stress testing for autonomous vehicles.\" 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2018.\n* What do the parameters mean in this experimental environment? How was the space of $\\Theta$ constructed? Also, why was $d = 5$ parameters chosen in these experiments? \n* The efficiency results only report how accuracy and error are impacted by dimensionality of the parameter space. However,I would like to see compute time comparisons of each method for the same scenario to understand the compute efficiency.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* The paper provides many ablations of the proposed method, and comparisons quantitative and qualitative with other methods. \n* The paper’s approach – using Gaussian Processes to infer the likelihood of failure under some scenario parameters – is, to the best of my knowledge, an original approach to finding failure / success scenarios. \n\nWeaknesses:\n* The mathematical clarity could be improved. Specifically, in the Gaussian Process section it was difficult to parse what was being modeled / inferred and from what data was being used to train the GP.  See more detailed comments below. \n* It was challenging to understand one of the key contributions – computational efficiency – of the proposed method from the main text. I would recommend including a metric like compute time on a standard computer for all methods. Additionally, even though efficiency is one of the proposed advantages of the method, the parameter dimensional still appears quite low – 5D at max – in the experiments. \n* I would like to see how the proposed method compares in terms of scalability and search performance to adaptive stress testing methods that utilize Deep RL / Monte Carlo Tree Search. These approaches seem to have a much stronger potential to scale to large state / parameter spaces than GPs. Detailed comments and specific methods are mentioned in the rebuttal section. \n\nOverall comments:\n* “The observed test samples $X = [\\theta_1;... ; \\theta_N], Y = [f^*(\\theta_1)...; f^*(\\theta_N)]$ then become observations of GP random variables” is a difficult sentence to parse, as is the math in section 3.2. Is X the observation or is Y the observation? And what is the final distribution you are modeling with the GP? It would be helpful to clearly write out 1) what are the random variables and how are they distributed (e.g., let $X = [\\theta_1, …\\theta_N] \\in R^{n \\times d} \\sim N(\\mu_\\theta, \\Sigma_\\theta)$ be a random variable of N parameter values. Let $Y = (f^*(\\theta_1), …f^*(\\theta_N)) \\in R^n$ be the corresponding scalar outputs of $f^*$ of each $\\theta$, …), and 2) what distribution is being modeled (e.g., saying something like `we seek the conditional distribution $P(Y | X)$ which models the probability of test outputs given a set of parameter values. We infer this distribution from…and model it as a Gaussian process with mean $\\mu$, and covariance $\\Sigma$…’’). \n* It is also unclear why the sets $P*, F*, P$ and $F$ in equation 3 are defined at the very start since most of the focus of this paper is on maintaining the full distribution, and these sets are only used in the evaluation metrics and not during the core of the method. It would be clearer to move these to Section 4. \n* In the main text, Section 4, it was difficult to understand 1) what simulator was being used?, 2) how was the data simulated for all the road agents? In particular, 2) is of concern because if the data for non-ego road agents was drawn from a static dataset, then the results would look different than from a road agent simulator where the agents can adapt/react to the ego vehicle. \n* The text writes  “Executing a test in simulation outputs continuous measures of safety or other performance metrics… $f^*(\\theta, A) \\in R$”. However, as noted in the Limitations section, in most AV driving scenarios, there is more than one performance metric – for example, one may be interested in a safety metric (e.g. the minimum distance between the vehicles) and a progress metric (e.g., time-to-destination). It would be helpful to make this assumption more clear upfront. For example, “We assume that a simulation test outputs a scalar indicating the level of safety (e.g., minimum distance to another roach agent). Mathematically, let  $f^*: \\theta \\times A \\rightarrow R$ be the test function which takes as input the simulation parameters, and autonomy software A and outputs a real-valued scalar.”", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "Xf-jAlIX6Fx", "reviewer": "Reviewer_yt5X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Autonomous vehicle (AV) driving scenarios are uncountable and depend on a huge number of factors, such as agents' states, intents, lane geometry, number and type of agents, weather, etc. Making a safety case for the AVs requires establishing an operational design domain (ODD) and proving safety over all scenarios that arise within this ODD, i.e., _coverage_.\n\nAs one can anticipate, the challenge to this task lies in testing these immense number of combinations. This paper provides a method for coverage testing by probabilistically modeling the safety outcomes of scenarios using Gaussian Process (GP). Scenarios that satisfy the safety threshold with high probability are deemed safe, while those that fail with high probability are deemed unsafe. The uncertain scenarios are slotted for further testing. This enables more scalable coverage testing by focusing on scenarios where the safety outcome is uncertain, instead of testing every possible combination.", "summary_of_recommendation": "I recommend strong accept. There are a couple of issues that I have listed in my review above, but they are minor. Overall, the method presented in this paper is sensible and it outperforms the existing methods.\n\n----------- Post-rebuttal -----------\n\nI have updated my score to Weak Accept due to the potentially limited impact of the paper.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please include more details of the simulator used in the tests. What simulator was used? How are the road agent reactions generated?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In my opinion, this is a strong paper. I truly did not find a lot to criticize here. The problem being addressed is pertinent, the proposed solutions makes sense, the results are thorough, and the writing is clear.\n\nOne limitation I would like to point out which is not mentioned in the paper is the requirement of a good realistic closed-loop simulator. Most existing simulators are either not closed-loop or the closed-loop behavior of other agents is unrealistic (such as nuPlan). Applying this approach on actual vehicle testing is not scalable. It is not practical to generate a scenario with some specific lane geometry and number of agents, etc., and then go out in the real-world to seek it. So from that perspective, this approach and others reliant on the development of good closed-loop AV simulators will not have a very significant impact in the near future.\n\nAnother point is that safety is viewed as merely collisions in this paper, measured with the distance from other agents. Establishing safety encompasses a lot more than that. Reasoning about the impact on AV's actions on other agents' behavior is also crucial. For instance, if the AV runs through a traffic light and manages to avoid any collisions because of the quick reaction of other road agents, this behavior should still be deemed unsafe even though no collision occurs. This is less of a gripe with the paper's core methodology, more with the safety metric used. \n\n----------- Post-rebuttal -----------\n\nThe key contribution of this paper is an approach to achieve improved \"targeted\" coverage of the ODD, which the paper demonstrates fairly well. I still view this paper positively and believe that it should be published. However, it seems that I share the same concern as other reviewers regarding the limited impact of the paper due to the need for good closed-loop simulators (as was raised in my initial review). So, to ensure that my score is aligned with the content of my comments, I have updated my score to Weak Accept.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "LFemo87Pk3", "reviewer": "Reviewer_WNoS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a framework for finding optimal test sequences such as found in testing autonomous driving components. For instance, AV are tested on parameterizable test scenarios. The dimensions of the search space for a given scenario are the number of parameters that the scenario has. Effectively the paper proposes an algorithm for efficiently characterizing the failing manifold of scenarios using a discrete amount of probes.", "summary_of_recommendation": "I would really rework the presentation of the paper.\nI think this paper has high impact potential but it's just not presented this way.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Could you address the focus of the paper?\n- Can you do a literature review on these kinds of algorithms?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "= Contribution\n\nThe algorithm proposed by the author is sound and shown to outperform several baselines. The paper is generally clear, though one weakness lies in the way the research idea is communicated. It seems to this reviewer that instead of focusing on the problem setup and experimental context, the authors should have simply addressed the problem in a more algorithmic way. This would have made the contribution much more clear.\n\n= Research Idea\n\n== Quality\n\nThe paper is very well executed. The contribution is well-situated and argued. The algorithmic solution is quite appropriate, it seems that this kind of algorithm should already exist in another context though. The idea is very similar to Bayesian optimization, with the key difference that the goal is to characterize a hypersurface. The paper presents a series of comparative experiments highlighting the benefit of the approach.\n\n== clarity\n\nThe paper is quite clear, though the focus is somewhat not ideally placed. The problem the authors are solving is algorithmic: How to sequence a discrete test campaign to best characterize the system binary performance on the benchmark? As such it would make the presentation much clearer to have the algorithm in the paper instead of in supplementary material. To also make the connection to Bayesian optimization stronger and to try to find in the literature, algorithms like this one applied in different contexts. Somehow the scope of the contribution should be reviewed to clarify the paper.\n\n== originality \n\nThe paper is rather original, and it is surprising that this idea was not investigated in prior work.\n\n== significance\n\nThis is quite significant and as said surprising that this was not investigated before. However a few limitations underly the approach. These are pointed out by the authors themselves.\n\n= Strength and Weakness\n\nStrengths\n-\tVery well-executed research idea\n-\tBreadth baseline comparison\n\nWeakness\n-\tDoes not focus with enough clarity on the algorithm underlying the method\n-\tThe algorithm is outlined throughout the paper but only made explicit in supplementary material\n-      The algorithm in the supplementary material lacks comments and is not easy to follow\n-\tMany hyperparameters are given but it is not clear why they are important or chosen\n-\tConnection to similar work is missing\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors' response has addressed many of the reviewers' concerns, and the general assessment is that this paper provides a good contribution to CoRL on a very timely topic, namely, testing of autonomous vehicles. I agree with this assessment and therefore recommend acceptance. That said, the reviewers had several suggestions for improvement -- in the final version of this paper, the authors should *carefully* revise the paper to address all reviewers' comments / suggestions.
Decision: Accept (Poster)