# Applying Heuristics to Generate Test Cases for Automated Driving Safety Evaluation
## 4 Assessment of Generated Cut-In Test Cases
### Comparison to Monte Carlo-Based Test Case Generation  
In this sub-chapter, a Monte Carlo-based test case generation was applied to the cut-in dataset, to enable an additional evaluation of the methodology proposed in the current paper, by comparing with a methodology typically applied to generate test cases in the field. As indicated in related work, no method is available which accounts for both multi-dimensional parameter dependencies and objective functions. A Monte Carlo simulation, which considers multi-dimensional dependencies, shows to be a suitable comparison method in terms of replicability and comparability.  
To ensure the comparability, the data subsets for severity and exposure (see Section 2.2) were reused, resulting in one test suite for severity and one test suite for exposure. First, the parameters were fitted to distributions independently for each subset, as follows: independent parameters were approximated by a normal distribution. These were \(\text{a}_{\text{x,VuT}}\), \(\text{v}_{\text{y,chall}}\), and \(\text{a}_{\text{x,chall}}\) for the severe data subset and \(\text{a}_{\text{x,VuT}}\), \(\text{v}_{\text{y,chall}}\), and \(\text{d}_{\text{x}}\) for the normal driving subset. The two dependent parameters, both times \(\text{v}_{\text{x,VuT}}\), \(\text{v}_{\text{x,chall}}\), and \(\text{d}_{\text{x}}\), were fitted with a multivariate Gaussian distribution. Thereby, the parameter dependencies were incorporated, which accounts for the comparability of the methodologies.  
Second, sampling was applied by generating randomized values or pairs of values, respectively, from the parameter distributions of each scenario subset and assigning them to the test cases.  
The number of test cases for each test suite was limited to the size of the heuristics test suites. The generated test suites for severity and exposure of the Monte Carlo simulation are provided in the Appendix A Table 11.  
In the following, the same evaluation methods as in section A and B were applied by comparing the heuristics test suites with the Monte Carlo test suites. For criticality, the test suites were assessed by their RPF values and the introduced independent criticality metric required deceleration, \(\text{a}_{\text{x}}\), in Figure 4.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \(\text{v}_{\text{x,VuT}}\) & \(\text{a}_{\text{x,VuT}}\) & \(\text{v}_{\text{x,chall}}\) & \(\text{v}_{\text{y,chall}}\) & \(\text{a}_{\text{x,chall}}\) & \(\text{d}_{\text{x}}\) \\ \hline \multicolumn{6}{c}{**Severity test cases**} \\ \hline \([46.8\%]\) & 93.5\% & \([30.5\%]\) & 43.1\% & 98.2\% & 41.5\% \\ \hline \multicolumn{6}{c}{**Exposure test cases (Normal driving)**} \\ \hline \([46.0\%]\) & 100\% & \([37.5\%]\) & 100\% & 100\% & 83.5\% \\ \hline \([]\): Parameters are set by their regression models and not independently. & & & & \\ \end{tabular}
\end{table}
Table 6: Coverage ratios of severity and exposure test suites.  
For both criticality metrics, the heuristics severity test suites showed significantly higher incorporated criticality than the generated Monte Carlo severity test suite. For exposure, a comparable distribution of the RPF values and \(\mathrm{a_{x}}\) required can be observed for the heuristics and the Monte Carlo test suites. While the heuristics aims to find different test cases (by newness evaluation), the Monte Carlo methodology is more focused on the frequency of occurrence. Therefore, the corner-cases tends to be neglected by the native Monte Carlo sampling.  
Regarding coverage, Table 7 presents the coverage rations for the Monte Carlo test suites of severity and exposure. The comparison with the coverage results of the heuristics test suites reveals a slightly higher parameter coverage for the severe Monte Carlo test suite, in comparison to the severe heuristics test suite. A contrary tendency becomes visible for the exposure test cases with a higher coverage of the exposure heuristics test suite.