# Model Validation and Scenario Selection for Virtual-Based Homologation of Automated Vehicles
## 3 Methodology
### Model Validation  
After the assessment in the validation domain, the validation metric operator quantifies the difference between the minimum distance to line from the physical system on the road \(y_{s}^{o}\) and from the simulation model \(y_{m}^{o}\). One option would be to average over the test repetitions and then calculate the distance between the two averaged values. However, we decide against this validation metric, because averaging causes a loss of information. Instead, we calculate the area between the whole ECDFs. There are three possibilities for each validation scenario. Either the ECDF of the simulation lies completely on the left-hand side of the system ECDF, completely on the right-hand side, or both intersect. The first case is most critical from a safety perspective, as the simulation suggests safer behavior than would be the case in reality. Whereas the first case yields only a leftarea (and zero right area) and the second case only a right area (and zero left area), the third case yields a separate left and right area  
\[e_{l}^{v} =\int_{F(y_{m}^{v})\geq F(y_{s}^{v})}|F(y_{m}^{v})-F(y_{s}^{v})|dy\,, \tag{14}\] \[e_{r}^{v} =\int_{F(y_{m}^{v})\leq F(y_{s}^{v})}|F(y_{m}^{v})-F(y_{s}^{v})|dy\,. \tag{15}\]  
This occurs when simulation and measurement show a desired similar behavior, but the simulation typically has less scatter, so that its steeper ECDF crosses the flatter one of the system. The principle of two areas is visualized in Figure 6 for one scenario. Thus, calculating the validation metric for all the validation scenarios \(\mathbf{X}^{v}\) yields two vectors \(\mathbf{\mathrm{e}}_{l}^{v}\) and \(\mathbf{\mathrm{e}}_{r}^{v}\) for the left and right areas of the minimum distance to line. We aggregate this knowledge about the validity of the simulation in an error model in order to be able to infer it to new scenarios in the application domain. This is particularly important because it is risky to compare the deviations only with the permissible tolerances, but to neglect them if they are considered suitable. These modeling errors can lead to wrong type-approval decisions regarding the safety of the vehicle and ultimately to accidents in the real world. We use a multiple linear regression model based on ([28], p. 657) to model the left area  
\[\delta_{l}^{vu}=\mathbf{\mathrm{x}}\cdot\mathbf{\mathrm{w}}_{l}^{\mathrm{T}}= \begin{bmatrix} 1&v_{x}&a_{y,ref}\end{bmatrix} \cdot \begin{bmatrix}w_{l,0}&w_{l,1}&w_{l,2} \end{bmatrix} ^{\mathrm{T}} \tag{16}\]  
and the right area, respectively. The hat symbol emphasizes that the error model result is an estimation from validation to application scenarios. The regression weights \(\mathbf{\mathrm{w}}_{l}\) are calculated using a least square optimization with the validation metric results \(\mathbf{\mathrm{e}}_{l}^{v}\) as training data. Since the error model itself is not perfect, it remains a mean squared error \(s\) when comparing the estimations \(\hat{\mathbf{\mathrm{e}}}_{l}^{v}\) with the training data \(\mathbf{\mathrm{e}}_{l}^{v}\) at the validation scenarios. This mean squared error and a t-distribution with a confidence of \(\alpha=95\%\) can be used to calculate a non-simultaneous Bonferroni-type prediction interval function ([61], p. 115)  
\[g_{p}(\mathbf{\mathrm{x}}^{\alpha})=t_{N^{v}-(N_{x}+1)}^{s/2}\cdot s\cdot\sqrt{1+ \mathbf{\mathrm{x}}^{\alpha^{\mathrm{T}}}(\mathbf{X}^{v^{\mathrm{T}}}\mathbf{X}^{v})^{-1} \mathbf{\mathrm{x}}^{\alpha}}\,. \tag{17}\]  
The prediction interval (PI) contains the uncertainty of the error model--as does a confidence interval--and additionally the uncertainty associated with the prediction  
Figure 6: Area validation metric at the exemplary validation scenario \(v_{x}=118\,\mathrm{km}/\mathrm{h}\) and \(a_{y,ref}=0.65\cdot a_{y,smx}\). The left and right areas refer to the perspective of the model with the system on its left or right side. to an unseen application scenario \(\mathbf{x}^{a}\).