# Improved Surprise Adequacy Tools for Corner Case Data Description and Detection
## 4 Experiments and Evaluation
### Benchmark Dataset Analysis
Therefore, it qualitatively illustrates that high DSA is useful to reflect corner cases' behaviors further. (3) Quantitative analysis of DSA behaviors  
While, besides the conclusion that DSA can capture activation behaviors of data from Figure 6, we can discover other interesting phenomenon, namely DSA3 has a relatively lower accuracy in all these three layers, especially at the beginning period of curves. That means, under the same condition, high DSA3 values can capture more high-risk images than other DSAs to lower the testing accuracy, which can imply DSA3 has a better capability on corner case data detection. To verify this conjecture and to further quantitatively study different DSAs' performance on corner case data detection, the proposed corner-case data coverage in (18) is calculated. Results of corner-case data coverage based on four DSAs are shown in Figure 8.  
In Figure 8, the random sampling testing data is firstly studied. It is seen that as the amount of random testing data increases, the value of corner-case data coverage increases uniformly. To calculate the values of corner-case data coverage based on DSA, the testing data is also sorted as the descending direction of DSA values firstly, then a subset consisting of data with highest DSA values is considered for coverage study, similar with the process of accuracy study in Figure 6. According to (18), the number of wrongly classified data in the subset is counted for the calculation of corner-case data coverage. From the results of Figure 8, it's seen that experiments using DSAs achieve larger corner-case data coverage than random testing. Considering this coverage is the reflection of the percentage of identified misclassified data, so these results illustrate that, with the consideration of large DSA values, more data is identified as corner cases compared with random testing, namely verifying DSAs' feasibility on corner case detection. Moreover, by comparing curves of different DSA definitions, we can find the conclusion again, that DSA3 has the largest coverage values on the analysis of all three given layers, implying its better performance on corner case identification. On the other hand, we can further utilize the measurement of AUC-ROC [33] on corner case data detection to evaluate different DSAs' performance. By assuming the utilization of DSA values directly to distinguish normal data and corner case data, the AUC value of this binary classification process can be calculated. AUC-ROC values of four DSAs in three given layers are presented in Table 1.  
Figure 8: Corner case data coverage of testing data sorted as the descending DSA. (**a**) Layer1. (**b**) Layer2. (**c**) Layer3. Based on the result in Table 1, three conclusions can be obtained. First, for the constructed CNN model on MNIST, it indeed achieves good performances by using DSA to guide corner case data detection as all AUC values are larger than 0.9. Second, it is found that the deeper the layer is, the better DSA performs on corner case data detection, seeing AUC(Layer3) > AUC(Layer2) > AUC(Layer1), which is consistent with the ability of DL models on feature learning. Third, DSA3 can have a relatively better performance when compared to the other DSA definitions, due to its additive improvement. Therefore, based on these results, we can consider choosing the proposed DSA3 for corner case data detection in the following studies. (4) Adversarial testing analysis  
To further analyze the performance of DSA, adversarial testing, as an important AI testing technique, can also be applied here. Generally, based on the original testing data, small perturbations imperceptible to humans are added to generate adversarial samples which will lead the DL model to make wrong behaviors (outputs). There are many different attack strategies applied to generate adversarial testing set in literature.