# Improved Surprise Adequacy Tools for Corner Case Data Description and Detection
## 4 Experiments and Evaluation
### Benchmark Dataset Analysis
In this paper, we take four widely-studied adversarial attacks to generate new testing set on MNIST, such as Fast Gradient Sign Method (FGSM) [16], Basic Iterative Method (BIM) [34], Carlini & Wagner (C&W) [35], and Projected Gradient Descent (PGD) [36]. Then, based on these four adversarial testing datasets, the proposed DSAs are applied to detect the corner case data, namely misclassified data. Similar to the above analysis, we do not care about the exact selection of the DSA threshold, and just utilize the AUC-ROC to evaluate the performance of using DSA on corner case data detection. The results of ROC curves of four DSAs on different adversarial testing sets are depicted in Figure 6.0 \begin{table}
\begin{tabular}{c c c c} \hline \hline  & **Layer1** & **Layer2** & **Layer3** \\ \hline DSA0 & 0.9422 & 0.9418 & 0.9759 \\ DSA1 & 0.9657 & 0.9722 & 0.9833 \\ DSA2 & 0.9114 & 0.9662 & 0.9875 \\ DSA3 & 0.9814 & 0.9884 & 0.9966 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: AUC-ROC of corner case data detection based on DSA. Figure 9: ROC of using DSAs on corner case data detection on four adversarial testing data. (**a**) FGSM. (**b**) BIM. (**c**) CW. (**d**) PGD. Figure 9 shows the ROC curves on corner case data detection, where the x-axis represents the value of FPR, and the y-axis represents the value of TPR respectively. In details [37], TPR means true positive rate, also called sensitivity or recall, which can be used to reflect the test's ability to correctly detecting corner case data here; FPR means false positive ratio, also known as false alarm ratio, which is the probability of falsely rejecting the null hypothesis for a particular test. Their calculation are presented as below  
\[\text{TPR}=\frac{TP}{TP+FN} \tag{18}\]  
\[\text{FPR}=\frac{FP}{FP+TN} \tag{19}\]  
where, _TP_ (True Positive), _FP_ (False Positive), _FN_ (False Negative), _TN_ (True Negative) are four events in the confusion matrix of corner case data detection. Moreover, based on these ROC curves, we can quantitatively calculate the values of AUC as a KPI metric, shown in Table 2.  
According to the results of Table 2, the same foundation is obtained that DSA3 performs relatively better than the other DSA definitions on all these four adversarial testing sets. Moreover, their high values also verify that using DSA values as a reference on corner case detection can reach good performance. Therefore, we can make use of DSA, especially DSA3, in the following study related to corner case data and AI quality assurance. (5) Discussion  
While, besides the positive results from the above studies on using DSA and its modifications for corner case identification, there are also some negative points that should be noticed. The first one is the application limitation of DSA only for classification problems. When dealing with other machine learning applications, e.g., regression, how to copy with the corner cases would be a challenge for the research in this paper. The second one is the limitation in DSA's modification. Even though three modifications of DSA are proposed in this paper, and achieve relatively good performance on corner case detection, DSAs' abilities on discovering more novelty behaviors are limited since these modifications are additive to the original DSA definition. Therefore, more exploration on SA definitions can be studied, e.g., a novelty variant using silhouette coefficient was proposed in [24]. The third point is the time overhead in DSA computation. Extra time consumption is unavoidable in the corner case study. In the DSA computation process, the distances of testing points to the whole training set on activation trace should be computed, so its computation time is proportional to the training set size [38]. For the mentioned four DSAs in this paper, DSA0/1/3 have the same time complexity as the above description, except DSA2 whose computation time is proportional to the number of categories. Then, corner case detection in the whole testing set would be quadratic with the testing set and training set size.