# Improved Surprise Adequacy Tools for Corner Case Data Description and Detection
## 4 Experiments and Evaluation  
### Benchmark Dataset Analysis  
(1) Data source and DL system description  
To study behaviors of the proposed DSA on corner case data detection, in this paper we take the MNIST data [25] as a reference in experiment and analysis. MNIST is a widely-used dataset in machine learning research, which contains ten classes of images, and is divided into 60,000/10,000 training/testing samples. To recognize different numbers in MNIST, we adopt a commonly-used five-layer Convolutional Neural Network (CNN), including the convolutional, max-pooling, dropout, and full-connection layers. On the other hand, experiments in this paper are implemented on a machine equipped with Intel i7-9750 CPU, 32 GB RAM, NVIDIA GeForce RTX 2060, running in Windows 10 Pro 64-bit OS. Code of this paper is released [31], which is developed from SADL [32] on Keras and contains the proposed three kinds of modification of DSA. (2) Qualitative analysis on corner case data detection  
Firstly, through training the given CNN architecture above on MNIST data, we can achieve 99.17% accuracy on the original testing set. Then, based on the proposed DSA definition and the trained DL model, we can calculate the values of DSA on each testing sample, and subsequently, study DSA's behaviors in responding to corner case data detection. By sorting all testing samples as the DSA descending trend, we can first study the influence of DSA variation on the DL model's performance. To implement comparative analysis, four kinds of DSA are calculated on outputs of three selected layers, such as the  
Figure 5: The third modification of DSA. (**a**) specific case. (**b**) modification. convolutional layer 1 (Layer1), convolutional layer 2 (Layer2), and the final full-connection layer (Layer3) of the given CNN. Results are shown in Figure 6.  
Figure 6 shows the curves of accuracy vs. DSA variation, Firstly, by taking a subset containing the first \(k_{n}\) samples of the sorted dataset as testing, then its testing accuracy is calculated. DSA value and accuracy are compared in the graph in Figure 3. Here, the beginning point is \(k_{n}=100\), and we can see the accuracy is very low. That implies there is a certain percentage of erroneous data (corner case data) in the first testing subset causing CNN with wrong decisions. This phenomenon can be found in all curves during the beginning period where testing samples all have large DSA values, especially when compared with the testing accuracy of random sampling. On the other hand, at the ending point where \(k_{n}=10\),000 and DSA threshold is set as the least value, the whole testing dataset is considered, so the testing accuracy reaches the highest value equal to 99.17%. From these results, we can conclude that data having large DSA values is more possible to lead to misclassification than data with low DSA values. Moreover, since corner case data also could cause misclassification, it also implies that data with high DSA values can capture corner case behaviors to some extent. To vividly analyze this foundation, we can further draw out images of these samples with the high DSA values, as shown in Figure 7.  
In Figure 7, images with the highest DSA values are drawn separately based on four DSA definitions, such as DSA0- DSA3 representing the original DSA and other three modifications, respectively. Here, DSA3 adopts the second definition in (16) which is more  
Figure 6: Testing accuracy variation as the descending DSA. (**a**) Layer1. (**b**) Layer2. (**c**) Layer3. Figure 7: Images with the largest DSA values. (**a**) DSA0. (**b**) DSA1. (**c**) DSA2. (**d**) DSA3. convenient in computation (e.g., \(k=20\)). From these images, we can indeed find that these images with high DSA values have unrecognizable characteristics, such as unusual writing, thick chirography, large slant, and structure disequilibrium. It shows these data may have a high-risk to cause wrong decisions for both human beings and AIs, and be categorized as corner cases.