## 4 Result and Discussion
### Defect Detection Performance for Metal Transfer Abnormality
curve tells something about how well the samples of the abnormal class can be separated from the normal class, while the prediction accuracy hints on the actual performance of classifier. AUC is the area under the ROC curve. Hence, if all of the probabilities are above 0.5, an AUC of approximately 1 can still be achieved if all of the abnormal cases have higher the number of probabilities than all of the normal cases. Based on Table 2, the VGG16-PRETR model using preprocessed images shows a low average accuracy of 0.638 and a high average AUC of 0.990. One of these models classified all 647 abnormal cases as abnormal. Conversely, in the case of normal images, 136 are matched with normal, and the remaining 513 are misclassified as abnormal. It appears that the model is overfitting for abnormal predictions. Other overfitting models (Basic CNN for metal transfer, VGG16, VGG16-PRETR) also show general characteristics of low accuracy and high AUC. Figure 11 is the distribution of abnormal prediction rates for all individual data of the VGG16-PRETR model using preprocessed images. The probability of predicting the normal class as abnormal is widely distributed below 1. The probability of predicting abnormal class as abnormal is distributed close to 1. In this case, a decision threshold close to 1, which will yield an error rate (probability of misclassification abnormal) of close to zero. It is noteworthy that because the AUC only measures the ranking of the probabilities, it doesn't reveal whether the probabilities are well calibrated (e.g., a systematic bias does not exist). Therefore, all model performance values (AUC, sensitivity, specificity, and accuracy) must be considered. The basic CNN for metal transfer, VGG16, and VGG16-PRETR models cannot be used when all model performance values are considered. This result is confirmed via Grad-CAM as well. A comparison of the effectiveness among CNN models using preprocessed images is shown in Figure 10. As shown, except for the high-performance model VGG16-PRETR-FINETUNE, the other three models (basic CNN for metal transfer, VGG16, VGG16-PRETR) comprise a color area spread across the entire area. This means that the feature information is not well specified. For example, VGG16-PRETR with the widest colored area indicates the lowest accuracy. The basic CNN for metal transfer indicates the second highest accuracy among the CNN models with preprocessed images. When considering the Grad-CAM image, the red area is distributed in the lower part of the image, not the edge of the metal. In the case of the basic CNN for metal transfer, which is not a deep CNN structure, it appears that feature learning does not perform well. In the case of the VGG-16 PRETR-FINETUNE, which has the best performance, it seems that we learn the straight and curved edge features as intended. Figure 11: Distribution of abnormal prediction rates for all individual data of the VGG16-PRETR model using preprocessed images. Based on the results shown in Table 2, Figures 9b and 10, a high-performance model (VGG16-PRETR-FINETUNE) can be defined for all measurements (AUC, sensitivity, specificity accuracy, and feature information visualization) developed using preprocessing.