# Defect Detection on Rolling Element Surface Scans Using Neural Image Segmentation
## 6 Evaluation  
As mentioned in the introduction, defect elements are a rarity. Due to this, 10 non-defect rolling elements were manually damaged to create a test dataset. These parts were not used during training of the neural network. To test the system on numerous defects, each rolling element was damaged with an average of 5 defects. After scanning the parts, this resulted in a total test-set of 50 defect image regions. The system was then evaluated on how many of these defects were correctly found on the 10 test images. If the system's performance would be rated on a per part basis, it would suffice to only detect one of multiple defects on a given part. Even though the other defects could be missed, the resulting  
Figure 7: A flowchart of the inference procedure. A single scan at a time is preprocessed and fed through the neural network. The resulting segmentation is filtered and binarized via a threshold. The maximum in the resulting binary image, either one or zero, is the final classification. classification is correct. This would inflate the system's accuracy, as these false negatives are not factored into the resulting score. Instead, measuring the system's accuracy on individual defects within an image prevents this. Since there are more defects than parts, this also allows for a much finer accuracy measurement. In practice, the predicted binary segmentations are evaluated instead of the final binary classification of the part. The former highlights individual defects, while the latter only classifies the whole part. However, during the system's later deployment only the binary classifications of the individual images are of interest. To measure the accuracy of the network predictions, they were compared to manually created segmentations. The system was evaluated using an intersection over union (IoU) score between the labels and the network predictions, as is standard in image segmentation problems in literature [11]. IoU is a normalized measure of the overlap between two segmentation masks. It is defined as:  
\[\text{IoU}(y,\;\hat{y})=\frac{\left|y\cap\;\hat{y}\right|}{\left|y\cup\;\hat{y }\right|} \tag{3}\]  
where \(\hat{y}\) is the network prediction and \(y\) is the manually created target segmentation. Figure 8 visualizes the measures used in the IoU score. The pixels highlighted in green in the rightmost image are in the intersection of both binary masks to the left. Together with the pixels highlighted in red and yellow, they form the union of both masks. Evaluated on the test set, the system achieved an IoU score of 98.0%. Neither the manually drawn masks, nor the predicted masks can be expected to pixel perfectly overlay the defect. Achieving higher scores through further parameter tweaking thus seems unlikely. Big defect areas that are correctly classified contribute the most to a high IoU score. At the same time, these defects are easiest to detect. Due to this, we further measured the detection rates of defects. If the system detected at least one NOK pixel in a defect, the defect is counted as detected. It is crucial to detect all defects on a part in the given setting. Over the whole test dataset, the evaluation resulted in the scores listed in Table 2 for detection of single defects. Here, false negatives (FN) count the number of defects missed by the system. False positives (FP) count the number of wrong detection of defects, where the surface is actually intact. The true positives (TP) count the number of correctly detected defects. Additionally, the listed counts are combined into an \(F_{1}\) score, according to the following equation:  
\[F_{1}=\frac{2 TP}{2 TP+FP+FN}, \tag{4}\]  
Figure 8: (**a**) An excerpt of a rolling element scan with a defect. (**b**) The binarized network segmentation prediction corresponding to the excerpt in (**a**). (**c**) The manually created label corresponding to the excerpt in (**a**). (**d**) Both binary masks in the left images are overlaid on top of the original input image.