# A Dataset of Pulsed Thermography for Automated Defect Depth Estimation
## 4 Data Labeling
### Evaluation of the Labeling Quality  
Qualitative evaluation: Figure 6 presents the overlaid images for a qualitative comparison of the labeling results obtained through both methods. For rectangular defects, the labeled masks were largely similar, with minor discrepancies observed in the manually labeled images. However, for round defects, a significant difference was noted between the masks generated by the two methods. The manual approach tended to generate larger masks, likely due to the difficulty in identifying precise reference points for round defects, as opposed to the more-straightforward four-corner identification in rectangles. This often resulted in the inclusion of adjacent regions in the mask, enlarging it beyond the actual size of the defect. Figure 5: PCT images (**top**) generated from 3D files and the depth images (**bottom**). While qualitative comparisons can be intuitively made, achieving an objective evaluation of the labeling quality is not straightforward. The actual defects lie beneath the surface of the specimen, making direct observation of their precise locations in the thermal imagery challenging. To overcome this, a reference distance was established to correlate the image dimensions with real-world measurements. The dimension of the holder of the measurement can be derived directly from the CAD file, as shown in Figure 7. The measured distance on the upper part is 96 mm, corresponding to a pixel distance of 185 in the image. Equation (6) defines a linear transformation. Therefore, the distance between points A and B in the image can be used as a reference for evaluating the labeled sizes. To quantify the accuracy of defect size labeling, the Mean Absolute Error (MAE) and Mean Squared Error (MSE) were used. The MAE can be calculated using the following formula:  
\[MAE=\frac{\sum_{i}^{n}|x_{i}-\hat{x_{i}}|}{n} \tag{5}\]  
Here, \(x_{i}\) represents the size of the region in the labeled image and \(\hat{x_{i}}\) represents the reference size of the same region derived from the reference distance and the 3D-CAD file. The MAE is a widely used metric for evaluating the model's performance, which is robust to outliers, particularly effective in scenarios with numerous outliers. This metric exhibits less sensitivity to large errors. Nevertheless, large errors are critically informative in this context. The MSE becomes relevant due to its characteristic of penalizing larger errors strongly. It can be calculated using the following formula:  
\[MSE=\frac{\sum_{i}^{n}(x_{i}-\hat{x_{i}})^{2}}{n} \tag{6}\]  
Table 1 provides the results of the objective labeling quality evaluation for both the manual labeling method and the CAD-based labeling method. To be specific, labels for defects in varying dimensions were evaluated. The dimensio  
Figure 6: Comparison of mask images labeled by the manual method and the 3D-CAD-based approach, where the green masks are labeled by the manual method and the red masks are labeled by the 3D CAD-based method. Yellow areas highlight the overlap between these two masks. Figure 7: Reference distance from point A to point B. The left image shows this distance on the CAD image. The right image illustrates the corresponding distance on the PCT image. from 2 mm to 10 mm for round defects and equivalent side lengths for square defects. On the one hand, the area for each defect was calculated in pixels based on the labeling data. On the other hand, each area was also calculated by employing the relative diameter or side length, which was derived using the reference length measured between points A and B. The results, obtained using the MAE and MSE, clearly demonstrated that the CAD-based method significantly outperformed the manual labeling method, as evidenced by the reduced magnitude of the errors observed. This suggests that relying solely on labeling methods based on PCT images, which is currently the most-popular approach, may frequently result in inadequate levels of accuracy.