## 2 Methodology
### Evaluation Methods  
After training the segmentation model, the performance needs to be analysed. Although machine learning technology is advancing, every prediction result has errors. There are some types of evaluation methods that can used to perform a quantitative error consideration of the trained model. A better model can be attained only if the error analysis is continuously combined with the adjustment of the model parameters. In the field of machine learning and statistical classification problems, the confusion matrix is a visualization tool especially for supervised learning. When evaluating standard machine learning models, the confusion matrix typically is used to divide predictions into four categories: true positives (\(TP\)), false positives (\(FP\)), true negatives (\(TN\)) and false negatives (\(FN\)). Taking the segmentation result 1 from Section 4.1, Table 1 results as an example of a confusion matrix. With this matrix, it is easy to see if the machine is confusing different classes. Each row of the confusion matrix represents the predicted category and each column represents the true category of the data. Using these categories, the Intersection over Union (IoU) can be calculated as the overlap rate between the prediction area generated by the model and the area of ground truth. This overlap rate can comprehensively account for the correctly predicted and incorrectly predicted pixels in the prediction results, which fully shows the credibility of the predicted results. It is therefore often used as an important evaluation criterion in the \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **Seam Shape Defect** & **Background** & **Pores** & **Cracks** & **Weld Metal** \\ \hline Seam shape defect & 601.17 pixels & 7327 pixels & 446 pixels & 13,065 pixels & 170 pixels \\ \hline Background & 4415 pixels & 1,285,278 pixels & 0 pixels & 2 pixels & 1009 pixels \\ \hline Pores & 1793 pixels & 41 pixels & 218 pixels & 1541 pixels & 94 pixels \\ \hline Cracks & 5413 pixels & 21 pixels & 3011 pixels & 15,000 pixels & 8 pixels \\ \hline Weld metal & 878 pixels & 14,913 pixels & 0 pixels & 155 pixels & 67,812 pixels \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Confusion matrix example for segmentation result 1; see Section 4.1. \(TP=\) blue, \(FP=\) yellow, \(FN=\) green and \(TN=\) red. Figure 3: Ambiguous defects between weld metal and background (**a**), between porosity and cracks and seam shape defects and background (**b**). field of image semantic segmentation. The model can be evaluated using the following equations [16; 17]:  
\[Accuracy\ =\ \frac{TP+TN}{FP+TP+FN+TN} \tag{1}\]  
\[Precision\ =\ \frac{TP}{FP+TP} \tag{2}\]  
\[IoU\ =\ \frac{TP}{FP+TP+FN} \tag{3}\]  
As there are different methods used to evaluate the model, the results can be seen in Table 2. Equation (1) calculates the ratio of all correctly predicted pixels to all pixels. Equation (2) calculates the probability of being correct among all outcomes predicted for a specific class. Equation (3) calculates the ratio of the intersection of the predicted result and the ground truth to the union of the predicted result and the ground truth for a specific class, but it only considers all cases related to a specific class (TP, FP and FN), and it does not consider the positive effects brought by other classes (TN). As shown in Table 1 with colored cells (TP = blue, FP = yellow, FN = green and TN = red), the use of Equation (3) only considers the results in relation to a particular class. In order to evaluate the welds, the calculation method of Equation (3) is chosen. An example is given in Figure 4 as a visual representation. It shows the ground truth and the predicted area as well as the intersection and the union between these two areas. As multiple images are considered in general, the IoU of each image in the test set is averaged. Therefore, the \"mean IoU\" (\(mIoU\)) is defined as shown in Equation (4).