# Automating the Analysis of Negative Test Verdicts:
## 3 Results and Discussion
### Cross-Validation  
After fitting the model to the training data, it is imperative to verify whether the trained model also performs adequately when exposed to real-world data. It is essential to confirm that the model is well acquainted with patterns within the data and does not exhibit excessive noise. The performance of the machine learning model was assessed through cross-validation. This involved training the model on a subset of input data and testing it on an unseen dataset [34]. A cross-validation of the investigated algorithms was performed using the entire dataset used in previous studies: a real embedded test dataset as the train data, and the Test Data representing new, unseen data collected in the form of logs from the real test system while performing a distinct test case from the preceding cases. In this case, the training dataset contains 231,880 samples (including 24,081 outliers, which is 10%), while the test dataset contains 146,178 samples (including 12,040 outliers, which is 10% as well). The results of the algorithm evaluation are presented in the form of quality indicators in Tables 15 and 16, MSE and MSLE in Tables 17 and 18, and the confusion matrices in Tables 19 and 20. MLP, SVM, XGB+, LGB and CatB+ algorithms sustained high performance on the test data, achieving high scores in ROC AUC, precision, R2 score, MCC, and balanced accuracy (BA). Although there were slight performance decreases compared to the training set. ResNet performance remained non-optimal on the test set, highlighting the challenges in generalization and capturing the data nuances. FTTransformer showed contrasting mixed results, with improvements in some metrics, but still lagged behind other algorithms. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Algorithm** & **Roc\_Auc\_Score** & **Precision** & **R2\_Score** & **MCC** & **BA** \\ \hline MLP & 0.9998 & 0.9787 & 0.958 & 0.981 & 0.984 \\ SVM & 1.0000 & 0.9949 & 0.991 & 0.994 & 0.997 \\ XGB+ & 1.0000 & 1.0000 & 0.999 & 1.000 & 1.000 \\ LGB & 0.9996 & 0.9603 & 0.946 & 0.958 & 0.964 \\ CatB+ & 0.9997 & 0.9734 & 0.958 & 0.969 & 0.993 \\ ResNEt & 0.474 & 0.0 & \(-\)0.12 & \(-\)0.07 & 0.499 \\ FTTransformer & 0.6688 & 0.25 & \(-\)0.408 & 0.098 & 0.537 \\ \hline \hline \end{tabular}
\end{table} 
Table 15: Results of cross-validation: algorithm’s performance metrics on train data. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Algorithm** & **MSE** & **MSLE** \\ \hline MLP & 0.004 & 0.002 \\ SVM & 0.0 & 0.0 \\ XGB+ & 0.0 & 0.0 \\ LGB & 0.005 & 0.002 \\ CatB+ & 0.004 & 0.002 \\ ResNet & 0.10 & 0.05 \\ FTTransformer & 0.13 & 0.068 \\ \hline \hline \end{tabular}
\end{table} 
Table 16: Results of cross-validation: algorithm’s performance metrics on test data. MLP, SVM, XGB+, LGB, and CatB+ consistently performed well in terms of regression metrics on both training and test datasets, suggesting their reliability for regression tasks. The choice of algorithm may depend on the specific requirements of the application, considering the trade-off between computational efficiency and the magnitude of regression errors. MLP, SVM, XGB, LGB, and CatB+ algorithms demonstrated strong performance on the train data, with high true positive (Tp) counts and relatively low false positive (Fp) and false negative (Fn) counts. SVM and XGB stood out with perfect true negative (Tn) counts, indicating precise classification of negative instances. ResNet and FTTransformer showed distinctive characteristics. ResNet had a perfect true positive count but misclassified all negative instances, resulting in high false negative and low true negative counts. FTTransformer exhibited a higher false positive count, indicating challenges in distinguishing negative instances. MLP, SVM, XGB, LGB, and CatB+ algorithms continued to demonstrate good performance on the test data, with high Tp counts and relatively low Fp and Fn counts. ResNet performed poorly on the test set, misclassifying all positive instances, leading to high false negative counts.