# Automating the Analysis of Negative Test Verdicts:
## 3 Results and Discussion
### Experimental Results
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithm** & **Roc\_Auc\_Score** & **Precision** & **MSE** \\ \hline CatB+ & 1.0 & 0.9985 & 0 \\ LGB & 1.0 & 0.9984 & 0 \\ MLP & 0.9955 & 0.9855 & 0.002 \\ SVM & 0.9915 & 0.984 & 0.002 \\ XGB+ & 1.0 & 1.0 & 0 \\ ResNet & 0.9959 & 0.9855 & 0.001 \\ FTTransformer & 1.0 & 0.9971 & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Results of PyOD supervised algorithms on test data (ODDS dataset). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithm** & **Roc\_Auc\_Score** & **Precision** & **MSE** \\ \hline CatB+ & 0.9998 & 0.9777 & 0.003 \\ LGB & 0.9997 & 0.9709 & 0.004 \\ MLP & 0.9983 & 0.9256 & 0.04 \\ SVM & 1.0 & 0.9944 & 0 \\ XGB+ & 1.0 & 1.0 & 0 \\ ResNet & 0.5357 & 0.1551 & 0.11 \\ FTTransformer & 0.8935 & 0.3405 & 0.11 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Results of PyOD supervised algorithms on train data (real embedded test dataset). \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Algorithm** & **Roc\_Auc\_Score** & **Precision** & **MSE** \\ \hline CatB+ & 0.7796 & 0.1314 & 0.06 \\ LGB & 0.7907 & 0.1203 & 0.06 \\ MLP & 1.0 & 0.9917 & 0.02 \\ SVM & 1.0 & 1.0 & 0.01 \\ XGB+ & 0.7602 & 0 & 0.06 \\ ResNet & 1.0 & 0.9977 & 0.04 \\ FTTransformer & 0.9957 & 0.985 & 0.06 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Results of PyOD supervised algorithms on test data (real embedded test dataset). and test data, reflecting the ability of the models to correctly identify positive instances. SVM, XGB+, and CatB+ algorithms stand out with precision values approaching or equal to 1.0000 on the test data. R2 scores, indicating the percentage of variance explained by the models, are generally high for all algorithms on the test data. XGB+ demonstrates perfect R2 scores, suggesting an excellent fit to the data. The MCC values, which measure the quality of binary classifications, are consistently high across algorithms and datasets. The XGB+ and SVM algorithms show particularly high MCC scores, indicating reliable classification performance. Balanced accuracy scores are consistently high, indicating a well-balanced performance between sensitivity and specificity for all algorithms. The XGB+ and SVM algorithms show the highest balanced accuracy scores on the test data. The performance of various algorithms is assessed through both Mean Squared Error (MSE) and Mean Squared Logarithmic Error (MSLE), which is presented in Tables 13 and 14. Each algorithm's ability to minimize these metrics provides insights into their effectiveness in capturing the underlying patterns within the dataset. Multilayer Perceptron (MLP) demonstrates competitive performance, with an MSE of 0.0082 and an MSLE of 0.0035. These values indicate reasonable precision of the model, although further optimization may be explored to potentially enhance its performance. Support Vector Machine (SVM) outperforms other models in terms of precision, exhibiting the lowest MSE (0.0008) and MSLE (0.0004). This suggests that SVM is highly effective in minimizing errors and capturing variability within the data. Ensemble models, represented by XGB+, LGB, and CatB+, consistently outperform individual models, presenting MSE and MSLE values that are significantly lower than the MLP. This highlights the effectiveness of ensemble methods in improving predictive accuracy. The lower MSLE values across all algorithms compared to MSE values suggest that the models perform well across different magnitudes of predictions. This is particularly important in scenarios where predictions cover a wide range of values. The varying performance of different algorithms indicates sensitivity to algorithmic choices. Understanding the strengths and weaknesses of each algorithm is crucial for selecting the most suitable model based on the specific requirements of the task. In summary, the evaluated algorithms consistently perform well across a range of metrics, showcasing their effectiveness in handling the classification task.