# Automating the Analysis of Negative Test Verdicts:
## 3 Results and Discussion
### Experimental Results  
The first stage of the study focuses on the analysis of unsupervised algorithms. In the first step, to validate their performance and assess their quality, the algorithms are tested on the ODDS dataset. Results of this stage are presented in Tables 1 and 2. In general, the evaluation of multiple anomaly detection algorithms on both training and test datasets reveals varying degrees of effectiveness across different metrics. The results highlight the diversity in performance, emphasizing the importance of selecting an algorithm based on specific priorities and the nature of the data. It is noteworthy that the Isolation Forest (IForest) algorithm stands out for its robust performance, consistently achieving high Roc_Auc_Score, precision and low mean square error (MSE) scores. Other algorithms, such as ECOD, PCA and INNE, also show commendable performance across multiple metrics. The next stage of the experimentation is to implement the aforementioned algorithms using the best configured hyperparameter settings on a real embedded test dataset. Results are shown in Tables 3 and 4. The obtained results can disappointingly be described as unsatisfactory. Modifications to the hyperparameters did not result in an improved performance of anomaly detection in the logs from the real embedded systems tested. It is decided to return to the first stage of the study to carry it out for supervised algorithms. Results of this stage are presented in Tables 5 and 6. The obtained excellent ROC AUC results (1.0) with zero mean squared error for the CatB+, LGB, XGB+, ResNet and FTTransformer algorithms demonstrate the high performance of anomaly detection in the tested ODDS dataset for supervised algorithms. An evaluation of these algorithms is also performed for test data obtained from real embedded systems. Results are presented in Tables 7 and 8. To summarize the obtained results, some algorithms perform exceptionally well on the training set, but there are some indications of potential over-fitting (LGB, CatB+) considering the performance of these algorithms on the test data. The decrease in performance observed for several models when moving from the learning set to the test set suggests that some models may have difficulties with generalizing to the new, unseen data. Mean squared error (MSE) values provide insight into the accuracy of regression predictions. In general, low MSE values indicate accurate predictions, and values are relatively consistent across algorithms. The SVM algorithm consistently achieves excellent performance in both Roc_Auc_Score and precision on both the training and test datasets, implying robust generalization capabilities. MSE values are consistently low, indicating accurate predictions. Similar results are reported for the MLP algorithm, but its lower performance on the training set suggests potential over-fitting or high sensitivity to the training data. In the next step of this study, the K-fold cross-validation technique is employed, entailing the partitioning of the learning dataset into K equally sized segments. K-1 subsets of data are utilized for model training (train data), while the remaining subset serves for validation (test data). This technique involves K iterations, each time reserving a specified subset for validation. Exclusion of training samples from those used to evaluate candidate parameter values reduces the likelihood of overfitting, thereby enhancing the generalization of the classifier [33]. The real embedded test dataset is divided into 10 subsets, and the model training and testing processes are repeated 10 times. Subsequently, the average values of the parameters determining the model's quality are computed. The dataset contains 231,880 samples.