## 4 Discussion  
The model that provided the highest accuracy from all the studied possibilities was the one generated by the subset obtained from up-sampling with the ADASYN method and corresponding to the Random Forest technique. Such a data subset was formed by the original values of the commercial tools and was annotated by three evaluators.  
When no sampling was performed, the accuracy highest values ranged from 0.52 to 0.72. Such accuracies were not much improved when under-sampling (0.49 to 0.72). There was a trend in the accuracy to present larger values when the dataset was balanced via over-sampling the minority class (0.66 to 0.9). The three highest values of the accuracy were 0.9, 0.87 and 0.87, obtained by the three up-sampling methods (Random Forests with ADASYN and C5.0 with SMOTE obtained a mean of ROC values of 0.9; Random Forests with SMOTE got 0.89; and C5.0 with ADASYN got 0.87).  
Most of the best results were performed when the dataset was annotated manually by the three evaluators. When the annotation was carried out by the five experts the accuracy mainly decreased. There is just one case when the five evaluators' annotations worked better (SVM without sampling) and five cases when the accuracy was equal to the one obtained by the three evaluators' annotations. Nevertheless, still in those situations, the accuracy was not very high (0.49, 0.52, 0.60, 0.64, 0.66 and 0.76). This can be explained by the disagreement of the three experts versus the five. When considering the five experts, as there were two of them who did not agree with the other three, the dataset increased  
\begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **Random** & **Multilayer** & & & & \\  & **Forests** & **Perceptron** & **C5.0** & **Logit Boost** & **BayesGLM** & **SVM** \\ \hline No sampling & 0.7 & 0.72 & 0.64 & 0.64 & 0.7 & 0.52 \\ Down-sampling & 0.665 & 0.695 & 0.64 & 0.63 & 0.665 & 0.59 \\ Up-sampling & 0.89 & 0.78 & 0.84 & 0.79 & 0.71 & 0.7 \\
**Global average** & **0.75** & **0.73** & **0.71** & **0.69** & **0.69** & **0.6** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Mean accuracy values for the different sampling subsets, corresponding to each ML method. ML tools are shown from left to right according to higher values of the global average accuracy.  
Figure 3: Highest mean accuracy values in up-sampling subset, corresponding to each ML method. Each color bar represents the different classification models. The texture of the bar represents the input predictor from which each model has been generated. The three different up-sampling methods are represented grouped in the X axis.  
its noise levels, and therefore, the accuracy would tend to be lower. On the other hand, adapted values as input to generate models, did not perform as well as the original or the combined ones.  
Random Forests was one of the models that obtained better accuracy values for the different sampled subsets (0.7 when no sampling, 0.61 for down-clustering, 0.72 for down-random, 0.9 for ADASYN, 0.89 for SMOTE and 0.87 for up-random). On the contrary, SVM usually got the lower values (0.52 when no sampling, 0.49 for down-clustering, 0.69 for down-random, 0.73 for ADASYN, 0.7 for SMOTE and 0.66 for up-random). The Multilayer Perceptron and C5.0 also obtained high accuracies (the first one when no sampling and for down-clustering, and the second one for SMOTE).  
There was a trend in the global accuracy average from the different ML methods: while Random Forests performed the best when up-sampling and regarding the overall accuracy, the Multilayer Perceptron showed the highest values of ROC means when no sampling and down-sampling. Analogously, whilst LogitBoost performed better than BayesGLM when up-sampling, BayesGLM obtained higher results in conditions of no sampling and down-sampling. The same global accuracy average was derived in both cases.