## 3 Results  
The principal objective of the present work is the generation of sentiment classification models learned from annotated vaccine-related tweets. Six different supervised learning methods were used for this task: C5.0, Logit Boost, Bayes GLM, Multilayer Perceptron, Random Forests and SVM. Each algorithm was run in each of the sampling subsets (non-sampling, down-sampling and up-sampling) and considering (i) the different numbers of evaluators (3 or 5) and (ii) the distinct input predictors (original, adapted or both). In the random modalities of the up-sampling and down-sampling, the techniques were executed a total of 10 times to analyze the independent results to guarantee their significance. The full tables with all the mean values and standard deviations of the ROC curves derived from 10-fold cross validation of the previous analysis are included in Supplementary Materials ([[https://medal.ctb.upm.es/internal/gitlab/mavis/mavis/blob/master/SA_ASC/MAVIS_tables_by_sampling.xlsx](https://medal.ctb.upm.es/internal/gitlab/mavis/mavis/blob/master/SA_ASC/MAVIS_tables_by_sampling.xlsx)]([https://medal.ctb.upm.es/internal/gitlab/mavis/mavis/blob/master/SA_ASC/MAVIS_tables_by_sampling.xlsx](https://medal.ctb.upm.es/internal/gitlab/mavis/mavis/blob/master/SA_ASC/MAVIS_tables_by_sampling.xlsx))). In the provided.xlsx file, each sheet corresponds to each sampling method (\"NO sampling\", \"DOWN-random\", \"DOWN-clustering\", \"UP-random\", \"UP-smote\" and \"UP-ADASYN\"). All the results are shown for every Machine Learning (ML) method and every combination of the number of evaluators and predictors. For the highest values of ROC curves' mean in each sampling subset and obtained by each ML method, results have been visualized ([[https://medal.ctb.upm.es/internal/gitlab/mavis/mavis/blob/master/SA_ASC/MAVIS_SA_best_results.xlsx](https://medal.ctb.upm.es/internal/gitlab/mavis/mavis/blob/master/SA_ASC/MAVIS_SA_best_results.xlsx)]([https://medal.ctb.upm.es/internal/gitlab/mavis/mavis/blob/master/SA_ASC/MAVIS_SA_best_results.xlsx](https://medal.ctb.upm.es/internal/gitlab/mavis/mavis/blob/master/SA_ASC/MAVIS_SA_best_results.xlsx))). We have represented three figures: Figure 1 represents the highest accuracy of the models in the initial subset without performing sampling, Figure 2 in the two under-sampling subsets and Figure 3 in the three over-sampling subsets. The colors of the bars stand for the different classification methods and the textures represent the input predictor from which the model has been generated. All the best generated models have been obtained from either the original predictor (\"ORIG\") or the combination of the original and the adapted (\"BOTH\"), but none of the best models in any subset have been generated from the adapted predictor (\"ADAP\"). There were two cases (1 for non-sampling subset and 1 for up-sampling) in which the highest mean of ROC values was the same for both predictors. In those cases, for simplicity, the predictor has been represented as the original one. The width of the bars in Figure 1 represents the number of evaluators, being 3 in the thinner bars and 5 in the larger one. For the other two figures, all the bars are representing models coming from three evaluators annotations. There were five cases (1 for non-sampling subset, 2 for down-clustering and 2 for up-sampling) in which the highest mean of ROC values was the same for both numbers of evaluators. For the sake of clarity in those cases, the number of evaluators has been set to three. Overall, the result of averaging the accuracy mean of the generated models with the ML methods in the different sampling subsets, led to the results presented in Table 1. In this table, the mean values of the different ML methods' accuracies from each of the sets have been represented. Results are displayed ranking the applied ML tools from left to right so that the higher global accuracy averages are shown in the left while the lower ones appear to the right.