## 2 Materials and Methods
### Different Datasets Creation  
To get the different datasets that would be input to the machine learning methods to generate the classification models, a process to annotate the tweets was implemented. Such annotations were performed in two ways: (1) using different commercial tools and (2) being manually revised by expert evaluators.  
* Annotation with commercial toolsThree tools were chosen to automatically annotate the whole set of tweets and quotations: IBM Watson ([[https://www.ibm.com/watson/services/tone-analyzer/](https://www.ibm.com/watson/services/tone-analyzer/)]([https://www.ibm.com/watson/services/tone-analyzer/](https://www.ibm.com/watson/services/tone-analyzer/))) (now called Watson Tone Analyzer), Google Cloud Natural Language ([[https://cloud.google.com/natural-language](https://cloud.google.com/natural-language)]([https://cloud.google.com/natural-language](https://cloud.google.com/natural-language))) and Meaning Cloud ([[https://www.meaningcloud.com/es](https://www.meaningcloud.com/es)]([https://www.meaningcloud.com/es](https://www.meaningcloud.com/es))). The three of them returned different formats of the polarity of tweets. IBM and Google returned numerical values (a score between \(-1\) and \(1\)) while Meaning Cloud returned a class (a class value between \(6\) classes: P+, P, NEU, N, N+ and NONE). The analysis was simplified so that tweets were discretized to either \"negative\" or \"non-negative\". This way, \"non-negative\" would include other classes such as \"neutral\". Models were generated considering (i) the original values without discretizing (\"original\"), (ii) the adapted discretized values (\"adapted\") and (iii) both of them (\"both\").  
* Manual annotation by experts  
Five experts in the field classified the tweets' polarity manually, by determining if each tweet contained a \"negative\" or a \"non-negative opinion\". An iterative process of annotation was cyclically performed three times: in each iteration a set of 100 tweets were annotated classifying the sentiment expressed in tweets. A total of 300 tweets were annotated. From those, a very low number of tweets were identified as negative, leading to a class imbalance challenge that could impact on the quality of the models that would be later generated.  
To increase the number of negative tweets, a sample that contained words with negative sentiments was extracted from the original dataset. Such words were selected from Meaning Cloud platform, since it allows retrieving the polarity of the words extracted in the sentiment analysis process. A subset of 459 tweets containing words from this list was obtained, ensuring they had not been previously selected for the first three iterations. Therefore, the total number of tweets that were classified by the five evaluators amounted to 759.  
Analyzing the resulting manually annotated dataset, it was found that two of the experts had a high grade of disagreement with the other three evaluators. Annotations of those three and of the total five evaluators were compared to the annotations performed by the commercial tools. The level of agreement was higher between the three evaluators and the commercial tools than between the five evaluators and the commercial tools. For the three evaluators, there were 142 tweets classified as negative and 617 as non-negative; and, for the three evaluators, there were 128 tweets annotated as negative and 631 as non-negative. Nevertheless, as will be stated in the next subsection, models were generated with both the three and the five expert's annotations, as the difference between them was sparse and the five evaluators scenario increased the number of negative tweets.