Sanremo's winner is... Category-driven Selection Strategies for Active Learning
Authors: Anne-Lyse Minard, Manuela Speranza, Mohammed R. H. Qwaider, Bernardo Magnini (Fondazione Bruno Kessler, 2017)
Topic: Sentiment, Emotion, Irony, Hate

Abstract
This paper compares Active Learning selection strategies for sentiment analysis of Twitter data. We focus mainly on category-driven strategies, which select training instances taking into consideration the confidence of the system as well as the category of the tweet (e.g. positive or negative). We show that this combination is particularly effective when the performance of the system is unbalanced over the different categories. This work was conducted in the framework of automatically ranking the songs of ``Festival di Sanremo 2017'' based on sentiment analysis of the tweets posted during the contest.

Introduction
Active Learning (AL) is a well known technique for the selection of training samples to be annotated by a human when developing a supervised machine learning system. AL allows for the collection of more useful training data, while at the same time reducing the annotation effort AUTHOR. In the AL framework samples are usually selected according to several criteria, such as informativeness, representativeness, and diversity AUTHOR. This paper investigates AL selection strategies that consider the categories the current classifier assigns to samples, combined with the confidence of the classifier on the same samples. We are interested in understanding whether these strategies are effective, particularly when category distribution and category performance are unbalanced. By comparing several options, we show that selecting low confidence samples of the category with the highest performance is a better strategy than selecting high confidence samples of the category with the lowest performance. The context of our study is the development of a sentiment analysis system that classifies tweets in Italian. We used the system to automatically rank the songs of Sanremo 2017 based on the sentiment of the tweets posted during the contest. The paper is structured as follows. In Section sec:relatedWork we give an overview of the state-of-the-art in selection strategies for AL. Then we present our experimental setting (Section sec:baseline) before detailing the tested selection strategies (Section sec:selectionStrat). Finally, we describe the results of our experiment in Section sec:evaluation and the application of the system to ranking Sanremo's songs in Section sec:sanremo.

Conclusion
We have presented a comparative study of three AL selection strategies. We have shown that a strategy that takes into account both the automatically assigned category and the system's confidence performs well in the case of unbalanced performance over the different classes. To complete our study it would be interesting to perform further experiments on other multi-classification problems. Unfortunately this work required intensive annotation work and so its replication on other tasks would be very expensive. A lot of work on Active Learning has been done using existing annotated corpora, but we think that it is too far from a real annotation situation as the datasets used are generally limited in tems of size. In order to test different selection strategies, we have evaluated the sentiment analysis system against a gold standard, but we have also performed an application-oriented evaluation by ranking the songs participating in Sanremo 2017. As future work, we want to explore the possibility of automatically adapting the selection strategies while annotating. For example, if the performance of the classifier of one class is low, the strategy in use could be changed in order to select the samples needed to improve on that class.