Is This an Effective Way to Annotate Irony Activators?
Authors: Alessandra Teresa Cignarella, Manuela Sanguinetti, Cristina Bosco, Paolo Rosso (Università di Torino, Universitat Politècnica de València, 2019)
Topic: Sentiment, Emotion, Irony, Hate

Abstract
In this article, we describe the first steps of the annotation process of specific irony activators in TWITTIRò-UD, a treebank of Italian tweets annotated with fine-grained labels for irony on one hand, and according to the Universal Dependencies scheme on the other. We discuss in particular the annotation scheme adopted to identify irony activators and some of the issues that emerged during the first annotation phase. This helped us in the design of the guidelines and allowed us to draw future research directions.

Introduction
In the last decade, several efforts have been devoted to address the challenges of sentiment analysis and related tasks, working mainly in English and other languages such as Italian, Spanish, or French. Provided that most of the existing approaches in NLP are based on supervised semantic shallow analysis and machine learning techniques, there has been a strong push towards the development of resources from where related knowledge can be learned. In particular, the detection of irony is among the tasks currently considered as especially challenging since its presence in a text can reverse the polarity of the opinion expressed, that is using positive words for intending a negative meaning or – less often – the other way around. This can significantly undermine systems’ accuracy and makes it crucial to develop irony-aware systems (Bosco et al., 2013; Reyes et al., 2013; Riloff et al., 2013; Wang, 2013; Barbieri et al., 2014; Joshi et al., 2015; Hernández Farías et al., 2015; Hernández Farías et al., 2016). Additionally, the challenge is further complicated when there is a co-occurrence with sarcasm or satire (Hernández Farías and Rosso, 2016; Joshi et al., 2017; Ravi and Ravi, 2017). The growing interest in irony detection is also attested by the proposal of shared tasks focusing on this topic within NLP evaluation campaigns. For instance, the pilot task on irony detection proposed for Italian in SENTIPOLC at EVALITA, in 2014 and 2016 (Barbieri et al., 2016; Basile et al., 2014), and the related task proposed for French at DEFT at TALN 2017 (Benamara et al., 2017). For what concerns English, after a first task at SemEval-2015 focusing on figurative language in Twitter (Ghosh et al., 2015), a shared task on irony detection in tweets has been proposed in 2018 (Van Hee et al., 2018). Concerning Spanish, the most recent shared task about irony in social media has been organized at IberLEF 2019 Irony Detection in Spanish Variants (IroSvA 2019), exploring the differences among varieties of Spanish from Spain, Cuba and Mexico (Ortega et al., 2019) in which the organizers also proposed a focus on context, stressing the importance of contextual semantics in ironic productions. While the majority of the participating systems in the above-mentioned shared-tasks are based on classical machine learning techniques (Cignarella and Bosco, 2019; Frenda and Patti, 2019), researchers have recently started to exploit approaches based on neural networks. Among these, Huang et al. (2017) applied attentive recurrent neural networks (RNNs) that capture specific words which are helpful in detecting the presence of irony in a tweet, while Wu et al. (2018) exploited densely connected LSTMs in a multitask learning strategy, adding PoS tag features, and Zhang et al. (2019) took advantage of recent advancements in transfer learning techniques. These settings are a clear indication of the growing interest for a deeper analysis of the linguistic phenomena underlying ironic expressions. Such kind of analysis naturally calls for the exploitation of finer-grained features and resources in order to improve the performance of automatic systems. For instance, an especially fine-grained annotation format for irony is the one proposed in Karoui et al. (2017), concerning French, Italian and English. The same scheme has later been applied on a new Italian corpus: TWITTIRò (Cignarella et al., 2018a). The resulting annotated corpus was used as reference dataset in the IronITA 2018 shared task on Irony and Sarcasm Detection in Italian Tweets (Cignarella et al., 2018b).

Conclusion
In this article we described the preliminary steps of the annotation process of irony activators in the TWITTIRò-UD corpus, a novel Italian treebank of ironic tweets. In particular, we described the problems that emerged during the first annotation phase, the strengths and weaknesses of the scheme itself, in order to highlight future research directions. Being a preliminary study, and having no benchmark to compare with, the results obtained in the observed agreement are rather promising; moreover, the tweets included in TWITTIRò were retrieved from different pre-existing Italian corpora (as described in Cignarella et al. (2017)): the heterogeneous sources the data were gathered from thus represents a signal of the potential portability of the scheme and paves the way for a more systematic annotation process of the whole dataset. The next steps will then consist in the guidelines improvement and the annotation of the remaining part of TWITTIRò-UD accordingly. Furthermore, the availability of English and French datasets annotated with the same scheme described in Section 2.1 (see Karoui et al. (2017)) allows the direct applicability of the annotation of irony activators in other languages than Italian. While this can be considered a further validation step to test the overall validity and portability of the scheme, it may also provide useful insights into the linguistic mechanisms underlying verbal irony in different languages. The actual usability of this kind of resources will be finally tested when training NLP tools for irony detection, in both mono- and multi-lingual settings.