SlowGuess/ACL_2020_OCR huggingface dataset:
Among the contributors who filled our survey and provided the tweets,  $56\%$  are from the UK and  $41\%$  from the US, while  $3\%$  are from other countries such as Canada and Australia.  $51\%$  are females, and over  $72\%$  are less than 35 years old. Figure 2 shows the age and gender distributions across contributors.
In iSarcasm, we investigated the presence of the hashtags #sarcasm, #sarcastic, and others often used to mark sarcasm in previous distant supervision datasets. None of our tweets contains any of those tags, which confirms one of our discussed limitations of this approach, that the lack of tags should not be associated with lack of sarcasm, and that these tags might capture only one flavor of
sarcasm, not sarcasm present on social media in general.
Regarding the categories of sarcasm, assigned by the linguist to the sarcastic tweets, Table 3 shows the distribution of the tweets into these categories. As shown, sarcasm and irony are the largest two categories (73%), while understatement is the smallest one (with only 12 tweets). Table 4 shows examples of the sarcastic tweets, along with the explanations and rephrases provided by the authors.
iSarcasm is published as two files, a training set and a test set, containing  $80\%$  and  $20\%$  of the examples chosen at random, respectively. Each file contains tweet IDs along with corresponding intended sarcasm labels. For sarcastic tweets we also provide the category of ironic speech they be
Figure 1: Tweet length distribution across iSarcasm.
Figure 2: Age and gender distributions across the Twitter users who provided tweets in iSarcasm.
long to. This is in accordance with the consent form that the contributors have agreed to, whose privacy we take seriously. Nonetheless, we still offer the tweets text along with the explanations and rephrases of the sarcastic tweets provided by the authors for free for research purposes, under an agreement that protects the privacy of our contributors.
5.2 Third-Party Labels
As we mentioned earlier, we collected three third-party labels for each tweet in the test set of iSarcasm. Using Cohen's kappa ( $\kappa$ ; Cohen (1960)) as a measure, the pairwise inter-annotator agreement (IAA) scores were  $\kappa_{12} = 0.37$ ,  $\kappa_{13} = 0.39$  and  $\kappa_{23} = 0.36$ , which highlights the high subjectivity of the task. We used majority voting to select the final perceived sarcasm label for each tweet. Table 5 shows the disagreement between the intended and perceived labels. As shown,  $30\%$  of the sarcastic tweets were unrecognised by the annotators, while  $45\%$  of the tweets perceived as sarcastic were actually not intended to be sarcastic
perc. sarc.	perc. non-sarc.
int. sarc.	61	26
int. non-sarc.	50	322
Table 5: The agreement between intended labels (int.), provided by the authors, and perceived labels, provided by third-party annotators, (perc.) on iSarcasm test set.
by their authors. This supports our argument that third-party annotation for sarcasm should not be trusted.
6 Detecting Intended Sarcasm
In the following, we examine the effectiveness of state-of-the-art sarcasm detection models on iSarcasm. We aim to investigate their ability to detect intended sarcasm rather than sarcasm labeled using distant supervision or manual annotation. As we have shown, these labelling methods could produce noisy labels. We experiment with those models that have achieved state-of-the-art results on previous benchmark datasets for sarcasm detection.
6.1 Baseline Datasets
We consider four previously published datasets. Two of them, Riloff (Riloff et al., 2013) and SemEval-2018 (Van Hee et al., 2018), were labeled via a hybrid approach of distant supervision for initial collection and manual annotation for actual labelling. The other two datasets, Ptáček et al., 2014) and SARC (Khodak et al., 2018), are labeled using distant supervision. As mentioned earlier, we managed to collect 1,832 tweets from the Riloff dataset. SemEval-2018 is a balanced dataset consisting of 4,792 tweets. For the Ptacket dataset, we collected 27,177 tweets out of the 50K published tweet IDs. Finally, The SARC datasets consists of Reddit comments. In a setting similar to Hazarika et al. (2018) who publish state-of-the-art results on this dataset, we consider two variants of SARC. SARC-balanced contains 154,702 comments with the same number of sarcastic and non-sarcastic comments, while SARC-imbalanced contains 103,135 comments with a ratio of about 20:80 between sarcastic and non-sarcastic comments.
6.2 Sarcasm Detection Models
Riloff and Ptacek datasets We replicate the models implemented in (Tay et al., 2018), who report state-of-the-art results on Riloff and Ptacek. These models are: LSTM first encodes the
tweet with a recurrent neural network with long-term short memory units (LSTM; Hochreiter and Schmidhuber (1997)), then adds a binary softmax layer to output a probability distribution over labels (sarcastic or non-sarcastic) and assigns the most probable label. It has one hidden layer of dimension 100. Att-LSTM adds an attention mechanism on top of the LSTM, in the setting specified by Yang et al. (2016). In particular, it uses the attention mechanism introduced by Bahdanau et al. (2014) of dimension 100. CNN encodes the tweet with a convolutional neural network (CNN) with 100 filters of size 3 and provides the result to feed-forward network with a final binary softmax layer, choosing the most probable label. SIARN (Single-Dimension Intra-Attention Network; Tay et al. (2018)) is the model that yields the best published performance on the Riloff dataset. It relies on the assumption that sarcasm is caused by linguistic incongruity between words. It uses an intra-attention mechanism (Shen et al., 2018) between each pair or words to detect this incongruity. MIARN (Multi-Dimension Intra-Attention Network; Tay et al. (2018)) reports the best results on the Ptacek dataset. In addition to SIARN, MIARN allows multiple intra-attention scores for each pair of words to account for multiple possible meanings of a word when detecting incongruity. We use an implementation of MIARN similar to that described by its authors. We set the dimension of all hidden layers of SIARN and MIARN to 100.
SARC datasets Hazarika et al. (2018) report the best results on SARC-balanced and SARC-imbalanced, to our knowledge. However, they model both the content of the comments as well as contextual information available about the authors. In this paper we only focus on content modelling, using a convolutional network (3CNN) in a setting similar to what they describe. 3CNN uses three filter types of sizes 3, 4, and 5, with 100 filters for each size.
SemEval-2018 dataset The SemEval dataset contains two types of labels for each tweet: binary labels that specify whether the tweet is sarcastic or not; and labels with four possible values, specifying the type of sarcasm present<sup>4</sup>. Wu et al. (2018) report the best results on both tasks with their Dense-LSTM model. Given a tweet, the
model uses a sequence of four LSTM layers to compute a hidden vector  $H$ .  $H$  is then concatenated with a tweet embedding  $S$  computed in advance by averaging embeddings of all words inside using the pre-trained embeddings provided by Bravo-Marquez et al. (2016).  $H$  and  $S$  are further concatenated with a sentiment feature vector of the tweet computed in advance using the weka toolkit (Mohammad and Bravo-Marquez, 2017), by applying the TweetToLexiconFeatureVector (Bravo-Marquez et al., 2014) and TweetToSentiStrengthFeatureVector (Thelwall et al., 2012) filters. The authors of Dense-LSTM train the network in a multitask setting on the SemEval dataset (Van Hee et al., 2018) to predict three components: the binary sarcasm label, one of the four types of sarcasm, and the corresponding hashtag, if any, that was initially used to mark the tweet as sarcastic, out of #sarcasm, #sarcastic, #irony and #not. Wu et al. (2018) report an F-score of 0.674 using a fixed dropout rate of 0.3 in all layers. They further report an F-score of 0.705 by averaging the performance of 10 Dense-LSTM models, varying the dropout rate to random values between 0.2 and 0.4. We implement and train it to only predict the binary sarcasm label, to make it applicable to iSarcasm and make the results on SemEval-2018 and iSarcasm comparable.
For each previous dataset, we implemented the models reported previously to achieve the best performance on that dataset, and made sure our implementations achieve similar performance to the published one. This is confirmed in Table 6, providing confidence in the correctness of our implementations.
6.3 Results and Analysis
Table 7 reports precision, recall and f-score results on the test set of iSarcasm using the detection models discussed, alongside third-party annotator performance. As shown, all the models perform significantly worse than humans, who achieve an F-score of only 0.616. MIARN is the best performing model with a considerably low F-score of 0.364, compared to its performance on the Riloff and Ptacek datasets (0.741 and 0.874 F-scores respectively). 3CNN achieves the lowest performance on iSarcasm with an F-Score of 0.286 compared to 0.675 and 0.788 on SARC balanced and imbalanced, respectively. Similarly, Dense-LSTM achieves 0.318, compared to 0.666 on SemEval2018.
4We use "sarcasm" to mean what they refer to as "verbal irony".
Dataset	Model	published	our impl.
Riloff	LSTM	0.673	0.669
Att-LSTM	0.687	0.679
CNN	0.686	0.681
SIARN	0.732	0.741
MIARN	0.701	0.712
Ptacek	LSTM	0.837	0.837
Att-LSTM	0.837	0.841
CNN	0.804	0.810
SIARN	0.846	0.864
MIARN	0.860	0.874
SARC-balanced	3CNN	0.660	0.675
SARC-unbalanced	3CNN	0.780	0.788
SemEval-2018	Dense-LSTM	0.674	0.666
Table 6: F-score yielded by our implementations of state-of-the-art models on previous datasets, compared to published results on those datasets.
Model	Precision	Recall	F-score
Manual Labelling	0.550	0.701	0.616
LSTM	0.217	0.747	0.336
Att-LSTM	0.260	0.436	0.325
CNN	0.261	0.563	0.356
SIARN	0.219	0.782	0.342
MIARN	0.236	0.793	0.364
3CNN	0.250	0.333	0.286
Dense-LSTM	0.375	0.276	0.318
Table 7: Experimental results on iSarcasm. Manual Labelling shows the results using the perceived sarcasm labels provided by third-party human annotators.
Previous models that achieved high performance in detecting sarcasm on datasets sampling perceived sarcasm (third-party labels) or hash-tagged sarcasm (distant supervision) have failed dramatically to detect sarcasm as meant by its author. This motivates the need to develop more effective methods for detecting intended sarcasm. Potentially, building models that account for sociocultural traits of the authors (available on, or inferred from, their Twitter profiles), or consider other contextual elements to judge the sarcasm in our dataset (Rockwell and Theriot, 2001). Previous research has considered certain contextual elements (Bamman and Smith, 2015b; Amir et al., 2016; Hazarika et al., 2018; Oprea and Magdy, 2019), but only on sarcasm captured by previous labelling methods.
We believe the iSarcasm dataset, with its novel method of sampling sarcasm as intended by its
authors, shall revolutionise research in sarcasm detection in the future; and open the direction for new sub-tasks, such as sarcasm category prediction, and sarcasm decoding/encoding, using information found both in the tweets themselves, and in the explanations and rephrases provided by the authors, available with each sarcastic tweet in the dataset.
In this paper, we presented iSarcasm, a dataset of intended sarcasm consisting of 4,484 tweets labeled and explained by their authors, and further revised and categorised by an expert linguistic. We believe this dataset will allow future work in sarcasm detection to progress in a setting free of the noise found in existing datasets. We saw that computational models perform poorly in detecting sarcasm in the new dataset, indicating that the sarcasm detection task might be more challenging compared to how it was seen in earlier research. We aim to promote research in sarcasm detection, and to encourage future investigations into sarcasm in general and how it is perceived across cultures.
This work was supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded by the UK Engineering and Physical Sciences Research Council (grant EP/L016427/1); the University of Edinburgh; and The Financial Times.
Gavin Abercrombie and Dirk Hovy. 2016. Putting Sarcasm Detection into Context: The Effects of Class Imbalance and Manual Labelling on Supervised Machine Classification of Twitter Conversations. In Proceedings of the ACL 2016 Student Research Workshop, pages 107-113. ACL.
Silvio Amir, Byron C. Wallace, Hao Lyu, Paula Carvalho, and Mario J. Silva. 2016. Modelling context with user embeddings for sarcasm detection in social media. In CoNLL, pages 167-177. ACL.
Salvatore Attardo. 2002. Talk is cheap: Sarcasm, alienation, and the evolution of language. Journal of Pragmatics, 34.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR.
David Bamman and Noah A. Smith. 2015a. Contextualized sarcasm detection on twitter. In ICWSM, pages 574-577. AAAI Press.
David Bamman and Noah A. Smith. 2015b. Contextualized sarcasm detection on twitter. In ICWSM, pages 574-577. AAAI Press.
Francesco Barbieri, Francesco Ronzano, and Horacio Saggion. 2014a. Italian irony detection in twitter: a first approach. In *CLiC-it*, page 28. AILC.
Francesco Barbieri, Horacio Saggion, and Francesco Ronzano. 2014b. Modelling sarcasm in twitter, a novel approach. In WASSA, pages 50-58. ACL.
Farah Benamara, Cyril Grouin, Jihen Karoui, Véronique Moriceau, and Isabelle Robba. 2017. Analyse d'opinion et langage figuratif dans des tweets: presentation et résultats du début fouille de textes deft2017.
S. K. Bharti, K. S. Babu, and S. K. Jena. 2015. Parsing-based sarcasm sentiment recognition in twitter data. In ASONAM, pages 1373-1380. ACM.
S. K. Bharti, K. S. Babu, and S. K. Jena. 2015. Parsing-based sarcasm sentiment recognition in twitter data. In ASONAM, pages 1373-1380. ACM.
Mondher Bouazizi and Tomoaki Ohtsuki. 2015. Opinion mining in twitter how to make use of sarcasm to enhance sentiment analysis. In *ASONAM*, pages 1594-1597. ACM.
F. Bravo-Marquez, E. Frank, S. M. Mohammad, and B. Pfahringer. 2016. Determining word-emotion associations from tweets by multi-label classification. In WI, pages 536-539. IEEE.
Felipe Bravo-Marquez, Marcelo Mendoza, and Barbara Poblete. 2014. Meta-level sentiment models for big social data analysis. Knowledge-Based Systems, 69:86 - 99.
Reynier Ortega Bueno, Francisco M. Rangel Pardo, Delia Irazú Hernández Farias, Paolo Rosso, Manuel Montes y Gómez, and José Medina-Pagola. 2019. Overview of the task on irony detection in spanish variants. In IberLEF@SEPLN.
John D Campbell and Albert N Katz. 2012. Are there necessary conditions for inducing a sense of sarcastic irony? Discourse Processes, 49(6):459-480.
Alessandra Teresa Cignarella, Simona Frenda, Valerio Basile, Cristina Bosco, Viviana Patti, Paolo Rosso, et al. 2018. Overview of the EVALITA 2018 task on irony detection in Italian tweets (IronITA). In Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 2018), volume 2263, pages 1-6. CEUR-WS.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1):37-46.
Dmitry Davidov, Oren Tsur, and Ari Rappoport. 2010. Semi-supervised recognition of sarcasm in twitter and amazon. In CoNLL, pages 107-116. ACL.
Megan L. Dress, Roger J. Kreuz, Kristen E. Link, and Gina M. Caucci. 2008. Regional variation in the use of sarcasm. JLS, 27(1):71-85.
Elena Filatova. 2012. Irony and sarcasm: Corpus generation and analysis using crowdsourcing. In LREC. ELRA.
Roberto González-Ibanez, Smaranda Muresan, and Nina Wacholder. 2011. Identifying sarcasm in twitter: A closer look. In HLT, pages 581-586. ACL.
Devamanyu Hazarika, Soujanya Poria, Sruthi Gorantla, Erik Cambria, Roger Zimmermann, and Rada Mihalcea. 2018. Cascade: Contextual sarcasm detection in online discussion forums. In COLING, pages 1837-1848. ACL.
Delia Irazú Hernández Farias, Emilio Sulis, Viviana Patti, Giancarlo Ruffo, and Cristina Bosco. 2015. Valento: Sentiment analysis of figurative language tweets with irony and sarcasm. In SemEval, pages 694-698. ACL.
Sepp Hochreiter and Jrgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Julia Jorgensen, George A Miller, and Dan Sperber. 1984. Test of the mention theory of irony. Journal of Experimental Psychology: General, 113(1):112-120.
Aditya Joshi, Pushpak Bhattacharyya, Mark Carman, Jaya Saraswati, and Rajita Shukla. 2016a. How do cultural differences impact the quality of sarcasm annotation?: A case study of indian annotators and american text. In *LaTeCH*, pages 95–99. ACL.
Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya. 2015. Harnessing context incongruity for sarcasm detection. In IJCNLP, pages 757-762. ACL.
Aditya Joshi, Vaibhav Tripathi, Kevin Patel, Pushpak Bhattacharyya, and Mark Carman. 2016b. Are word embedding-based features useful for sarcasm detection? In EMNLP, pages 1006-1011. ACL.
David S. Kaufer. 1981. Understanding ironic communication. Journal of Pragmatics, 5(6):495 - 510.
Mikhail Khodak, Nikunj Saunshi, and Kiran Vodrahalli. 2018. A large self-annotated corpus for sarcasm. In LREC. ELRA.
Quoc Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. In ICML, pages 1188-1196. PMLR.
John S Leggitt and Raymond W Gibbs Jr. 2000. Emotional reactions to verbal irony.
Christine Liebrecht, Florian Kunneman, and Antal Van den Bosch. 2013. The perfect solution for detecting sarcasm in tweets #not. In WASSA, pages 29-37. ACL.
Diana Maynard and Mark Greenwood. 2014. Who cares about sarcastic tweets? investigating the impact of sarcasm on sentiment analysis. In LREC. ELRA.
Saif Mohammad and Felipe Bravo-Marquez. 2017. WASSA-2017 shared task on emotion intensity. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 34-49. Association for Computational Linguistics.
Silviu Oprea and Walid Magdy. 2019. Exploring author context for detecting intended vs perceived sarcasm. In ACL, pages 2854-2859. ACL.
Silviu Oprea and Walid Magdy. 2020. The effect of sociocultural variables on sarcasm communication online. Proceedings of The 23rd ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW).
Tomáš Ptáček, Ivan Habernal, and Jun Hong. 2014. Sarcasm detection on czech and english twitter. In COLING, pages 213-223. ACL.
Antonio Reyes and Paolo Rosso. 2012. Making objective decisions from subjective data: Detecting irony in customer reviews. Decision Support Systems, 53(4):754-760.
Ellen Riloff, Ashequl Qadir, Prafulla Survey, Lalindra De Silva, Nathan Gilbert, and Ruihong Huang. 2013. Sarcasm as contrast between a positive sentiment and negative situation. In *EMNLP*, pages 704-714. ACL.
Patricia Rockwell and Evelyn M. Theriot. 2001. Culture, gender, and gender mix in encoders of sarcasm: A self-assessment analysis. Communication Research Reports, 18(1):44-52.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. Disan: Directional self-attention network for rnn/cnn-free language understanding. In AAAI. AAAI Press.
Yi Tay, Anh Tuan Luu, Siu Cheung Hui, and Jian Su. 2018. Reasoning with sarcasm by reading in between. In ACL, pages 1010-1020. ACL.
Mike Thelwall, Kevan Buckley, and Georgios Paltoglou. 2012. Sentiment strength detection for the social web. J. Am. Soc. Inf. Sci. Technol., 63(1):163-173.
Cynthia Van Hee, Els Lefever, and Veronique Hoste. 2018. SemEval-2018 task 3: Irony detection in English tweets. In SemEval, pages 39-50. ACL.
Deirdre Wilson. 2006. The pragmatics of verbal irony: Echo or pretence? Lingua, 116(10):1722-1743.
Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhigang Yuan, and Yongfeng Huang. 2018. THU_NGN at SemEval-2018 task 3: Tweet irony
detection with densely connected LSTM and multitask learning. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 51-56, New Orleans, Louisiana. Association for Computational Linguistics.