SlowGuess/ACL_2020_OCR huggingface dataset:
Silviu Vlad Oprea
silviu.oprea@ed.ac.uk
Walid Magdy
wmagdy@inf.ed.ac.uk
We consider the distinction between intended and perceived sarcasm in the context of textual sarcasm detection. The former occurs when an utterance is sarcastic from the perspective of its author, while the latter occurs when the utterance is interpreted as sarcastic by the audience. We show the limitations of previous labelling methods in capturing intended sarcasm and introduce the iSarcasm dataset of tweets labeled for sarcasm directly by their authors. Examining the state-of-the-art sarcasm detection models on our dataset showed low performance compared to previously studied datasets, which indicates that these datasets might be biased or obvious and sarcasm could be a phenomenon under-studied computationally thus far. By providing the iSarcasm dataset, we aim to encourage future NLP research to develop methods for detecting sarcasm in text as intended by the authors of the text, not as labeled under assumptions that we demonstrate to be sub-optimal.
Sarcasm is a form of irony that occurs when there is some discrepancy between the literal and intended meanings of an utterance. This discrepancy is used to express dissociation towards a previous proposition, often in the form of contempt or derogation (Wilson, 2006). Sarcasm is omnipresent in social media text and can be highly disruptive of systems that harness this data for sentiment and emotion analysis (Maynard and Greenwood, 2014). It is therefore imperative to devise models for sarcasm detection. The effectiveness of such models depends on the availability and quality of labelled data used for training. Collecting such data is challenging due to the subjective nature of sarcasm. For instance, Dress et al. (2008) notice a lack of consistence in how sarcasm is used by people of different socio-cultural backgrounds. As a result, an utterance intended sarcastic by its author might
not be perceived as such by audiences of different backgrounds (Rockwell and Theriot, 2001; Oprea and Magdy, 2020).
There are two methods used so far to label texts for sarcasm: distant supervision, where texts are considered sarcastic if they meet predefined criteria, such as including specific hashtags; and manual labelling by human annotators. We believe both methods are sub-optimal for capturing the sarcastic intention of the authors of the texts. As a result, existing models trained on such datasets might be optimized to capture the noise induced by these labelling methods.
In this paper, we present the iSarcasm dataset of tweets labelled for sarcasm by their authors. To our knowledge, this is the first attempt to create noise-free examples of intended sarcasm. In a survey, we asked Twitter users to provide both sarcastic and non-sarcastic tweets that they had posted in the past. For each sarcastic tweet, we asked them to explain why it was sarcastic and how they would convey the same meaning non-sarcastically. Labels were thus implicitly specified by the authors themselves. We implemented restrictive quality control to exclude spurious survey responses. We then asked a trained linguist to manually check the sarcastic tweets and further label them into the subcategories of sarcasm defined by Leggitt and Gibbs Jr. (2000).
We further collected third-party sarcasm labels for the tweets in iSarcasm from workers on a crowdsourcing platform. Third-party annotation for sarcasm has been conducted before (Filatova, 2012; Riloff et al., 2013; Abercrombie and Hovy, 2016), but no studies checked the ability of the annotators to capture the actual sarcasm meant by the authors. On iSarcasm, annotators recognise author labels with an F-score of 0.616. This indicates that sarcasm is a subjective phenomenon, challenging even for humans to detect. Further, it demonstrates that using third-party annotators to label texts for sarcasm can lead to inaccurate labels.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1279-1289 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
We implemented state-of-the-art sarcasm detection models (Tay et al., 2018; Hazarika et al., 2018; Van Hee et al., 2018) and tested them on iSarcasm, to investigate their effectiveness in capturing sarcasm as intended by the authors. While these models achieve F-scores reaching 0.874 on existing datasets, they yield a maximum F-score of 0.364 on iSarcasm, suggesting that previous datasets might be biased or obvious. This highlights the importance of developing new approaches for sarcasm detection that are more effective at capturing author intention.
iSarcasm contains 4,484 English tweets, each with an associated intended sarcasm label provided by its author, with a ratio of roughly 1:5 of sarcastic to non-sarcastic tweets. Each sarcastic tweet has an extra label indicating the category of sarcasm it belongs to. We publish the dataset publicly for research purposes<sup>1</sup>.
2.1 Intended and Perceived Sarcasm
The way sarcasm is used can vary across sociocultural backgrounds. Dress et al. (2008) notice that members of collectivist cultures tend to express sarcasm in a more subtle way than individualists. They also point out gender differences. Females seem to have a more self-deprecating attitude when using sarcasm than males. Rockwell and Theriot (2001) find some cultures to associate sarcasm with humour more than others. There are also cultures who do not use sarcasm at all, such as the Hua, a group of New Guinea Highlanders (Attardo, 2002). Because of these differences, an utterance intended sarcastic by its author might not be perceived as such by the audience (Jorgensen et al., 1984). Conversely, the audience could perceive the utterance as sarcastic, even if it was not intended as such.
The distinction between intended and perceived sarcasm, also referred to as encoded and decoded sarcasm, respectively, has been pointed out in previous research (Kaufer, 1981; Rockwell and Theriot, 2001). However, it has not been considered in a computational context thus far when building datasets for textual sarcasm detection. We believe accounting for it is essential, especially nowadays. Consider social media posts that can reach audiences of unprecedented sizes. It is important to consider both the communicative intention of the author, for tasks such as opinion mining, as well
as possible interpretations by audiences of different sociocultural backgrounds, for tasks such as hate-speech detection.
2.2 Sarcasm Datasets
Two methods were used so far to label texts for sarcasm: distant supervision and manual labelling.
Distant supervision This is by far the most common method. Texts are considered positive examples (sarcastic) if they meet predefined criteria, such as containing specific tags, such as #sarcasm for Twitter data (Páček et al., 2014), and /s for Reddit data (Khodak et al., 2018), or being posted by specific social media accounts (Barbieri et al., 2014a). Negative examples are usually random posts that do not match the criteria. Table 1 gives an overview of datasets constructed this way, along with tags or accounts they associate with sarcasm.
The main advantage of distant supervision is that it allows building large labelled datasets with no manual effort. However, as we discuss in Section 3, the labels produced can be very noisy.
Manual labelling An alternative to distant supervision is collecting texts and presenting them to human annotators for labelling. Filatova (2012) asks annotators to find pairs of Amazon reviews where one is sarcastic and the other one is not, collecting 486 positive and 844 negative examples. Abercrombie and Hovy (2016) annotate 2,240 Twitter conversations, ending up with 448 positive and 1,732 negative labels, respectively. Riloff et al. (2013) use a hybrid approach, where they collect a set of 1,600 tweets that contain #sarcasm or #sarcastic, and another 1,600 without these tags. They remove such tags from all tweets and present the tweets to a group of human annotators for final labelling. We call this the *Riloff* dataset. A similar approach is employed by Van Hee et al. (2018) who recently presented their dataset as part of a SemEval shared task for sarcasm detection. It is a balanced dataset of 4,792 tweets. We call it the *SemEval-2018* dataset.
2.3 Sarcasm Detection Models
Based on the information considered when classifying a text as sarcastic or non-sarcastic, we identify two classes of models across literature: text-based models and contextual models.
Text-based models These models only consider information available within the text being clas
$^{1}$ https://github.com/silviu-oprea/iSarcasm
Sarcasm labeling method	Source	Details / Tags / Accounts
Distant supervision
Davidov et al. (2010)	Twitter	#sarcasm, #sarcastic, #not
Barbieri et al. (2014b)	Twitter	#sarcasm, #education, #humor, #irony, #politics
Ptáček et al. (2014)	Twitter	#sarcasm, #sarcastic, #irony, #satire
Bamman and Smith (2015a); Joshi et al. (2015)	Twitter	#sarcasm, #sarcastic
González-Ibáñez et al. (2011); Reyes and Rosso (2012); Liebrecht et al. (2013); Bouazizi and Ohtsuki (2015); Bharti et al. (2015)	Twitter	#sarcasm
Barbieri et al. (2014a)	Twitter	tweets posted by @spinozait or @LiveSpinoza
Khodak et al. (2018)	Reddit	/s
Manual annotation / Hybrid
Riloff et al. (2013); Benamara et al. (2017); Cignarella et al. (2018); Van Hee et al. (2018); Bueno et al. (2019)	Twitter	tweets
Abercrombie and Hovy (2016)	Twitter	tweet-reply pairs
Filatova (2012)	Amazon	product reviews
Table 1: Datasets previously suggested for sarcasm detection, all annotated using either distant supervision or manual labelling, as discussed in Section 2.2.
sified. Most work in this direction considers linguistic incongruity (Campbell and Katz, 2012) to be a marker of sarcasm. Riloff et al. (2013) look for a positive verb in a negative sentiment context. Bharti et al. (2015) search for a negative phrase in a positive sentence. (Hernández Farias et al., 2015) measure semantic relatedness between words using WordNet-based similarity. Joshi et al. (2016b) use the cosine similarity between word embeddings. Recent work (Tay et al., 2018) uses a neural intraattention mechanism to capture incongruity.
Contextual models These models utilize information from both the text and the context of its disclosure, such as author information. There is a limited amount of work in this direction. Using Twitter data, Bamman and Smith (2015a) represent author context as manually-curated features extracted from their historical tweets. Amir et al. (2016) merge all historical tweets into one document and use the Paragraph Vector model (Le and Mikolov, 2014) to build an embedding of that document. Building on this, Hazarika et al. (2018) extract additional personality features from the merged historical tweets with a model pre-trained on a personality detection corpus. Using the same strategy, Oprea and Magdy (2019) build separate embeddings for each historical tweet and identify author context with their the weighted average.
Despite reporting encouraging results, all previous models are trained and tested on datasets annotated via manual labelling, distant supervision, or a mix between them. We believe both labelling meth
ods are limited in their ability to capture sarcasm in texts as intended by the authors of the texts without noise. We now discuss how noise can occur.
3 Limitations of Current Labelling Methods
In this section, we discuss limitations of current labelling methods that make them sub-optimal for capturing intended sarcasm. We demonstrate them empirically on the Riloff dataset (Riloff et al., 2013), which uses a hybrid approach for labelling.
3.1 Limitations of Distant Supervision
Since it is based on signals provided by the authors, distant supervision might seem like a candidate for capturing intended sarcasm. However, we identify a few fundamental limitations with it. First, the tags may not mark sarcasm, but may constitute the subject or object of conversation, e.g. #sarcasm annoys me!. This could lead to false positives. Second, when using tags such as #politics and #education (Barbieri et al., 2014b), there is a strong underlying assumption that these tags are accompanied by sarcasm, potentially generating further false positives. The assumption that some accounts always generate sarcasm (Barbieri et al., 2014a) is similarly problematic. In addition, the intended sarcasm that distant supervision does capture might be of a specific flavor, such that, for instance, the inclusion of a tag would be essential to ensure inferability. Building a model trained on such a dataset might, therefore, be biased to a specific flavour of sarcasm, being unable to capture other flavours,
with tag	without tag
annot. sarcastic	345	26
annot. non-sarcastic	486	975
Table 2: The agreement between manual annotation and the presence of sarcasm tags in the Riloff dataset, as discussed in Section 3.2.
increasing the risk of false negatives and limiting the ability of trained models to generalise. Finally, if a text does not contain the predefined tags, it is considered non-sarcastic. This is a strong and problematic assumption that can lead to false negatives.
3.2 Limitations of Manual labelling
The main limitation of manual labelling is the absence of evidence on the intention of the author of the texts that are being labelled. Annotator perception may be different to author intention, in light of studies that point out how sarcasm perception varies across socio-cultural contexts (Rockwell and Theriot, 2001; Dress et al., 2008).
Joshi et al. (2016a) provide more insight into this problem on the Riloff dataset. They present the dataset, initially labelled by Americans, to be labelled by Indians who are trained linguists. They find higher disagreement between Indian and American annotators, than between annotators of the same nationality. Furthermore, they find higher disagreement between pairs of Indian annotators, indicating higher uncertainty, than between pairs of American annotators. They attribute these results to socio-cultural differences between India and the United States. They conclude that sarcasm annotation expands beyond linguistic expertise and is dependent on considering such factors.
Labels provided by third-party annotators might therefore not reflect the sarcastic intention of the authors of the texts that are being labelled, making this labelling method sub-optimal for capturing intended sarcasm. To investigate this further, we looked at the Riloff dataset, which is published as a list of labelled tweet IDs. We could only retrieve 1,832 tweets, the others being removed from Twitter. We looked at the agreement between the presence of tags and manual annotation. Table 2 shows the results. We notice that  $58\%$  of the tweets that contained the predefined hashtags were labeled non-sarcastic. This disagreement between distant supervision and manual annotation provides further evidence to doubt the ability of the latter to cap
ture intended sarcasm, at least not the flavor that distant supervision might capture. We could not perform the same analysis on the SemEval-2018 dataset because only the text of the tweets is provided, hashtags are filtered out, and tweet IDs are not available.
As we have shown, both labelling methods use a proxy for labelling sarcasm, in the form of predefined tags, predefined sources, or third-party annotators. As such, they are unable to capture the sarcastic intention of the authors of the texts they label, generating both false positives and false negatives. Our objective is to create a noise-free dataset of texts labelled for sarcasm, where labels reflect the sarcastic intention of the authors.
4.1 Collecting Sarcastic Tweets
We designed an online survey where we asked Twitter users to provide links to one sarcastic and three non-sarcastic tweets that they had posted in the past, on their timeline, or as replies to other tweets. We made it clear that the tweets had to be their own and no retweets were allowed. We further required that the tweets should not include references to multimedia content or, if such content was referred, it should not be informative in judging sarcasm.
For each sarcastic tweet, users had to provide, in full English sentences, an explanation of why it was sarcastic and a rephrase that would convey the same message non-sarcastically. This way, we aimed to prevent them from misjudging the sarcastic nature of their previous tweets under experimental bias. Finally, we asked for their age, gender, birth country and region, and current country and region. We use the term response to refer to all data collected from one submission of the survey.
To ensure genuine responses, we implemented the following quality control steps:
- The provided links should point to tweets posted no sooner than 48 hours before the submission, to prevent users from posting and providing tweets on the spot;
- All tweets in a response should come from the same account;
- Tweets cannot be from verified accounts or accounts with more than 30K followers to avoid getting tweets from popular accounts and claiming to be personal tweets  ${}^{2}$  .
2The initial number was set to 5K, but some workers asked
- Tweets should contain at least 5 words, excluding any hashtags and URLs;
- Links to tweets should not have been submitted in a previous response;
- Responses submitted in less than three minutes are discarded.
Each contributor agreed on a consent form before entering the survey, which informed them that only the IDs of the tweets they provide will be made public, to allow them to delete a tweet anytime and thus be in control of their own privacy in the future. They have agreed that we may collect public information from their profile, which is accessible via the Twitter API as long as the tweets pointed to by the provided IDs are not removed.
We published our survey on multiple crowdsourcing platforms, including Figure-Eight (F8), Amazon Mechanical Turk (AMT) and Prolific Academic  $(\mathrm{PA})^3$ . We could not get any quality responses from F8. In fact, most of our quality control steps were developed over multiple iterations on F8. On AMT, we retrieved some high quality responses, but, unfortunately, AMT stopped our job, considering that getting links to personal tweets of participants violates their policy. We collected the majority of responses on PA.
4.2 Labelling Sarcasm Categories
We then asked a trained linguist to inspect each collected sarcastic tweet, along with the explanation provided by the author and the non-sarcastic rephrase, in order to validate the quality of the response and further assign the tweet to one of the following categories of ironic speech defined by Leggitt and Gibbs Jr. (2000):
1. sarcasm: tweets that contradict the state of affairs and are critical towards an addressee;
2. irony: tweets that contradict the state of affairs but are not obviously critical towards an addressee;
3. satire: tweets that appear to support an addressee, but contain underlying disagreement and mocking;
4. understatement: tweets that undermine the importance of the state of affairs they refer to;
5. overstatement: tweets that describe the state of affairs in obviously exaggerated terms;
6. rhetorical question: tweets that include a question whose invited inference (implicature) is obviously contradicting the state of affairs;
7. invalid: tweets for which the explanation provided by their authors is unclear/unjustified. These were excluded from the dataset.
4.3 Collecting Third-Party Labels
In this part, we decided to replicate the manual annotation approach presented in previous research (Riloff et al., 2013; Abercrombie and Hovy, 2016; Van Hee et al., 2018) on part of our dataset, which we consider later as the test set, and compare the resulting perceived sarcasm labels to the intended sarcasm labels collected from the authors of the tweets. Our aim was to estimate the human performance in detecting sarcasm as intended by the authors.
When collecting perceived sarcasm labels, we aimed to reduce noise caused by variations in how sarcasm is defined across socio-cultural backgrounds. Previous studies have shown gender (Dress et al., 2008) and country (Joshi et al., 2016a) to be the variables that are most influential on this definition. Based on their work, we made sure all annotators shared the same values for these variables. We used PA to collect three annotations for each tweet in the iSarcasm dataset, and considered the dominant one as the label, which follows the same procedure as with building the Riloff dataset (Riloff et al., 2013).
5 Data Statistics and Analysis
5.1 iSarcasm Dataset
We received 1,236 responses to our survey. Each response contained four tweets labelled for sarcasm by their author, one sarcastic and three non-sarcastic. As such, we received 1,236 sarcastic and 3,708 non-sarcastic tweets. We filtered tweets using the quality control steps described in Section 4, and further disregarded all tweets that fall under the invalid category. The resulting dataset is what we call iSarcasm, containing 777 sarcastic and 3,707 non-sarcastic tweets. For each sarcastic tweet, we have its author's explanation as to why it is sarcastic, as well as how they would rephrase the tweet to be non-sarcastic. The average length of a tweet is around 20 words. Figure 1 shows the tweet length distribution across iSarcasm. The average length of explanations 21 words, and of rephrases 14 words. Over  $46\%$  of the tweets were posted in 2019, over
us to raise it since they had more followers.
3AMT: www.mturk.com, PA: prolific.ac, F8:
www_figure-eight.com
overall	sarcasm category
sarcastic	non-sarcastic	sarcasm	irony	satire	underst.	overst.	rhet. question
777	3,707	324	245	82	12	64	50
Table 3: Distribution of sarcastic tweets into the categories introduced in Section 4.2.
category	tweet text	explanation	rephrased
sarcasm	Thank @user for being so entertain- ing at the Edinburgh signings! You did not disappoint! I made my flight so will have plenty time to read @user	I went to a book signing and the au- thor berated me for saying I was lying about heading to Singapore straight after the signing	I would have said 'here is the proof of my travel, I am mad you embarrassed me in front of a large audience'!
irony	Staring at the contents of your fridge but never deciding what to eat is a cool way to diet	I wasn't actually talking about a real diet. I was making fun of how you never eat anything just staring at the contents of your fridge full of inde- cision.	I'm always staring at the con- tents of my fridge and then walk- ing away with nothing cause I can never decide.
satire	@mizzieashitey @PCDPhotog- raphy Totally didnt happen, its a big conspiracy, video can be faked...after all, they've been faking the moon landings for years	It's an obvious subversion of known facts about mankind's space explo- ration to date that are nonetheless disputed by conspiracy theorists.	It's not a conspiracy, the video is real... after all, we've known for years that the moon landings happened.
underst.	@user @user @user Still made 5 grand will do him for a while	The person I was tweeting to cashed out 5k in a sports accumulator - how- ever he would've won 295k. "Still made 5k will do him for a while" is used to underplay the devastation of losing out.	He made 5 grand, but that will only last him a month.
overst.	the worst part about quitting cigarettes is running into people you went to high school with at a vape shop	There are many things that are actu- ally harder about quitting cigarettes than running into old classmates.	Running into old classmates at a vape shop is one of the eas- ier things you have to deal with when you quit cigarettes.
rhetorical question	@user do all your driver's take a course on how to #tailgate!	Drivers don't have to take a course on how to tailgate its just bad driving on their part.	Could you ask your drivers not to tailgate other people on the roads please?
Table 4: Examples of sarcastic tweets from our datasets along with the explanations that authors gave to what makes their tweets sarcastic (explanation) and how they can rephrase them to be non-sarcastic (rephrased).
83% starting with 2017, and the earliest in 2008.