Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue

Abstract: The use of irony and sarcasm in social media allows us to study them at scale for the first time. However, their diversity has made it difficult to construct a high-quality corpus of sarcasm in dialogue. Here, we describe the process of creating a large- scale, highly-diverse corpus of online debate forums dialogue, and our novel methods for operationalizing classes of sarcasm in the form of rhetorical questions and hyperbole. We show that we can use lexico-syntactic cues to reliably retrieve sarcastic utterances with high accuracy. To demonstrate the properties and quality of our corpus, we conduct supervised learning experiments with simple features, and show that we achieve both higher precision and F than previous work on sarcasm in debate forums dialogue. We apply a weakly-supervised linguistic pattern learner and qualitatively analyze the linguistic differences in each class.

Irony and sarcasm in dialogue constitute a highly creative use of language signaled by a large range of situational, semantic, pragmatic and lexical cues. Previous work draws attention to the use of both hyperbole and rhetorical questions in conversation as distinct types of lexico-syntactic cues defining diverse classes of sarcasm BIBREF0 .
Theoretical models posit that a single semantic basis underlies sarcasm's diversity of form, namely "a contrast" between expected and experienced events, giving rise to a contrast between what is said and a literal description of the actual situation BIBREF1 , BIBREF2 . This semantic characterization has not been straightforward to operationalize computationally for sarcasm in dialogue. Riloffetal13 operationalize this notion for sarcasm in tweets, achieving good results. Joshietal15 develop several incongruity features to capture it, but although they improve performance on tweets, their features do not yield improvements for dialogue.
Previous work on the Internet Argument Corpus (IAC) 1.0 dataset aimed to develop a high-precision classifier for sarcasm in order to bootstrap a much larger corpus BIBREF3 , but was only able to obtain a precision of just 0.62, with a best F of 0.57, not high enough for bootstrapping BIBREF4 , BIBREF5 . Justoetal14 experimented with the same corpus, using supervised learning, and achieved a best precision of 0.66 and a best F of 0.70. Joshietal15's explicit congruity features achieve precision around 0.70 and best F of 0.64 on a subset of IAC 1.0.
We decided that we need a larger and more diverse corpus of sarcasm in dialogue. It is difficult to efficiently gather sarcastic data, because only about 12% of the utterances in written online debate forums dialogue are sarcastic BIBREF6 , and it is difficult to achieve high reliability for sarcasm annotation BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 . Thus, our contributions are:
There has been relatively little theoretical work on sarcasm in dialogue that has had access to a large corpus of naturally occurring examples. Gibbs00 analyzes a corpus of 62 conversations between friends and argues that a robust theory of verbal irony must account for the large diversity in form. He defines several subtypes, including rhetorical questions and hyperbole:
Other categories of irony defined by Gibbs00 include understatements, jocularity, and sarcasm (which he defines as a critical/mocking form of irony). Other work has also tackled jocularity and humor, using different approaches for data aggregation, including filtering by Twitter hashtags, or analyzing laugh-tracks from recordings BIBREF11 , BIBREF12 .
Previous work has not, however, attempted to operationalize these subtypes in any concrete way. Here we describe our methods for creating a corpus for generic sarcasm (Gen) (Sec. SECREF11 ), rhetorical questions (RQ), and hyperbole (Hyp) (Sec. SECREF15 ) using data from the Internet Argument Corpus (IAC 2.0). Table TABREF9 provides examples of sarcastic and not-sarcastic posts from the corpus we create. Table TABREF10 summarizes the final composition of our sarcasm corpus.
We first replicated the pattern-extraction experiments of LukinWalker13 on their dataset using AutoSlog-TS BIBREF13 , a weakly-supervised pattern learner that extracts lexico-syntactic patterns associated with the input data. We set up the learner to extract patterns for both sarcastic and not-sarcastic utterances. Our first discovery is that we can classify not-sarcastic posts with very high precision, ranging between 80-90%.
Because our main goal is to build a larger, more diverse corpus of sarcasm, we use the high-precision not-sarcastic patterns extracted by AutoSlog-TS to create a "not-sarcastic" filter. We did this by randomly selecting a new set of 30K posts (restricting to posts with between 10 and 150 words) from IAC 2.0 BIBREF14 , and applying the high-precision not-sarcastic patterns from AutoSlog-TS to filter out any posts that contain at least one not-sarcastic cue. We end up filtering out two-thirds of the pool, only keeping posts that did not contain any of our high-precision not-sarcastic cues. We acknowledge that this may also filter out sarcastic posts, but we expect it to increase the ratio of sarcastic posts in the remaining pool.
We put out the remaining 11,040 posts on Mechanical Turk. As in LukinWalker13, we present the posts in "quote-response" pairs, where the response post to be annotated is presented in the context of its “dialogic parent”, another post earlier in the thread, or a quote from another post earlier in the thread BIBREF15 . In the task instructions, annotators are presented with a definition of sarcasm, followed by one example of a quote-response pair that clearly contains sarcasm, and one pair that clearly does not. Each task consists of 20 quote-response pairs that follow the instructions. Figure FIGREF13 shows the instructions and layout of a single quote-response pair presented to annotators. As in LukinWalker13 and Walkeretal12d, annotators are asked a binary question: Is any part of the response to this quote sarcastic?.
To help filter out unreliable annotators, we create a qualifier consisting of a set of 20 manually-selected quote-response pairs (10 that should receive a sarcastic label and 10 that should receive a not-sarcastic label). A Turker must pass the qualifier with a score above 70% to participate in our sarcasm annotations tasks.
Our baseline ratio of sarcasm in online debate forums dialogue is the estimated 12% sarcastic posts in the IAC, which was found previously by Walker et al. by gathering annotations for sarcasm, agreement, emotional language, attacks, and nastiness from a subset of around 20K posts from the IAC across various topics BIBREF6 . Similarly, in his study of recorded conversation among friends, Gibbs cites 8% sarcastic utterances among all conversational turns BIBREF0 .
We choose a conservative threshold: a post is only added to the sarcastic set if at least 6 out of 9 annotators labeled it sarcastic. Of the 11,040 posts we put out for annotation, we thus obtain 2,220 new posts, giving us a ratio of about 20% sarcasm – significantly higher than our baseline of 12%. We choose this conservative threshold to ensure the quality of our annotations, and we leave aside posts that 5 out of 9 annotators label as sarcastic for future work – noting that we can get even higher ratios of sarcasm by including them (up to 31%). The percentage agreement between each annotator and the majority vote is 80%.
We then expand this set, using only 3 highly-reliable Turkers (based on our first round of annotations), giving them an exclusive sarcasm qualification to do additional HITs. We gain an additional 1,040 posts for each class when using majority agreement (at least 2 out of 3 sarcasm labels) for the additional set (to add to the 2,220 original posts). The average percent agreement with the majority vote is 89% for these three annotators. We supplement our sarcastic data with 2,360 not-sarcastic posts from the original data by BIBREF3 that follow our 150-word length restriction, and complete the set with 900 posts that were filtered out by our not-sarcastic filter – resulting in a total of 3,260 posts per class (6,520 total posts).
Rows 1 and 2 of Table TABREF9 show examples of posts that are labeled sarcastic in our final generic sarcasm set. Using our filtering method, we are able to reduce the number of posts annotated from our original 30K to around 11K, achieving a percentage of 20% sarcastic posts, even though we choose to use a conservative threshold of at least 6 out of 9 sarcasm labels. Since the number of posts being annotated is only a third of the original set size, this method reduces annotation effort, time, and cost, and helps us shift the distribution of sarcasm to more efficiently expand our dataset than would otherwise be possible.
The goal of collecting additional corpora for rhetorical questions and hyperbole is to increase the diversity of the corpus, and to allow us to explore the semantic differences between sarcastic and not-sarcastic utterances when particular lexico-syntactic cues are held constant. We hypothesize that identifying surface-level cues that are instantiated in both sarcastic and not sarcastic posts will force learning models to find deeper semantic cues to distinguish between the classes.
Using a combination of findings in the theoretical literature, and observations of sarcasm patterns in our generic set, we developed a regex pattern matcher that runs against the 400K unannotated posts in the IAC 2.0 database and retrieves matching posts, only pulling posts that have parent posts and a maximum of 150 words. Table TABREF16 only shows a small subset of the “more successful” regex patterns we defined for each class.
Cue annotation experiments. After running a large number of retrieval experiments with our regex pattern matcher, we select batches of the resulting posts that mix different cue classes to put out for annotation, in such a way as to not allow the annotators to determine what regex cues were used. We then successively put out various batches for annotation by 5 of our highly-qualified annotators, in order to determine what percentage of posts with these cues are sarcastic.
Table TABREF16 summarizes the results for a sample set of cues, showing the number of posts found containing the cue, the subset that we put out for annotation, and the percentage of posts labeled sarcastic in the annotation experiments. For example, for the hyperbolic cue "wow", 977 utterances with the cue were found, 153 were annotated, and 44% of those were found to be sarcastic (i.e. 56% were found to be not-sarcastic). Posts with the cue "oh wait" had the highest sarcasm ratio, at 87%. It is the distinction between the sarcastic and not-sarcastic instances that we are specifically interested in. We describe the corpus collection process for each subclass below.
It is important to note that using particular cues (regex) to retrieve sarcastic posts does not result in posts whose only cue is the regex pattern. We demonstrate this quantitatively in Sec. SECREF4 . Sarcasm is characterized by multiple lexical and morphosyntactic cues: these include the use of intensifiers, elongated words, quotations, false politeness, negative evaluations, emoticons, and tag questions inter alia. Table TABREF17 shows how sarcastic utterances often contain combinations of multiple indicators, each playing a role in the overall sarcastic tone of the post.
Rhetorical Questions. There is no previous work on distinguishing sarcastic from non-sarcastic uses of rhetorical questions (RQs). RQs are syntactically formulated as a question, but function as an indirect assertion BIBREF16 . The polarity of the question implies an assertion of the opposite polarity, e.g. Can you read? implies You can't read. RQs are prevalent in persuasive discourse, and are frequently used ironically BIBREF17 , BIBREF18 , BIBREF0 . Previous work focuses on their formal semantic properties BIBREF19 , or distinguishing RQs from standard questions BIBREF20 .
We hypothesized that we could find RQs in abundance by searching for questions in the middle of a post, that are followed by a statement, using the assumption that questions followed by a statement are unlikely to be standard information-seeking questions. We test this assumption by randomly extracting 100 potential RQs as per our definition and putting them out on Mechanical Turk to 3 annotators, asking them whether or not the questions (displayed with their following statement) were rhetorical. According to majority vote, 75% of the posts were rhetorical.
We thus use this "middle of post" heuristic to obviate the need to gather manual annotations for RQs, and developed regex patterns to find RQs that were more likely to be sarcastic. A sample of the patterns, number of matches in the corpus, the numbers we had annotated, and the percent that are sarcastic after annotation are summarized in Table TABREF16 .
We extract 357 posts following the intermediate question-answer pairs heuristic from our generic (Gen) corpus. We then supplement these with posts containing RQ cues from our cue-annotation experiments: posts that received 3 out of 5 sarcastic labels in the experiments were considered sarcastic, and posts that received 2 or fewer sarcastic labels were considered not-sarcastic. Our final rhetorical questions corpus consists of 851 posts per class (1,702 total posts). Table TABREF18 shows some examples of rhetorical questions and self-answering from our corpus.
Hyperbole. Hyperbole (Hyp) has been studied as an independent form of figurative language, that can coincide with ironic intent BIBREF21 , BIBREF22 , and previous computational work on sarcasm typically includes features to capture hyperbole BIBREF23 . KreuzRoberts95 describe a standard frame for hyperbole in English where an adverb modifies an extreme, positive adjective, e.g. "That was absolutely amazing!" or "That was simply the most incredible dining experience in my entire life."
ColstonObrien00b provide a theoretical framework that explains why hyperbole is so strongly associated with sarcasm. Hyperbole exaggerates the literal situation, introducing a discrepancy between the "truth" and what is said, as a matter of degree. A key observation is that this is a type of contrast BIBREF24 , BIBREF1 . In their framework:
An event or situation evokes a scale;
An event can be placed on that scale;
The utterance about the event contrasts with actual scale placement.
Fig. FIGREF22 illustrates that the scales that can be evoked range from negative to positive, undesirable to desirable, unexpected to expected and certain to uncertain. Hyperbole moves the strength of an assertion further up or down the scale from the literal meaning, the degree of movement corresponds to the degree of contrast. Depending on what they modify, adverbial intensifiers like totally, absolutely, incredibly shift the strength of the assertion to extreme negative or positive.
Table TABREF23 shows examples of hyperbole from our corpus, showcasing the effect that intensifiers have in terms of strengthening the emotional evaluation of the response. To construct a balanced corpus of sarcastic and not-sarcastic utterances with hyperbole, we developed a number of patterns based on the literature and our observations of the generic corpus. The patterns, number matches on the whole corpus, the numbers we had annotated and the percent that are sarcastic after annotation are summarized in Table TABREF16 . Again, we extract a small subset of examples from our Gen corpus (30 per class), and supplement them with posts that contain our hyperbole cues (considering them sarcastic if they received at least 3/5 sarcastic labels, not-sarcastic otherwise). The final hyperbole dataset consists of 582 posts per class (1,164 posts in total).
To recap, Table TABREF10 summarizes the total number of posts for each subset of our final corpus.
Our primary goal is not to optimize classification results, but to explore how results vary across different subcorpora and corpus properties. We also aim to demonstrate that the quality of our corpus makes it more straightforward to achieve high classification performance. We apply both supervised learning using SVM (from Scikit-Learn BIBREF25 ) and weakly-supervised linguistic pattern learning using AutoSlog-TS BIBREF13 . These reveal different aspects of the corpus.
Supervised Learning. We restrict our supervised experiments to a default linear SVM learner with Stochastic Gradient Descent (SGD) training and L2 regularization, available in the SciKit-Learn toolkit BIBREF25 . We use 10-fold cross-validation, and only two types of features: n-grams and Word2Vec word embeddings. We expect Word2Vec to be able to capture semantic generalizations that n-grams do not BIBREF26 , BIBREF27 . The n-gram features include unigrams, bigrams, and trigrams, including sequences of punctuation (for example, ellipses or "!!!"), and emoticons. We use GoogleNews Word2Vec features BIBREF28 .
Table TABREF25 summarizes the results of our supervised learning experiments on our datasets using 10-fold cross validation. The data is balanced evenly between the sarcastic and not-sarcastic classes, and the best F-Measures for each class are shown in bold. The default W2V model, (trained on Google News), gives the best overall F-measure of 0.74 on the Gen corpus for the sarcastic class, while n-grams give the best not-sarcastic F-measure of 0.73. Both of these results are higher F than previously reported for classifying sarcasm in dialogue, and we might expect that feature engineering could yield even greater performance.
On the RQ corpus, n-grams provide the best F-measure for sarcastic at 0.70 and not-sarcastic at 0.71. Although W2V performs well, the n-gram model includes features involving repeated punctuation and emoticons, which the W2V model excludes. Punctuation and emoticons are often used as distinctive feature of sarcasm (i.e. "Oh, really?!?!", [emoticon-rolleyes]).
For the Hyp corpus, the best F-measure for both the sarcastic and not-sarcastic classes again comes from n-grams, with F-measures of 0.65 and 0.68 respectively. It is interesting to note that the overall results of the Hyp data are lower than those for Gen and RQs, likely due to the smaller size of the Hyp dataset.
To examine the effect of dataset size, we compare F-measure (using the same 10-fold cross-validation setup) for each dataset while holding the number of posts per class constant. Figure FIGREF26 shows the performance of each of the Gen, RQ, and Hyp datasets at intervals of 100 posts per class (up to the maximum size of 582 posts per class for Hyp, and 851 posts per class for RQ). From the graph, we can see that as a general trend, the datasets benefit from larger dataset sizes. Interestingly, the results for the RQ dataset are very comparable to those of Gen. The Gen dataset eventually gets the highest sarcastic F-measure (0.74) at its full dataset size of 3,260 posts per class.
Weakly-Supervised Learning. AutoSlog-TS is a weakly supervised pattern learner that only requires training documents labeled broadly as sarcastic or not-sarcastic. AutoSlog-TS uses a set of syntactic templates to define different types of linguistic expressions. The left-hand side of Table TABREF28 lists each pattern template and the right-hand side illustrates a specific lexico-syntactic pattern (in bold) that represents an instantiation of each general pattern template for learning sarcastic patterns in our data. In addition to these 17 templates, we added patterns to AutoSlog for adjective-noun, adverb-adjective and adjective-adjective, because these patterns are frequent in hyperbolic sarcastic utterances.
The examples in Table TABREF28 show that Colston's notion of contrast shows up in many learned patterns, and that the source of the contrast is highly variable. For example, Row 1 implies a contrast with a set of people who are not your mother. Row 5 contrasts what you were asked with what you've (just) done. Row 10 contrasts chapter 12 and chapter 13 BIBREF30 . Row 11 contrasts what I am allowed vs. what you have to do.
AutoSlog-TS computes statistics on the strength of association of each pattern with each class, i.e. P(sarcastic INLINEFORM0 INLINEFORM1 ) and P(not-sarcastic INLINEFORM2 INLINEFORM3 ), along with the pattern's overall frequency. We define two tuning parameters for each class: INLINEFORM4 , the frequency with which a pattern occurs, INLINEFORM5 , the probability with which a pattern is associated with the given class. We do a grid-search, testing the performance of our patterns thresholds from INLINEFORM6 = {2-6} in intervals of 1, INLINEFORM7 ={0.60-0.85} in intervals of 0.05. Once we extract the subset of patterns passing our thresholds, we search for these patterns in the posts in our development set, classifying a post as a given class if it contains INLINEFORM8 ={1, 2, 3} of the thresholded patterns. For more detail, see BIBREF13 , BIBREF31 .
An advantage of AutoSlog-TS is that it supports systematic exploration of recall and precision tradeoffs, by selecting pattern sets using different parameters. The parameters have to be tuned on a training set, so we divide each dataset into 80% training and 20% test. Figure FIGREF30 shows the precision (x-axis) vs. recall (y-axis) tradeoffs on the test set, when optimizing our three parameters for precision. Interestingly, the subcorpora for RQ and Hyp can get higher precision than is possible for Gen. When precision is fixed at 0.75, the recall for RQ is 0.07 and the recall for Hyp is 0.08. This recall is low, but given that each retrieved post provides multiple cues, and that datasets on the web are huge, these P values make it possible to bootstrap these two classes in future.
Here we aim to provide a linguistic characterization of the differences between the sarcastic and the not-sarcastic classes. We use the AutoSlog-TS pattern learner to generate patterns automatically, and the Stanford dependency parser to examine relationships between arguments BIBREF13 , BIBREF32 . Table TABREF31 shows the number of sarcastic patterns we extract with AutoSlog-TS, with a frequency of at least 2 and a probability of at least 0.75 for each corpus. We learn many novel lexico-syntactic cue patterns that are not the regex that we search for. We discuss specific novel learned patterns for each class below.
Generic Sarcasm. We first examine the different patterns learned on the Gen dataset. Table TABREF29 show examples of extracted patterns for each class. We observe that the not-sarcastic patterns appear to capture technical and scientific language, while the sarcastic patterns tend to capture subjective language that is not topic-specific. We observe an abundance of adjective and adverb patterns for the sarcastic class, although we do not use adjective and adverb patterns in our regex retrieval method. Instead, such cues co-occur with the cues we search for, expanding our pattern inventory as we show in Table TABREF31 .
Rhetorical Questions. We notice that while the not-sarcastic patterns generated for RQs are similar to the topic-specific not-sarcastic patterns we find in the general dataset, there are some interesting features of the sarcastic patterns that are more unique to the RQs.
Many of our sarcastic questions focus specifically on attacks on the mental abilities of the addressee. This generalization is made clear when we extract and analyze the verb, subject, and object arguments using the Stanford dependency parser BIBREF32 for the questions in the RQ dataset. Table TABREF32 shows a few examples of the relations we extract.
Hyperbole. One common pattern for hyperbole involves adverbs and adjectives, as noted above. We did not use this pattern to retrieve hyperbole, but because each hyperbolic sarcastic utterance contains multiple cues, we learn an expanded class of patterns for hyperbole. Table TABREF33 illustrates some of the new adverb adjective patterns that are frequent, high-precision indicators of sarcasm.
We learn a number of verbal patterns that we had not previously associated with hyperbole, as shown in Table TABREF34 . Interestingly, many of these instantiate the observations of CanoMora2009 on hyperbole and its related semantic fields: creating contrast by exclusion, e.g. no limit and no way, or by expanding a predicated class, e.g. everyone knows. Many of them are also contrastive. Table TABREF33 shows just a few examples, such as though it in no way and so much knowledge.
We have developed a large scale, highly diverse corpus of sarcasm using a combination of linguistic analysis and crowd-sourced annotation. We use filtering methods to skew the distribution of sarcasm in posts to be annotated to 20-31%, much higher than the estimated 12% distribution of sarcasm in online debate forums. We note that when using Mechanical Turk for sarcasm annotation, it is possible that the level of agreement signals how lexically-signaled the sarcasm is, so we settle on a conservative threshold (at least 6 out of 9 annotators agreeing that a post is sarcastic) to ensure the quality of our annotations.
We operationalize lexico-syntactic cues prevalent in sarcasm, finding cues that are highly indicative of sarcasm, with ratios up to 87%. Our final corpus consists of data representing generic sarcasm, rhetorical questions, and hyperbole.
We conduct supervised learning experiments to highlight the quality of our corpus, achieving a best F of 0.74 using very simple feature sets. We use weakly-supervised learning to show that we can also achieve high precision (albeit with a low recall) for our rhetorical questions and hyperbole datasets; much higher than the best precision that is possible for the Generic dataset. These high precision values may be used for bootstrapping these two classes in the future.
We also present qualitative analysis of the different characteristics of rhetorical questions and hyperbole in sarcastic acts, and of the distinctions between sarcastic/not-sarcastic cues in generic sarcasm data. Our analysis shows that the forms of sarcasm and its underlying semantic contrast in dialogue are highly diverse.
In future work, we will focus on feature engineering to improve results on the task of sarcasm classification for both our generic data and subclasses. We will also begin to explore evaluation on real-world data distributions, where the ratio of sarcastic/not-sarcastic posts is inherently unbalanced. As we continue our analysis of the generic and fine-grained categories of sarcasm, we aim to better characterize and model the great diversity of sarcasm in dialogue.
This work was funded by NSF CISE RI 1302668, under the Robust Intelligence Program.

Figure and table captions:
Table 1: Examples of different types of SARCASTIC (S) and NOT-SARCASTIC (NS) Posts
Table 2: Total number of posts in each subcorpus (each with a 50% split of SARCASTIC and NOTSARCASTIC posts)
Figure 1: Mechanical Turk Task Layout
Table 3: Annotation Counts for a Subset of Cues
Table 4: Utterances with Multiple Sarcastic Cues
Figure 2: Hyperbole shifts the strength of what is said from literal to extreme negative or positive (Colston and O’Brien, 2000)
Table 5: Examples of Rhetorical Questions and Self-Answering
Table 6: Examples of Hyperbole and the Effects of Intensifiers
Table 7: Supervised Learning Results for Generic (Gen: 3,260 posts per class), Rhetorical Questions (RQ: 851 posts per class) and Hyperbole (Hyp: 582 posts per class)
Figure 3: Plot of Dataset size (x-axis) vs Sarc. FMeasure (y-axis) for the three subcorpora, with ngram features
Table 8: AutoSlog-TS Templates and Example Instantiations
Table 9: Examples of Characteristic Patterns for Gen using AutoSlog-TS Templates
Figure 4: Plot of Precision (x-axis) vs Recall (yaxis) for three subcorpora with AutoSlog-TS parameters, aimed at optimizing precision
Table 10: Total number of patterns passing threshold of Freq ≥ 2, Prob ≥ 0.75
Table 13: Verb Patterns in Hyperbole
Table 11: Attacks on Mental Ability in RQs
Table 12: Adverb Adjective Cues in Hyperbole

Questions about this paper:
What are the linguistic differences between each class?

Answers:
Each class has different patterns in adjectives, adverbs and verbs for sarcastic and non-sarcastic classes