Introduction
Affective computing has raised a great deal of interest in the last years. Picard picard1995affective introduced it as a computing paradigm that relates to, arises from, or influences emotions, letting computers be both more effective in assisting humans and successful in making decisions.
Language, as a conceptual process, plays a key role in the perception of verbal irony and sarcasm, two well-known forms of figurative language (FL) BIBREF0 Traditionally, irony as a figure of speech can be intended as “saying something while meaning something else” BIBREF1 . A comprehensive overview of different theories of irony has been illustrated in Attardo attardo07. Understanding if irony and sarcasm are the same linguistic phenomenon or not is still an unresolved question in literature BIBREF2 . Some authors consider irony a more general form of sarcasm, while others tend to consider it a separate linguistic issue BIBREF3 , BIBREF4 . According to the theory of sarcastic irony, sarcasm and irony are very similar, but sarcasm has a specific victim who is the object of the sarcastic statement, while irony does not have such a target BIBREF5 . More commonly, the noun “sarcasm” is understood as “saying the opposite of what one is thinking”, usually with a negative intention. Henceforth, due to the different nuances of irony and sarcasm, and the multiple interpretations of these two concepts, we do not differentiate between them, and, like many researchers, e.g., BIBREF6 , we will use the term “sarcasm” to refer to both verbal irony and sarcasm.
A sarcastic sentence may include features that characterize a positive sentiment, but that insinuates a negative sentiment BIBREF7 , BIBREF8 . It is clear that sarcastic sentences are more difficult to process by an algorithm than non-sarcastic assertions; as a matter of fact, both the situation and the mental state of the speaker are factors that can determine a sarcastic content in a sentence.
A system capable of detecting sarcasm correctly would greatly improve the performance of sentiment analysis systems BIBREF9 , BIBREF10 , BIBREF6 , BIBREF11 , especially considering the big data available nowadays due to the exponential growth of social platforms. Unfortunately, sarcasm detection in written texts is a difficult task even for humans BIBREF12 .
Moreover, some people usually do not understand sarcasm, and there are sentences meant as being sarcastic by the author that are not recognized as such by the readers.
We focus our attention on the possibility of detecting sarcastic sentences automatically from written text only, and from the reader's point of view. Managing this task without any knowledge of relevant contextual features, like prosody, is very hard.
The problem of sarcasm detection has been tackled with machine learning approaches, made possible by the availability of several annotated corpora. In the literature we can find two main categories of such corpora: automatically annotated and manually annotated.
The automatically annotated corpora are usually collected from the microblogging platform Twitter BIBREF13 , BIBREF14 by exploiting the final hashtag of tweets. For instance, a tweet is labeled as sarcastic only if it ends with a hashtag such as #sarcasm or #irony. The same cue is used in Davidov, Tsur and Rappoport davidov2010semi to produce a silver standard for evaluating their model.
Manually annotated corpora are collected from a more diversified range of social media, such as Amazon reviews BIBREF15 , Reddit (Wallace et al. 2014) or online forums BIBREF16 , BIBREF17 , and then labeled by hiring people in the Amazon Mechanical Turk portal. When using crowdsourcing, the annotation procedures are complex and involve, among others, a stage for ensuring that the workers understood the task and they are performing correctly, and a quality assurance stage for removing texts for which a high discrepancy between the annotators arises.
In this work we have tackled the problem of sarcasm detection by trying to use an entirely data-driven approach, exploiting a distributional semantics representation by inducing a semantic space and then applying a set of classifiers to classify the texts as being sarcastic or not sarcastic. With “fully data-driven” we mean approaches that are capable of finding connections between input text and class labels without using any a priori knowledge about the features that characterize a sarcastic statement.
In particular, we do not define “irony” or “sarcasm”, neither use any definition. We simply rely on sets of sentences binary labeled for sarcasm detection taking for granted that the labels correctly identify a sarcastic sentence.
It is worthwhile to point out that in this work we do not create any dataset: we simply exploit the labels of datasets that have already been produced by others, trying to give a baseline for the sarcasm detection task.
The contribution of this work can be summed up in three key points:
To reach these goals, we exploit a Distributional Semantics approach, whose aim is to give a representation of words in a continuous vector space BIBREF18 , BIBREF19 , where word similarity is coded in an unsupervised manner. This representation is useful for building models with little, or no, a-priori knowledge about the task BIBREF20 .
Distributional semantics is a research field that concerns methodologies aimed at determining semantic similarities between linguistic items. The key idea is based on the hypothesis that words co-occurring in similar contexts tend to have similar meaning BIBREF21 , BIBREF22 . Distributional semantics deals with the automatic construction of semantic models induced from large unstructured textual corpora, and it exploits vector space models to represent the meaning of a word BIBREF23 . Many methods can be applied to construct distributional models. They range from the statistical models to machine learning ones BIBREF24 , BIBREF19 , BIBREF25 , BIBREF26 . Among these techniques, Latent Semantic Analysis (LSA) is a methodology for building distributional semantic spaces that extract statistical relations between words which co-occurr in a given context though the use of the Truncated Singular value decomposition (T-SVD). In this work we explored and studied the possibility of building a data-driven model in the field of sarcasm detection exploiting the well-known Latent Semantic Analysis (LSA) paradigm both in its traditional formulation given by Landauer, Foltz and Laham landauer1998introduction and by using the Truncated Singular Value Decomposition (T-SVD) as a statistical estimator as illustrated in Pilato and Vassallo pilato2015tsvd.
Both approaches have been used to create data-driven semantic spaces where documents and, generally, text chunks can be mapped.
The theory behind LSA states that the “psychological similarity between any two words is reflected in the way they co-occur in small sub-samples of language” (Landauer et al. 1998).
We have chosen to exploit the LSA paradigm since it is a well-known distributional semantics paradigm capable of modeling many human cognitive abilities; furthermore, it has many potential practical applications BIBREF27 , BIBREF18 , BIBREF28 , BIBREF29 . Moreover, it has been demonstrated in Pilato and Vassallo pilato2015tsvd that Truncated Singular Value Decomposition (T-SVD), as used in LSA, can be interpreted as a statistical estimator, giving a robust theoretical interpretation to the Latent Semantic Analysis paradigm. Many researchers have successfully applied this technique for typical Semantic Computing applications, such as natural language understanding, cognitive modeling, speech recognition, smart indexing, anti-spam filters, dialogue systems, and other Statistical Natural Language processing problems BIBREF30 , BIBREF31 , BIBREF32 . Moreover, Latent Semantic Analysis has been successfully used for inducing data-driven “conceptual” spaces BIBREF33 . For the aforementioned reasons, we have chosen this approach as a baseline for the detection of sarcasm in texts.
Furthermore, our study makes use of four machine learning methods that have been used on four manually annotated, publicly available corpora.
The experimental results show that our data-driven approach consisting of LSA followed by a classifier can establish models that outperform the published results on two of the corpora; additionally, it produces competitive results for the other corpora that we used for our evaluation.
The next section describes the state of the art in the field, Section SECREF3 describes the Semantic Representation and the Machine Learning methods used in the study. Section SECREF4 introduces the datasets used for the experiments. Section SECREF5 summarizes the experimental results, Section SECREF6 is for the final conclusions and remarks.
The code and the datasets used for the experiments are available on github.
Related works
The problem of sarcasm detection has been tackled using a wide range of supervised or semi-supervised techniques applied to corpora from different social media sources.
In the present work, we do not collect a new corpus for sarcasm detection, but sarcastic corpus annotation has received much attention in the literature. Most of the works have used unsupervised or semi-supervised approaches in order to reduce the cost of the annotation, while partially sacrificing the data quality. One of the first approaches was introduced by Tsur, Davidov and Rappoport tsur2010icwsm for a corpus extracted from Twitter and further developed in Davidov et al. davidov2010semi with a corpus consisting of Amazon reviews. This semi-supervised approach uses “YAHOO! BOSS” API web search for collecting INLINEFORM0 utterances similar to the ones in a small initial labeled seed set. It was the first work to show that automatically-crawled data are useful for the task of sarcasm detection. Most of the works have been pursued using data extracted from Twitter, as it is relatively easy to extract ironic or sarcastic tweets using the search by hashtag. In fact, in Twitter, the restricted number of characters allowed encourages to mark the ironic intent with a hashtag like #irony or #sarcasm to prevent ambiguities. The hashtag is usually removed from the tweets and used as a label for the silver standard. Moreover, the first studies on Twitter data showed that the task is quite difficult also for human beings. González-Ibánez et al. gonzalez2011identifying collected a corpus of INLINEFORM1 tweets balanced between sarcastic, positive sentiment and negative sentiment. They presented a part of the corpus to human judges, who achieved low agreement and low accuracy. Reyes et al. reyes2013multidimensional collected a corpus using 4 hashtags that identify four different categories, irony, education, humor, and politics, with INLINEFORM2 tweets each. The same corpus was used in a later work BIBREF34 . Their results suggest that detecting sarcasm in full documents is easier than in single sentences because of the presence of a context, but in both cases, it remains a difficult task also for humans that often have a low agreement. The specific case of positive sentiment and a negative situation, which is the most typical sarcastic situation, has also been analyzed BIBREF35 . In particular, authors have found that less than half of the tweets ending with the hashtag #sarcastic are recognized as sarcastic by humans after removing the hashtag. Bharti, Babu, and Jena bharti2015parsing proposed two algorithms with the goal to find, respectively, tweets with contrast in sentiment and situation, and tweets starting with interjections. They also found that the label distribution does not correlate perfectly with the hashtag distribution, e.g., only INLINEFORM3 out of INLINEFORM4 tweets ending with #sarcastic are actually sarcastic. Farias, Patti and Rosso farias16 proposed a method that uses affective content to classify sarcastic tweets, and show that it outperforms preceding methods in several Twitter benchmarks. Since classifying tweets by using only the text is a difficult task also for humans, other works proposed new methods capable of exploiting other kind of data, like the identity of the author or the thread of the tweet. Bamman and Smith bamman2015contextualized augmented the feature vectors with features describing the author of the tweet and the user to which the tweet is addressed, obtaining significant improvements in accuracy. They also found that the hashtags #sarcasm and #sarcastic are mainly used when the audience is not known. Wang, Wu, Wang and Ren wang2015twitter use a sequential classifier for classifying tweets taking into account the previous responses, thus improving the performance concerning a simple multi-class classifier.
Amir, Wallace, Lyu, Carvalho and Silva amir2016modelling used the dataset collected in Bamman et al. bamman2015contextualized (which was not completely available) for training a deep learning model that could represent users with user embeddings and this method seems to outperform the method from Bamman and colleagues. Sarcasm classification on Twitter involves different modelling techniques that perform better when taking into account the user and the thread history of a Tweet. Our work focuses on the task of classifying a single document written by a single author. Thus, we focus mainly on different kinds of datasets. Buschmeier, Cimiano and Klinger buschmeier2014impact have studied the corpus introduced in Filatova filatova2012irony by extracting a high number of features about typographic cues that can represent sarcasm, and used different classification methods obtaining results that vary significantly according to the classifier. They found that the single most important feature is the star rating of the review, and this happens because sarcastic reviews are more probable when a user did not like the product.
Wallace et al. wallace2014humans created a corpus from Reddit posts, for which they also stored context information, such as the post that is answered. The authors proposed a method that uses the bag of words and other features from previous studies for building an SVM classifier that gets very low results. Moreover, a correlation is found between posts for which the humans require the context and sarcastic posts. This can be explained by considering that the chosen sub-reddits are about religion or politics, and they are thus very prone to controversial discussions. Consequently, to understand the ironic intent of a post it is quite important to know the author position on the topic and also the posts they are answering to.
Joshi, Sharma and Bhattacharyya joshi-sharma-bhattacharyya:2015:ACL-IJCNLP used features for capturing intrinsic and extrinsic incongruity in texts and outperforms two previous methods both in tweets and in forum posts. These works represent valuable means of comparison for the present work. We show that an approach based only on distributional semantics is competitive with other approaches using more elaborated feature engineering, even when the data amount is quite small. Distributional semantics became popular in NLP thanks to the availability of good quality word embeddings BIBREF19 , and are introduced by design in deep learning models. In sarcasm detection, distributional semantics has been used to serve different roles. Ghosh, Guo, and Muresan ghosh2015sarcastic have adopted word embeddings to disambiguate a literal use of single words from a sarcastic use. Joshi, Tripathi, Patel, Bhattacharyya and Carman joshi2016word use word embeddings to compute incongruities among words using them as additional features for methods selected from the literature. Our work differs from these as we use LSA instead of word embeddings, and distributional semantics is the only kind of features we use. Ghosh and Veale ghosh2016 use LSA to extend the list of hashtags to find more sarcastic tweets on Twitter and use a deep neural network to perform the actual classification. Our work differs from theirs as we use LSA to compute the vectorial representation of documents and we do not perform tweet crawling. Poria, Cambria, Hazarika and Vij cambria2016 train a convolutional neural network to classify sarcasm in tweets. They extend the neural network with features extracted from other datasets for sentiment, emotion and personality classification, as these features are considered to be useful for the task of sarcasm detection.
Data-Driven Induction of Semantic Spaces and Traditional Classifiers
We focused our research on the role that fully data-driven models can play in detecting sarcasm. To reach this goal, we exploited the Latent Semantic Analysis paradigm both in its traditional formulation (Landauer et al. 1998) and by using the Truncated Singular Value Decomposition (T-SVD) as a statistical estimator as shown in Pilato et al. pilato2015tsvd. We have chosen to use the LSA paradigm to exploit a well-known and well-founded approach for inducing semantic spaces that have been effectively used in natural language understanding, cognitive modeling, speech recognition, smart indexing, and other statistical natural language processing problems. The sub-symbolic codings of documents obtained by the aforementioned LSA-based approaches are then used as inputs by a set of classifiers to evaluate the differences of performances obtained by using different machine learning approaches and testing them on different sarcasm-detection datasets.
The full work-flow, composed of the following steps:
does not require any expert or domain knowledge.
Preprocessing of text
The first step of preprocessing for texts is the tokenization using spaces, punctuation and special characters (e.g., $, , @) as separators. Thus one token is a sequence of alphanumeric characters or of punctuation symbols. The set of all the extracted tokens constitutes a “vocabulary” named INLINEFORM0 .
The sequences of tokens, each representing a single document in the training set, are used to generate a word-document co-occurrence raw matrix INLINEFORM0 , where each INLINEFORM1 cell contains the number of times the token INLINEFORM2 appears in the document INLINEFORM3 . Let INLINEFORM4 be the number of tokens, i.e., INLINEFORM5 , and let INLINEFORM6 be the number of documents of the corpus used for computing the matrix INLINEFORM7 ; the dimensionality of INLINEFORM8 is INLINEFORM9 .
Data driven induction of semantic spaces by means of LSA-oriented paradigms
The matrix INLINEFORM0 is used and further processed to induce proper Semantic Spaces where terms and documents can be mapped. To generate these semantic spaces, we have used both the traditional LSA algorithm (Deerwester et al. 1990, Landauer et al. 1998) and the approach which uses T-SVD as a statistical estimator as proposed in Pilato et al. pilato2015tsvd. For the sake of brevity, we call this last approach Statistical LSA to differentiate it by the Traditional LSA. It is worthwhile to point out that, in the Latent Semantic Analysis paradigm (i.e., both “general” and “statistical”), the corpus used for building the semantic space plays a key role in performances. As a matter of fact, large and heterogeneous corpora may give more noise or too much specific information from a single domain, decreasing the accuracy of the induced models BIBREF36 .
The traditional LSA is a procedure that has been used mainly for information retrieval (Deerwester et al. 1990). The previously described matrix INLINEFORM0 is used for computing a Tf-Idf (Term-Frequency Inverse-document frequency) matrix INLINEFORM1 BIBREF37 . Let INLINEFORM2 be the rank of INLINEFORM3 . The following factorization, called Singular Value Decomposition (SVD) holds for the matrix INLINEFORM4 : DISPLAYFORM0
where INLINEFORM0 is a INLINEFORM1 orthogonal matrix, INLINEFORM2 is a INLINEFORM3 orthogonal matrix and INLINEFORM4 is a INLINEFORM5 diagonal matrix, whose diagonal elements INLINEFORM6 are called singular values of INLINEFORM7 . It can be shown that the singular value decomposition of INLINEFORM8 is unique up to the order of the singular values and of the corresponding columns of INLINEFORM9 and INLINEFORM10 , so there is no loss of generality if we suppose that INLINEFORM11 are ranked in decreasing order.
Let INLINEFORM0 be an integer such that INLINEFORM1 , let INLINEFORM2 be the matrix obtained from INLINEFORM3 by removing its last INLINEFORM4 columns, INLINEFORM5 the matrix obtained from INLINEFORM6 in the same manner and INLINEFORM7 the diagonal matrix obtained from INLINEFORM8 by suppressing both its last INLINEFORM9 rows and INLINEFORM10 columns. INLINEFORM11 is the matrix containing the INLINEFORM12 -dimensional vector representation of the words and INLINEFORM13 is the matrix containing the INLINEFORM14 -dimensional vector representation of the documents. It can be shown (Deerwester et al. 1990) that the matrix: DISPLAYFORM0
is the best rank INLINEFORM0 approximation to INLINEFORM1 according to the Frobenius distance. INLINEFORM6 is called the reconstructed matrix. The process by which INLINEFORM7 is obtained from INLINEFORM8 is called Truncated Singular Value Decomposition (T-SVD). The book by Golub and Van Loan golub1996matrix provide further details about the Singular Value Decomposition technique.
The traditional Latent Semantic Analysis based on T-SVD is one of the possible methods to infer data-driven models. Furthermore, one of its major drawbacks, which is the lack of a sound statistical interpretation, has been recently overcome in Pilato et al. pilato2015tsvd, where authors have been presented a statistical explanation of this paradigm.
According to this interpretation, the T-SVD algorithm, as used in the Latent Semantic Analysis paradigm, acts as an estimator, which conveys statistically significant information from the sample to the model.
To briefly sum-up the procedure, we recall here the concepts of probability amplitude and probability distribution associated with a matrix as they have been defined in Pilato et al. pilato2015tsvd.
Let INLINEFORM0 , INLINEFORM1 be two positive integers and let INLINEFORM2 be the set of real numbers. Given a INLINEFORM3 matrix INLINEFORM4 with INLINEFORM5 , INLINEFORM6 , INLINEFORM7 where at least one of its components INLINEFORM8 is positive, we define a set INLINEFORM9 , composed of all the pairs INLINEFORM10 that identify the positive components of INLINEFORM11 , i.e.: DISPLAYFORM0
Subsequently, we define the probability amplitude associated with INLINEFORM0 , the INLINEFORM1 matrix INLINEFORM2 resulting from the mapping INLINEFORM3 : DISPLAYFORM0
whose elements INLINEFORM0 are computed as: DISPLAYFORM0
so that INLINEFORM0 it is INLINEFORM1 and INLINEFORM2 .
We define also the probability distribution associated with a matrix INLINEFORM0 the INLINEFORM1 matrix resulting from the mapping INLINEFORM2 : DISPLAYFORM0
whose elements are the squares of the elements of INLINEFORM0 , i.e. INLINEFORM1 . The method starts with a raw data matrix INLINEFORM2 consisting of positive values. In our study the raw data matrix INLINEFORM3 is the term-document co-occurrence matrix. From INLINEFORM4 a real-valued normalized matrix INLINEFORM5 is computed by dividing every element for the sum of all elements of INLINEFORM6 . DISPLAYFORM0
If we call INLINEFORM0 the matrix: DISPLAYFORM0
The matrix INLINEFORM0 can be decomposed with the SVD technique: DISPLAYFORM0
and its best rank-r decomposition INLINEFORM0 is obtained by applying the T-SVD technique, which minimizes the Frobenius distance INLINEFORM1 , given INLINEFORM2 : DISPLAYFORM0
Even if INLINEFORM0 is not a probability distribution, the computation of INLINEFORM1 makes it possible to identify, without any further addition of external information, the probability distribution we are looking for. As shown in Pilato et al. pilato2015tsvd, it theoretically suffices computing the probability amplitude associated to INLINEFORM2 , i.e. INLINEFORM3 , and consequently calculating the probability distribution INLINEFORM4 associated to INLINEFORM5 . The aforementioned Frobenius distance INLINEFORM6 constitutes an upper bound to the Hellinger distance between the sample probability INLINEFORM11 and the probability distribution estimated by the procedure.
Mapping new documents to the semantic space
Both LSA approaches illustrated in the previous subsections provide us with the three, obviously different for each approach, matrices INLINEFORM0 , INLINEFORM1 and INLINEFORM2 .
The INLINEFORM0 and the INLINEFORM1 matrices can be used for computing the vector representation of the new documents into the induced semantic space. The INLINEFORM2 matrix contains in its diagonal the singular values; INLINEFORM3 is composed by rows that represent the r-dimensional sub-symbolic, i.e., numerical, mapping in the semantic space of the tokens constituting the vocabulary INLINEFORM4 . Then, given a text chunk INLINEFORM5 , INLINEFORM6 is sub-symbolically represented by a INLINEFORM7 -dimensional word occurrence vector INLINEFORM8 , from which it is computed a vector INLINEFORM9 with two different procedures depending on which LSA paradigm has been chosen.
In the case of Traditional LSA, it is the Tf-Idf representation BIBREF38 of INLINEFORM0 by using the same parameters learned during training.
In the case of the Statistical LSA, the INLINEFORM0 vector is transformed into INLINEFORM1 similarly as the matrix INLINEFORM2 is transformed into the matrix INLINEFORM3 : DISPLAYFORM0
Once the appropriate coding of INLINEFORM0 has been computed, an r-dimensional vector INLINEFORM1 representing the sub-symbolic coding of INLINEFORM2 is then obtained from the vector INLINEFORM3 by means of the following mapping formula: DISPLAYFORM0
Supervised learning
The training and test documents are mapped into the semantic spaces induced at the previous step. These vectors, sub-symbolic coding of the documents, are therefore used as inputs to different classifiers to train or test on them. Such classifiers will finally solve a binary classification problem assigning the label 1 (sarcastic) or 0 (nonsarcastic) to a generic document. For this study we have used Support Vector Machines, Logistic Regression, Random Forests, and Gradient boosting as they represent the state of the art for most of the binary classification problems with small datasets. In the following, we recall a brief description of them.
The logistic regressor (LR) is a generalized linear model suitable for binary responses BIBREF39 . In LR the following log-linear model is adopted: DISPLAYFORM0
where INLINEFORM0 represents the probability of the success outcome. A suitable way of minimizing the so called empirical risk is the numerical estimation of the INLINEFORM1 s coefficient by a maximum likelihood procedure: DISPLAYFORM0
where INLINEFORM0 is the training set, INLINEFORM1 is the norm of the weights vector used for regularization, and can be either the INLINEFORM2 or the INLINEFORM3 norm, and INLINEFORM4 is the weight to give to the regularization factor. The function in formula EQREF33 is convex, so it can be minimized even with the simple gradient descent algorithm, but more complex algorithms can be used in order to reduce the convergence time. In this work we use the trust region Newton method proposed by Lin, Weng and Keerthy lin2008trust, as provided by the LIBLINEAR library BIBREF40 .
A kernel INLINEFORM0 is any mapping satisfying DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 are elements in the input space, INLINEFORM2 is a mapping from the input space to a new representation space INLINEFORM3 where an inner product is defined. The function INLINEFORM4 is chosen to be nonlinear, and the dimension of the feature space is taken intentionally greater than the dimension of the input space. These choices could give the chance to make the classification problem linearly separable in INLINEFORM5 . Support vector machines (SVMs), also called kernel machines BIBREF41 are binary linear classifiers that make use of kernels. They search for the optimal hyperplane INLINEFORM6 in the feature space that maximizes the geometric margin, which is the distance of the hyperplane to the nearest training data point of any class. The main advantage of SVM is that it provides a solution to the global optimization problem, thereby reducing the generalization error of the classifier. The formulation of SVM can be easily extended to build a nonlinear classifier by incorporating a kernel of the class H DISPLAYFORM0
No systematic tools have been developed to automatically identify the optimal kernel for a particular application.
Decision trees BIBREF42 are rooted trees that can be used successfully as classifiers BIBREF43 . Each node of the three represents a binary rule that splits the feature space according to the value of a predictive feature and a path from the root to leaf nodes represents a series of rules that are used to recursively divide the feature space into smaller subspaces, where a class label is assigned. The structure of the tree in terms of split nodes can be learned from data by using several approaches. Random forests BIBREF44 are an ensemble of decision trees, found using the bootstrap sampling technique on the training set. In particular, a fixed number of random samples are extracted with replacement from the training set, and each of them is used as a training set to fit a decision tree. The forest is composed by each of these decision trees, and the final predictions are made by averaging the predictions from all the individual decision trees.
Boosting is another ensemble strategy with the special purpose of improving the combination of a set of weak classifiers. These are chosen to be of very low model complexity such as the case of decision trees with a single split. The general framework of boosting sequentially adds a tree to an ensemble, the new one with the goal of correcting its predecessor. Gradient boosting BIBREF45 uses a gradient-descent like procedure to sequentially improve a tree classifier. This is done by adding to the actual classifier a new decision tree learned from the residual errors made by the predecessor. The final predictions are made by the tree classifier resulting after a fixed number of iterations of the procedure.
Datasets
We have chosen 4 corpora for our experiments, all of them are publicly available and treating the problem as a binary classification: “SarcasmCorpus” (Filatova 2012) , “IAC-Sarcastic” BIBREF46 , which is a subset of Internet Argument Corpus1.0 prepared for sarcasm detection, “irony-context” (Wallace et al. 2014), and “IAC-Sarcastic-v2” (Oraby et al. 2016), which is extracted from the second version of Internet Argument Corpus BIBREF47 . In order to provide a more complete evaluation, we also use the corpus of the shared task “Semeval2018 Task 3A” BIBREF48 .
SarcasmCorpus
Filatova filatova2012irony collected 1254 reviews from Amazon for different kinds of products, of which 437 are sarcastic, and 817 are not sarcastic. The dataset is unbalanced toward the “regular” texts, and this is due both to the policy of Amazon, which explicitly requires sincere reviews and to the peculiarity of sarcasm itself, which is used only in some cases, especially because of the difficulty for humans to recognize it over the internet.
Each review in the corpus consists of the title, author, product name, review text and number of stars, and the review is a stand-alone document referring to a single product. This corpus, like all the others considered in this work, has been entirely hand-labeled by the Amazon Mechanical Turkers, who were asked whether each review contains sarcasm in it. Each text has been presented to 5 Turkers and has been classified as sarcastic when at least three among five workers agreed. The corpus contains INLINEFORM0 distinct tokens, with INLINEFORM1 occurring only in sarcastic reviews, INLINEFORM2 occurring only in regular reviews and INLINEFORM3 occurring in both categories. Buschmeier et al. buschmeier2014impact made an interesting analysis of the corpus by collecting some statistics and publishing the only classification results that are available for it up to now. They extracted 29 task-specific features and combined them with the bag-of-words representation and multiple classifiers. The bag of words resulted to be important for the classification. In fact, for example, they get a poor 50.9% F-score value with logistic regressor without bag-of-words, which is increased to 74% by using it. This result is surely related to the difference in terms used by the two classes, but it also shows that information about the words used in the document is needed for the task.
IAC-Sarcastic
The second dataset we used is the IAC-Sarcastic sub-corpus, which consists of 1995 posts coming from 4forums.com, a classical forum where several topics are discussed. This corpus is actually extracted from the larger Internet Argument Corpus (IAC), containing INLINEFORM0 discussions, INLINEFORM1 posts and INLINEFORM2 words. In IAC there are INLINEFORM3 Quote-Response (Q-R) pairs and INLINEFORM4 three-posts chains that have been manually labeled for several HITs (Human-Intelligence Tasks) by Amazon Mechanical Turk. For each Q-R item, the Turkers were asked to evaluate the response section by considering the quote as a context. One of the HITs regarded the identification of a sarcastic response. As a result, the IAC-Sarcastic Corpus consists of 1995 responses, without any quote, with a binary label that indicates the presence of sarcasm. 998 texts are labeled as sarcastic, and 997 are not, so this is one of the rare balanced datasets for this task. To the best of our knowledge, only the work by Justo, Corcoran, Lukin, Walker, and Torres justo2014 published results on the sarcastic task of the IAC dataset, but the authors made a different sampling of the documents from the one used for IAC-Sarcastic. Thus, our results for this corpus are not comparable with the ones reported in that work.
Irony-context
A third dataset is the one collected in Wallace et al. wallace2014humans. The main goal of that study was to highlight the role of the context of a text to make irony understandable by humans. The dataset is extracted from Reddit by collecting comments from the following six sub-reddits: politics, progressive, conservative, atheism, Christianity, technology, with their respective size of 873, 573, 543, 442, 312 and 277 samples. Each comment has been labeled by three university undergraduates using a browser interface which let them see the context of the comment in the form of previous comments or related pages under request. The label of a comment was selected with a simple majority of 2 out of 3 labelers. For each comment and each labeler, they stored whether the context has been requested and if the labeler changed his mind after having seen it. This allowed the authors to study the correlation between the sarcastic label and the requests for context.
The results allowed the authors to infer that the machines would also need the context for detecting sarcasm, as their model did not predict correctly the texts for which the humans required the context. This is an important cue that should be considered while developing sarcasm detection methods, even though we do not explicitly consider the context of our method. As a result, we cannot expect to obtain high absolute results for this dataset by letting the model observe only the single text.
IAC-Sarcastic-v2
In 2016 a new version of IAC was made available (IACv2) (Abbot et al. 2016), and after some months also the sarcastic sub-corpus was released (Oraby et al. 2016), which is bigger than the first version. It consists of three sub-corpora, among which the bigger one is called “generic”, and it is made of INLINEFORM0 posts per class collected from IACv2. For the creation of this sub-corpus, the authors produced a high-precision classifier for the non-sarcastic class, which helped to filter out many non-sarcastic posts from the original corpus and lower the labeling costs. Then, to have high-quality labeling, they required a majority of 6 out of 9 sarcastic annotations to label a post as sarcastic.
To produce a more diverse corpus, they built two more corpora focused on particular rhetorical figures often associated with sarcasm: rhetorical questions and hyperboles. For both of the sub-corpora, the authors used patterns to recognize posts containing the chosen rhetorical figure from IACv2. Each of the collected posts has been subsequently shown to five AMTs for the sarcastic/not sarcastic annotation. The label is given with simple majority.
The purpose of these two focused sub-corpora is to force classifiers to find some semantic cues which can distinguish sarcastic posts even in the presence of rhetorical figures usually associated with sarcasm. In fact, the presence of hyperboles has been used before as a feature for detecting sarcasm BIBREF49 .
Semeval-2018 Task3 Corpus of Tweets
The International Workshop on Semantic Evaluation Semeval-2018 featured a shared task on verbal irony detection in tweets (Van Hee et al. 2018). The corpus contains a class-balanced training set consisting of INLINEFORM0 tweets, and a test set with 784 tweets. In the test set, only 40% of the instances are ironic. The corpus has been collected from Twitter searching for tweets with the hashtags #irony, #sarcasm and #not. The corpus has been annotated by three students in linguistics who showed a high inter-annotator agreement. After the annotation, INLINEFORM1 tweets out of INLINEFORM2 were ironic and only 604 were not. Thus, an additional set of INLINEFORM3 non-ironic tweets was added to the corpus. Finally, the corpus was split randomly in class-balanced training and test set, but an additional cleaning step for removing ambiguous sentences modified the proportion to 40% ironic.
Experimental setup
We ran three groups of experiments, to assess both the effectiveness of our approach when compared with the approaches we found in literature and its capability of extracting features that are relevant for sarcasm in a cross-domain scenario. In both cases, we denote with the word model one of the possible combinations of classic/statistical LSA and a classifier. The used classifiers are Support Vector Machine (SVM), Logistic regression (Log.Reg), Random Forest (RF) and gradient boosting (XGB).
For the first group of experiments, we evaluated the performance of each of our models in every corpus. We use 10-fold cross-validation and report the mean values of INLINEFORM0 -score, precision, and recall among all the folds. The proportion of the two classes in each fold is equal to the proportion in the whole corpus. Where applicable, we compare our results with existing results in the literature. Besides, we compare with the method presented in Poira et al. cambria2016.
The second group of experiments has been performed on the Semeval 2018 Task 3 dataset (Van Hee et al. 2018). We first find the best LSA dimensionality by 10-fold cross-validation in the training set. Then, we trained again the models in the whole dataset and evaluated them in the test set for comparison with the participants to the shared task.
The third group of experiments is inter-corpora. For each experiment, we have chosen one corpus as a training set and another one as a test set. This process is performed for all the models and all the corpora pairs. We aim to find whether sarcasm detection is domain-dependent.
Finally, in the fourth group of experiments (union experiments) we perform another 10-fold in which all the corpora are concatenated. Each fold contains samples from every corpus proportionally to the size of that corpus. The goal of this experiment is to understand whether simply adding more data, but from different domains, improves the classification performance.
The hyperparameters of the classifiers have been chosen by grid search on SarcasmCorpus with LSA dimensionality 40, and then used for all the reported experiments. We use SVM with Gaussian kernel, C value of 100, INLINEFORM0 logistic regression with penalty L1 and C=10 and decision tree with entropy loss. SVM and logistic regression both have balanced class weights to cope with unbalanced datasets.
In-corpus Experiments
In SarcasmCorpus each sample consists of a review title, a review text, a product name and the number of stars given to the product ranging from 1 to 5. Buschmeier et al. buschmeier2014impact showed that the star rating is the most discriminative feature. Thus we performed the experiment both including and not including it. In Table TABREF48 , we refer to “SarcasmCorpus” when the star rating is not used, and “SarcasmCorpus*” when it is used. We use the star rating by simply concatenating it to the document vector produced by LSA. The document vector is computed only from the review texts because in our preliminary experiments we found that the other parts are not useful for the task. Accuracy and F-score values of all classifiers for SarcasmCorpus and SarcasmCorpus* are plotted in Figures FIGREF72 and FIGREF73 , and the best F-scores, with the relative precision and recall, are reported in the two columns SarcasmCorpus and SarcasmCorpus* of Table TABREF48 . The best result from the logistic regression in SarcasmCorpus is INLINEFORM0 which represents a INLINEFORM1 % relative improvement concerning the INLINEFORM2 reported in the above-mentioned work by Buschmeier et al. buschmeier2014impact. The results from Poira et al. cambria2016 are even higher in terms of F-score, with a relative improvement of INLINEFORM3 , which is due mostly to a much higher recall.
Note that the method by Poira et al. cambria2016 uses also features extracted from other datasets for sentiment, emotion and personality classification, as these features are considered to be useful for the task of sarcasm detection. Moreover, as our goal is to propose a baseline, the training time in the order of minutes is an advantage of our model. We report such results as an upper bound considering that our model does not use additional information from external data.
The best results are obtained using the star labels. In this setting, our best-performing classifiers are better than the INLINEFORM0 F-score value reported by Buschmeier, and our best INLINEFORM1 -score of INLINEFORM2 represents a INLINEFORM3 relative improvement. In this single case of SarcasmCorpus*, the results with the Traditional LSA are all higher than their counterparts with Statistical LSA.
For IAC-Sarcastic we do not have any previously published result to compare with. The only related result is reported in Joshi et al. joshi-sharma-bhattacharyya:2015:ACL-IJCNLP, which use a corpus randomly extracted from IAC containing 752 sarcastic and 752 not sarcastic texts. They report an F-score of INLINEFORM0 (average over a 5-fold), but the text sampling procedure is not specified in the paper. Thus, we prefer to use the sarcastic selection given by the Internet Argument Corpus website which is also a bit larger (998 sarcastic and 997 non-sarcastic texts).
Accuracies and F-scores of all the classifiers at varying T-SVD size are plotted in Figure FIGREF74 , best values of F-score, precision and recall are reported in column IAC-Sarcastic of Table TABREF49 . The best result (F= INLINEFORM0 ) is lower than in SarcasmCorpus, despite IAC-Sarcastic being balanced and larger than SarcasmCorpus. With Traditional LSA the INLINEFORM1 -scores are generally slightly lower, but the precision values are higher.
The results from Poira et al. cambria2016 are significantly higher, suggesting that in this dataset the sarcasm can be detected in most cases with the linguistic features used by their network independently from the context.
For the irony-context corpus, we used the same 1949 documents selected for the experiments reported in Wallace et al. wallace2014humans. To allow fair comparisons, we used only the texts of the comments, without any contextual information.
The authors report a mean F-score over the five-fold of 0.383 by using a bag-of-words representation with 50.000 tokens, plus some other binary features that have proven useful in other works, and an SVM classifier with a linear kernel. Our results are plotted in Figure FIGREF78 and reported in column irony-context of Table TABREF49 , where it is shown how our classifiers clearly outperform the baseline. Our maximum F-score of INLINEFORM0 represents a relative improvement of 20%. Moreover, it is important to highlight the incredibly low values obtained in this corpus when compared with the results from the previous corpora. This is certainly due to the high skewness between the classes; in fact, the positive samples are just 537 over 1949 (27.5%). If we consider that in SarcasmCorpus the sarcastic texts are only 33% of the total, we suppose there are other causes. Another reason that can explain the poor results can be found in the diversity of topics, as the texts are extracted from six different forums, and the words used for sarcasm can be highly specific to a given context, both cultural and topical. In Wallace et al. wallace2014humans it is explicitly said that the request of the context from an annotator is high for the sarcastic texts. As a consequence, classifying correctly the texts without a context is difficult even for humans. Moreover, the forums from which the posts were extracted are highly controversial, as they regard politics or religion. As a consequence, it is difficult to grasp the sarcasm of a text without knowing the author's opinions.
The results with Traditional LSA are very similar to Statistical LSA, and the real surprise is the incredibly low scores obtained by the random forest and gradient boosting methods.
In this case, we wanted to compare our results against those from Oraby et al. oraby2016creating, which deal with the three sub-corpora separately. However, they are not directly comparable because at the moment in which we report these results only half of the corpus has been released, consisting of 3260 posts in the generic sub-corpus, 582 in the Hyperbole side and 850 for rhetorical questions. The three sub-corpora are all balanced.
Results computed on the three subcorpora are plotted in Figures FIGREF75 , FIGREF76 , FIGREF77 and reported in the last three columns of table TABREF50 . Despite the difference in data availability, the results are quite encouraging. In fact, we can see that our method reaches the INLINEFORM0 - score of INLINEFORM1 in the generic sub-corpus, slightly better than the previous study. Moreover, it improves over Oraby et al. (2016) also in the other two sub-corpora but using Traditional LSA.
Nonetheless, these results show that it is possible to achieve very good performance when high-quality labeled corpora are available, even with a limited number of examples.
For the CNN, we have results only in the generic sub-corpus, and this is the only case in which at least one of our models can outperform it in terms of F-score.
SemEval 2018 Task 3A
The last experiment on a single dataset was performed on the settings of SemEval 2018 Task 3A (Van Hee et al. 2018), which is a shared task on a binary classification of irony, which we introduced in Section SECREF47 .
We start by performing 10-fold cross-validation with our classifiers over varying LSA dimensionality to choose the best setting. We used the same set of hyper-parameters used for the previous experiments.
Once we have found the best setting, we train again the model with all the data and predict the classes of the test tweets. We found that we obtain the best results in cross-validation with LSA vectors of size 20, and the results are presented in Table TABREF59 . We list results for four different classifiers, namely logistic regression, support vector machine, gradient boosting and random forest. In this case, we get the best results using random forests, followed by gradient boosting. In particular, random forest obtains a F INLINEFORM0 -score of INLINEFORM1 , which is higher than the 6-th submission. It is worth noting that the submissions that we listed in the Table, except for the baseline, all use approaches based on deep learning. Compared to the unigram SVM baseline used for the shared task (row 11 in table 4), our model with the random forest is clearly better according to all the metrics, while our model with SVM is better in terms of F INLINEFORM2 score but not accuracy.
Surely the model we provide is not the best one in terms of accuracy, and showing its superiority among all the others does not represent the goal of this work, but the best performers, i.e. deep learning networks, involve an high number of parameters and high computational training cost. Moreover, there are additional interesting notes. First, the submission by BIBREF50 also makes use of deep neural networks but does not get a higher score than our best. Second, the submission by BIBREF51 is using SVMs over syntactic, semantic, and affective features, but still is not better than our best score. The models that showed a clear superiority use deep networks pre-trained on external data to extract more meaningful features. Thus, while the advantage is real, the number of parameters and the amount of data used is much higher.
Inter-corpora Experiments
The second group of experiments is aimed at finding whether the sarcasm is domain-dependent, or the knowledge acquired over one dataset can be transferred to another. We evaluate the similarity among the datasets by training a model over all the data of a corpus and using a second corpus as a test set. Our best results for every corpus pair are listed in Tables TABREF62 and TABREF63 , where the rows indicate the training set and the columns the test set. Quite interestingly, unlike the in-corpus experiments where the logistic regression works better in some cases, all the top scores that we report for these experiments are obtained by using the SVM classifier.
In Table TABREF62 we find the results for SarcasmCorpus and IAC-Sarcastic used as test sets. For the case of SarcasmCorpus, the F-scores are quite low compared to the in-corpus experiments. In fact, here we obtain the best result of only INLINEFORM0 when IAC-Sarcastic is the training set, which is much lower than the scores of about 70 that we get in the in-corpus experiments (column SarcasmCorpus in table TABREF48 ). The low results suggest us that the sarcasm conveyed by the texts in SarcasmCorpus is somehow different from what we can observe in the other corpora.
When we use IAC-Sarcastic as a test set, we can observe higher scores (column IAC-Sarcastic in table TABREF62 ), and the F-score of INLINEFORM0 that we obtain by training in IAC-Sarcastic-v2 is comparable to the INLINEFORM1 , which is the best result in the in-corpus experiments. Also, the lower result, which we obtain when training on irony-context, is quite close to the result obtained for the in-corpus experiment, and unexpected since the poor results obtained in the in-corpus experiments for irony-context (column Irony-Context in table TABREF49 ). When irony-context is the test set (first three columns of table TABREF63 ), we can observe again that the F-score obtained by training in IAC-Sarcastic-v2 is higher than the score obtained in the in-corpus experiment. Nonetheless, all the scores for this test set are lower than INLINEFORM2 with high recalls and low precisions.
When using IAC-Sarcastic-v2 as the test set (see last three columns of Table TABREF63 ) we can observe F-scores between INLINEFORM0 and INLINEFORM1 and are characterized by a high recall and lower precision. The top F1 score is obtained when using IAC-Sarcastic as a training set, which also corresponds to the highest precision. This represents a further proof in favor of the similarity of the two corpora. The top recall score of INLINEFORM2 is obtained by training on SarcasmCorpus, but the precision is much lower than the other two cases.
Overall, it is worth noting that, for all the experiments, the top results are obtained by training on either IAC-Sarcastic or IAC-Sarcastic-v2, while SarcasmCorpus is always better than irony-context. Considering that the quality of the features depends on the quality of the data and of the annotation, we suppose that the quality of the first two datasets is higher than the quality of irony-context, while the data contained in SarcasmCorpus are too different from the other corpora. A deeper analysis of the corpora can be found in the discussion (Section SECREF71 ).
Union Experiments
The last group of experiments we ran has the goal of understanding whether the combination of data coming from different sources can influence positively the final score. For this purpose, as anticipated in Section SECREF51 , we computed 10-folds of each of the four corpora used for the first group of experiments, and used as a training set the concatenation of 9 folds of every corpus, and as a validation set the remaining single folds of each corpus.
From Tables TABREF64 , TABREF65 we can observe that these results are not higher overall with respect to the inter-corpora results. The only exceptions are SarcasmCorpus, where the results are almost 20 F-score points higher than those obtained in the inter-corpora; and IAC-v2, where the gradient boosting (XGB) obtains 2 F-score points more than the top score in the inter-corpora results.
The results on SarcasmCorpus are still lower than the in-corpus results, and the scores of random forest and gradient boosting are much lower than the other two methods. This is further evidence that adding diverse data is not helpful, or is actually harmful, for classifying SarcasmCorpus.
The general trend of this block of experiments is that our classifiers are not able to leverage data from different domains in order to improve global results. In-domain data represent the best choice even if the data amount is lower.
Discussion
In this section, we discuss our results from a more general point of view. We start by briefly discussing the content of the different corpora. Then we try to relate the results of the different types of experiments. Finally, we detect the limits of our experiments for the type of documents we worked with.
The corpora we used for our experiments are characterized by high internal variability in style, as each corpus consists of texts from thousands of different authors. Despite the number of authors, there are some factors that depend on the type of text and the medium. For instance, the irony-context, IAC Sarcastic, and IAC Sarcastic v2 corpora are made of posts collected from online forums, which are mostly about politics. Most of the texts are extracted from longer arguments, and thus the style is informal and in general with aggressive tones.
In Tables TABREF67 , TABREF68 and TABREF69 we show some randomly selected samples from these corpora. As it is apparent from the samples, the posts have a target to attack, who can be another user or the subject of the discussion. Table TABREF67 shows some examples from IAC-Sarcastic. In all the examples the author attacks another user or his opinions. For instance, the first and the third sarcastic examples make sarcasm about the Bible to attack another user's religious ideas, while in the second example the author uses sarcasm to expose a fallacious position of another user and do not appear rude on his side. By contrast, the non-sarcastic examples are much more direct about their meaning. A similar pattern can be found in the examples from IAC Sarcastic v2 (table TABREF69 ). Sarcasm is again used to attack a person (first example) or his/her opinions (second example), maybe religious. The third example shows that also in this corpus some sentences are hard to classify. In this case, the information that we get is that the target has ultraconservative ideas, but it is not easy to grasp the sarcasm. The examples from irony-context (in table TABREF68 ) are much more difficult to grasp without knowing contextual information. For instance, the first sarcastic example can be either sarcastic or regular according to the political opinion of the author. It is sarcastic if the author is a Republican, it is not sarcastic (but would appear strange to write) if the author is a Democrat. The second and the third examples are hard to classify without knowing the subject of the conversation. The same issue of missing a broader context also appears in the non-sarcastic examples, and the third examples can easily be interpreted as sarcastic by humans. In SarcasmCorpus the situation is different as there is no argument ongoing, and the sarcasm is made against products that the author did not like. In this case, there are many references to the external world and the writing is more passionate in its negative stance. Some samples are shown in Table TABREF66 . The sarcastic examples in table TABREF66 all express a negative sentiment and also use negative words. Sarcasm is used within this negative reviews to attack the product in a more creative way and make the text more fun than a usual negative review. The non-sarcastic reviews, on the other side, give a description of the product and their experience with it, with regular forms of expressing the sentiment (“are also a great feature”, “It is a great little camera”). We suppose that this difference in style is the main obstacle to correct classification of SarcasmCorpus instances in the cross-corpora experiments.
We now discuss the relations among the results of the different experiments to gain some further insights into the sarcastic content of our corpora. From the in-corpus experiments, we obtain good results on SarcasmCorpus, which is the only corpus containing Amazon reviews. Unfortunately, when we train our models in a cross-corpora or all-corpora setting, our results drop dramatically, especially in the cross-corpora case. These results mean that the sarcasm in SarcasmCorpus is conveyed through features that are not present in the other corpora. This is especially true when considering that in the inter-corpora experiments, using SarcasmCorpus as a training set in all cases yields results that are only better than the ones obtained when using irony-context as a training set.
The results on irony-context show that this corpus is much more difficult to classify than the others, as it was pointed out also in the paper that presented it (Wallace et al. 2014), which highlights how the human annotators needed to read the contexts to be sure about the sarcastic posts. In the inter-corpora experiments, the results when training on irony-context are the worst for all the test sets, but only for a few points of F-score, while at first, we could expect dramatically lower results. For us, the previous are strong suggestions that the types of texts present in irony-context are similar to the ones present in IAC-Sarcastic-v2, but the quality is lower. As a consequence, this is a further proof that the datasets annotators do not consider sarcasm and irony two different linguistic phenomena.
The two versions of IAC-Sarcastic have proved to be the easiest to classify when using other corpora for training. The best result in IAC-Sarcastic is obtained in the Union experiment (see Tables TABREF64 , TABREF65 ), and thus it benefits from the higher amount of data, especially from the data from IAC-Sarcastic-v2, as can be observed from the cross-corpora results (Table TABREF62 ).
By contrast, the best results on IAC-Sarcastic-v2 are obtained with the in-corpus experiments, while all the results obtained in the inter-corpora experiments are clearly worse. Among the inter-corpora, training the model with IAC-Sarcastic results in a F_score of INLINEFORM0 , which means a relative decrement of INLINEFORM1 concerning the top score for the intra-corpus experiments of IAC-Sarcastic-v2. It is interesting to note that one cause of the decrement can also be the size of corpora, in fact, IAC-Sarcastic contains only 1995 texts, while IAC-Sarcastic-v2 contains 3260.
One final remark is about the absolute scores obtained in the in-corpus experiments. In fact, we can notice that in SarcasmCorpus the F_score can go beyond INLINEFORM0 , and up to INLINEFORM1 by adding the star rating as a feature. The high result can be explained by the peculiarity of this corpus, where sarcasm is present mostly in negative reviews, and the star label is the single best indicator of sarcasm BIBREF49 . The other corpora consist of texts that belong to a thread of forum posts. Sometimes it is reasonable to classify such posts as sarcastic or not out of context, but in many cases, it is impossible also for humans (see examples in Table TABREF68 ). In fact, the low F_score in irony-context is due to low precision, which is an indicator of high similarity between the positive and negative classes. Moreover, low precision and higher recall is a pattern that is present in most of the experiments, even if with higher absolute numbers. The combination of high recall and lower precision suggests that the dubious texts are classified as sarcastic more often than not sarcastic.
Conclusions
In this work, we have tackled the problem of automatic sarcasm detection from a data-driven point of view. More in details, we have used a set of labeled dataset and applied distributional semantics followed by some machine learning approaches in order to give a baseline for the literature in managing such a problem. We do not differentiate between sarcasm and irony because they are not so easily distinguishable even for human experts. Experiments have been carried out on four different corpora containing texts from online reviews or forums, and the corpus used for the shared task on irony detection on Twitter proposed in SemEval 2018. We have shown experimentally that some basic methods can outperform in all the datasets other methods based on bag of words and linguistic features, thus representing a solid baseline. With our experiments that train the models with one corpus and test them by using the other corpora, we have confirmed experimentally that also the annotators tend to not distinguish the distinction between irony and sarcasm. By contrast, major differences can be found according to the text domains, i.e., review vs. political forum. The domain difference can also prevent the method from taking benefits from more data when they are too diverse from the test data. As a future work, we will try to improve distributional semantics approaches with linguistic features in order to perform more fair comparisons with more recent and advanced methods. Furthermore, we will exploit more classical AI methodologies (e.g., by using ontologies, reasoners, common-sense reasoning techniques, etc.) to deduce the context, understanding the concepts expressed in a sentence, exploiting also features like hashtags and emojis to improve the overall performance of the approach.

Question: Do they evaluate only on English datasets?
Answer: Unanswerable

Question: What baseline models are used?
Answer: Unanswerable

Question: What classical machine learning algorithms are used?
Answer: Support Vector Machine (SVM), Logistic regression (Log.Reg), Random Forest (RF), gradient boosting (XGB)

Question: What are the different methods used for different corpora?
Answer: Support Vector Machine (SVM), Logistic regression (Log.Reg), Random Forest (RF) and gradient boosting (XGB)

Question: In which domains is sarcasm conveyed in different ways?
Answer: Amazon reviews