prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. Which Facebook pages did they look at?
Context 1: Introduction
This work is licenced under a Creative Commons Attribution 4 0 International Licence  Licence details: http://creativecommons org/licenses/by/4 0/
In the spirit of the brevity of social media's messages and reactions, people have got used to express feelings minimally and symbolically, as with hashtags on Twitter and Instagram  On Facebook, people tend to be more wordy, but posts normally receive more simple “likes” than longer comments  Since February 2016, Facebook users can express specific emotions in response to a post thanks to the newly introduced reaction feature (see Section SECREF2 ), so that now a post can be wordlessly marked with an expression of say “joy" or “surprise" rather than a generic “like” 
It has been observed that this new feature helps Facebook to know much more about their users and exploit this information for targeted advertising BIBREF0 , but interest in people's opinions and how they feel isn't limited to commercial reasons, as it invests social monitoring, too, including health care and education BIBREF1   However, emotions and opinions are not always expressed this explicitly, so that there is high interest in developing systems towards their automatic detection  Creating manually annotated datasets large enough to train supervised models is not only costly, but also—especially in the case of opinions and emotions—difficult, due to the intrinsic subjectivity of the task BIBREF2 , BIBREF3   Therefore, research has focused on unsupervised methods enriched with information derived from lexica, which are manually created BIBREF3 , BIBREF4   Since go2009twitter have shown that happy and sad emoticons can be successfully used as signals for sentiment labels, distant supervision, i e  using some reasonably safe signals as proxies for automatically labelling training data BIBREF5 , has been used also for emotion recognition, for example exploiting both emoticons and Twitter hashtags BIBREF6 , but mainly towards creating emotion lexica  mohammad2015using use hashtags, experimenting also with highly fine-grained emotion sets (up to almost 600 emotion labels), to create the large Hashtag Emotion Lexicon  Emoticons are used as proxies also by hallsmarmulti, who use distributed vector representations to find which words are interchangeable with emoticons but also which emoticons are used in a similar context 
We take advantage of distant supervision by using Facebook reactions as proxies for emotion labels, which to the best of our knowledge hasn't been done yet, and we train a set of Support Vector Machine models for emotion recognition  Our models, differently from existing ones, exploit information which is acquired entirely automatically, and achieve competitive or even state-of-the-art results for some of the emotion labels on existing, standard evaluation datasets  For explanatory purposes, related work is discussed further and more in detail when we describe the benchmarks for evaluation (Section SECREF3 ) and when we compare our models to existing ones (Section SECREF5 )  We also explore and discuss how choosing different sets of Facebook pages as training data provides an intrinsic domain-adaptation method 
Facebook reactions as labels
For years, on Facebook people could leave comments to posts, and also “like” them, by using a thumbs-up feature to explicitly express a generic, rather underspecified, approval  A “like” could thus mean “I like what you said", but also “I like that you bring up such topic (though I find the content of the article you linked annoying)" 
In February 2016, after a short trial, Facebook made a more explicit reaction feature available world-wide  Rather than allowing for the underspecified “like” as the only wordless response to a post, a set of six more specific reactions was introduced, as shown in Figure FIGREF1 : Like, Love, Haha, Wow, Sad and Angry  We use such reactions as proxies for emotion labels associated to posts 
We collected Facebook posts and their corresponding reactions from public pages using the Facebook API, which we accessed via the Facebook-sdk python library  We chose different pages (and therefore domains and stances), aiming at a balanced and varied dataset, but we did so mainly based on intuition (see Section SECREF4 ) and with an eye to the nature of the datasets available for evaluation (see Section SECREF5 )  The choice of which pages to select posts from is far from trivial, and we believe this is actually an interesting aspect of our approach, as by using different Facebook pages one can intrinsically tackle the domain-adaptation problem (See Section SECREF6 for further discussion on this)  The final collection of Facebook pages for the experiments described in this paper is as follows: FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney 
Note that thankful was only available during specific time spans related to certain events, as Mother's Day in May 2016 
For each page, we downloaded the latest 1000 posts, or the maximum available if there are fewer, from February 2016, retrieving the counts of reactions for each post  The output is a JSON file containing a list of dictionaries with a timestamp, the post and a reaction vector with frequency values, which indicate how many users used that reaction in response to the post (Figure FIGREF3 )  The resulting emotion vectors must then be turned into an emotion label 
In the context of this experiment, we made the simple decision of associating to each post the emotion with the highest count, ignoring like as it is the default and most generic reaction people tend to use  Therefore, for example, to the first post in Figure FIGREF3 , we would associate the label sad, as it has the highest score (284) among the meaningful emotions we consider, though it also has non-zero scores for other emotions  At this stage, we didn't perform any other entropy-based selection of posts, to be investigated in future work 
Emotion datasets
Three datasets annotated with emotions are commonly used for the development and evaluation of emotion detection systems, namely the Affective Text dataset, the Fairy Tales dataset, and the ISEAR dataset  In order to compare our performance to state-of-the-art results, we have used them as well  In this Section, in addition to a description of each dataset, we provide an overview of the emotions used, their distribution, and how we mapped them to those we obtained from Facebook posts in Section SECREF7   A summary is provided in Table TABREF8 , which also shows, in the bottom row, what role each dataset has in our experiments: apart from the development portion of the Affective Text, which we used to develop our models (Section SECREF4 ), all three have been used as benchmarks for our evaluation 
Affective Text dataset
Task 14 at SemEval 2007 BIBREF7 was concerned with the classification of emotions and valence in news headlines  The headlines where collected from several news websites including Google news, The New York Times, BBC News and CNN  The used emotion labels were Anger, Disgust, Fear, Joy, Sadness, Surprise, in line with the six basic emotions of Ekman's standard model BIBREF8   Valence was to be determined as positive or negative  Classification of emotion and valence were treated as separate tasks  Emotion labels were not considered as mututally exclusive, and each emotion was assigned a score from 0 to 100  Training/developing data amounted to 250 annotated headlines (Affective development), while systems were evaluated on another 1000 (Affective test)  Evaluation was done using two different methods: a fine-grained evaluation using Pearson's r to measure the correlation between the system scores and the gold standard; and a coarse-grained method where each emotion score was converted to a binary label, and precision, recall, and f-score were computed to assess performance  As it is done in most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , we also treat this as a classification problem (coarse-grained)  This dataset has been extensively used for the evaluation of various unsupervised methods BIBREF2 , but also for testing different supervised learning techniques and feature portability BIBREF10  
Fairy Tales dataset
This is a dataset collected by alm2008affect, where about 1,000 sentences from fairy tales (by B  Potter, H C  Andersen and Grimm) were annotated with the same six emotions of the Affective Text dataset, though with different names: Angry, Disgusted, Fearful, Happy, Sad, and Surprised  In most works that use this dataset BIBREF3 , BIBREF4 , BIBREF9 , only sentences where all annotators agreed are used, and the labels angry and disgusted are merged  We adopt the same choices 
ISEAR
The ISEAR (International Survey on Emotion Antecedents and Reactions BIBREF11 , BIBREF12 ) is a dataset created in the context of a psychology project of the 1990s, by collecting questionnaires answered by people with different cultural backgrounds  The main aim of this project was to gather insights in cross-cultural aspects of emotional reactions  Student respondents, both psychologists and non-psychologists, were asked to report situations in which they had experienced all of seven major emotions (joy, fear, anger, sadness, disgust, shame and guilt)  In each case, the questions covered the way they had appraised a given situation and how they reacted  The final dataset contains reports by approximately 3000 respondents from all over the world, for a total of 7665 sentences labelled with an emotion, making this the largest dataset out of the three we use 
Overview of datasets and emotions
We summarise datasets and emotion distribution from two viewpoints  First, because there are different sets of emotions labels in the datasets and Facebook data, we need to provide a mapping and derive a subset of emotions that we are going to use for the experiments  This is shown in Table TABREF8 , where in the “Mapped” column we report the final emotions we use in this paper: anger, joy, sadness, surprise  All labels in each dataset are mapped to these final emotions, which are therefore the labels we use for training and testing our models 
Second, the distribution of the emotions for each dataset is different, as can be seen in Figure FIGREF9  
In Figure FIGREF9 we also provide the distribution of the emotions anger, joy, sadness, surprise per Facebook page, in terms of number of posts (recall that we assign to a post the label corresponding to the majority emotion associated to it, see Section SECREF2 )  We can observe that for example pages about news tend to have more sadness and anger posts, while pages about cooking and tv-shows have a high percentage of joy posts  We will use this information to find the best set of pages for a given target domain (see Section SECREF5 ) 
Model
There are two main decisions to be taken in developing our model: (i) which Facebook pages to select as training data, and (ii) which features to use to train the model, which we discuss below  Specifically, we first set on a subset of pages and then experiment with features  Further exploration of the interaction between choice of pages and choice of features is left to future work, and partly discussed in Section SECREF6   For development, we use a small portion of the Affective data set described in Section SECREF4 , that is the portion that had been released as development set for SemEval's 2007 Task 14 BIBREF7 , which contains 250 annotated sentences (Affective development, Section SECREF4 )  All results reported in this section are on this dataset  The test set of Task 14 as well as the other two datasets described in Section SECREF3 will be used to evaluate the final models (Section SECREF4 ) 
Selecting Facebook pages
Although page selection is a crucial ingredient of this approach, which we believe calls for further and deeper, dedicated investigation, for the experiments described here we took a rather simple approach  First, we selected the pages that would provide training data based on intuition and availability, then chose different combinations according to results of a basic model run on development data, and eventually tested feature combinations, still on the development set 
For the sake of simplicity and transparency, we first trained an SVM with a simple bag-of-words model and default parameters as per the Scikit-learn implementation BIBREF13 on different combinations of pages  Based on results of the attempted combinations as well as on the distribution of emotions in the development dataset (Figure FIGREF9 ), we selected a best model (B-M), namely the combined set of Time, The Guardian and Disney, which yields the highest results on development data  Time and The Guardian perform well on most emotions but Disney helps to boost the performance for the Joy class 
Features
In selecting appropriate features, we mainly relied on previous work and intuition  We experimented with different combinations, and all tests were still done on Affective development, using the pages for the best model (B-M) described above as training data  Results are in Table TABREF20   Future work will further explore the simultaneous selection of features and page combinations 
We use a set of basic text-based features to capture the emotion class  These include a tf-idf bag-of-words feature, word (2-3) and character (2-5) ngrams, and features related to the presence of negation words, and to the usage of punctuation 
This feature is used in all unsupervised models as a source of information, and we mainly include it to assess its contribution, but eventually do not use it in our final model 
We used the NRC10 Lexicon because it performed best in the experiments by BIBREF10 , which is built around the emotions anger, anticipation, disgust, fear, joy, sadness, and surprise, and the valence values positive and negative  For each word in the lexicon, a boolean value indicating presence or absence is associated to each emotion  For a whole sentence, a global score per emotion can be obtained by summing the vectors for all content words of that sentence included in the lexicon, and used as feature 
As additional feature, we also included Word Embeddings, namely distributed representations of words in a vector space, which have been exceptionally successful in boosting performance in a plethora of NLP tasks  We use three different embeddings:
Google embeddings: pre-trained embeddings trained on Google News and obtained with the skip-gram architecture described in BIBREF14   This model contains 300-dimensional vectors for 3 million words and phrases 
Facebook embeddings: embeddings that we trained on our scraped Facebook pages for a total of 20,000 sentences  Using the gensim library BIBREF15 , we trained the embeddings with the following parameters: window size of 5, learning rate of 0 01 and dimensionality of 100  We filtered out words with frequency lower than 2 occurrences 
Retrofitted embeddings: Retrofitting BIBREF16 has been shown as a simple but efficient way of informing trained embeddings with additional information derived from some lexical resource, rather than including it directly at the training stage, as it's done for example to create sense-aware BIBREF17 or sentiment-aware BIBREF18 embeddings  In this work, we retrofit general embeddings to include information about emotions, so that emotion-similar words can get closer in space  Both the Google as well as our Facebook embeddings were retrofitted with lexical information obtained from the NRC10 Lexicon mentioned above, which provides emotion-similarity for each token  Note that differently from the previous two types of embeddings, the retrofitted ones do rely on handcrafted information in the form of a lexical resource 
Results on development set
We report precision, recall, and f-score on the development set  The average f-score is reported as micro-average, to better account for the skewed distribution of the classes as well as in accordance to what is usually reported for this task BIBREF19  
From Table TABREF20 we draw three main observations  First, a simple tf-idf bag-of-word mode works already very well, to the point that the other textual and lexicon-based features don't seem to contribute to the overall f-score (0 368), although there is a rather substantial variation of scores per class  Second, Google embeddings perform a lot better than Facebook embeddings, and this is likely due to the size of the corpus used for training  Retrofitting doesn't seem to help at all for the Google embeddings, but it does boost the Facebook embeddings, leading to think that with little data, more accurate task-related information is helping, but corpus size matters most  Third, in combination with embeddings, all features work better than just using tf-idf, but removing the Lexicon feature, which is the only one based on hand-crafted resources, yields even better results  Then our best model (B-M) on development data relies entirely on automatically obtained information, both in terms of training data as well as features 
Results
In Table TABREF26 we report the results of our model on the three datasets standardly used for the evaluation of emotion classification, which we have described in Section SECREF3  
Our B-M model relies on subsets of Facebook pages for training, which were chosen according to their performance on the development set as well as on the observation of emotions distribution on different pages and in the different datasets, as described in Section SECREF4   The feature set we use is our best on the development set, namely all the features plus Google-based embeddings, but excluding the lexicon  This makes our approach completely independent of any manual annotation or handcrafted resource  Our model's performance is compared to the following systems, for which results are reported in the referred literature  Please note that no other existing model was re-implemented, and results are those reported in the respective papers 
Discussion, conclusions and future work
We have explored the potential of using Facebook reactions in a distant supervised setting to perform emotion classification  The evaluation on standard benchmarks shows that models trained as such, especially when enhanced with continuous vector representations, can achieve competitive results without relying on any handcrafted resource  An interesting aspect of our approach is the view to domain adaptation via the selection of Facebook pages to be used as training data 
We believe that this approach has a lot of potential, and we see the following directions for improvement  Feature-wise, we want to train emotion-aware embeddings, in the vein of work by tang:14, and iacobacci2015sensembed  Retrofitting FB-embeddings trained on a larger corpus might also be successful, but would rely on an external lexicon 
The largest room for yielding not only better results but also interesting insights on extensions of this approach lies in the choice of training instances, both in terms of Facebook pages to get posts from, as well as in which posts to select from the given pages  For the latter, one could for example only select posts that have a certain length, ignore posts that are only quotes or captions to images, or expand posts by including content from linked html pages, which might provide larger and better contexts BIBREF23   Additionally, and most importantly, one could use an entropy-based measure to select only posts that have a strong emotion rather than just considering the majority emotion as training label  For the former, namely the choice of Facebook pages, which we believe deserves the most investigation, one could explore several avenues, especially in relation to stance-based issues BIBREF24   In our dataset, for example, a post about Chile beating Colombia in a football match during the Copa America had very contradictory reactions, depending on which side readers would cheer for  Similarly, the very same political event, for example, would get very different reactions from readers if it was posted on Fox News or The Late Night Show, as the target audience is likely to feel very differently about the same issue  This also brings up theoretical issues related more generally to the definition of the emotion detection task, as it's strongly dependent on personal traits of the audience  Also, in this work, pages initially selected on availability and intuition were further grouped into sets to make training data according to performance on development data, and label distribution  Another criterion to be exploited would be vocabulary overlap between the pages and the datasets 
Lastly, we could develop single models for each emotion, treating the problem as a multi-label task  This would even better reflect the ambiguity and subjectivity intrinsic to assigning emotions to text, where content could be at same time joyful or sad, depending on the reader 
Acknowledgements
In addition to the anonymous reviewers, we want to thank Lucia Passaro and Barbara Plank for insightful discussions, and for providing comments on draft versions of this paper 
Context 2: Introduction
Microblogging such as Twitter and Weibo is a popular social networking service, which allows users to post messages up to 140 characters  There are millions of active users on the platform who stay connected with friends  Unfortunately, spammers also use it as a tool to post malicious links, send unsolicited messages to legitimate users, etc  A certain amount of spammers could sway the public opinion and cause distrust of the social platform  Despite the use of rigid anti-spam rules, human-like spammers whose homepages having photos, detailed profiles etc  have emerged  Unlike previous "simple" spammers, whose tweets contain only malicious links, those "smart" spammers are more difficult to distinguish from legitimate users via content-based features alone BIBREF0  
There is a considerable amount of previous work on spammer detection on social platforms  Researcher from Twitter Inc  BIBREF1 collect bot accounts and perform analysis on the user behavior and user profile features  Lee et al  lee2011seven use the so-called social honeypot by alluring social spammers' retweet to build a benchmark dataset, which has been extensively explored in our paper  Some researchers focus on the clustering of urls in tweets and network graph of social spammers BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , showing the power of social relationship features As for content information modeling, BIBREF6 apply improved sparse learning methods  However, few studies have adopted topic-based features  Some researchers BIBREF7 discuss topic characteristics of spamming posts, indicating that spammers are highly likely to dwell on some certain topics such as promotion  But this may not be applicable to the current scenario of smart spammers 
In this paper, we propose an efficient feature extraction method  In this method, two new topic-based features are extracted and used to discriminate human-like spammers from legitimate users  We consider the historical tweets of each user as a document and use the Latent Dirichlet Allocation (LDA) model to compute the topic distribution for each user  Based on the calculated topic probability, two topic-based features, the Local Outlier Standard Score (LOSS) which captures the user's interests on different topics and the Global Outlier Standard Score (GOSS) which reveals the user's interests on specific topic in comparison with other users', are extracted  The two features contain both local and global information, and the combination of them can distinguish human-like spammers effectively 
To the best of our knowledge, it is the first time that features based on topic distributions are used in spammer classification  Experimental results on one public dataset and one self-collected dataset further validate that the two sets of extracted topic-based features get excellent performance on human-like spammer classification problem compared with other state-of-the-art methods  In addition, we build a Weibo dataset, which contains both legitimate users and spammers 
To summarize, our major contributions are two-fold:
In the following sections, we first propose the topic-based features extraction method in Section 2, and then introduce the two datasets in Section 3  Experimental results are discussed in Section 4, and we conclude the paper in Section 5  Future work is presented in Section 6 
Methodology
In this section, we first provide some observations we obtained after carefully exploring the social network, then the LDA model is introduced  Based on the LDA model, the ways to obtain the topic probability vector for each user and the two topic-based features are provided 
Observation
After exploring the homepages of a substantial number of spammers, we have two observations  1) social spammers can be divided into two categories  One is content polluters, and their tweets are all about certain kinds of advertisement and campaign  The other is fake accounts, and their tweets resemble legitimate users' but it seems they are simply random copies of others to avoid being detected by anti-spam rules  2) For legitimate users, content polluters and fake accounts, they show different patterns on topics which interest them 
Legitimate users mainly focus on limited topics which interest him  They seldom post contents unrelated to their concern 
Content polluters concentrate on certain topics 
Fake accounts focus on a wide range of topics due to random copying and retweeting of other users' tweets 
Spammers and legitimate users show different interests on some topics e g  commercial, weather, etc 
To better illustrate our observation, Figure  1 shows the topic distribution of spammers and legitimate users in two employed datasets(the Honeypot dataset and Weibo dataset)  We can see that on both topics (topic-3 and topic-11) there exists obvious difference between the red bars and green bars, representing spammers and legitimate users  On the Honeypot dataset, spammers have a narrower shape of distribution (the outliers on the red bar tail are not counted) than that of legitimate users  This is because there are more content polluters than fake accounts  In other word, spammers in this dataset tend to concentrate on limited topics  While on the Weibo dataset, fake accounts who are interested in different topics take large proportion of spammers  Their distribution is more flat (i e  red bars) than that of the legitimate users  Therefore we can detect spammers by means of the difference of their topic distribution patterns 
LDA model
Blei et al blei2003latent first presented Latent Dirichlet Allocation(LDA) as an example of topic model 
Each document $i$ is deemed as a bag of words $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $ and $M$ is the number of words  Each word is attributable to one of the document's topics $Z=\left\lbrace  z_{i1},z_{i2},   ,z_{iK}\right\rbrace $ and $K$ is the number of topics  $\psi _{k}$ is a multinomial distribution over words for topic $k$   $\theta _i$ is another multinomial distribution over topics for document $i$   The smoothed generative model is illustrated in Figure  2   $\alpha $ and $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $0 are hyper parameter that affect scarcity of the document-topic and topic-word distributions  In this paper, $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $1 , $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $2 and $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $3 are empirically set to 0 3, 0 01 and 15  The entire content of each Twitter user is regarded as one document  We adopt Gibbs Sampling BIBREF8 to speed up the inference of LDA  Based on LDA, we can get the topic probabilities for all users in the employed dataset as: $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $4 , where $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $5 is the number of users  Each element $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $6 is a topic probability vector for the $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $7 document  $W=\left\lbrace  w_{i1},w_{i2},   ,w_{iM}\right\rbrace $8 is the raw topic probability vector and our features are developed on top of it 
Topic-based Features
Using the LDA model, each person in the dataset is with a topic probability vector $X_i$   Assume $x_{ik}\in X_{i}$ denotes the likelihood that the $\emph {i}^{th}$ tweet account favors $\emph {k}^{th}$ topic in the dataset  Our topic based features can be calculated as below 
Global Outlier Standard Score measures the degree that a user's tweet content is related to a certain topic compared to the other users  Specifically, the "GOSS" score of user $i$ on topic $k$ can be calculated as Eq ( 12 ):
$$\centering \begin{array}{ll} \mu \left(x_{k}\right)=\frac{\sum _{i=1}^{n} x_{ik}}{n},\\ GOSS\left(x_{ik}\right)=\frac{x_{ik}-\mu \left(x_k\right)}{\sqrt{\underset{i}{\sum }\left(x_{ik}-\mu \left(x_{k}\right)\right)^{2}}}  \end{array}$$   (Eq  12)
The value of $GOSS\left(x_{ik}\right)$ indicates the interesting degree of this person to the $\emph {k}^{th}$ topic  Specifically, if $GOSS\left(x_{ik}\right)$ > $GOSS\left(x_{jk}\right)$ , it means that the $\emph {i}^{th}$ person has more interest in topic $k$ than the $\emph {j}^{th}$ person  If the value $GOSS\left(x_{ik}\right)$ is extremely high or low, the $\emph {i}^{th}$ person showing extreme interest or no interest on topic $k$ which will probably be a distinctive pattern in the fowllowing classfication  Therefore, the topics interested or disliked by the $\emph {k}^{th}$0 person can be manifested by $\emph {k}^{th}$1 , from which the pattern of the interested topics with regarding to this person is found  Denote $\emph {k}^{th}$2 our first topic-based feature, and it hopefully can get good performance on spammer detection 
Local Outlier Standard Score measures the degree of interest someone shows to a certain topic by considering his own homepage content only  For instance, the "LOSS" score of account $i$ on topic $k$ can be calculated as Eq ( 13 ):
$$\centering \begin{array}{ll} \mu \left(x_{i}\right)=\frac{\sum _{k=1}^{K} x_{ik}}{K},\\ LOSS\left(x_{ik}\right)=\frac{x_{ik}-\mu \left(x_i\right)}{\sqrt{\underset{k}{\sum }\left(x_{ik}-\mu \left(x_{i}\right)\right)^{2}}}  \end{array}$$   (Eq  13)
$\mu (x_i)$ represents the averaged interesting degree for all topics with regarding to $\emph {i}^{th}$ user and his tweet content  Similarly to $GOSS$ , the topics interested or disliked by the $\emph {i}^{th}$ person via considering his single post information can be manifested by $f_{LOSS}^{i}=[LOSS(x_{i1})\cdots LOSS(x_{iK})]$ , and $LOSS$ becomes our second topic-based features for the $\emph {i}^{th}$ person 
Dataset
We use one public dataset Social Honeypot dataset and one self-collected dataset Weibo dataset to validate the effectiveness of our proposed features 
Social Honeypot Dataset: Lee et al  lee2010devils created and deployed 60 seed social accounts on Twitter to attract spammers by reporting back what accounts interact with them  They collected 19,276 legitimate users and 22,223 spammers in their datasets along with their tweet content in 7 months  This is our first test dataset 
Our Weibo Dataset: Sina Weibo is one of the most famous social platforms in China  It has implemented many features from Twitter  The 2197 legitimate user accounts in this dataset are provided by the Tianchi Competition held by Sina Weibo  The spammers are all purchased commercially from multiple vendors on the Internet  We checked them manually and collected 802 suitable "smart" spammers accounts 
Preprocessing: Before directly performing the experiments on the employed datasets, we first delete some accounts with few posts in the two employed since the number of tweets is highly indicative of spammers  For the English Honeypot dataset, we remove stopwords, punctuations, non-ASCII words and apply stemming  For the Chinese Weibo dataset, we perform segmentation with "Jieba", a Chinese text segmentation tool  After preprocessing steps, the Weibo dataset contains 2197 legitimate users and 802 spammers, and the honeypot dataset contains 2218 legitimate users and 2947 spammers  It is worth mentioning that the Honeypot dataset has been slashed because most of the Twitter accounts only have limited number of posts, which are not enough to show their interest inclination 
Evaluation Metrics
The evaluating indicators in our model are show in 2   We calculate precision, recall and F1-score (i e  F1 score) as in Eq  ( 19 )  Precision is the ratio of selected accounts that are spammers  Recall is the ratio of spammers that are detected so  F1-score is the harmonic mean of precision and recall 
$$precision =\frac{TP}{TP+FP}, recall =\frac{TP}{TP+FN}\nonumber \\ F1-score = \frac{2\times precision \times recall}{precision + recall}$$   (Eq  19)
Performance Comparisons with Baseline
Three baseline classification methods: Support Vector Machines (SVM), Adaboost, and Random Forests are adopted to evaluate our extracted features  We test each classification algorithm with scikit-learn BIBREF9 and run a 10-fold cross validation  On each dataset, the employed classifiers are trained with individual feature first, and then with the combination of the two features  From 1 , we can see that GOSS+LOSS achieves the best performance on F1-score among all others  Besides, the classification by combination of LOSS and GOSS can increase accuracy by more than 3% compared with raw topic distribution probability 
Comparison with Other Features
To compare our extracted features with previously used features for spammer detection, we use three most discriminative feature sets according to Lee et al  lee2011seven( 4 )  Two classifiers (Adaboost and SVM) are selected to conduct feature performance comparisons  Using Adaboost, our LOSS+GOSS features outperform all other features except for UFN which is 2% higher than ours with regard to precision on the Honeypot dataset  It is caused by the incorrectly classified spammers who are mostly news source after our manual check  They keep posting all kinds of news pieces covering diverse topics, which is similar to the behavior of fake accounts  However, UFN based on friendship networks is more useful for public accounts who possess large number of followers  The best recall value of our LOSS+GOSS features using SVM is up to 6% higher than the results by other feature groups  Regarding F1-score, our features outperform all other features  To further show the advantages of our proposed features, we compare our combined LOSS+GOSS with the combination of all the features from Lee et al  lee2011seven (UFN+UC+UH)  It's obvious that LOSS+GOSS have a great advantage over UFN+UC+UH in terms of recall and F1-score  Moreover, by combining our LOSS+GOSS features and UFN+UC+UH features together, we obtained another 7 1% and 2 3% performance gain with regard to precision and F1-score by Adaboost  Though there is a slight decline in terms of recall  By SVM, we get comparative results on recall and F1-score but about 3 5% improvement on precision 
Conclusion
In this paper, we propose a novel feature extraction method to effectively detect "smart" spammers who post seemingly legitimate tweets and are thus difficult to identify by existing spammer classification methods  Using the LDA model, we obtain the topic probability for each Twitter user  By utilizing the topic probability result, we extract our two topic-based features: GOSS and LOSS which represent the account with global and local information  Experimental results on a public dataset and a self-built Chinese microblog dataset validate the effectiveness of the proposed features 
Future Work
In future work, the combination method of local and global information can be further improved to maximize their individual strengths  We will also apply decision theory to enhancing the performance of our proposed features  Moreover, we are also building larger datasets on both Twitter and Weibo to validate our method  Moreover, larger datasets on both Twitter and Weibo will be built to further validate our method 
Context 3: Introduction
Sarcasm is defined as “a sharp, bitter, or cutting expression or remark; a bitter gibe or taunt”  As the fields of affective computing and sentiment analysis have gained increasing popularity BIBREF0 , it is a major concern to detect sarcastic, ironic, and metaphoric expressions  Sarcasm, especially, is key for sentiment analysis as it can completely flip the polarity of opinions  Understanding the ground truth, or the facts about a given event, allows for the detection of contradiction between the objective polarity of the event (usually negative) and its sarcastic characteristic by the author (usually positive), as in “I love the pain of breakup”  Obtaining such knowledge is, however, very difficult 
In our experiments, we exposed the classifier to such knowledge extracted indirectly from Twitter  Namely, we used Twitter data crawled in a time period, which likely contain both the sarcastic and non-sarcastic accounts of an event or similar events  We believe that unambiguous non-sarcastic sentences provided the classifier with the ground-truth polarity of those events, which the classifier could then contrast with the opposite estimations in sarcastic sentences  Twitter is a more suitable resource for this purpose than blog posts, because the polarity of short tweets is easier to detect (as all the information necessary to detect polarity is likely to be contained in the same sentence) and because the Twitter API makes it easy to collect a large corpus of tweets containing both sarcastic and non-sarcastic examples of the same event 
Sometimes, however, just knowing the ground truth or simple facts on the topic is not enough, as the text may refer to other events in order to express sarcasm  For example, the sentence “If Hillary wins, she will surely be pleased to recall Monica each time she enters the Oval Office :P :D”, which refers to the 2016 US presidential election campaign and to the events of early 1990's related to the US president Clinton, is sarcastic because Hillary, a candidate and Clinton's wife, would in fact not be pleased to recall her husband's alleged past affair with Monica Lewinsky  The system, however, would need a considerable amount of facts, commonsense knowledge, anaphora resolution, and logical reasoning to draw such a conclusion  In this paper, we will not deal with such complex cases 
Existing works on sarcasm detection have mainly focused on unigrams and the use of emoticons BIBREF1 , BIBREF2 , BIBREF3 , unsupervised pattern mining approach BIBREF4 , semi-supervised approach BIBREF5 and n-grams based approach BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 with sentiment features  Instead, we propose a framework that learns sarcasm features automatically from a sarcasm corpus using a convolutional neural network (CNN)  We also investigate whether features extracted using the pre-trained sentiment, emotion and personality models can improve sarcasm detection performance  Our approach uses relatively lower dimensional feature vectors and outperforms the state of the art on different datasets  In summary, the main contributions of this paper are the following:
The rest of the paper is organized as follows: Section SECREF2 proposes a brief literature review on sarcasm detection; Section SECREF4 presents the proposed approach; experimental results and thorough discussion on the experiments are given in Section SECREF5 ; finally, Section SECREF6 concludes the paper 
Related Works
NLP research is gradually evolving from lexical to compositional semantics BIBREF10 through the adoption of novel meaning-preserving and context-aware paradigms such as convolutional networks BIBREF11 , recurrent belief networks BIBREF12 , statistical learning theory BIBREF13 , convolutional multiple kernel learning BIBREF14 , and commonsense reasoning BIBREF15   But while other NLP tasks have been extensively investigated, sarcasm detection is a relatively new research topic which has gained increasing interest only recently, partly thanks to the rise of social media analytics and sentiment analysis  Sentiment analysis BIBREF16 and using multimodal information as a new trend BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , BIBREF14 is a popular branch of NLP research that aims to understand sentiment of documents automatically using combination of various machine learning approaches BIBREF21 , BIBREF22 , BIBREF20 , BIBREF23  
An early work in this field was done by BIBREF6 on a dataset of 6,600 manually annotated Amazon reviews using a kNN-classifier over punctuation-based and pattern-based features, i e , ordered sequence of high frequency words  BIBREF1 used support vector machine (SVM) and logistic regression over a feature set of unigrams, dictionary-based lexical features and pragmatic features (e g , emoticons) and compared the performance of the classifier with that of humans  BIBREF24 described a set of textual features for recognizing irony at a linguistic level, especially in short texts created via Twitter, and constructed a new model that was assessed along two dimensions: representativeness and relevance  BIBREF5 used the presence of a positive sentiment in close proximity of a negative situation phrase as a feature for sarcasm detection  BIBREF25 used the Balanced Window algorithm for classifying Dutch tweets as sarcastic vs  non-sarcastic; n-grams (uni, bi and tri) and intensifiers were used as features for classification 
BIBREF26 compared the performance of different classifiers on the Amazon review dataset using the imbalance between the sentiment expressed by the review and the user-given star rating  Features based on frequency (gap between rare and common words), written spoken gap (in terms of difference between usage), synonyms (based on the difference in frequency of synonyms) and ambiguity (number of words with many synonyms) were used by BIBREF3 for sarcasm detection in tweets  BIBREF9 proposed the use of implicit incongruity and explicit incongruity based features along with lexical and pragmatic features, such as emoticons and punctuation marks  Their method is very much similar to the method proposed by BIBREF5 except BIBREF9 used explicit incongruity features  Their method outperforms the approach by BIBREF5 on two datasets 
BIBREF8 compared the performance with different language-independent features and pre-processing techniques for classifying text as sarcastic and non-sarcastic  The comparison was done over three Twitter dataset in two different languages, two of these in English with a balanced and an imbalanced distribution and the third one in Czech  The feature set included n-grams, word-shape patterns, pointedness and punctuation-based features 
In this work, we use features extracted from a deep CNN for sarcasm detection  Some of the key differences between the proposed approach and existing methods include the use of a relatively smaller feature set, automatic feature extraction, the use of deep networks, and the adoption of pre-trained NLP models 
Sentiment Analysis and Sarcasm Detection
Sarcasm detection is an important subtask of sentiment analysis BIBREF27   Since sarcastic sentences are subjective, they carry sentiment and emotion-bearing information  Most of the studies in the literature BIBREF28 , BIBREF29 , BIBREF9 , BIBREF30 include sentiment features in sarcasm detection with the use of a state-of-the-art sentiment lexicon  Below, we explain how sentiment information is key to express sarcastic opinions and the approach we undertake to exploit such information for sarcasm detection 
In general, most sarcastic sentences contradict the fact  In the sentence “I love the pain present in the breakups" (Figure FIGREF4 ), for example, the word “love" contradicts “pain present in the breakups”, because in general no-one loves to be in pain  In this case, the fact (i e , “pain in the breakups") and the contradictory statement to that fact (i e , “I love") express sentiment explicitly  Sentiment shifts from positive to negative but, according to sentic patterns BIBREF31 , the literal sentiment remains positive  Sentic patterns, in fact, aim to detect the polarity expressed by the speaker; thus, whenever the construction “I love” is encountered, the sentence is positive no matter what comes after it (e g , “I love the movie that you hate”)  In this case, however, the sentence carries sarcasm and, hence, reflects the negative sentiment of the speaker 
In another example (Figure FIGREF4 ), the fact, i e , “I left the theater during the interval", has implicit negative sentiment  The statement “I love the movie" contradicts the fact “I left the theater during the interval"; thus, the sentence is sarcastic  Also in this case the sentiment shifts from positive to negative and hints at the sarcastic nature of the opinion 
The above discussion has made clear that sentiment (and, in particular, sentiment shifts) can largely help to detect sarcasm  In order to include sentiment shifting into the proposed framework, we train a sentiment model for sentiment-specific feature extraction  Training with a CNN helps to combine the local features in the lower layers into global features in the higher layers  We do not make use of sentic patterns BIBREF31 in this paper but we do plan to explore that research direction as a part of our future work  In the literature, it is found that sarcasm is user-specific too, i e , some users have a particular tendency to post more sarcastic tweets than others  This acts as a primary intuition for us to extract personality-based features for sarcasm detection 
The Proposed Framework
As discussed in the literature BIBREF5 , sarcasm detection may depend on sentiment and other cognitive aspects  For this reason, we incorporate both sentiment and emotion clues in our framework  Along with these, we also argue that personality of the opinion holder is an important factor for sarcasm detection  In order to address all of these variables, we create different models for each of them, namely: sentiment, emotion and personality  The idea is to train each model on its corresponding benchmark dataset and, hence, use such pre-trained models together to extract sarcasm-related features from the sarcasm datasets 
Now, the viable research question here is - Do these models help to improve sarcasm detection performance?' Literature shows that they improve the performance but not significantly  Thus, do we need to consider those factors in spotting sarcastic sentences? Aren't n-grams enough for sarcasm detection? Throughout the rest of this paper, we address these questions in detail  The training of each model is done using a CNN  Below, we explain the framework in detail  Then, we discuss the pre-trained models  Figure FIGREF6 presents a visualization of the proposed framework 
General CNN Framework
CNN can automatically extract key features from the training data  It grasps contextual local features from a sentence and, after several convolution operations, it forms a global feature vector out of those local features  CNN does not need the hand-crafted features used in traditional supervised classifiers  Such hand-crafted features are difficult to compute and a good guess for encoding the features is always necessary in order to get satisfactory results  CNN, instead, uses a hierarchy of local features which are important to learn context  The hand-crafted features often ignore such a hierarchy of local features 
Features extracted by CNN can therefore be used instead of hand-crafted features, as they carry more useful information  The idea behind convolution is to take the dot product of a vector of INLINEFORM0 weights INLINEFORM1 also known as kernel vector with each INLINEFORM2 -gram in the sentence INLINEFORM3 to obtain another sequence of features INLINEFORM4   DISPLAYFORM0
Thus, a max pooling operation is applied over the feature map and the maximum value INLINEFORM0 is taken as the feature corresponding to this particular kernel vector  Similarly, varying kernel vectors and window sizes are used to obtain multiple features BIBREF32   For each word INLINEFORM1 in the vocabulary, a INLINEFORM2 -dimensional vector representation is given in a look up table that is learned from the data BIBREF33   The vector representation of a sentence, hence, is a concatenation of vectors for individual words  Similarly, we can have look up tables for other features  One might want to provide features other than words if these features are suspected to be helpful  The convolution kernels are then applied to word vectors instead of individual words

ground_truth: FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney

FoxNews, CNN, ESPN, New York Times, Time magazine, Huffington Post Weird News, The Guardian, Cartoon Network, Cooking Light, Home Cooking Adventure, Justin Bieber, Nickelodeon, Spongebob, Disney.

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. LDA is an unsupervised method; is this paper introducing an unsupervised approach to spam detection?
Context 3: 
We use these features to train higher layers of the CNN, in order to represent bigger groups of words in sentences  We denote the feature learned at hidden neuron INLINEFORM0 in layer INLINEFORM1 as INLINEFORM2   Multiple features may be learned in parallel in the same CNN layer  The features learned in each layer are used to train the next layer: DISPLAYFORM0
where * indicates convolution and INLINEFORM0 is a weight kernel for hidden neuron INLINEFORM1 and INLINEFORM2 is the total number of hidden neurons  The CNN sentence model preserves the order of words by adopting convolution kernels of gradually increasing sizes that span an increasing number of words and ultimately the entire sentence  As mentioned above, each word in a sentence is represented using word embeddings 
We employ the publicly available word2vec vectors, which were trained on 100 billion words from Google News  The vectors are of dimensionality 300, trained using the continuous bag-of-words architecture BIBREF33   Words not present in the set of pre-trained words are initialized randomly  However, while training the neural network, we use non-static representations  These include the word vectors, taken as input, into the list of parameters to be learned during training 
Two primary reasons motivated us to use non-static channels as opposed to static ones  Firstly, the common presence of informal language and words in tweets resulted in a relatively high random initialization of word vectors due to the unavailability of these words in the word2vec dictionary  Secondly, sarcastic sentences are known to include polarity shifts in sentimental and emotional degrees  For example, “I love the pain present in breakups" is a sarcastic sentence with a significant change in sentimental polarity  As word2vec was not trained to incorporate these nuances, we allow our models to update the embeddings during training in order to include them  Each sentence is wrapped to a window of INLINEFORM0 , where INLINEFORM1 is the maximum number of words amongst all sentences in the dataset  We use the output of the fully-connected layer of the network as our feature vector 
We have done two kinds of experiments: firstly, we used CNN for the classification; secondly, we extracted features from the fully-connected layer of the CNN and fed them to an SVM for the final classification  The latter CNN-SVM scheme is quite useful for text classification as shown by Poria et al  BIBREF18   We carry out n-fold cross-validation on the dataset using CNN  In every fold iteration, in order to obtain the training and test features, the output of the fully-connected layer is treated as features to be used for the final classification using SVM  Table TABREF12 shows the training settings for each CNN model developed in this work  ReLU is used as the non-linear activation function of the network  The network configurations of all models developed in this work are given in Table TABREF12  
Sentiment Feature Extraction Model
As discussed above, sentiment clues play an important role for sarcastic sentence detection  In our work, we train a CNN (see Section SECREF5 for details) on a sentiment benchmark dataset  This pre-trained model is then used to extract features from the sarcastic datasets  In particular, we use Semeval 2014 BIBREF34 Twitter Sentiment Analysis Dataset for the training  This dataset contains 9,497 tweets out of which 5,895 are positive, 3,131 are negative and 471 are neutral  The fully-connected layer of the CNN used for sentiment feature extraction has 100 neurons, so 100 features are extracted from this pre-trained model  The final softmax determines whether a sentence is positive, negative or neutral  Thus, we have three neurons in the softmax layer 
Emotion Feature Extraction Model
We use the CNN structure as described in Section SECREF5 for emotional feature extraction  As a dataset for extracting emotion-related features, we use the corpus developed by BIBREF35   This dataset consists of blog posts labeled by their corresponding emotion categories  As emotion taxonomy, the authors used six basic emotions, i e , Anger, Disgust, Surprise, Sadness, Joy and Fear  In particular, the blog posts were split into sentences and each sentence was labeled  The dataset contains 5,205 sentences labeled by one of the emotion labels  After employing this model on the sarcasm dataset, we obtained a 150-dimensional feature vector from the fully-connected layer  As the aim of training is to classify each sentence into one of the six emotion classes, we used six neurons in the softmax layer 
Personality Feature Extraction Model
Detecting personality from text is a well-known challenging problem  In our work, we use five personality traits described by BIBREF36 , i e , Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism, sometimes abbreviated as OCEAN (by their first letters)  As a training dataset, we use the corpus developed by BIBREF36 , which contains 2,400 essays labeled by one of the five personality traits each 
The fully-connected layer has 150 neurons, which are treated as the features  We concatenate the feature vector of each personality dimension in order to create the final feature vector  Thus, the personality model ultimately extracts a 750-dimensional feature vector (150-dimensional feature vector for each of the five personality traits)  This network is replicated five times, one for each personality trait  In particular, we create a CNN for each personality trait and the aim of each CNN is to classify a sentence into binary classes, i e , whether it expresses a personality trait or not 
Baseline Method and Features
CNN can also be employed on the sarcasm datasets in order to identify sarcastic and non-sarcastic tweets  We term the features extracted from this network baseline features, the method as baseline method and the CNN architecture used in this baseline method as baseline CNN  Since the fully-connected layer has 100 neurons, we have 100 baseline features in our experiment  This method is termed baseline method as it directly aims to classify a sentence as sarcastic vs non-sarcastic  The baseline CNN extracts the inherent semantics from the sarcastic corpus by employing deep domain understanding  The process of using baseline features with other features extracted from the pre-trained model is described in Section SECREF24  
Experimental Results and Discussion
In this section, we present the experimental results using different feature combinations and compare them with the state of the art  For each feature we show the results using only CNN and using CNN-SVM (i e , when the features extracted by CNN are fed to the SVM)  Macro-F1 measure is used as an evaluation scheme in the experiments 
Sarcasm Datasets Used in the Experiment
This dataset was created by BIBREF8   The tweets were downloaded from Twitter using #sarcasm as a marker for sarcastic tweets  It is a monolingual English dataset which consists of a balanced distribution of 50,000 sarcastic tweets and 50,000 non-sarcastic tweets 
Since sarcastic tweets are less frequently used BIBREF8 , we also need to investigate the robustness of the selected features and the model trained on these features on an imbalanced dataset  To this end, we used another English dataset from BIBREF8   It consists of 25,000 sarcastic tweets and 75,000 non-sarcastic tweets 
We have obtained this dataset from The Sarcasm Detector  It contains 120,000 tweets, out of which 20,000 are sarcastic and 100,000 are non-sarcastic  We randomly sampled 10,000 sarcastic and 20,000 non-sarcastic tweets from the dataset  Visualization of both the original and subset data show similar characteristics 
A two-step methodology has been employed in filtering the datasets used in our experiments  Firstly, we identified and removed all the “user", “URL" and “hashtag" references present in the tweets using efficient regular expressions  Special emphasis was given to this step to avoid traces of hashtags, which might trigger the models to provide biased results  Secondly, we used NLTK Twitter Tokenizer to ensure proper tokenization of words along with special symbols and emoticons  Since our deep CNNs extract contextual information present in tweets, we include emoticons as part of the vocabulary  This enables the emoticons to hold a place in the word embedding space and aid in providing information about the emotions present in the sentence 
Merging the Features
Throughout this research, we have carried out several experiments with various feature combinations  For the sake of clarity, we explain below how the features extracted using difference models are merged 
In the standard feature merging process, we first extract the features from all deep CNN based feature extraction models and then we concatenate them  Afterwards, SVM is employed on the resulted feature vector 
In another setting, we use the features extracted from the pre-trained models as the static channels of features in the CNN of the baseline method  These features are appended to the hidden layer of the baseline CNN, preceding the final output softmax layer 
For comparison, we have re-implemented the state-of-the-art methods  Since BIBREF9 did not mention about the sentiment lexicon they use in the experiment, we used SenticNet BIBREF37 in the re-implementation of their method 
Results on Dataset 1
As shown in Table TABREF29 , for every feature CNN-SVM outperforms the performance of the CNN  Following BIBREF6 , we have carried out a 5-fold cross-validation on this dataset  The baseline features ( SECREF16 ) perform best among other features  Among all the pre-trained models, the sentiment model (F1-score: 87 00%) achieves better performance in comparison with the other two pre-trained models

ground_truth: No

No

prompt_front: You are given a scientific article and a question. Answer the question as concisely as you can, using a single phrase or sentence if possible. If the question cannot be answered based on the information in the article, write "unanswerable". If the question is a yes/no question, answer "yes", "no", or "unanswerable". Do not provide any explanation.

Please answer the questions based on the text provided below. What are the network's baseline features?
Context 3:  Interestingly, when we merge the baseline features with the features extracted by the pre-trained deep NLP models, we only get 0 11% improvement over the F-score  It means that the baseline features alone are quite capable to detect sarcasm  On the other hand, when we combine sentiment, emotion and personality features, we obtain 90 70% F1-score  This indicates that the pre-trained features are indeed useful for sarcasm detection  We also compare our approach with the best research study conducted on this dataset (Table TABREF30 )  Both the proposed baseline model and the baseline + sentiment + emotion + personality model outperform the state of the art BIBREF9 , BIBREF8   One important difference with the state of the art is that BIBREF8 used relatively larger feature vector size ( INLINEFORM0 500,000) than we used in our experiment (1,100)  This not only prevents our model to overfit the data but also speeds up the computation  Thus, we obtain an improvement in the overall performance with automatic feature extraction using a relatively lower dimensional feature space 
In the literature, word n-grams, skipgrams and character n-grams are used as baseline features  According to Ptacek et al  BIBREF8 , these baseline features along with the other features (sentiment features and part-of-speech based features) produced the best performance  However, Ptacek et al  did not analyze the performance of these features when they were not used with the baseline features  Pre-trained word embeddings play an important role in the performance of the classifier because, when we use randomly generated embeddings, performance falls down to 86 23% using all features 
Results on Dataset 2
5-fold cross-validation has been carried out on Dataset 2  Also for this dataset, we get the best accuracy when we use all features  Baseline features have performed significantly better (F1-score: 92 32%) than all other features  Supporting the observations we have made from the experiments on Dataset 1, we see CNN-SVM outperforming CNN on Dataset 2  However, when we use all the features, CNN alone (F1-score: 89 73%) does not outperform the state of the art BIBREF8 (F1-score: 92 37%)  As shown in Table TABREF30 , CNN-SVM on the baseline + sentiment + emotion + personality feature set outperforms the state of the art (F1-score: 94 80%)  Among the pre-trained models, the sentiment model performs best (F1-score: 87 00%) 
Table TABREF29 shows the performance of different feature combinations  The gap between the F1-scores of only baseline features and all features is larger on the imbalanced dataset than the balanced dataset  This supports our claim that sentiment, emotion and personality features are very useful for sarcasm detection, thanks to the pre-trained models  The F1-score using sentiment features when combined with baseline features is 94 60%  On both of the datasets, emotion and sentiment features perform better than the personality features  Interestingly, using only sentiment, emotion and personality features, we achieve 90 90% F1-score 
Results on Dataset 3
Experimental results on Dataset 3 show the similar trends (Table TABREF30 ) as compared to Dataset 1 and Dataset 2  The highest performance (F1-score 93 30%) is obtained when we combine baseline features with sentiment, emotion and personality features  In this case, also CNN-SVM consistently performs better than CNN for every feature combination  The sentiment model is found to be the best pre-trained model  F1-score of 84 43% is obtained when we merge sentiment, emotion and personality features 
Dataset 3 is more complex and non-linear in nature compared to the other two datasets  As shown in Table TABREF30 , the methods by BIBREF9 and BIBREF8 perform poorly on this dataset  The TP rate achieved by BIBREF9 is only 10 07% and that means their method suffers badly on complex data  The approach of BIBREF8 has also failed to perform well on Dataset 3, achieving 62 37% with a better TP rate of 22 15% than BIBREF9   On the other hand, our proposed model performs consistently well on this dataset achieving 93 30% 
Testing Generalizability of the Models and Discussions
To test the generalization capability of the proposed approach, we perform training on Dataset 1 and test on Dataset 3  The F1-score drops down dramatically to 33 05%  In order to understand this finding, we visualize each dataset using PCA (Figure FIGREF17 )  It depicts that, although Dataset 1 is mostly linearly separable, Dataset 3 is not  A linear kernel that performs well on Dataset 1 fails to provide good performance on Dataset 3  If we use RBF kernel, it overfits the data and produces worse results than what we get using linear kernel  Similar trends are seen in the performance of other two state-of-the-art approaches BIBREF9 , BIBREF8   Thus, we decide to perform training on Dataset 3 and test on the Dataset 1  As expected better performance is obtained with F1-score 76 78%  However, the other two state-of-the-art approaches fail to perform well in this setting  While the method by BIBREF9 obtains F1-score of 47 32%, the approach by BIBREF8 achieves 53 02% F1-score when trained on Dataset 3 and tested on Dataset 1  Below, we discuss about this generalizability issue of the models developed or referred in this paper 
As discussed in the introduction, sarcasm is very much topic-dependent and highly contextual  For example, let us consider the tweet “I am so glad to see Tanzania played very well, I can now sleep well :P"  Unless one knows that Tanzania actually did not play well in that game, it is not possible to spot the sarcastic nature of this sentence  Thus, an n-gram based sarcasm detector trained at time INLINEFORM0 may perform poorly to detect sarcasm in the tweets crawled at time INLINEFORM1 (given that there is a considerable gap between these time stamps) because of the diversity of the topics (new events occur, new topics are discussed) of the tweets  Sentiment and other contextual clues can help to spot the sarcastic nature in this kind of tweets  A highly positive statement which ends with a emoticon expressing joke can be sarcastic 
State-of-the-art methods lack these contextual information which, in our case, we extract using pre-trained sentiment, emotion and personality models  Not only these pre-trained models, the baseline method (baseline CNN architecture) performs better than the state-of-the-art models in this generalizability test setting  In our generalizability test, when the pre-trained features are used with baseline features, we get 4 19% F1-score improvement over the baseline features  On the other hand, when they are not used with the baseline features, together they produce 64 25% F1-score 
Another important fact is that an n-grams model cannot perform well on unseen data unless it is trained on a very large corpus  If most of the n-grams extracted from the unseen data are not in the vocabulary of the already trained n-grams model, in fact, the model will produce a very sparse feature vector representation of the dataset  Instead, we use the word2vec embeddings as the source of the features, as word2vec allows for the computation of similarities between unseen data and training data 
Baseline Features vs Pre-trained Features
Our experimental results show that the baseline features outperform the pre-trained features for sarcasm detection  However, the combination of pre-trained features and baseline features beats both of themselves alone  It is counterintuitive, since experimental results prove that both of those features learn almost the same global and contextual features  In particular, baseline network dominates over pre-trained network as the former learns most of the features learned by the latter  Nonetheless, the combination of baseline and pre-trained classifiers improves the overall performance and generalizability, hence proving their effectiveness in sarcasm detection  Experimental results show that sentiment and emotion features are the most useful features, besides baseline features (Figure FIGREF36 )  Therefore, in order to reach a better understanding of the relation between personality features among themselves and with other pre-trained features, we carried out Spearman correlation testing  Results, displayed in Table TABREF39 , show that those features are highly correlated with each other 
Conclusion
In this work, we developed pre-trained sentiment, emotion and personality models for identifying sarcastic text using CNN, which are found to be very effective for sarcasm detection  In the future, we plan to evaluate the performance of the proposed method on a large corpus and other domain-dependent corpora  Future work will also focus on analyzing past tweets and activities of users in order to better understand their personality and profile and, hence, further improve the disambiguation between sarcastic and non-sarcastic text 

ground_truth:  The features extracted from CNN.