Introduction
Nowadays, people increasingly tend to use social media like Facebook and Twitter as their primary source of information and news consumption. There are several reasons behind this tendency, such as the simplicity to gather and share the news and the possibility of staying abreast of the latest news and updated faster than with traditional media. An important factor is also that people can be engaged in conversations on the latest breaking news with their contacts by using these platforms. Pew Research Center's newest report shows that two-thirds of U.S. adults gather their news from social media, where Twitter is the most used platform. However, the absence of a systematic approach to do some form of fact and veracity checking may also encourage the spread of rumourous stories and misinformation BIBREF0 . Indeed, in social media, unverified information can spread very quickly and becomes viral easily, enabling the diffusion of false rumours and fake information.
Within this scenario, it is crucial to analyse people attitudes towards rumours in social media and to resolve their veracity as soon as possible. Several approaches have been proposed to check the rumour veracity in social media BIBREF1 . This paper focus on a stance-based analysis of event-related rumours, following the approach proposed at SemEval-2017 in the new RumourEval shared task (Task 8, sub-task A) BIBREF2 . In this task English tweets from conversation threads, each associated to a newsworthy event and the rumours around it, are provided as data. The goal is to determine whether a tweet in the thread is supporting, denying, querying, or commenting the original rumour which started the conversation. It can be considered a stance classification task, where we have to predict the user's stance towards the rumour from a tweet, in the context of a given thread. This task has been defined as open stance classification task and is conceived as a key step in rumour resolution, by providing an analysis of people reactions towards an emerging rumour BIBREF0 , BIBREF3 . The task is also different from detecting stance towards a specific target entity BIBREF4 .
Contribution We describe a novel classification approach, by proposing a new feature matrix, which includes two new groups: (a) features exploiting the conversational structure of the dataset BIBREF2 ; (b) affective features relying on the use of a wide range of affective resources capturing different facets of sentiment and other affect-related phenomena. We were also inspired by the fake news study on Twitter in BIBREF5 , showing that false stories inspire fear, disgust, and surprise in replies, while true stories inspire anticipation, sadness, joy, and trust. Meanwhile, from a dialogue act perspective, the study of BIBREF6 found that a relationship exists between the use of an affective lexicon and the communicative intention of an utterance which includes AGREE-ACCEPT (support), REJECT (deny), INFO-REQUEST (question), and OPINION (comment). They exploited several LIWC categories to analyse the role of affective content.
Our results show that our model outperforms the state of the art on the Semeval-2017 benchmark dataset. Feature analysis highlights the contribution of the different feature groups, and error analysis is shedding some light on the main difficulties and challenges which still need to be addressed.
Outline The paper is organized as follows. Section 2 introduces the SemEval-2017 Task 8. Section 3 describes our approach to deal with open stance classification by exploiting different groups of features. Section 4 describes the evaluation and includes a qualitative error analysis. Finally, Section 5 concludes the paper and points to future directions.
SemEval-2017 Task 8: RumourEval
The SemEval-2017 Task 8 Task A BIBREF2 has as its main objective to determine the stance of the users in a Twitter thread towards a given rumour, in terms of support, denying, querying or commenting (SDQC) on the original rumour. Rumour is defined as a “circulating story of questionable veracity, which is apparently credible but hard to verify, and produces sufficient skepticism and/or anxiety so as to motivate finding out the actual truth” BIBREF7 . The task was very timing due to the growing importance of rumour resolution in the breaking news and to the urgency of preventing the spreading of misinformation.
Dataset The data for this task are taken from Twitter conversations about news-related rumours collected by BIBREF3 . They were annotated using four labels (SDQC): support - S (when tweet's author support the rumour veracity); deny -D (when tweet's author denies the rumour veracity); query - Q (when tweet's author ask for additional information/evidence); comment -C (when tweet's author just make a comment and does not give important information to asses the rumour veracity). The distribution consists of three sets: development, training and test sets, as summarized in Table TABREF3 , where you can see also the label distribution and the news related to the rumors discussed. Training data consist of 297 Twitter conversations and 4,238 tweets in total with related direct and nested replies, where conversations are associated to seven different breaking news. Test data consist of 1049 tweets, where two new rumourous topics were added.
Participants Eight teams participated in the task. The best performing system was developed by Turing (78.4 in accuracy). ECNU, MamaEdha, UWaterloo, and DFKI-DKT utilized ensemble classifier. Some systems also used deep learning techniques, including Turing, IKM, and MamaEdha. Meanwhile, NileTRMG and IITP used classical classifier (SVM) to build their systems. Most of the participants exploited word embedding to construct their feature space, beside the Twitter domain features.
Proposed Method
We developed a new model by exploiting several stylistic and structural features characterizing Twitter language. In addition, we propose to utilize conversational-based features by exploiting the peculiar tree structure of the dataset. We also explored the use of affective based feature by extracting information from several affective resources including dialogue-act inspired features.
Structural Features
They were designed taking into account several Twitter data characteristics, and then selecting the most relevant features to improve the classification performance. The set of structural features that we used is listed below.
Retweet Count: The number of retweet of each tweet.
Question Mark: presence of question mark "?"; binary value (0 and 1).
Question Mark Count: number of question marks present in the tweet.
Hashtag Presence: this feature has a binary value 0 (if there is no hashtag in the tweet) or 1 (if there is at least one hashtag in the tweet).
Text Length: number of characters after removing Twitter markers such as hashtags, mentions, and URLs.
URL Count: number of URL links in the tweet.
Conversation Based Features
These features are devoted to exploit the peculiar characteristics of the dataset, which have a tree structure reflecting the conversation thread.
Text Similarity to Source Tweet: Jaccard Similarity of each tweet with its source tweet.
Text Similarity to Replied Tweet: the degree of similarity between the tweet with the previous tweet in the thread (the tweet is a reply to that tweet).
Tweet Depth: the depth value is obtained by counting the node from sources (roots) to each tweet in their hierarchy.
Affective Based Features
The idea to use affective features in the context of our task was inspired by recent works on fake news detection, focusing on emotional responses to true and false rumors BIBREF5 , and by the work in BIBREF6 reflecting on the role of affect in dialogue acts BIBREF6 . Multi-faceted affective features have been already proven to be effective in some related tasks BIBREF9 , including the stance detection task proposed at SemEval-2016 (Task 6).
We used the following affective resources relying on different emotion models.
Emolex: it contains 14,182 words associated with eight primary emotion based on the Plutchik model BIBREF10 , BIBREF11 .
EmoSenticNet(EmoSN): it is an enriched version of SenticNet BIBREF12 including 13,189 words labeled by six Ekman's basic emotion BIBREF13 , BIBREF14 .
Dictionary of Affect in Language (DAL): includes 8,742 English words labeled by three scores representing three dimensions: Pleasantness, Activation and Imagery BIBREF15 .
Affective Norms for English Words (ANEW): consists of 1,034 English words BIBREF16 rated with ratings based on the Valence-Arousal-Dominance (VAD) model BIBREF17 .
Linguistic Inquiry and Word Count (LIWC): this psycholinguistic resource BIBREF18 includes 4,500 words distributed into 64 emotional categories including positive (PosEMO) and negative (NegEMO).
Dialogue-Act Features
We also included additional 11 categories from bf LIWC, which were already proven to be effective in dialogue-act task in previous work BIBREF6 . Basically, these features are part of the affective feature group, but we present them separately because we are interested in exploring the contribution of such feature set separately. This feature set was obtained by selecting 4 communicative goals related to our classes in the stance task: agree-accept (support), reject (deny), info-request (question), and opinion (comment). The 11 LIWC categories include:
Agree-accept: Assent, Certain, Affect;
Reject: Negate, Inhib;
Info-request: You, Cause;
Opinion: Future, Sad, Insight, Cogmech.
Experiments, Evaluation and Analysis
We used the RumourEval dataset from SemEval-2017 Task 8 described in Section SECREF2 . We defined the rumour stance detection problem as a simple four-way classification task, where every tweet in the dataset (source and direct or nested reply) should be classified into one among four classes: support, deny, query, and comment. We conducted a set of experiments in order to evaluate and analyze the effectiveness of our proposed feature set..
The results are summarized in Table TABREF28 , showing that our system outperforms all of the other systems in terms of accuracy. Our best result was obtained by a simple configuration with a support vector classifier with radial basis function (RBF) kernel. Our model performed better than the best-performing systems in SemEval 2017 Task 8 Subtask A (Turing team, BIBREF19 ), which exploited deep learning approach by using LTSM-Branch model. In addition, we also got a higher accuracy than the system described in BIBREF20 , which exploits a Random Forest classifier and word embeddings based features.
We experimented with several classifiers, including Naive Bayes, Decision Trees, Support Vector Machine, and Random Forest, noting that SVM outperforms the other classifiers on this task. We explored the parameter space by tuning the SVM hyperparameters, namely the penalty parameter C, kernel type, and class weights (to deal with class imbalance). We tested several values for C (0.001, 0.01, 0.1, 1, 10, 100, and 1000), four different kernels (linear, RBF, polynomial, and sigmoid) and weighted the classes based on their distribution in the training data. The best result was obtained with C=1, RBF kernel, and without class weighting.
An ablation test was conducted to explore the contribution of each feature set. Table TABREF32 shows the result of our ablation test, by exploiting several feature sets on the same classifier (SVM with RBF kernel) . This evaluation includes macro-averages of precision, recall and INLINEFORM0 -score as well as accuracy. We also presented the scores for each class in order to get a better understanding of our classifier's performance.
Using only conversational, affective, or dialogue-act features (without structural features) did not give a good classification result. Set B (conversational features only) was not able to detect the query and deny classes, while set C (affective features only) and D (dialogue-act features only) failed to catch the support, query, and deny classes. Conversational features were able to improve the classifier performance significantly, especially in detecting the support class. Sets E, H, I, and K which utilize conversational features induce an improvement on the prediction of the support class (roughly from 0.3 to 0.73 on precision). Meanwhile, the combination of affective and dialogue-act features was able to slightly improve the classification of the query class. The improvement can be seen from set E to set K where the INLINEFORM0 -score of query class increased from 0.52 to 0.58. Overall, the best result was obtained by the K set which encompasses all sets of features. It is worth to be noted that in our best configuration system, not all of affective and dialogue-act features were used in our feature vector. After several optimization steps, we found that some features were not improving the system's performance. Our final list of affective and dialogue-act based features includes: DAL Activation, ANEW Dominance, Emolex Negative, Emolex Fear, LIWC Assent, LIWC Cause, LIWC Certain and LIWC Sad. Therefore, we have only 17 columns of features in the best performing system covering structural, conversational, affective and dialogue-act features.
We conducted a further analysis of the classification result obtained by the best performing system (79.50 on accuracy). Table TABREF30 shows the confusion matrix of our result. On the one hand, the system is able to detect the comment tweets very well. However, this result is biased due to the number of comment data in the dataset. On the other hand, the system is failing to detect denying tweets, which were falsely classified into comments (68 out of 71). Meanwhile, approximately two thirds of supporting tweets and almost half of querying tweets were classified into the correct class by the system.
In order to assess the impact of class imbalance on the learning, we performed an additional experiment with a balanced dataset using the best performing configuration. We took a subset of the instances equally distributed with respect to their class from the training set (330 instances for each class) and test set (71 instances for each class). As shown in Table TABREF31 , our classifier was able to correctly predict the underrepresented classes much better, although the overall accuracy is lower (59.9%). The result of this analysis clearly indicates that class imbalance has a negative impact on the system performance.
Error analysis
We conducted a qualitative error analysis on the 215 misclassified in the test set, to shed some light on the issues and difficulties to be addressed in future work and to detect some notable error classes.
Denying by attacking the rumour's author. An interesting finding from the analysis of the Marina Joyce rumour data is that it contains a lot of denying tweets including insulting comments towards the author of the source tweet, like in the following cases:
Rumour: Marina Joyce
Misclassified tweets:
(da1) stfu you toxic sludge
(da2) @sampepper u need rehab
Misclassification type: deny (gold) INLINEFORM0 comment (prediction)
Source tweet:
(s1) Anyone who knows Marina Joyce personally knows she has a serious drug addiction. she needs help, but in the form of rehab #savemarinajoyce
Tweets like (da1) and (da2) seem to be more inclined to show the respondent's personal hatred towards the s1-tweet's author than to deny the veracity of the rumour. In other words, they represent a peculiar form of denying the rumour, which is expressed by personal attack and by showing negative attitudes or hatred towards the rumour's author. This is different from denying by attacking the source tweet content, and it was difficult to comprehend for our system, that often misclassified such kind of tweets as comments.
Noisy text, specific jargon, very short text. In (da1) and (da2) (as in many tweets in the test set), we also observe the use of noisy text (abbreviations, misspellings, slang words and slurs, question statements without question mark, and so on) that our classifier struggles to handle . Moreover, especially in tweets from the Marina Joyce rumour's group, we found some very short tweets in the denying class that do not provide enough information, e.g. tweets like “shut up!", “delete", and “stop it. get some help".
Argumentation context. We also observed misclassification cases that seem to be related to a deeper capability of dealing with the argumentation context underlying the conversation thread.
Rumour: Ferguson
Misclassified tweet:
(arg1)@QuadCityPat @AP I join you in this demand. Unconscionable.
Misclassification type: deny (gold) INLINEFORM0 comment (prediction)
Source tweet:
(s2) @AP I demand you retract the lie that people in #Ferguson were shouting “kill the police", local reporting has refuted your ugly racism
Here the misclassified tweet is a reply including an explicit expression of agreement with the author of the source tweet (“I join you”). Tweet (s2) is one of the rare cases of source tweets denying the rumor (source tweets in the RumourEval17 dataset are mostly supporting the rumor at issue). Our hypothesis is that it is difficult for a system to detect such kind of stance without a deeper comprehension of the argumentation context (e.g., if the author's stance is denying the rumor, and I agree with him, then I am denying the rumor as well). In general, we observed that when the source tweet is annotated by the deny label, most of denying replies of the thread include features typical of the support class (and vice versa), and this was a criticism.
Mixed cases. Furthermore, we found some borderline mixed cases in the gold standard annotation. See for instance the following case:
Rumour: Ferguson
Misclassified tweet:
(mx1) @MichaelSkolnik @MediaLizzy Oh do tell where they keep track of "vigilante" stats. That's interesting.
Misclassification type: query (gold) INLINEFORM0 comment (prediction)
Source tweet:
(s3) Every 28 hours a black male is killed in the United States by police or vigilantes. #Ferguson
Tweet (mx1) is annotated with a query label rather than as a comment (our system prediction), but we can observe the presence of a comment (“That's interesting”) after the request for clarification, so it seems to be a kind of mixed case, where both labels make sense.
Citation of the source's tweet. We have noticed many misclassified cases of replying tweets with error pattern support (gold) INLINEFORM0 comment (our prediction), where the text contains a literal citation of the source tweet, like in the following tweet: THIS HAS TO END “@MichaelSkolnik: Every 28 hours a black male is killed in the United States by police or vigilantes. #Ferguson” (the text enclosed in quotes is the source tweet). Such kind of mistakes could be maybe addressed by applying some pre-processing to the data, for instance by detecting the literal citation and replacing it with a marker.
Figurative language devices. Finally, the use of figurative language (e.g., sarcasm) is also an issue that should be considered for the future work. Let us consider for instance the following misclassified tweets:
Rumour: Hillary's Illness
Misclassified tweets:
(fg1) @mitchellvii True, after all she can open a pickle jar.
(fg2) @mitchellvii Also, except for having a 24/7 MD by her side giving her Valium injections, Hillary is in good health! https://t.co/GieNxwTXX7
(fg3) @mitchellvii @JoanieChesnutt At the very peak yes, almost time to go down a cliff and into the earth.
Misclassification type: support (gold) INLINEFORM0 comment (prediction)
Source tweet:
(s4) Except for the coughing, fainting, apparent seizures and "short-circuits," Hillary is in the peak of health.
All misclassified tweets (fg1-fg3) from the Hillary's illness data are replies to a source tweet (s4), which is featured by sarcasm. In such replies authors support the rumor by echoing the sarcastic tone of the source tweet. Such more sophisticated cases, where the supportive attitude is expressed in an implicit way, were challenging for our classifier, and they were quite systematically misclassified as simple comments.
Conclusion
In this paper we proposed a new classification model for rumour stance classification. We designed a set of features including structural, conversation-based, affective and dialogue-act based feature. Experiments on the SemEval-2017 Task 8 Subtask A dataset show that our system based on a limited set of well-engineered features outperforms the state-of-the-art systems in this task, without relying on the use of sophisticated deep learning approaches. Although achieving a very good result, several research challenges related to this task are left open. Class imbalance was recognized as one the main issues in this task. For instance, our system was struggling to detect the deny class in the original dataset distribution, but it performed much better in that respect when we balanced the distribution across the classes.
A re-run of the RumourEval shared task has been proposed at SemEval 2019 and it will be very interesting to participate to the new task with an evolution of the system here described.
Acknowledgements
Endang Wahyu Pamungkas, Valerio Basile and Viviana Patti were partially funded by Progetto di Ateneo/CSP 2016 (Immigrants, Hate and Prejudice in Social Media, S1618_L2_BOSC_01).

Question: Is this an English-language dataset?
Answer: Yes

Question: What affective-based features are used?
Answer: affective features provided by different emotion models such as Emolex, EmoSenticNet, Dictionary of Affect in Language, Affective Norms for English Words and Linguistics Inquiry and Word Count

Question: What conversation-based features are used?
Answer: Text Similarity to Source Tweet, Text Similarity to Replied Tweet, Tweet Depth