We present a data-driven method for determining the veracity of a set of rumorous claims on social media data. Tweets from different sources pertaining to a rumor are processed on three levels: first, factuality values are assigned to each tweet based on four textual cue categories relevant for our journalism use case; these amalgamate speaker support in terms of polarity and commitment in terms of certainty and speculation. Next, the proportions of these lexical cues are utilized as predictors for tweet certainty in a generalized linear regression model. Subsequently, lexical cue proportions, predicted certainty, as well as their time course characteristics are used to compute veracity for each rumor in terms of the identity of the rumor-resolving tweet and its binary resolution value judgment. The system operates without access to extralinguistic resources. Evaluated on the data portion for which hand-labeled examples were available, it achieves .74 F1-score on identifying rumor resolving tweets and .76 F1-score on predicting if a rumor is resolved as true or false. 1 Background and Task Definition A growing amount of studies investigate how rumors and memes spread and change on social media platforms (Leskovec et al., 2009; Qazvinian et al., 2011; Procter et al., 2013); given the amount of usergenerated content, the need for automatic fact checking and claim verification procedures is obvious. To compute veracity, systems have been created recently for assessing the credibility of sources and claims (Berti-Équille and Borge-Holthoefer, 2015). Upcoming initiatives endorsed veracity detection in social media content as a shared task, calling for targeted applications and releasing benchmark data1. To tackle this challenge, we implemented a system that seeks to achieve three goals: (i) to compute a judgment indicating how factual a claim is, based on textual cues and predicted speaker certainty, (ii) to identify which tweet is resolving a rumor, in a set of tweets that discuss this rumor, and (iii) to predict the resolution value for the rumor, i.e., whether the rumor is verified as true or false. Veracity computation is based on information from three information layers related to rumorousness: (1) lexical-level factuality cues, (2) temporal patterns, and (3) speaker certainty. The system is purely data-driven and operates without building claim source profiles for the analyzed content. Below we introduce our motivation in the context of previous and related work. The means by which factuality is conveyed are largely but not exclusively encoded on linguistic levels and are tightly related to the notion of certainty. Certainty and other extra-propositional aspects of meaning have prominently been investigated in terms of modality, negation and speculative language phenomena (Morante and Blanco, 2012; Morante and Sporleder, 2012). Benchmark corpora with annotations emerged (Saurı́ and Pustejovsky, 2009; Farkas et al., 2010), and systems have been built (Saurı́ and Pustejovsky, 2012; de Marneffe et al., 2012; Velldal and Read, 2012) to process texts from the genres of literature, newswire, biomedicine and online encyclopedia, typically drawing on lexical and syntactic cues. (Szarvas et al., 2012) propose a method for porting uncertainty detection across genres and domains. (Kilicoglu et al., 2015) present a full-fledged, compositional approach to factuality ∗UDR is supported by an Alexander von Humboldt Society grant. †PL is supported by the PHEME FP7 project (Grant No. 611233). http://alt.qcri.org/semeval2017/task8/ ar X iv :1 61 1. 02 59 0v 1 [ cs .C L ] 8 N ov 2 01 6 modeling and detection on texts from the domain of biomedicine based on fine-grained typology and dictionary-based classification of extra-propositional phenomena. Several components of the model are motivated by the nature of scientific communication that serves to track hypothesis building processes with tentative results, analogously to journalistic reports about breaking news. (Soni et al., 2014) focus on factuality framing in social media data in quoted claims with a small set of cues, whereas (Finn et al., 2014) implement keyword-based negation detection without providing quantitative evaluation. Next to linguistically expressed uncertainty, extralinguistic information such as the temporal distribution of claims is shown to be an important aspect of veracity computation. Previous studies that investigated temporal patterns of linguistic cues tied to claims emerging in real-world events focus on keywords related to sentiment, named entities and domain terms (Temnikova et al., 2014), but not factuality-conveying cues. (Wei et al., 2013) report on the first uncertainty corpus based on tweets, as well as on classification results for uncertain tweets. Next to platform-specific metadata, they utilized cue phrases in annotated uncertain tweets and an algorithm to detect peaks in the data. (Kwon and Cha, 2014) and (Ma et al., 2015) show for rumor detection that accuracy can be improved by not only looking at message-related properties but also at how these properties change over time. (Ma et al., 2015) propose a time series structure for features and their deltas as the input for classification. On the full PHEME dataset, (Lukasik et al., 2016) report on stance detection in the context of temporal dynamics. They utilize textual information via language modeling but do not evaluate the contribution of textual as opposed to other features. On the same dataset, (Zubiaga et al., 2016) analyzed labeled certainty values in dependence of claim resolution, and found that tweeters post messages with statistically similar certainty before and after a claim is resolved, moreover, irrespective of the resolution value. In (Lendvai et al., 2016) we analyzed and validated a subset of the PHEME data on English and German data that temporal distribution and polarity of lexical markers can be used to represent and quantify changes in factuality framing in a rumor’s lifecycle. Our current study furthers this research by incorporating, evaluating, and visualizing temporally anchored features for claim resolution point as well as claim resolution value prediction in English language rumors discussed in potentially noisy, user-generated content. The paper is structured as follows. In Section 2 we introduce the underlying data and certainty annotations, and describe the automatic extension of lexical cues assigned to four levels of factuality. In Section 3 the relation between certainty and each of the factuality levels is assessed, and regression analysis is used for predicting certainty values by cue-type ratios. In Section 4 we quantify trend discontinuities in time series data of lexical cue ratios and predicted certainty scores to describe rumor resolution points. Cue ratios, certainty, as well as their time course characteristics are exploited in Section 5, where we train classifiers to identify claim-resolving tweets within series of tweets spanning a claim’s lifetime, and additionally predict the claim’s resolution value. The findings are discussed in Section 6.
Output: Veracity Computing from Lexical Cues and Perceived Certainty Trends