A Framework for Automated Fact-Checking for Real-Time Validation of Emerging Claims on the Web

Literally minutes after the Las Vegas shooting, rumors about the identity and the motives of the perpetrator started to circulate on Twitter. Some of the most persistent ones were that the perpetrator was a Muslim convert [1] or a member of Antifa [2]. Moreover, also alleged reactions of people to the shooting received much attention. On a fake Twitter account, a liberal teacher was supposedly hoping that Trump supports were among the victims ([3]). When all the attention is on the event and not all details about the incident are known, propagandists have the opportunity to instrumentalize the event to promote a certain worldview. As the problem of false information being distributed on the web became more severe in the past couple of years, there is also an increased interest in information validation. Thus, fact-checking websites like politifact.com, fullfact.org, and snopes.com are becoming more popular. On these websites, journalists or professional factcheckers are manually resolving controversial claims, by providing a verdict, which is backed up by evidence. Nevertheless, even though manual fact-checking blossoms from the spread of fake news, the approach is rather mitigating the influence of false information rather than solving the problem. The resolution is often done subsequently after a fake news article has spread, although most of the damage is caused when the fake news article is distributed through social networks. In fact, many of the news consumers are not going to review the facts on a story once the spotlight of the media has shifted to a different topic. Thus, real-time fact-checking techniques are required, which would be able to intervene in the proliferation process in the early stages, before the false information goes viral. Many of the issues of manual claim validation can be addressed by automated fact-checking, as it would be possible to validate a large number of articles as they appear on the web automatically. To address the problem, a number of different approaches have been suggested, many of which are based on knowledge bases ([4, 5, 6]). These methods are validating a claim by verifying whether it is consistent with the knowledge base, that is, whether predicates can be found which basically restate the claim or contradict it. Nevertheless, knowledge bases only represent a small portion of all the information available on the web and newly generated knowledge is rare since the updating process requires some time. Thus, in particular for real-time fact-checking, methods based on raw text are more suitable, as they would allow recently published web documents to be incorporated into the validation process. However, claim validation on the basis of raw text has not yet received much attention and only a few studies address these issue. The task 8 in SemEval-2017 [7] was concerned with the problem of validating claims on Twitter. The claim itself was represented by a tweet and the problem was approach in two different settings: In the closed setting, the validation was done only on the basis of the features of the claim tweet itself. In the open setting, external information, in form of related Wikipedia articles and web documents, was provided. Both problem settings turned out to be too challenging for the applied methods since the majority baseline could not be beaten. The participants suggest implementing additional more discriminative features, like those used in sentiment analysis, or discriminative rules to further increase performance. A method for the identification and validation of simple claims about 16 statistical properties of countries is presented in [8]. The authors introduce a distantly supervised approach which is based on a knowledge base, as well as raw text input. The method is able to identify statistical claims with 60% precision and to validate these claims without explicit supervision. The framework for claim validation presented in [9] is to our knowledge the most comprehensive. The authors extracted 4856 claims and the verdicts for these claim from the fact-checking website snopes.com [10]. In order to collect external information for the resolution of the claims, the Google search engine was used. The developed system is able to determine the stance of a text with respect to a given claim, the credibility of sources and the validity of the claim. The authors report 80% accuracy for the claim validation task.