Question: Does the modeling of the context yield a significant boost in performance?
Evidence: The current coverage of the political landscape in the press and in social media has led to an unprecedented situation. Like never before, a statement in an interview, a press release, a blog note, or a tweet can spread almost instantaneously and reach the public in no time. This proliferation speed has left little time for double-checking claims against the facts, which has proven critical in politics, e.g., during the 2016 presidential campaign in the USA, which was arguably impacted by fake news in social media and by false claims.
Investigative journalists and volunteers have been working hard trying to get to the root of a claim and to present solid evidence in favor or against it. Manual fact-checking has proven very time-consuming, and thus automatic methods have been proposed as a way to speed-up the process. For instance, there has been work on checking the factuality/credibility of a claim, of a news article, or of an information source BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7. However, less attention has been paid to other steps of the fact-checking pipeline, which is shown in Figure FIGREF1.
The process starts when a document is made public. First, an intrinsic analysis is carried out in which check-worthy text fragments are identified. Then, other documents that might support or rebut a claim in the document are retrieved from various sources. Finally, by comparing a claim against the retrieved evidence, a system can determine whether the claim is likely true or likely false. For instance, BIBREF8 do this on the basis of a knowledge graph derived from Wikipedia. The outcome could then be presented to a human expert for final judgment.
In this paper, we focus on the first step: predicting check-worthiness of claims. Our contributions can be summarized as follows:
New dataset: We build a new dataset of manually-annotated claims, extracted from the 2016 US presidential and vice-presidential debates, which we gathered from nine reputable sources such as CNN, NPR, and PolitiFact, and which we release to the research community.
Modeling the context: We develop a novel approach for automatically predicting which claims should be prioritized for fact-checking, based on a rich input representation. In particular, we model not only the textual content, but also the context: how the target claim relates to the current segment, to neighboring segments and sentences, and to the debate as a whole, and also how the opponents and the public react to it.
State-of-the-art results: We achieve state-of-the-art results, outperforming a strong rivaling system by a margin, while also demonstrating that this improvement is due primarily to our modeling of the context.
We model the problem as a ranking task, and we train both Support Vector Machines (SVM) and Feed-forward Neural Networks (FNN) obtaining state-of-the-art results. We also analyze the relevance of the specific feature groups and we show that modeling the context yields a significant boost in performance. Finally, we also analyze whether we can learn to predict which facts are check-worthy with respect to each of the individual media sources, thus capturing their biases. It is worth noting that while trained on political debates, many features of our model can be potentially applied to other kinds of information sources, e.g., interviews and news.
The rest of the paper is organized as follows: Section SECREF2 discusses related work. Section SECREF3 describes the process of gathering and annotating our political debates dataset. Section SECREF4 presents our supervised approach to predicting fact-checking worthiness, including the explanation of the model and the information sources we use. Section SECREF5 presents the evaluation setup and discusses the results. Section SECREF6 provides further analysis. Finally, Section SECREF7 presents the conclusions and outlines some possible directions for future research.
We also analyze the relevance of the specific feature groups and we show that modeling the context yields a significant boost in performance.
Answer: Yes, it does.